SECOND CHAYYAH · GROUNDING VALIDATOR
SHOR

Classifies whether an agent's output is grounded in its source material. Runs before the downstream action. No LLM.

When an agent generates a summary, drafts a response, or prepares a tool call that depends on prior context, SHOR checks whether the output is actually traceable to the provided sources. Classification is deterministic — no model call, no network dependency, same input yields same result.

Install

npm install @reshimu/shor
yarn add @reshimu/shor
pip install reshimu-shor

No native dependencies. No compilation step. Works in any Node.js ≥ 18 or Python ≥ 3.9 runtime.

Quick start

TypeScript / JavaScript

import { classify } from '@reshimu/shor'

const result = classify({
  output: "The meeting was scheduled for Thursday at 2pm in Conference Room B.",
  sources: [
    "Per the invite, the sync is Thursday 2:00 PM, CR-B.",
    "Attendees: Priya, Marcus, Leila."
  ]
})

console.log(result.level)      // 'GROUNDED'
console.log(result.score)      // 0.94
console.log(result.unmatched)  // []

Partial grounding

const result = classify({
  output: "The Q3 revenue was $4.2M, up 18% from Q2.",
  sources: [
    "Q3 revenue: $4.2M."
    // No Q2 comparison in source material
  ]
})

console.log(result.level)      // 'PARTIAL'
console.log(result.unmatched)  // ['up 18% from Q2']

Python

from reshimu_shor import classify

result = classify(
    output="The patient was prescribed 500mg twice daily.",
    sources=[
        "Dosage: 500mg BID.",
        "Medication: amoxicillin."
    ]
)

print(result.level)      # 'GROUNDED'
print(result.score)      # 0.97
print(result.unmatched)  # []

Classification levels

GROUNDED
All factual claims in the output have direct support in the source material. Score meets or exceeds threshold.
PARTIAL
Some claims are supported. Others have no traceable source span. Score is below threshold but above zero.
UNGROUNDED
Output contains claims with no support in the provided sources. May indicate hallucination or out-of-scope synthesis.
INDETERMINATE
Source material is insufficient to evaluate grounding. Usually means sources were empty or the output contained no extractable claims.
On PARTIAL vs UNGROUNDED. PARTIAL means the agent got some things right from the sources and added something it didn't. UNGROUNDED means the sources don't support any of the substantive claims. In most pipelines, PARTIAL warrants a log-and-flag; UNGROUNDED warrants a block.

API reference

classify(options)

Parameters:

Field Type Required Description
output string yes The agent output to evaluate
sources string[] yes Source documents or context the output is expected to draw from
threshold number no Minimum span-match ratio for GROUNDED. Default: 0.85
strict boolean no If true, any unmatched claim returns UNGROUNDED regardless of threshold. Default: false

Returns:

Field Type Description
level 'GROUNDED' | 'PARTIAL' | 'UNGROUNDED' | 'INDETERMINATE' Classification result
score number Span-match ratio, 0–1
unmatched string[] Claims in output with no traceable source span
matched string[] Claims with at least one source match

Integration guide

SHOR sits between an agent's output and any downstream action that depends on it. The pattern is consistent regardless of orchestration framework.

import { classify } from '@reshimu/shor'

async function agentAction(agentOutput: string, sourceContext: string[]) {
  const grounding = classify({ output: agentOutput, sources: sourceContext })

  if (grounding.level === 'UNGROUNDED') {
    return {
      blocked: true,
      reason: 'ungrounded output',
      unmatched: grounding.unmatched
    }
  }

  if (grounding.level === 'PARTIAL') {
    // Log and escalate, or continue with reduced-trust flag
    console.warn('Partial grounding. Unmatched:', grounding.unmatched)
  }

  return { blocked: false, output: agentOutput }
}

With NESHER (irreversibility gate)

SHOR and NESHER compose. The recommended order is: classify the action's reversibility first (NESHER), then check grounding of the output that produced it (SHOR). This way, an UNGROUNDED output cannot propose an irreversible action without both gates flagging.

import { classify as grounding } from '@reshimu/shor'
import { classify as reversibility } from '@reshimu/nesher'

async function intercept(action: string, output: string, sources: string[]) {
  const rev  = reversibility({ action, target: output })
  const gnd  = grounding({ output, sources })

  if (rev.level === 'CRITICAL' || gnd.level === 'UNGROUNDED') {
    return { blocked: true, rev, gnd }
  }

  return { blocked: false, rev, gnd }
}

Python agent loop

from reshimu_shor import classify

def before_tool_call(output: str, sources: list[str]) -> dict:
    result = classify(output=output, sources=sources)

    if result.level == "UNGROUNDED":
        raise ValueError(f"Blocked: ungrounded output. Unmatched: {result.unmatched}")

    return {"level": result.level, "score": result.score}

Design constraints

No LLM calls. Using a language model to evaluate the output of another language model is a circular dependency. The evaluator shares the generator's failure modes — it can be persuaded by fluent writing, it fills gaps rather than flags them, and it lacks a stable reference for what "support" means across runs. SHOR uses deterministic entity extraction and span-level matching. Same input, same classification, every time.
ConstraintReason
No LLM calls Circular evaluation — the grader shares the generator's failure modes
No native dependencies Runs in any runtime without compilation; no binding maintenance burden
Synchronous by default Sub-5ms on typical payloads; sits in the execution path, not a sidecar
Deterministic Same input must yield same output across runs; grounding is not a probability

For the full reasoning behind the no-LLM constraint, see Why we don't trust LLMs to classify the call they're about to make.

Runtime stack

SHOR is the second of four runtime integrity validators — the Chayyot — that form the Reshimu agent governance stack.

ValidatorRoleStatus
NESHER Irreversibility classifier — stops unrecoverable actions before execution v0.1.0 live
SHOR Grounding validator — checks that outputs trace to source material v0.1.0 live
ARYEH Scope classifier — detects out-of-bounds tool calls and capability overreach in development
PANIM ADAM Gray-zone escalator — routes ambiguous actions to the right handler in development

Each validator is a zero-dependency library, usable independently. The Atzmut OS MCP server integrates all four into a single intercept layer.

For the architectural pattern underlying all four, see Bearers of the Throne.