The Model Quotes, Python Decides: An NDA Playbook Reviewer With a Live Agent Trace#

The first person to try the live mode of this app uploaded a PDF, looked at the screen for eleven seconds, and went back to the queue.

Nothing was broken. The model was doing exactly what it should: reading a 4,000-character NDA and writing out ten verbatim quotes as JSON. That takes about a minute on a MacBook. For that minute the screen showed a row of step names with one of them highlighted, and nothing else. Eleven seconds in, "nothing else" reads as "stuck." Leaving the page closed the stream. Closing the stream cancelled the review. The matter sat in the queue marked running, forever, with no route.

This post is about the app, and about the fix, which ended up being the most interesting part of it: a review that runs as a background job with a replayable event log, and a trace view that shows which steps the model did and which steps plain Python did.

The code is on GitHub: MinhQuanBuiSco/Agent-workflows.

The Problem, and the One Rule#

Picture one in-house counsel and a queue of vendor NDAs. Most are standard paper and should clear the same day. A few break the company playbook — a perpetual term, a residuals clause, a right to train models on your data — and those should land on the desk with the sentence that broke the rule, not a paragraph of chatbot prose.

An LLM is good at the first half of that job: finding the clause about confidentiality term in a messy PDF. It is not a good place to put the second half: deciding that four years is a fallback and five years and one day is a reject. That decision is company policy. It should be diffable, testable, and boring.

So the app has one rule, and the rest of the design follows from it:

The model is not allowed to assign green, yellow, or red.

The model proposes. Python decides. A quote that is not actually in the contract cannot clear anything.

Route	Meaning
Green	Every position is an accept, and every quote is a verbatim span of the contract.
Yellow	A fallback, or a non-critical reject. Counsel glances.
Red	A critical position is a reject, is ungrounded, or is missing where absence is a reject.
Out of playbook	The document is not an NDA. It is not scored.

The queue: eight sample contracts, a Use demo PDF button, and Fixture and Live modes

Six Steps, Two Lanes#

A review is six steps. Two of them call the model. Four of them are Python.

#	Step	Lane	What it does
1	Read	Python	Text from the PDF (`pdfplumber`) or TXT
2	Extract	Model	One call at temperature 0: is this an NDA, and for each playbook position, a verbatim excerpt plus a few typed fields
3	Citation gate	Python	Is each excerpt a real span of the contract? If not, the position is `ungrounded`
4	Playbook rules	Python	One function per position reads the fields and returns accept, fallback, or reject
5	Draft comments	Model	Only for fallbacks and rejects: phrase a redline. Cannot change a verdict
6	Route	Python	Sum the verdicts into green, yellow, red, or out of playbook

The playbook is YAML: ten positions for a vendor NDA, each with a requirement, a critical flag, what absence means (if_absent), the name of a Python rule, the fields the model must fill, and the company's standard fallback sentence.

- id: term
  title: Confidentiality term
  critical: true
  if_absent: reject
  rule: term
  requirement: >
    Accept a survival period of two or three years. Four or five years is a
    fallback, including when it covers all information rather than trade
    secrets only. Reject a perpetual term, or any term longer than five years.
    A term shorter than two years is a fallback, not an accept.
  fallback_comment: >
    Please revise the survival period to three years from disclosure.
  fields:
    - name: years
      type: number
      description: Length of the confidentiality survival period in years. Null when the term is perpetual or unstated.
    - name: perpetual
      type: bool
      description: True when confidentiality does not expire.
    - name: trade_secret_only
      type: bool
      description: True when the longer survival applies only to trade secrets.

The requirement text is written for the model, so it knows what to look for. The rule is written for Python, so it decides. They describe the same policy from two sides.

Step 2: An Extraction Contract With No Field for a Verdict#

The extraction prompt is rendered from the playbook. The schema it asks for has no place to put an opinion:

Schema:
{
  "document_type": "nda" or "other",
  "counterparty": "the party that is not Acme Industries, or null",
  "positions": [
    {"id": "<id>", "present": true or false, "excerpt": "verbatim span or null", "fields": {}}
  ]
}

Rules:
- Include exactly one object for every position id below.
- When present is false, excerpt must be null and fields must be {}.
- When present is true, excerpt must be copied verbatim from the contract.
  Same words, contiguous span. Do not paraphrase.
- Do not add a verdict, a risk level, or a route. You do not decide acceptability.

The last line is belt and braces. Even if the model adds a "risk": "high" key, Pydantic ignores it (extra = "ignore"), and nothing downstream reads it. If the JSON is invalid, the extractor retries once with the parse error appended. A second failure fails the review loudly. It does not fall back to anything.

Step 3: The Citation Gate Is Four Lines#

def normalize_ws(text: str) -> str:
    return " ".join(text.split())


def is_verbatim(excerpt: str | None, document: str) -> bool:
    if excerpt is None or not excerpt.strip():
        return False
    return normalize_ws(excerpt) in normalize_ws(document)

Whitespace is collapsed so a PDF line break inside a sentence does not fail an otherwise exact quote. Nothing else is forgiven. A paraphrase fails. A quote stitched from two sentences fails. A quote with one word changed fails.

A failed quote makes the position ungrounded, and on a critical position ungrounded routes red. That is the property I care about most: a hallucinated quote can never make a contract greener. The worst it can do is send a clean contract to a human.

The repo keeps a fixture that proves it. It takes a clean, green NDA and paraphrases every quote in the recorded extraction. The fields are all still "correct." Every quote fails the gate, and the route is red. It runs in the test suite, next to the eight labeled samples.

Step 4: Rules Are a Registry, Not a Prompt#

Each position maps to a plain function from fields to a verdict and a one-line reason:

def rule_term(fields: dict) -> tuple[str, str]:
    if fields.get("perpetual") is True:
        return "reject", "Confidentiality does not expire. This playbook rejects a perpetual term."
    years = _years(fields)
    if years is None:
        return "reject", "No usable term length was extracted."
    if years > 5:
        return "reject", f"{years:g} years is longer than the five-year ceiling."
    if years >= 4:
        return "fallback", f"{years:g} years is outside the two-to-three-year accept band."
    if years >= 2:
        return "accept", f"{years:g} years is inside the two-to-three-year accept band."
    ...

RULES = {
    "mutuality": rule_mutuality,
    "purpose": rule_purpose,
    "term": rule_term,
    # ... one per playbook position
}

Two things fall out of this for free.

The rules are testable without a model. One table-driven test feeds fields into the rules — 23 cases, from term(years=7) to governing_law(jurisdiction="England") — and checks the verdict. Changing "four years is a fallback" to "four years is a reject" is a one-line diff that a lawyer can read and a test can pin.

The rules disagree with you when you are wrong. When I wrote the demo NDA, I put in a compelled-disclosure clause with no duty to notify, and I told the person trying it that it would come back as a fallback. It came back as a reject, and the rule was right: the playbook says silence on compelled disclosure is a fallback, but a clause that is present and requires no notice is a reject. My summary of the policy was wrong. The code had the policy.

Absent positions never reach a rule. They take the playbook's if_absent verdict directly: no residuals clause is an accept, no personal-data carve-out is a reject.

Step 6: A Router the Model Never Touches#

for position in positions:
    if position.verdict in {"reject", "ungrounded"} and position.critical:
        reds.append(line)
    elif position.verdict == "fallback":
        yellows.append(line)
    elif position.verdict in {"reject", "ungrounded"}:
        yellows.append(line)

Any critical reject or ungrounded quote is red. Any fallback or non-critical reject is yellow. Otherwise green. The reasons shown on the result banner are the rule outputs, word for word.

The finished trace: every quote verified, every rule call visible, and the router tally 5 accept, 4 fallback, 1 reject, which routes yellow

The Model: Qwen2.5-14B on the Laptop#

The model runs locally with mlx_lm.server, which speaks the OpenAI chat API, so the backend uses the plain openai client pointed at http://127.0.0.1:8080/v1. The app never loads MLX itself — MLX needs Metal, and the API also has to run in a Linux container.

I picked mlx-community/Qwen2.5-14B-Instruct-8bit for a boring reason: it is an instruct model, already quantized for MLX, and it copies text faithfully into JSON. The machine is an M4 Pro with 24 GB. The 8-bit weights take about 15 GB of that. The vision model and the reasoning distillations I also had on disk are the wrong tools for "copy this sentence exactly and put it in a string."

Here is what that costs, from two live runs on the same machine:

Step	Demo NDA (Bluefin)	Residuals sample (Oakline)
Read	< 1 ms	< 1 ms
Extract (model)	50.9 s	63 s
Citation gate	1 ms	3 ms
Playbook rules	< 1 ms	< 1 ms
Draft comments (model)	~10 s, 5 comments	3.7 s, 1 comment
Route	< 1 ms	3 ms
Route	Yellow	Red

Almost all the wall-clock time is the model. The whole decision — ten quotes checked, ten rules applied, one route — takes a few milliseconds. That asymmetry is the argument of the app, and it's what the trace view ended up showing.

The 11-Second Bug#

Back to the person who left after eleven seconds. There were two bugs hiding in that one moment.

Bug one: the review lived inside the HTTP request. The endpoint started the review as an asyncio task and streamed its progress. When the client disconnected, a finally block cancelled the task. That is a reasonable default for a chat reply. It is the wrong default for a document review that someone will want to come back to.

Bug two: cancellation left the matter stuck. The review's error handler was except Exception. In Python 3.8+, asyncio.CancelledError is a BaseException, not an Exception, so the handler that resets the status never ran. The matter stayed running with no route, and opening it showed an empty page.

The fix for the second bug is one line and a comment:

except BaseException as exc:
    # BaseException, so a cancelled task does not leave the matter stuck on running.
    failed = store.get(matter_id)
    failed.status = "needs_review" if failed.model_route else "new"
    ...
    raise

The fix for the first bug changed the architecture.

A review is a run with an event log#

A review is now a Run owned by a small in-process hub, not by the request that started it. Every step appends an event to the run's log. Any number of readers can replay the log from the start and then follow it live.

class Run:
    def emit(self, kind: str, **data: Any) -> None:
        self._flush_delta()
        self.events.append({"seq": len(self.events), "t": self.elapsed_ms(), "type": kind, **data})
        self._wake()

    async def delta(self, call: str, text: str) -> None:
        """Model tokens arrive one at a time. Batch them so the log stays small."""
        ...

Two endpoints read it:

POST /api/matters/{id}/review starts a run, or attaches to the one already in flight, and streams it.
GET /api/matters/{id}/events?after=N replays the log after sequence N, then follows it.

Closing the tab now only closes a reader. The run keeps going. Reopening the matter, or pressing F5 halfway through, replays everything that already happened and picks up the live stream where it is.

async def follow(self, run: Run, after: int = -1) -> AsyncIterator[dict[str, Any]]:
    index = after + 1
    # Not logged: tells a reader who joins late how long the run has been going.
    yield {"type": "clock", "t": run.elapsed_ms(), "done": run.done}
    while True:
        while index < len(run.events):
            yield run.events[index]
            index += 1
        if run.done:
            return
        try:
            await asyncio.wait_for(run._changed.wait(), timeout=KEEPALIVE_SECONDS)
        except TimeoutError:
            yield {"type": "keepalive"}

The clock frame came from a bug I found while testing the replay. After a reload, every timer on the page restarted from zero, because the browser was timing from when it joined. One unlogged frame with the server's elapsed time fixed that: the client sets clockStart = Date.now() - t, and a stage that has been running for 56 seconds says 56 seconds.

It is still Server-Sent Events, not WebSockets. Data only flows one way, and SSE over fetch handles POST, reconnect-by-replay, and proxies without a connection lifecycle to manage.

Events carry data, not just step names#

The old stream said {"type": "progress", "step": "extracting"} and nothing more. The new vocabulary carries what each step actually produced:

run_started   mode, model, the ten playbook positions
progress      step, lane (model | python), status, message, ms
model_call    extract | draft, start | end, tokens, ms
model_delta   a batch of streamed tokens
citation      position, excerpt, found: true | false | null (absent)
rule          position, rule name, fields, verdict, rationale
comment       position, text, source: model | playbook
route         route, reasons, tally
result        the final matter

The frontend is one reducer over this vocabulary. Replaying a finished run and following a live one go through the same code path.

Designing the Trace#

The trace has four parts. Each exists to answer one question someone asked while watching a demo.

"Is it doing anything?" A swimlane with two rows, Model — proposes and Python — decides. Each step sits in its lane with a live timer. The running step pulses. You can see at a glance that the purple lane is where the time goes.

"What is the model actually saying?" A console that streams the model's raw output token by token. Before the first token, it says what is happening instead of staying blank: "Reading the contract. The first token comes after the model has read the whole prompt." Prefill on a 4,000-character contract is the slowest silent stretch of the whole run, and one honest sentence covers it.

"How did it decide?" A table with one row per playbook position: did the quote pass the gate, which Python rule ran with which fields, and what came out. The rule column is the literal call, term(years=4, perpetual=false, trade_secret_only=false) → FALLBACK, because showing the function call makes the point better than any caption. Rows fade in as rule events arrive, and they arrive in the same millisecond.

"Why yellow?" A router strip: the tally of accept, fallback, reject, and ungrounded, an arrow, and the route chip, with the routing rule written underneath in one sentence. Then the result banner, the contract with the quote highlighted, and one card per position.

The result desk: a yellow banner with the five reasons, the contract text, and a card for each playbook position

Fixture mode replays a recorded extraction through the same trace in about three seconds, and the badge says Replay · model not called. That keeps the demo honest when the laptop is on battery and the model is off.

What Counsel Can and Cannot Do#

A lawyer can override any verdict, with a note. The route is recomputed from the new verdicts, and the banner says so. But the matter also keeps model_route, the route the system produced, and that field never moves. Approving a yellow or red matter is a separate action, approve with exceptions, which needs a note and does not turn the route green.

That split matters for measurement. The eval compares model_route against eight hand-labeled contracts and reports false-green: contracts the system painted green that were not green. On the recorded fixture set that number is 0/8. The number is only meaningful because a human override cannot quietly improve it.

What This Leaves Out#

This is a local demo, not legal advice, and it is deliberately small:

One playbook, vendor NDAs. An order form is out of playbook on purpose.
No auth and no database. Matters and traces live in memory.
No Word redlines. Comments are text, not tracked changes.
The gate checks the quote, not the field. The remaining risk in live mode is a real quote paired with a field the quote does not support — the model quotes the four-year clause and says years: 3. The gate stops invented quotes. It does not yet check that the field follows from the quote. That is the next thing I would build: a second, deterministic pass that re-derives the simple fields (years, jurisdiction) from the verified excerpt and flags a disagreement.

Takeaways#

Give the model a job with a checkable output. A verbatim quote is checkable in four lines of Python. A risk score is not checkable at all.
Put policy in code the lawyer can read. The rule registry was right about compelled disclosure when I was not.
A long agent step needs a run, not a request. Background task, append-only event log, replay from any sequence number. Closing a tab should never cancel work.
Show which lane did what. Fifty seconds of model and three milliseconds of Python, side by side, explains the architecture better than a diagram.

The full code, including the demo PDF, the eight labeled samples, and the 45 tests that run without a model, is at github.com/MinhQuanBuiSco/Agent-workflows.