The AI judge

Running soft rules — inferential checks a regular expression cannot make.

Soft rules are the inferential tier: judgements about whether a sentence means something, which no pattern can settle. They are evaluated by the AI judge, and only when you ask.

lawlint --judge draft.md

Requires a configured model — see Setting up AI features. Without one, --judge exits with guidance rather than running.

Choosing a model

lawlint --judge draft.md                    # the model from lawlint init
lawlint --judge=anthropic:<model> draft.md  # override for this run

Bare --judge uses the configured model. --judge=MODEL overrides it, taking the same specs as init:

To run the judge on every lint without the flag, set judge.enabled in .lawlint/config.json.

How a document is judged

The document is cut into sections, and each soft rule is judged against them. How that is batched follows the backend, so the defaults track the model rather than a fixed constant:

Each rule sees the whole document, so a rule can reason across sections. Every rule request over one document shares a cacheable prompt prefix, which is what keeps one-request-per-rule affordable.

If you are pointing lawlint at a small self-hosted model, lower contextChars and set perRule: false — a small model does better on short sections with the rubrics bundled.

Every one of those is overridable — see judge configuration. The one to know is contextChars, which is also the cache granule: bigger sections mean fewer requests, and also that an edit anywhere in a section re-runs that whole section.

When the judge fails

A section that fails is skipped, not fatal — the run continues on the deterministic tiers and tells you what it lost:

lawlint: warning: judge failed on 6 of 6 chunks; those chunks used tiers 1-2 only
lawlint: warning: first failure: malformed judge response: model generated no
  output before hitting its token cap — raise `judge.maxTokens`

  AI rules  2 run  0 findings  (incomplete: 6 of 6 sections failed)
  Human-likeness  100/100  (AI review incomplete)

Failed on every section, “no output before hitting its token cap”. A reasoning model spent its whole generation budget on hidden thinking and emitted nothing. Raise the budget:

// .lawlint/config.json
{ "judge": { "maxTokens": 32768 } }

Only tokens actually generated are billed, so headroom costs nothing on models that do not think first.

Failed on every section, some other message. The first failure’s cause is printed verbatim after the count. A 401/403 is a credential problem (lawlint init, or the *_API_KEY environment variable); a 404 usually means the deployment name in your model spec does not exist.

Failed on a few sections only. Usually transient — rate limits or a dropped connection. Each request is already retried once before its section is skipped; re-running picks up the sections that succeeded from cache and only re-sends the rest. Lowering judge.concurrency helps if you are hitting provider rate limits.

What it costs you

Judging sends the text being linted to the configured provider. That is the trade, stated plainly:

Soft-rule severity is capped at warning or suggestion, so an inferential judgement never fails a build on its own. Each soft finding also carries a confidence value in JSON output.

Judging versus revising

Two different things use a model, in opposite directions:

--format prompt needs no configuration and sends nothing anywhere — it just prints a brief for you to paste. See Fixes and tracked changes.

Writing soft rules

Soft rules use their Markdown body as the judge’s rubric, and require at least three flag examples and three pass examples. See Authoring rules.