Patterns
Keep code in control while the model judges narrow questions.
Start with a small decision and an explicit fallback. The model judges meaning; deterministic code checks eligibility, authorization, budgets, and side effects.
Useful patterns include confidence routing, a human-review queue, a companion true/false fit check, and bounded concurrent requests. None requires a second inference endpoint or hidden batch operation.
Evaluate the entire workflow: the chosen answer, the fraction sent to review, whether the review queue is staffed, and the effect of errors on users. Record the pinned model, rubric, threshold, and final human decision when your data policy permits it.
- Confidence routing: use the effective server policy.
- Human escalation: persist and resolve uncertain cases.
- Companion check: test whether an offered category fits.
- Batching: process independent cases with bounded concurrency.