System design answer
Question answering over long legal contracts
Long context with prompt caching for one contract, retrieval across many, strict citations to clauses, and lawyer review of risky answers.
The question“Design a tool where a legal team uploads contracts and asks questions such as "what is the termination notice period?"”
1Clarify0-5 min
Ask before you draw. These answers change the design.
- One contract at a time, or questions across hundreds?
- How long are contracts, and are they scanned?
- How exact must answers be, and who checks them?
- Confidentiality rules: may text leave the region?
2Requirements and numbers5-8 min
It must
- Upload and parse contracts with clause structure
- Answer with the exact clause cited
- Compare a clause across many contracts
And it must be
- Every answer cites a clause; no citation, no answer
- Documents stay in the firm's cloud region
- Answer under 10 s for one contract
Numbers to say out loud
- Contract size
- 40-120 pages, about 40,000-100,000 tokens
- Library
- 5,000 contracts
- Questions
- 2,000 a day
3High-level design8-12 min
- Upload → Parse with clause structure
- Parse with clause structure → Store clauses with numbers
- Question on one contract → fits the window → Whole contract, cached prefix
- Whole contract, cached prefix → Model
- Question across contracts → Retrieve matching clauses
- Retrieve matching clauses → Model
- Model → Answer citing clause numbers
- Answer citing clause numbers → Check quotes match
- Check quotes match → risky topic → Lawyer review
4Deep dives12-26 min
Long context or retrieval
For one contract, send it whole with the stable prefix cached, so follow-up questions are cheap and nothing is missed. Across the library, retrieve clauses by type (termination, liability) with metadata, then answer per contract in a table.
Citations
Clauses keep their numbers from parsing. The answer must quote the clause; code checks that the quote appears in that clause before showing it.
5Failures26-30 min
| What can go wrong | What stops it |
|---|---|
| Answer from the wrong clause | Quote check against the cited clause |
| Scanned contract with OCR errors | OCR confidence shown; low confidence goes to review |
| Confidential data leaves the region | Model in the firm's region through the gateway |
6Evals and monitoring30-33 min
- 200 questions over 20 contracts with lawyer-written answers
- Citation accuracy and quote match rate
7Trade-offs33-35 min
- Long context is simpler and more complete for one document but costs more; caching closes much of the gap
- Requiring citations means some refusals, which lawyers prefer to guesses
Follow-ups to expect
- How would you find every contract with an unusual liability cap?
- What if two clauses contradict each other?