How AI obligation extraction works — and where it fails

What the task actually is

An obligation is a commitment: a party must do something, by some time, possibly if some condition holds. "The Supplier shall deliver the Monthly Report within ten Business Days of each Month End" is one. So is "the Customer shall not disclose Confidential Information for three years following termination", which is a negative obligation and behaves differently.

The output people want is a register: obligation, owner, counterparty, trigger, due date, source clause. That last field matters more than it looks. An extracted obligation without a clause reference is an assertion nobody can check, and the first thing a reviewer does is check.

The pipeline, honestly described

Under any vendor's framing, the same steps have to happen:

• Get text. Native .docx or a text-layer PDF is straightforward. A scanned document needs OCR, and everything downstream inherits its errors.
• Recover structure. Clause numbering, headings, schedules and tables carry meaning. A contract flattened into one text blob loses the hierarchy that tells you clause 9.2 sits under Confidentiality.
• Segment into clauses. Obligations are scoped to clauses, and getting the boundaries wrong splits one duty into two or merges two into one.
• Classify. Which segments create an obligation at all? Recitals, definitions and governing-law clauses do not, and a naive model tags them anyway.
• Extract the fields. Who is bound, what is owed, what triggers it, when is it due.
• Normalise. "Within ten Business Days of each Month End" has to become something a calendar can hold, which requires knowing the Effective Date, the business-day convention and the month-end definition.
• Review. A human confirms or corrects, and the corrections are what make the register trustworthy.

Where it fails

These are not edge cases. On a real portfolio you will hit most of them in the first hundred contracts.

• Relative dates with no anchor. "Within 30 days of termination" has no absolute due date until termination happens. A system that writes a date anyway is inventing one; a system that drops the obligation has lost it. The right behaviour is to store it as conditional and wake it on the trigger.
• Defined terms. The obligation is in clause 9, the meaning of "Deliverable" is in clause 1, and the actual list is in Schedule 3. Extraction that reads clause 9 alone produces something technically accurate and operationally useless.
• Annexures, SOWs and side letters. A large share of real obligations live outside the main agreement. If the pipeline only ingests the executed master, the register is confidently incomplete — the worst failure, because it looks complete.
• Amendments. Contract three supersedes half of contract one. Unless amendments are linked to their parent and applied in order, the register will carry obligations that were renegotiated away two years ago.
• Negative and standing obligations. Non-compete, exclusivity, non-solicit: no due date, no deliverable, and yet they are the ones that cause disputes. Systems tuned to find dated deliverables under-report them badly.
• Deliberate vagueness. Sometimes the parties could not agree, so the drafting is soft on purpose. "Commercially reasonable efforts" is not sloppy writing, it is a negotiated outcome. AI will extract a crisp obligation from language chosen precisely because it was not crisp.
• OCR on bad scans. Stamped, signed, photocopied and re-scanned pages are common in Indian archives. A misread digit in a date or an amount propagates silently into the register.
• Precision against recall. Missing an obligation is worse than raising a false one — but a list with too many false positives stops being reviewed, and an unreviewed register is no better than no register. Tuning this is a product decision, and you should ask which way a vendor has tuned it.

What good practice looks like

Treat the first extraction as a draft register, not an answer. Budget review time proportionate to contract value: every obligation on the top twenty agreements gets checked by a person, the long tail gets sampled.

Keep the clause reference on every row, so a reviewer can verify in seconds rather than re-reading the contract. Keep the extracted text too, not just the normalised field, because that is what makes a disagreement resolvable.

Make conditional obligations first-class. Anything anchored to a future event should be stored as conditional and activated when the event is recorded, rather than being forced into a date field or dropped.

Re-extract after amendments rather than patching the register by hand. A register maintained manually diverges from the contracts within a year, which is the problem you bought the software to solve.

How to evaluate a vendor's claim

Accuracy percentages quoted in a deck are close to meaningless, because they depend entirely on the corpus. The way to find out is to make the test adversarial:

• Give them your worst contracts, not your best. A scanned 2016 MSA with handwritten amendments tells you more than a clean template.
• Include a contract with a side letter and one with two amendments, and check whether superseded obligations are correctly suppressed.
• Count what was missed, not just what was found. Have someone extract twenty contracts by hand first and compare. Recall is the number vendors quote least and you need most.
• Ask what happens to an obligation with no date. The answer tells you whether conditional obligations are modelled or discarded.
• Ask to see the clause reference and source text on every extracted row. If it is not there, the register cannot be audited.

Why this is still worth doing

Everything above is a reason to be sceptical of accuracy claims, not a reason to keep doing it manually. The realistic comparison is not AI extraction against a perfect register. It is AI extraction against the spreadsheet somebody stopped updating in 2023, or against no register at all.

A draft register produced in an afternoon and corrected over a fortnight is a categorically different starting point from a blank page, and the corrections compound: the reviewed rows become the reference the next extraction is measured against.