A historical procurement review can reveal whether an AI system understands the work your team actually does. It can also produce a misleading result if the system sees the answer before it makes the decision.
Imagine evaluating a late-delivery case using today's PO record. The record already contains the revised date your buyer negotiated last month. The system appears to know the right next step, but it has been given information that did not exist when the problem arose.
A useful replay reconstructs what was knowable at each step. It tests findings and proposed actions against that evidence, keeps uncertain cases visible, and does not confuse hypothetical actions with money recovered or orders rescued.
The method below is an offline evaluation: no supplier messages are sent and no operational records are changed. Any later live pilot needs its own scope and authorization.
Download the historical replay workbook (Excel) for the data manifest, event sequence, review sheet, and results log.
Choose a question small enough to answer
Start with a specific workflow. “Would the system identify incomplete acknowledgements and request the right missing information?” is testable. “How much could AI improve procurement?” leaves too much room for speculation.
Define the population, time window, trigger, permitted next actions, and expected result. For acknowledgement work, the population might be issued purchase orders with a defined supplier response deadline. For invoice discrepancies, it might be invoice lines connected to an agreement and a receipt record.
Write the decision the review will inform. It may be whether to run a limited live pilot, whether a workflow needs better source data first, or which supplier conventions need explicit instructions.
Do not require the replay to establish benefits it cannot observe. It can test whether a proposed follow-up is appropriate. It cannot prove how quickly a supplier would have replied to a message that was never sent.
Select cases without selecting only the problems
Identify the full eligible population before choosing examples. Record the count and selection method. Include ordinary completed orders, genuine unresolved issues, apparent discrepancies with legitimate explanations, and cases already handled by the team.
A sample made only from known claims tests issue handling but cannot measure unnecessary claims raised against clean invoices. A sample made only from successful historical recoveries excludes the cases where evidence or supplier acceptance was missing.
Stratify where the operation varies materially: supplier, document format, unit convention, order complexity, or workflow. Keep the strata in the report. If complex cases are deliberately overrepresented, describe the result as performance on that test mix rather than a production forecast.
Separate cases used to improve operating instructions from cases reserved for assessment. Keep a dated record of the instruction version used. A result after repeated case-specific coaching belongs in the development log.
Gather the records that connect the conversation to the order
Build a manifest rather than dropping an unexplained folder of attachments into the test. The reviewer needs to know what is present, what period it covers, and what is missing.
For each case, gather the relevant PO revisions, line and schedule identifiers, supplier messages, attachments, confirmations, receipts, invoices, contract terms, and approval records. Add supplier conventions only where there is evidence that the team knew them at the time.
Record each source's identifier, document date, received timestamp where available, version, and relationship to the case. Keep document date and availability date separate. A quote dated Monday but received Thursday could not have informed Tuesday's action.
Note whether an ERP export is a current snapshot or a historical version. A current snapshot may be useful for verifying the final result while being unsuitable as an input to an earlier decision.
Use the minimum access and fields needed for the evaluation. Assign an owner for the dataset, access, storage, and eventual retention decision according to your organization's rules. Keep credentials and unrelated personal correspondence out of the evaluation bundle.
Reconstruct a sequence of decision points
Turn each case into an ordered event stream. At a decision point, expose only records available by that point, then request a finding and proposed next action. Save the answer before revealing the next event.
Historical review
Reconstruct one order from start to finish.
When the delivery date changed.
What action was available at the time.
Whether the issue was actually resolved.
The review records proposed actions and identifies any outcomes supported by your records.
An illustrative acknowledgement case might run like this:
- Monday, 09:00: PO revision 1 is issued for 500 units required September 21. The response deadline is Wednesday at noon under the test's stated operating rule.
- Wednesday, 12:00: no acknowledgement is in the available records. The expected proposal is a targeted confirmation request, not a claim that the supplier has missed delivery.
- Thursday, 10:00: the supplier offers 300 units September 21 and 200 on September 25. The expected finding is a split proposal with a possible need-date gap.
- Thursday, 14:00: planning confirms that 400 units are required by September 21. This establishes a 100-unit gap against the proposed first delivery.
- Friday, 11:00: the buyer approves a revised plan after a separate supplier discussion. Only now is that approval available to the replay.
The system should not use Friday's approval to justify accepting Thursday's proposal. Nor should it infer a production shortage before the planning requirement becomes available.
Keep final outcomes out of the input packet
Store the answer sheet and final-outcome evidence separately from the material presented at each decision point. Check for indirect leakage in filenames, folder labels, summaries, and current ERP fields.
A file named “approved-credit-final” can reveal the result even when its body is hidden. A case selected under a “confirmed overcharges” heading tells the system that it is expected to find an overcharge. Neutral case IDs make the evaluation easier to interpret.
Where historical versions cannot be reconstructed, narrow the question. A current snapshot may support a document-reconciliation test but not a claim about what the system would have known earlier. Label that limitation in the results.
Ask a reviewer unfamiliar with the case to inspect a sample input packet. Their job is to look for missing context and accidentally revealed outcomes, not to judge the model's answer.
Write the reference judgment before reviewing the output
For each decision point, have the process owner record the supported finding, acceptable next actions, authority boundary, and evidence needed for closure. Allow more than one acceptable action where the workflow permits it.
Distinguish a fact from a judgment. “The supplier offered September 25” may be directly supported. “September 25 is acceptable” requires the relevant planning and approval context.
Record uncertain cases explicitly. If the signed agreement is missing, the correct reference may be “request agreement evidence” rather than “submit a price claim.” If two reviewers disagree, reconcile the difference or leave the disputed dimension unscored with a reason.
Historical buyer behavior is a comparison point, not unquestionable ground truth. The team may discover that a past action lacked documentation or left a record incomplete. Capture that as a process finding rather than changing the answer to favor either the system or the buyer.
Score findings, actions, and closure separately
For every scorable decision point, assess whether the system found the right issue, associated the right records, proposed an acceptable action, and respected the approval boundary. Also record unnecessary interventions on ordinary cases.
Report counts alongside rates. In an illustrative sample of 40 cases, six lack enough historical evidence for the chosen question. That leaves 34 scorable cases. If 12 of those contain an issue and the system correctly identifies nine, issue recall is 9/12, or 75%.
If it flags 11 cases in total, two are unnecessary flags and precision is 9/11, or 81.8%. The six unscorable cases belong in the report with their reasons; they do not become either successes or failures by convenience.
Then evaluate the proposed actions independently. A correctly identified discrepancy with an unsupported demand for a credit is a detection success and an action failure. Keep those results separate.
A replay can observe documented historical closure. It cannot attribute that closure to the tested system. Label outcomes as historical, proposed, or subsequently observed in a live pilot.
Inspect the errors before improving the instructions
Group failures by the work needed to correct them. Useful categories include missing source data, wrong record association, unit conversion, agreement interpretation, stale revision, authority error, and premature closure.
For each failure, record the source evidence, expected behavior, observed behavior, consequence, and proposed correction. Distinguish a reusable supplier convention from a hint that simply gives away a particular answer.
After revising instructions, test a held-out set. Preserve the original result and configuration in the report. Repeatedly testing the same example can demonstrate that a fix works for that example; it does not establish broader reliability.
Count assistance during evaluation. If a specialist supplied the applicable agreement, corrected the line mapping, or rewrote the follow-up, that work should appear in the effort record. Otherwise the review can hide the workload it was meant to measure.
Turn the findings into a limited live workflow
Choose a workflow with enough recurring volume, usable source records, and a clear authority boundary. The replay may show that incomplete acknowledgements are ready for a pilot while price disputes need a better agreement register.
Write down the pilot's population, systems, allowed actions, named approvers, review frequency, and pause conditions. Use the approval matrix to make the boundaries concrete. Define the finish of each case and how open work remains visible.
Establish a baseline using comparable work. In live operation, measure actual supplier responses, record updates, buyer interventions, and time to resolution. Those observations answer questions the historical replay cannot.
Mandel uses the context across sourcing and ordering work to carry out agreed actions and bring people the decisions that require them. A historical review should help you select where that is useful and what remains uncertain. The next step is a bounded evaluation, followed by evidence from the live workflow—not a universal savings estimate derived from a folder of old orders.
