Evidence-Backed Robot Acceptance: Building a Defensible Acceptance Record

Defensible acceptance evidence has four characteristics. It is specific: it captures the exact conditions under which a test was performed, not a general description of the type of test. It is primary: it was captured at the time of the test, not reconstructed afterward from memory or secondary sources.

It is attributable: it can be traced to the specific test run, the specific date, and the specific operator who performed the test. It is complete: it covers the full test scenario required by the acceptance criterion, not a subset that was convenient to capture. Video evidence that shows a specific test scenario in its entirety, with visible robot behavior and environmental conditions, is more defensible than a technician's verbal confirmation that the test passed.

A sensor log that captures the robot's state through the full duration of a performance test is more defensible than a summary table extracted from the log. The level of evidence required for defensibility depends on the consequence of the acceptance decision: acceptance criteria with significant safety, contractual, or operational consequences require higher-quality evidence than criteria with low consequence.

Different types of acceptance criteria have different minimum evidence requirements based on their nature. Navigation and localization criteria require evidence of the robot's position accuracy across the full operational zone, typically captured through comparison of planned and actual trajectories over a representative sample of journeys. Safety stop criteria require evidence that the robot stops within specified distances when approaching obstacles or people, captured through controlled safety test scenarios with measured stopping distances.

Integration criteria require evidence that the robot correctly exchanges data with facility systems under representative load and error conditions. Performance criteria require evidence of the robot's throughput or cycle time over a statistical sample that is large enough to be representative of normal operational variability. Configuration criteria require evidence that the robot's configuration matches the specification, captured through direct query of the running configuration state.

For each criterion type, the acceptance plan should specify the minimum evidence standard before testing begins, so that the evidence captured during testing meets the standard without requiring additional testing after the fact.

Evidence collected during acceptance testing is only useful if it is organized in a way that allows a reviewer to quickly find the evidence for any specific criterion. An evidence organization structure that maps each piece of evidence to the criterion it addresses, the test scenario it was captured in, and the pass or fail determination it supports is more useful than an unorganized collection of logs and videos. The acceptance package presented at the customer review meeting should allow a reviewer to ask about any criterion and navigate directly to the supporting evidence.

Evidence packages that are organized by artifact type rather than by criterion - all videos in one folder, all logs in another - require the reviewer to search for the relevant evidence during the review meeting, which is slow and creates the impression that the evidence record is incomplete. Organizing evidence by criterion from the start of testing, rather than reorganizing at the end, is the practice that makes the acceptance review efficient.

Evidence gaps arise when testing cannot be completed as specified before the planned acceptance date, when test conditions cannot be reproduced for evidence capture, or when evidence was captured but was lost, corrupted, or in a format that is not accessible for review. Each type of gap requires a different response. Incomplete testing should be addressed by completing the tests before acceptance or by formally accepting the criterion as untested with a condition for subsequent testing.

Non-reproducible test conditions should be documented with an explanation of why the conditions could not be reproduced and what alternative evidence is available. Lost or inaccessible evidence should be documented as a gap with the steps taken to recover it and the conclusion about recoverability. In each case, the gap should be documented explicitly rather than silently omitted from the evidence record.

A criterion without evidence is not a met criterion; it is an unevidenced criterion that should be treated as open until evidence is obtained or an exception is formally approved.

The post-acceptance value of an evidence-backed acceptance record is most visible when disputes arise about whether the system was performing as specified at the time of acceptance. A customer who argues that a failure mode was present at acceptance can be answered with the specific evidence records that demonstrate the relevant acceptance criteria were tested and passed under the conditions specified in the acceptance plan.

A deploying team that is challenged about whether a specific configuration was verified at acceptance can reference the configuration acceptance record and the evidence of the tests that were performed in that configuration. An insurance or regulatory inquiry about the safety characteristics of the deployed system at the time of acceptance can be answered with the safety test evidence that was captured during acceptance testing.

Each of these responses is only possible if the evidence exists and is organized in a way that makes it findable and readable. The investment in evidence-backed acceptance is an investment in the long-term defensibility of the acceptance decision.

• Define the minimum evidence standard for each acceptance criterion type before testing begins
• Organize evidence from the start of testing by criterion, not by artifact type
• Capture evidence at the time of each test rather than reconstructing it afterward
• Verify that each piece of captured evidence meets the minimum standard before marking the criterion as met
• Document evidence gaps explicitly with the reason for the gap and the steps taken to address it
• Present evidence organized by criterion at the customer acceptance review meeting
• Archive the complete evidence record in a format accessible to all authorized parties after acceptance