Robot Regression Testing in Deployments: A Practical Guide

Regression testing is required whenever a change to the robot system could plausibly affect a behavior that was previously verified at acceptance. The category of changes that trigger regression testing consideration includes firmware updates, software updates, hardware component replacements, configuration parameter changes, and facility environment changes that affect the robot's operating conditions. Not every change in these categories requires regression testing: a firmware security patch that does not modify any functional behavior, or a configuration parameter change that adjusts only a non-acceptance-critical function, may require only a basic confirmation that the system starts and operates normally.

The decision about whether a specific change requires regression testing, and at what scope, is the output of the change impact assessment. Without a change impact assessment, the regression testing decision is made informally, which typically means either no regression testing at all - the change is applied and assumed to have no behavioral impact - or excessive regression testing that covers functions unrelated to the change.

Both extremes are inefficient; proportionate risk-based regression testing based on a documented impact assessment is the practice that manages the risk at reasonable cost.

The scope of a regression test set for a specific change should be proportionate to the change's likely functional impact and the consequence of a behavioral regression in the affected functions. For a minor configuration change affecting a single operational parameter, the regression scope may be limited to the acceptance criteria directly governed by that parameter. For a major software update that modifies the navigation stack, the regression scope should cover all navigation-related acceptance criteria.

For a hardware component replacement, the regression scope should cover the acceptance criteria that depend on the replaced component's functional contribution. Proportionality also applies to the depth of retesting within each included criterion: a criterion that involves a complex multi-scenario test set may be addressed in a regression context with a representative subset of the original scenarios, if the subset is selected to cover the scenarios most sensitive to the type of change being made.

The scope rationale should be documented explicitly, not implicit in the selection of tests, so that a reviewer can assess whether the scope decision was appropriate.

Regression test evidence must meet the same primary evidence standards as original acceptance test evidence: captured at the time of the test, specific to the test conditions, attributable to a specific date and tester. The additional requirement for regression test evidence is that it must be comparable to the original acceptance test evidence for the same criteria, so that a reviewer can assess whether the performance in the regression test is consistent with the performance in the original acceptance test.

If the original acceptance test used a specific sensor log format for a navigation accuracy criterion, the regression test should use the same format. If the original acceptance test captured video for a safety stop criterion, the regression test should capture video of the same scenario. Where the regression test uses exactly the same test protocol as the original acceptance test, comparison is straightforward.

Where the regression test uses a modified protocol, the modifications should be documented with a rationale, and the evidence should note any limitations on comparability that the modification introduces.

Facility environment changes - changes to the physical layout, the operational workflow, the ambient conditions, or the facility systems - are a frequently overlooked trigger for robot regression testing. A robot that was accepted in a specific facility configuration may exhibit different behavior after a facility reconfiguration, not because the robot system has changed but because the operating environment has changed in ways that affect the robot's performance or behavior.

A major racking reconfiguration changes navigation paths and may affect localization accuracy in specific zones. A change in the lighting system may affect camera-based perception. A change in the floor surface in a specific area may affect navigation accuracy or traction.

For each significant facility change, the deployment team should conduct an impact assessment for the robot's deployment, using the same approach as for robot system changes, to determine whether the facility change is likely to affect any of the robot's acceptance-verified behaviors. Where an impact is identified, regression testing of the affected criteria should be conducted in the updated facility environment.

Each regression test that produces passing results for its targeted criteria should be integrated into the deployment's acceptance record as a dated evidence update for those criteria, linked to the change event that triggered the regression test. This integration keeps the acceptance record current as a representation of the system's verified state: a customer or auditor reviewing the acceptance record after multiple change events and regression tests can see not only the original acceptance evidence but also the subsequent regression tests that confirmed the acceptance criteria were still met after each change.

Regression tests that produce failing results for any criteria should be treated as new acceptance issues requiring investigation and resolution, not simply as test failures that block the change. The acceptance record should not be updated for the change until the regression failures have been investigated and either resolved or formally accepted as new exceptions. The goal is an acceptance record that accurately represents the verified state of the system at all times, not one that records only the original acceptance and implicitly assumes that the system has remained in that state through subsequent changes.

• Determine whether each proposed change triggers regression testing based on the change impact assessment output
• Scope regression tests proportionately: targeted for minor changes, comprehensive for major changes
• Document the scope rationale explicitly, including criteria included for specific reasons and criteria excluded with explanation
• Use the same evidence capture formats as the original acceptance test for directly comparable results
• Document evidence comparability limitations where the regression test protocol differs from the original acceptance protocol
• Assess facility environment changes as potential regression testing triggers, not only robot system changes
• Integrate passing regression test results into the acceptance record as dated evidence updates linked to the change event