Robotics trial and evidence record guide

1. State the question and the decision

Write the question the trial should answer and identify who needs the result. Checking a specified requirement and evaluating whether a system suits its intended user are related but different activities; NASA describes these as verification and validation. Identify which purpose applies before choosing measurements or arranging a demonstration.

• Link each trial objective to a requirement or an engineering uncertainty.
• Define the expected evidence and any pass or fail threshold in advance.
• Identify the decision the result will support and who can make it.

Reference: NASA Systems Engineering Handbook: Product Realization

2. Record what will actually be tested

Create a configuration list for the trial article, payload, software and supporting equipment. Include settings that can affect the result, along with any temporary changes. A trial label should let the team find the relevant drawings and release without relying on memory. Photograph the setup where that adds useful context.

• Record hardware, firmware, software and configuration revisions.
• Identify payload mass, mounting position and power arrangement.
• List instruments, measurement range and relevant calibration status.

3. Check readiness, roles and stopping conditions

Agree who operates the system, records evidence and supervises the trial area. Keep the stop instruction unambiguous. Review the applicable risk assessment, access permission and recovery arrangement before starting. ArduPilot-based vehicles include configurable failsafe responses, so record the reviewed settings and verify the required behaviour through an appropriate controlled check.

• Confirm boundaries, communications and access for recovery.
• List entry conditions and reasons to stop or postpone a run.
• Check that everyone understands their role and the agreed response to a fault.

Reference: ArduPilot Rover: Failsafes

4. Prepare and check the evidence capture

Choose records that answer the trial question: measurements, event notes, photographs, video or system logs. Align their time references and confirm that a short sample can be opened before the main run. ROS 2 rosbag2 can record and replay communication data; its documentation also describes quality-of-service settings and resource limits that can affect recording.

• Record time zone, clock source, file naming and available storage.
• Verify that the selected data streams and relevant metadata are present.
• Replay logs in an isolated analysis environment so recorded commands cannot drive live equipment.

Reference: ROS 2 rosbag2: Recording and Playback Documentation

5. Give every run its own record

Assign a run identifier before starting. Record conditions at the time, including any difference from the planned setup. Note events as they occur rather than reconstructing them from the most memorable video afterwards. Keep aborted, incomplete and unsuccessful runs in the record; they can explain the limits of an apparent improvement.

• Record the start and finish time, configuration and actual conditions.
• Separate measured values from operator observations.
• Log any change made between runs and its reason.

6. Review the results against the planned criteria

Compare the evidence with the criterion agreed before the trial. Describe uncertainty, missing data and conditions that restrict the conclusion. An unloaded run on one surface does not establish endurance with a different payload or on different ground. If a criterion changes, retain the original and explain why the new question needs another assessment.

• Use pass, fail, inconclusive or not performed where appropriate.
• Link anomalies to the affected run and evidence files.
• Record whether another run, design change or additional analysis is needed.

7. Keep a usable evidence package

Preserve the original records and identify any processed files separately. Add a short index explaining the question, configuration, result and next action. A checksum can help identify an unchanged file, but it does not establish whether the test itself was valid. Agree who may receive the evidence and remove unrelated personal or sensitive information from public extracts.

• Name an owner for each unresolved finding and its review date.
• State the limits of the result in the trial summary.
• Connect the evidence to the next development decision, rather than declaring the whole system proven.

Further reading

• NASA Systems Engineering Handbook: Product Realization
• ArduPilot Rover: Failsafes
• ROS 2 rosbag2: Recording and Playback Documentation