Robotics Training Data Types: Sensors, Actions & Metadata

Robotics Training Data Has 5 Necessary Parts

Source-backed context[1] Open X-Embodiment dataset and RT-X action structure[2] LeRobotDataset v3 multimodal format[3] DROID robot-manipulation dataset documentation

Robotics training data types are often presented as a hardware list: RGB, depth, IMU, LiDAR, force, tactile, and audio. That list is incomplete. Sensors primarily describe what a person or robot observed and how a system moved. Learning a behavior may also require the robot’s internal state, the action or command applied, the resulting change, and enough timing and validity information to connect those elements correctly.

A practical robotics example therefore combines five parts: instruction, observation, robot state, action, and result. Open X-Embodiment represents robot data as episodes and documents an example observation space containing an RGB image and task string plus a seven-dimensional action. LeRobotDataset exposes images, observation state, actions, timestamps, tasks, and episode metadata. DROID similarly distributes multi-camera recordings alongside action and proprioception trajectories. The exact fields vary, but the behavior-learning contract is consistent.

• Instruction or goal: what the system is expected to accomplish
• Observation: what cameras and other sensors make available to the model
• Robot or actor state: the configuration and motion of the body performing the task
• Action: the command, control target, trajectory, or intervention applied
• Result: the next state, object change, reward, success, failure, or safety outcome

1. Vision and Spatial Observation Data

Source-backed context[2] LeRobotDataset v3 multimodal format[3] DROID robot-manipulation dataset documentation[4] Project Aria multimodal data formats[6] NVIDIA Isaac Sim sensor documentation

Vision is the most common observation modality because it records objects, surfaces, tools, people, workspace conditions, and visible task progress. A robotics dataset may contain monocular RGB, stereo pairs, wrist cameras, head or chest cameras, external views, depth maps, infrared or thermal frames, event-camera streams, semantic masks, optical flow, point clouds, or reconstructed geometry.

Viewpoint is part of the data specification. An egocentric or wrist-mounted camera can preserve local hand-object activity while an external view can preserve body motion and workspace geometry. Depth and stereo can make distance and shape more explicit. LiDAR, radar, or ultrasonic sensing may be more appropriate for navigation, range, or difficult visual conditions. More views are useful only when their contribution is defined and their clocks and coordinate frames can be aligned.

• RGB and stereo video: appearance, objects, tools, state changes, and visual feedback
• Depth and RGB-D: surface distance, free space, occlusion, and 3D structure
• Point clouds and LiDAR: geometry, mapping, localization, and obstacle structure
• Thermal, infrared, radar, and event sensing: conditions ordinary RGB may not represent well
• Segmentation, optical flow, surface normals, and scene reconstruction: derived spatial supervision

2. IMU, Motion, and Localization Data

Source-backed context[4] Project Aria multimodal data formats[8] ROS 2 IMU message definition

An inertial measurement unit is a sensor package rather than a single measurement. A typical IMU contains a three-axis accelerometer and a three-axis gyroscope; some systems also combine magnetometer readings. The accelerometer measures specific force, including the effect of gravity. The gyroscope measures angular velocity. A magnetometer measures the local magnetic field and can contribute heading information, although indoor metal and electronics can disturb it.

For an egocentric camera, the IMU acts like an inner ear: RGB records visible appearance while high-rate inertial samples record how the device accelerates and rotates between frames. The streams can support visual-inertial odometry, orientation estimation, motion compensation, stabilization, localization, and motion-aware segmentation. They do not automatically provide ground-truth pose. Accelerometer bias, gravity, gyroscope drift, and integration error must be handled by a documented estimation or fusion method.

Other motion and localization streams can include GNSS or GPS, barometric pressure, wheel odometry, visual odometry, SLAM trajectories, motion capture, linear and angular velocity, and device-to-world pose. Every vector needs units, axes, coordinate frames, timestamps, calibration, uncertainty, and a measured-versus-estimated designation.

• Accelerometer: X/Y/Z specific force, commonly expressed in metres per second squared
• Gyroscope: X/Y/Z angular velocity, commonly expressed in radians per second
• Magnetometer: magnetic-field vector used as one possible heading reference
• Pose and trajectory: position and orientation produced by sensing, fusion, or external reference
• Odometry, GNSS, barometer, and SLAM: complementary location and motion evidence

3. Robot State and Proprioceptive Data

Source-backed context[2] LeRobotDataset v3 multimodal format[3] DROID robot-manipulation dataset documentation

Proprioceptive data describes the robot’s internal physical state: where its joints are, how they are moving, what configuration the end effector has reached, and what the actuators report. It is the machine equivalent of body awareness. LeRobot separates low-dimensional, high-frequency states and actions from camera video, while DROID includes proprioception trajectories alongside its visual recordings.

Common fields include joint position, velocity, acceleration, effort or torque, motor current, encoder values, end-effector position and orientation, gripper opening, gripper force, mobile-base velocity, and robot pose. Operational state such as controller mode, actuator temperature, battery condition, saturation, collision, emergency stop, and hardware-fault flags may also be important for filtering or safety analysis.

Names alone are insufficient. A joint vector needs the robot model, joint order, units, limits, zero convention, coordinate frames, control rate, and missing-value policy. Cross-embodiment datasets also need to state which fields are native to each robot and which were normalized or mapped into a shared representation.

4. Robot Actions and Control Data

Source-backed context[1] Open X-Embodiment dataset and RT-X action structure[2] LeRobotDataset v3 multimodal format[3] DROID robot-manipulation dataset documentation

Robot action data records what command was issued rather than merely what the robot looked like afterward. Depending on the controller, an action may be a target joint position, joint velocity, torque, Cartesian end-effector pose, relative pose change, gripper command, base velocity, whole-body command, or a higher-level skill invocation. The representation must identify whether values are absolute, relative, velocity-based, normalized, or expressed in another control convention.

This is the line between observation learning and direct policy supervision. Human egocentric video can teach task structure, object use, language grounding, affordances, and state changes. It does not inherently contain the executable commands for a particular robot. Behavior cloning or VLA policy training usually needs robot-native actions, teleoperation logs, retargeted trajectories, simulation controls, or another validated translation layer.

• Command value, action space, unit, coordinate frame, and control convention
• Command timestamp, control frequency, duration, latency, and execution status
• Teleoperator input, policy prediction, safety override, and human intervention
• Commanded action versus measured robot response
• Retargeting or normalization method when actions cross embodiments

5. Touch, Force, and Physical-Interaction Data

Source-backed context[5] MIT ActionSense multimodal human-activity dataset[6] NVIDIA Isaac Sim sensor documentation

Vision may show that a gripper reached an object without revealing whether the grasp was secure, slipping, overloaded, or making contact at the intended surface. Force, torque, pressure, tactile, vibration, and contact measurements add evidence about the physical interaction itself. They are particularly relevant to insertion, assembly, deformable objects, surface following, dexterous manipulation, and safety-sensitive contact.

Possible streams include wrist force-torque vectors, joint torque, fingertip pressure, tactile images or arrays, binary contact events, collision flags, slip estimates, vibration, temperature, deformation, and estimated object weight. ActionSense demonstrates how wearable tactile arrays and muscle sensing can complement first-person vision and inertial body tracking in human demonstrations. Simulators can also generate contact and effort signals, but synthetic values should remain clearly identified as simulated rather than measured in the physical world.

6. Human Demonstration and Wearable Data

Source-backed context[4] Project Aria multimodal data formats[5] MIT ActionSense multimodal human-activity dataset

Human demonstrations can broaden task, environment, object, strategy, language, mistake, and recovery coverage before robot-native collection is practical at the same scale. Egocentric RGB, synchronized external cameras, wearable IMUs, hand tracking, full-body motion capture, gaze, audio, narration, finger sensors, tactile gloves, and electromyography can describe different parts of the demonstration.

Project Aria illustrates a wearable data stack containing RGB and scene cameras, IMUs, microphones, eye tracking, barometer, magnetometer, GNSS, calibration, and derived trajectories or point clouds. ActionSense combines first-person eye tracking, body-worn inertial sensors, finger tracking, forearm muscle activity, tactile gloves, external cameras, depth, and audio. These examples show the possible design space, not a default shopping list.

Human motion still differs from robot embodiment. A useful program states whether the human data supports visual pretraining, task segmentation, language grounding, object-state prediction, pose or trajectory estimation, simulation reconstruction, retargeting, reward learning, or joint training with robot episodes. The transfer hypothesis determines which wearable signals are worth collecting.

7. Language, Task, Annotation, and Outcome Data

Source-backed context[1] Open X-Embodiment dataset and RT-X action structure[2] LeRobotDataset v3 multimodal format

A multimodal robotics dataset also needs semantic context. Language instructions describe the goal. Task and subtask labels describe temporal structure. Object and action annotations identify what happened. Outcome fields say whether the objective succeeded, failed, was interrupted, required intervention, or reached only part of the intended state.

Useful supervision can include task ID, natural-language instruction, episode and step boundaries, action segments, object identities, hand and body keypoints, contact, grasp state, affordances, object-state changes, preconditions, subgoals, reward, completion, failure reason, recovery, correction, and reviewer confidence. Outcome, recording quality, privacy eligibility, and release approval should remain separate: an unsuccessful task can be valuable evidence, while a completed task can still be unusable because the critical action was hidden or the rights are insufficient.

8. Object, Environment, and World Metadata

Source-backed context[3] DROID robot-manipulation dataset documentation[6] NVIDIA Isaac Sim sensor documentation

Robots operate in a physical context that may need explicit representation. Object metadata can include identity, category, dimensions, weight, material, articulation, CAD mesh, 6D pose, affordances, initial state, final state, and relationships to other objects. Environment records can describe workspace layout, lighting, clutter, surface, obstacles, safety zones, maps, scene graphs, weather, geography at an approved granularity, and controlled variations.

Some of these fields are directly measured, some are configured by the protocol, and others are annotated or estimated. Preserve the source of each value. A scanned mesh, manufacturer specification, model-estimated pose, and human-reviewed object label are not equivalent forms of evidence even when they appear in the same training example.

9. Synchronization, Calibration, and Coordinate Frames

Source-backed context[2] LeRobotDataset v3 multimodal format[4] Project Aria multimodal data formats

Multimodal does not mean multiple files from roughly the same session. Images, depth, audio, IMU, gaze, robot state, actions, force, and external cameras must resolve onto a trustworthy episode timeline. The delivery should define timestamp units, time domains, clock sources, exposure timing, offsets, drift, resets, gaps, interpolation, and measured synchronization error.

Calibration gives geometric meaning to the streams. Relevant records can include camera intrinsics, lens distortion, rolling-shutter timing, sensor extrinsics, robot kinematics, camera-to-IMU transforms, camera-to-robot transforms, axis order, handedness, quaternion convention, transform direction, calibration version, residual error, and validity interval. A position or acceleration vector without a named frame and unit is not safely reusable.

Different rates are normal. Video may arrive at tens of frames per second while IMU, force, state, or control measurements arrive at much higher rates. The schema should state how a training sample selects, windows, integrates, interpolates, or masks those asynchronous measurements without hiding gaps.

10. Dataset Governance, Rights, Quality, and Lineage

Source-backed context[7] MLCommons Croissant 1.1 metadata specification

Robotics dataset metadata must also explain whether an episode may be used, not only how it can be parsed. Release records can include capture authority, contributor consent, permitted use, license, commercial-training status, redistribution restrictions, privacy review, redaction, retention, deletion, security classification, reviewer decision, and buyer-specific eligibility.

Lineage connects raw evidence to every derived asset, label, correction, shard, split, and export. Croissant 1.1 provides machine-readable mechanisms for dataset structure, provenance, annotations, and usage conditions. A delivery still needs the underlying operational records: stable identifiers, checksums, transformation versions, known limitations, QA results, release approval, and a history of corrections or withdrawals.

• Dataset card, schema, data dictionary, manifest, checksums, and version history
• Per-modality validation, missing-data masks, uncertainty, and rejection reasons
• Provenance from original capture through processing, annotation, split, and export
• Consent, license, permitted use, privacy, retention, and distribution status
• Train, validation, and test splits designed around the claimed generalization target

Measured, Derived, Annotated, and Synthetic Signals Are Different

Source-backed context[4] Project Aria multimodal data formats[5] MIT ActionSense multimodal human-activity dataset[7] MLCommons Croissant 1.1 metadata specification

A clean schema should distinguish four origins. Measured data comes directly from a sensor or controller. Derived data is produced by an algorithm, such as visual-inertial pose, depth estimation, optical flow, gaze projection, or action retargeting. Annotated data is an interpretation produced or reviewed by people or models. Synthetic data is generated in simulation or another artificial process.

Derived and synthetic signals can be extremely useful, but they are not automatically ground truth. Store the input assets, method, model or simulator version, configuration, coordinate conventions, confidence, uncertainty, and validation evidence. Retain raw data where policy permits so improved processing can create a new version without erasing the original observation.

Recommended Modalities by Robotics Use Case

Source-backed context[1] Open X-Embodiment dataset and RT-X action structure[2] LeRobotDataset v3 multimodal format[3] DROID robot-manipulation dataset documentation[5] MIT ActionSense multimodal human-activity dataset

The right robotics data modalities follow the learning objective and deployment interface. More sensors are not automatically better. Every added stream creates hardware, synchronization, calibration, storage, privacy, review, and loader work. Add it when it resolves an ambiguity the model or evaluation actually faces.

• VLA policy learning: language instruction, RGB or RGB-D, robot state, action trajectory, timestamps, and outcome
• Manipulation: wrist and scene vision, object and hand state, proprioception, gripper command, contact or force, and object-state change
• Navigation: cameras, depth or LiDAR, IMU, odometry, pose or map, commands, route context, collision, and arrival outcome
• Humanoid and whole-body control: multi-view vision, joint and base state, motion capture or pose, balance and contact signals, whole-body actions, and safety events
• Human-to-robot learning: egocentric video, hand-object activity, task language, motion or pose, outcome, explicit translation method, and robot-native validation
• World models: synchronized temporal observations, actions, state transitions, valid masks, long-horizon context, and future-state targets
• Perception-only training: the required sensor observations, labels, calibration, and evaluation slices—without pretending that action supervision is present

Specify the Episode Contract Before Choosing Sensors

Implementation guidanceEGXO guidance for translating the research into a project specification.

Start with one representative training example and write down what the model receives, what it predicts, which measurements establish validity, and how success is judged. Then select the minimum sensor and annotation stack capable of producing that example across the required tasks, environments, objects, people, devices, and failure conditions.

A buyer should inspect complete synchronized episodes rather than a modality checklist. Confirm that observations, robot state, actions, task context, and outcomes replay together in the intended loader. Review raw, derived, rejected, repaired, and ambiguous examples. Measure clock alignment, calibration, critical-action visibility, missing signals, rights eligibility, and ingest success before collection scales.

• Which learning target does each modality support?
• Is every field measured, derived, annotated, or synthetic—and is that origin explicit?
• Are timestamps, units, axes, coordinate frames, calibration, and uncertainty documented?
• Can a complete episode be reconstructed without filename assumptions or manual repair?
• Do human demonstrations connect to a defined robot action or evaluation path?
• Are outcomes, quality, privacy, rights, and release approval represented separately?
• Has a representative release passed through the real preprocessing and training loader?