Robot Action Data

01

Action Data Is the Control Side of an Episode

Source-backed context[1] Open X-Embodiment action standardization[2] LeRobotDataset v3 episode and feature format

Robot learning examples often pair an observation with the action that followed it. The observation can include images, depth, robot state, or language. The action describes the target sent into a controller or an abstract skill selected by a higher-level policy. The definition must follow the actual control interface used during collection and deployment.

Actions are not interchangeable across robots. A seven-joint arm, a mobile manipulator, and a humanoid may represent the same task with different dimensions and constraints. Even two similar arms can use different joint ordering, gripper conventions, scaling, or control frequencies. The schema has to make these assumptions explicit.

02

Common Robot Action Spaces

Source-backed context[1] Open X-Embodiment action standardization[3] ROS 2 JointTrajectory message

Joint-space actions specify targets for individual joints, such as position, velocity, or torque. Cartesian actions specify a desired end-effector pose or pose change. Mobile robots may use linear and angular velocity. Other systems use waypoint sequences, discrete skills, or tokenized action chunks that a lower-level controller converts into motion.

The choice controls what the model learns and how portable the dataset is. Low-level actions preserve control detail but bind data closely to the robot and controller. Higher-level actions can transfer more easily across embodiments but rely on another system to execute the command safely and consistently.

• Joint position, velocity, acceleration, or torque targets
• End-effector position and orientation targets
• Delta actions relative to the current pose
• Gripper position, width, force, or open-close state
• Base velocity, waypoint, or discrete skill commands

03

Commanded Action and Executed Motion Are Different

Source-backed context[2] LeRobotDataset v3 episode and feature format[4] DROID action and observation dataset

A command may be clipped by safety limits, modified by a controller, delayed by communication, or interrupted by contact. Recording only the requested action can hide what the robot actually did. High-value datasets preserve commanded action alongside measured robot state and, when relevant, controller status, safety interventions, or execution error.

This distinction is essential for learning from teleoperation and for debugging policy behavior. If a model is trained against commands that were never executed, it may learn an incorrect relationship between observations and motion. Timestamped state and action streams allow the preprocessing pipeline to test that relationship.

04

Action Data Failure Modes

Source-backed context[1] Open X-Embodiment action standardization[2] LeRobotDataset v3 episode and feature format[3] ROS 2 JointTrajectory message

The most damaging defects are semantic, not cosmetic. A vector can load successfully while its joints are in the wrong order, angles use unexpected units, the reference frame changes between sessions, or actions are shifted by one timestep. Normalization applied during export can also make raw values impossible to reconstruct.

Validation should replay representative episodes through the intended loader and verify dimensions, ranges, timing, controller semantics, valid masks, and terminal conditions. Raw and normalized values should remain traceable through the release lineage.

• Missing action schema or joint ordering
• Radians and degrees mixed across sessions
• Absolute and delta actions combined without a flag
• Command timestamps shifted relative to observations
• Safety clipping or failed execution omitted

05

What a Buyer Should Specify

Implementation guidanceEGXO guidance for translating the definition into a project specification.

Document the complete action contract before collection. Include the target robot, controller, action dimensions, units, axes, frames, frequency, normalization, terminal behavior, and execution evidence. If human demonstrations are included, identify exactly how human motion becomes or complements robot action targets.

• Command type and controller interface
• Action vector names, ordering, units, and valid ranges
• Absolute, relative, normalized, or tokenized representation
• Timestamp and observation-alignment rules
• Commanded versus measured execution fields
• Pilot replay through the production training loader