Egocentric Data Metadata: Types, Annotation, Schema, and QA

Metadata Is the Contract Around the Egocentric Data

Source-backed context[1] Project Aria data formats[4] LeRobotDataset v3 format[5] RLDS episode and step structure[6] MLCommons Croissant 1.1 metadata specification

Egocentric data metadata is the structured information that makes first-person video, audio, depth, IMU, gaze, pose, and annotations interpretable and usable. It explains what each asset contains, how and when it was captured, which task or episode it belongs to, how streams align, what processing occurred, whether it passed quality and rights gates, and which release a buyer is allowed to use.

The media and sensor samples are the primary data. The records that describe their structure, timing, calibration, context, labels, quality, and lineage are the accompanying metadata. In practice, annotations and derived trajectories may be substantial datasets of their own. The important design decision is not the label applied to the table; it is preserving the relationship between raw evidence and every record created from it.

Organize Metadata at 4 Different Scopes

Source-backed context[4] LeRobotDataset v3 format[5] RLDS episode and step structure[6] MLCommons Croissant 1.1 metadata specification

A reliable schema states the scope of every field. Dataset-wide facts should not be repeated inconsistently on every frame, while sample-level timestamps cannot be hidden in a release note. Modern formats such as LeRobot, RLDS, and Croissant separate dataset structure, episode or record organization, features, and supporting metadata in different ways, but the same four scopes recur.

• Dataset or release level: purpose, version, schema, license, splits, statistics, known limitations, and compatibility
• Asset or stream level: file identity, modality, device, codec, calibration reference, time domain, and integrity
• Session, take, or episode level: task, environment, contributor cohort, attempt outcome, protocol, and boundaries
• Frame, sample, event, or annotation level: timestamp, sensor reading, label, confidence, quality flag, and source reference

1. Asset and Release Metadata

Source-backed context[4] LeRobotDataset v3 format[6] MLCommons Croissant 1.1 metadata specification

Asset metadata identifies the files and makes the delivery reproducible. At minimum, each media or tabular shard needs a stable identifier, path or object key, modality, container and codec, byte size, checksum, creation or export version, and relationships to the release and episode records it contains. Video-specific fields can include encoded and decoded duration, resolution, nominal and measured frame rate, pixel format, orientation, and audio-channel presence.

Release metadata describes the dataset as a versioned product: name, semantic version, publication date, schema version, task and environment coverage, splits, aggregate statistics, intended and unsupported uses, license or contract reference, loader compatibility, and known limitations. LeRobot v3, for example, records feature schema, frame rate, codebase version, file templates, task definitions, statistics, and episode offsets in its metadata layer rather than relying on filenames alone.

• Stable dataset, release, asset, stream, session, take, and episode IDs
• File path, MIME type, codec, dimensions, duration, size, and checksum
• Schema, dataset, exporter, ontology, and loader versions
• Split assignment, statistics, limitations, and compatibility notes

2. Capture Configuration Metadata

Source-backed context[1] Project Aria data formats

Capture metadata records how the source signal was produced. Useful fields include device family and unit reference, firmware, sensor or stream ID, mounting location, lens and field of view, resolution, target sampling rate, exposure or gain mode, stabilization state, audio configuration, and the capture application or protocol version. Project Aria illustrates this pattern by assigning identities to camera, IMU, audio, barometer, magnetometer, GPS, Wi-Fi, and Bluetooth streams and storing sensor-specific configuration records.

Configuration fields should describe the actual take, not only the hardware’s advertised specification. A camera configured for 30 FPS may deliver a variable or interrupted frame sequence. A head mount may slip after setup. Preserve measured facts and QA results separately from requested settings so buyers can tell what was intended from what occurred.

• Device, firmware, capture application, and protocol version
• Viewpoint and mount: glasses, head, chest, wrist, robot, or exocentric reference
• Sensor configuration: resolution, nominal rate, lens, exposure, gain, and orientation
• Session setup events: calibration, restart, battery change, remount, or clock reset

3. Temporal and Synchronization Metadata

Source-backed context[2] Project Aria timestamp definitions[3] Project Aria online calibration format

Every time-dependent stream needs timestamps plus an explicit time domain. Record the timestamp unit, epoch or origin, clock source, sampling policy, monotonicity expectation, frame or sample index, exposure timing where available, known offsets, estimated drift, dropped-sample behavior, and any mapping used to align devices. Project Aria distinguishes device timestamps and shared-clock mappings because a number without its time domain cannot safely align multiple sensors.

For RGB plus IMU, audio, gaze, depth, or paired ego/exo capture, include the method and evidence used to synchronize streams. The start of a recording is not enough: clocks can drift during long takes, frame rates can vary, and device restarts can create discontinuities. Store residual alignment error or pass/fail results across the episode rather than a vague synced flag.

• Timestamp value, unit, time domain, clock source, and sample or frame index
• Nominal and observed rate, gaps, duplicates, resets, and dropped-sample indicators
• Per-stream offset, drift model, alignment transform, and uncertainty
• Episode start and end boundaries referenced to the canonical timeline

4. Calibration and Coordinate-Frame Metadata

Source-backed context[1] Project Aria data formats[3] Project Aria online calibration format

Geometry-dependent use cases need enough calibration metadata to interpret pixels, rays, motion, pose, and sensor measurements in named coordinate frames. That can include camera intrinsics, lens-distortion model and coefficients, sensor extrinsics, image readout time, IMU axes, handedness, gravity convention, units, calibration timestamp, calibration method, and validity interval.

Project Aria’s calibration outputs include camera and IMU intrinsics, extrinsics, time offsets, and timestamped online-calibration results. This is a useful model for a broader rule: never publish a pose, gaze point, acceleration vector, or object location without defining the coordinate frame, transform direction, units, and confidence or validity. The numbers may parse while their meaning remains dangerously ambiguous.

• Camera matrix, distortion model, image size, rolling-shutter or readout parameters
• Sensor-to-device and device-to-world transforms with named source and target frames
• Axis order, handedness, gravity convention, angular units, and distance units
• Calibration method, date, residual error, version, and applicable device or time interval

5. Task, Session, and Episode Metadata

Source-backed context[4] LeRobotDataset v3 format[5] RLDS episode and step structure

Task metadata gives the sequence its learning context. Useful fields include task ID and language, instruction version, goal state, environment or station, object set, scenario variables, session and take IDs, attempt number, start and end times, success, failure reason, interruption, retry relationship, and terminal-state definition. RLDS and LeRobot both organize sequential data around episodes and tasks because file boundaries are not a reliable substitute for behavioral structure.

Do not compress success, failure, interruption, and rejection into one boolean. An unsuccessful attempt can be valuable training evidence, while a technically completed task can still fail visibility, privacy, or format acceptance. Represent the task outcome, capture disposition, and release eligibility as separate fields with separate reasons.

• Task, instruction, protocol, environment, object-set, and scenario IDs
• Session, take, episode, attempt, retry, and parent-child relationships
• Start, end, terminal state, completion status, interruption, and failure reason
• Sampling strata and split keys needed to prevent actor, room, object, or session leakage

6. Sensor and Derived-Signal Metadata

Source-backed context[1] Project Aria data formats[3] Project Aria online calibration format

Additional modalities may include IMU, audio, depth, gaze, GPS, barometer, magnetometer, hand pose, body pose, object pose, trajectories, point clouds, or robot state and actions. The actual samples are data; their accompanying metadata describes stream identity, units, rate, axes, calibration, time domain, missing-value behavior, confidence, and whether the signal is measured, estimated, manually labeled, or transformed.

This distinction prevents a common procurement failure: presenting an estimated signal as sensor ground truth. A six-degree-of-freedom trajectory derived from cameras and IMU should identify the algorithm or service, version, inputs, coordinate frames, and quality metrics. Gaze should identify the calibration and validity status. Pose should identify the model or annotation method and confidence. Every derived stream needs lineage back to its inputs.

• Modality, feature name, shape, type, unit, rate, axes, and valid range
• Measured, estimated, interpolated, fused, manually labeled, or synthetic origin
• Algorithm, model, configuration, and transformation version for derived signals
• Confidence, validity, covariance or uncertainty, and missing-signal reason

7. Annotation Metadata

Source-backed context[5] RLDS episode and step structure[7] EPIC-KITCHENS dataset and annotation overview

Annotation metadata describes labels and how they were produced. Egocentric programs may use clip classes, temporal action segments, narrations, transcriptions, object identities, bounding boxes, masks, hand-object contact, gaze targets, hand or body pose, object state changes, affordances, task steps, failures, and recovery events. EPIC-KITCHENS demonstrates that one first-person corpus can support action, narration, hand-object, mask, audio, and 3D camera annotations with different structures and purposes.

Each annotation should identify its target asset or time interval, ontology term, guideline version, creator type, creation method, confidence or validity, review state, and supersession relationship. For model-assisted work, store the model and configuration version plus whether the suggestion was accepted, edited, rejected, or independently produced. Preserve disagreement and adjudication rather than overwriting uncertainty with a clean-looking final row.

• Annotation ID, target asset or episode, start and end time, frame range, and label
• Ontology, guideline, tool, model, prompt or configuration, and reviewer-role version
• Confidence, abstention, disagreement, review, adjudication, and supersession status
• Source evidence and relationships between actions, objects, contacts, narrations, and outcomes

8. Quality and Acceptance Metadata

Implementation guidanceEGXO guidance for translating the research into a project specification.

Quality metadata records what was tested, against which threshold, on which population, and with what result. Technical checks can cover decode, dimensions, duration, timestamps, synchronization, required fields, duplicates, calibration, and referential integrity. Human checks can cover task correctness, critical-action visibility, occlusion, framing, label fitness, privacy, and reviewer agreement.

Avoid a single quality score. It hides why an asset passed and makes new buyer thresholds impossible to apply. Store individual check IDs, metric values, thresholds, rule versions, sample populations, pass, fail, warning or unverified status, reviewer role, timestamp, and disposition. Recorded, uploaded, technically valid, annotation-approved, privacy-eligible, and release-approved should remain distinct states.

• Automated metric, observed value, threshold, validator version, and evidence artifact
• Human review decision, rubric version, reviewer role, and adjudication outcome
• Rejection, quarantine, repair, repeat-capture, exception, and release status
• Coverage and error statistics sliced by task, environment, device, cohort, and modality

9. Contributor and Environment Metadata

Implementation guidanceEGXO guidance for translating the research into a project specification.

Contributor and environment context can support coverage analysis, leakage-safe splits, ergonomics, and bias evaluation, but it also increases privacy and governance risk. Collect only purpose-bound attributes that materially affect the learning or evaluation claim. Prefer pseudonymous internal references and approved cohort or sampling fields over direct identity in the training release.

Environment metadata may describe country or region at an approved granularity, environment type, workspace layout, lighting condition, noise, clutter, surface, object inventory, safety constraints, and controlled variations. Record whether a value was instructed, self-reported, sensed, inferred, or reviewed. Unknown should be a valid value; invented precision is worse than an explicit gap.

• Pseudonymous contributor, cohort, eligibility, experience, and dominant-hand fields when justified
• Approved environment category, station, layout, lighting, clutter, noise, and object-set fields
• Sampling stratum, recruitment channel, session history, and split-exclusion key
• Collection method, permissible granularity, retention, access tier, and export policy for sensitive context

10. Governance, Rights, and Lineage Metadata

Source-backed context[6] MLCommons Croissant 1.1 metadata specification

Governance metadata connects each released asset to the authority and decisions that permit its use. Relevant records can include consent or other collection-authority reference, notice and agreement versions, permitted use, license, buyer or distribution tier, geographic or vendor restrictions, retention, deletion state, privacy review, redaction, security classification, and final release approval. Store references and decisions—not unnecessary identity—in the training dataset.

Lineage records every transformation from raw capture to buyer release: transcoding, clipping, synchronization, calibration, redaction, filtering, annotation, normalization, sharding, split assignment, and export. Croissant 1.1 provides ways to represent machine-readable provenance and data-use restrictions, but a standard format does not create the underlying evidence. The operator still needs stable IDs, transformation records, governing documents, and versioned approvals.

• Collection authority, notice, consent or agreement, permitted-use, and license references
• Privacy review, redaction, access, retention, deletion, and distribution-tier status
• Source release, parent assets, transformation graph, code or operator version, and derived outputs
• Release approver, approval date, manifest, checksum set, corrections, withdrawals, and superseded versions

Build a Minimum Viable Metadata Contract

Source-backed context[4] LeRobotDataset v3 format[5] RLDS episode and step structure[6] MLCommons Croissant 1.1 metadata specification

Before collection scales, publish a machine-readable schema and a human-readable data dictionary. For every field define the name, scope, type, unit, time domain, coordinate frame, allowed values, null behavior, source, sensitivity, owner, validation rule, and version. Use stable keys instead of encoding meaning in filenames. Separate canonical internal records from buyer-specific exports so a format change does not erase provenance.

The minimum contract should be exercised with representative and adversarial records: missing modalities, interrupted tasks, retries, multiple identical objects, ambiguous temporal boundaries, dropped frames, clock resets, redacted intervals, sensor replacement, annotation correction, and post-release withdrawal. Decide whether each condition is represented, rejected, quarantined, or explicitly unsupported.

• Data dictionary and typed schema with required, optional, nullable, and conditional fields
• Identifier map and relationship rules across release, asset, stream, session, episode, and annotation
• Timebase, coordinate-frame, unit, ontology, and missing-value conventions
• Validation suite, worked examples, edge cases, compatibility policy, and migration notes

What a Buyer Should Receive

Source-backed context[4] LeRobotDataset v3 format[6] MLCommons Croissant 1.1 metadata specification

A delivery is not complete when the files arrive. The buyer should be able to inspect the release, resolve every relationship, load representative episodes, reproduce the documented splits, and trace a derived record to its source and method. Validate the actual export in the buyer’s target stack; a schema that passes internally can still fail on codec support, timestamp semantics, unsupported nulls, or incorrect episode boundaries.

The practical standard is simple: no field should require reverse engineering, and no important claim should depend on a field merely being populated. The delivery should distinguish what was captured, inferred, annotated, measured, reviewed, approved, and left unverified.

• Dataset card, release manifest, file checksums, schema, data dictionary, and version history
• Episode, task, stream, annotation, QA, governance, and lineage manifests at the documented scopes
• Ontology and annotation guidelines, calibration and synchronization reports, and known limitations
• Reference loader, small worked examples, validation results, and proof of buyer-side ingest