Egocentric Data Annotation and Metadata

Do Not Mix Capture Facts With Interpretation

A durable dataset distinguishes raw capture metadata from labels created later. Device, timestamp, resolution, frame rate, and file checksum are capture facts. Task steps, object interactions, narrations, poses, and quality decisions are derived records with their own method and version. Mixing them in one mutable table makes corrections hard to audit.

Use 4 Metadata Layers

The storage format may change, but the logical separation should remain stable.

• Capture: asset ID, device, modality, timestamps, codec, dimensions, and checksums
• Context: task, environment, session, viewpoint, protocol version, and allowed attributes
• Annotation: temporal segments, actions, objects, language, pose, contacts, and label version
• Governance: rights scope, consent reference, privacy review, QA status, lineage, and release version

Choose Temporal Granularity From the Consumer

Clip-level labels are efficient for retrieval and coarse training. Step segments support procedural understanding. Frame or event timestamps support contact, pose, and fine action learning. Dense labels cost more and are not automatically more useful; annotate at the minimum granularity required by the target loss or evaluation.

Make Uncertainty and Disagreement Explicit

Real-world activity contains ambiguous boundaries, occlusions, and overlapping actions. Store annotator confidence, adjudication status, guideline version, and disagreement where relevant. A forced single label can hide uncertainty that the model team needs to understand.

Export Without Losing Provenance

Buyer formats such as RLDS or LeRobot organize sequential data around episodes, steps, video, signals, and metadata. Build exports from a canonical internal representation and retain stable IDs and lineage back to the source release. Validate episode boundaries, timestamp alignment, schema types, missing values, and sample decoding in the buyer’s actual loader.

Validate Schemas With Adversarial Records

A schema that works only for the happy path is not finished. Test missing modalities, interrupted tasks, multiple objects of the same type, overlapping actions, uncertain boundaries, dropped frames, redacted segments, and corrections issued after release. Decide whether each case is represented explicitly, rejected, or documented as unsupported.

Automated validation should check types, ranges, required relationships, monotonic timestamps, episode boundaries, referential integrity, vocabulary versions, and media existence. Human review should sample semantic correctness and ensure validators are not merely confirming that a field is populated.

Give Buyers a Compact Data Contract

The delivery should include a machine-readable schema and a short human guide explaining units, coordinate frames, timestamp conventions, null behavior, categorical vocabularies, transforms, split logic, and version compatibility. Include several small worked examples and known edge cases. A buyer should not have to reverse-engineer meaning from filenames or loader code.

• Field name, type, unit, and allowed null behavior
• Stable key and relationship definitions
• Timebase and coordinate conventions
• Ontology and annotation guideline versions
• Release compatibility and migration notes

Treat Annotation Operations as a Measured System

Track throughput alongside rework, disagreement, abstention, adjudication, and error by task family. A fast annotation queue can create expensive downstream noise when guidelines are vague or the interface hides temporal context. Calibration sets should include easy, ambiguous, and rare examples, with feedback tied to specific guideline rules rather than a single aggregate score.

Model-assisted annotation needs its own lineage. Store the model and prompt or configuration version, confidence, human action, and whether the suggestion was accepted, edited, rejected, or never shown. Periodically sample apparent agreements because reviewers can inherit systematic model errors instead of independently validating them.

• Measure quality by task and label type, not only overall
• Separate guideline ambiguity from reviewer error
• Recalibrate after ontology, tooling, or model changes
• Keep adjudication decisions available to future reviewers