Egocentric Dataset Specification Template

A Dataset Specification Is a Data Contract

Source-backed context[1] Ego4D video and modality documentation[2] RLDS dataset structure[3] LeRobotDataset v3 format

An egocentric dataset specification connects a model objective to collection, annotation, governance, and delivery requirements. It states what the buyer needs to observe, how each episode is produced, which records accompany it, what causes rejection, and how the final release will be loaded. Without that contract, teams tend to optimize for upload count while definitions drift across operations, QA, labeling, and engineering.

Use the downloadable brief before requesting price or volume. Suppliers can then identify assumptions, validate feasibility, and propose a pilot against the same requirements. The document should remain versioned with the protocol and dataset release so a buyer can trace which rules produced which assets.

1. Define the Learning or Evaluation Objective

Implementation guidanceEGXO guidance for translating the research into a project specification.

Write one specific model-facing outcome. Examples include detecting hand-object contact, learning procedural structure, grounding task language, retrieving prior activity, estimating pose, generating future states, or evaluating failure recovery. Avoid broad goals such as better robot intelligence; they do not determine what the camera, labels, or acceptance checks must capture.

• Target model, training stage, or evaluation workflow
• Observable signal expected from the dataset
• Known missing signals and how other data sources provide them
• Representative baseline and the failure the new data should address

2. Specify Tasks, Environments, and Coverage

Implementation guidanceEGXO guidance for translating the research into a project specification.

Define the unit of work as an episode with a start condition, terminal condition, allowed variation, and failure policy. List objects, tools, layouts, lighting, clutter, participant criteria, geography where relevant, and the slices that need deliberate balance. State which rare or unsafe conditions must be simulated, excluded, or handled through another source.

Coverage should be measurable. A task name alone cannot tell a reviewer whether critical contacts, intermediate states, corrections, or negative examples appear. Include a coverage matrix and assign an owner to approve changes after the pilot.

3. Define Viewpoint, Hardware, and Modalities

Source-backed context[1] Ego4D video and modality documentation

Specify camera placement, resolution, frame rate, field of view, orientation, stabilization policy, audio behavior, and required sensors. For depth, IMU, gaze, pose, or synchronized cameras, define sampling, shared timebase, calibration, drift tolerance, missing-signal policy, and how alignment is validated across a full take.

Treat placement as a pilot result rather than a preference. The selected mount must keep model-critical actions visible across people, body shapes, dominant hands, workspace heights, and natural movement.

4. Define Episode, Metadata, and Annotation Structure

Source-backed context[2] RLDS dataset structure[3] LeRobotDataset v3 format

State how sessions, takes, episodes, steps, retries, interruptions, and failures are represented. Separate capture facts from context, annotations, QA, and governance. Define stable keys, field types, units, null behavior, timestamps, coordinate frames, ontologies, confidence, adjudication, and every version needed to interpret a record.

Annotate at the minimum granularity required by the target loss or evaluation. Clip-level, segment-level, event-level, and frame-level labels have different costs and failure modes. Dense annotation without a consumer is waste, not quality.

5. Set Acceptance and Rejection Rules

Implementation guidanceEGXO guidance for translating the research into a project specification.

Acceptance should combine automated checks and human review. Define tolerances for decode, dimensions, frame rate, timestamps, alignment, required fields, duplication, task completion, critical-action visibility, framing, motion, privacy, label fitness, and release eligibility. Record whether a failed asset is repeated, repaired, quarantined, or permanently excluded.

• Media and metadata integrity thresholds
• Critical visibility moments and maximum tolerated occlusion
• Task completion, failure, and interruption policy
• Privacy and rights release gate
• Annotation agreement, adjudication, and rework thresholds

6. Define Governance and Permitted Use

Source-backed context[4] Ego4D privacy and ethics

Document contributor and bystander handling, consent or other lawful basis, compensation, sensitive-environment rules, retention, deletion, security, reviewer access, redaction, incident response, and release approval. Keep permission to collect, permission to annotate, buyer delivery, model training, redistribution, and public marketing as distinct decisions.

The commercial agreement should state training, evaluation, derived-model, vendor-access, geographic, retention, exclusivity, and redistribution rights. Do not infer those rights from the fact that a file can be downloaded.

7. Define Delivery and Buyer Ingest

Source-backed context[2] RLDS dataset structure[3] LeRobotDataset v3 format

Specify media layout, manifest schema, dataset card, checksums, splits, release notes, transformation history, license packet, and target loader. Names such as RLDS or LeRobot are not sufficient; define the exact version, fields, episode semantics, video encoding, timestamp conventions, and extensions.

The pilot should end with an ingest test in the buyer’s real environment. Decode every required modality, traverse episode boundaries, construct batches, inspect representative records, and record failures. Scale approval should depend on this evidence.

8. Turn the Specification Into a Pilot Decision

Implementation guidanceEGXO guidance for translating the research into a project specification.

Choose the smallest pilot that exercises the hardest tasks, viewpoint risks, privacy conditions, annotation decisions, delivery format, and buyer loader. The decision record should list what passed, what failed, open assumptions, protocol changes, new acceptance thresholds, and whether scale is approved, revised, or stopped.

A good specification gets more precise after the pilot. Update the document, increment its version, and connect the approved version to every subsequent capture and release.