Dataset Provenance

01

Provenance, Metadata, and Lineage

Source-backed context[1] W3C PROV overview[2] MLCommons Croissant metadata specification

Metadata describes data. Provenance focuses on origin and history. Lineage often emphasizes movement and transformation through systems. In practice these concepts overlap, but a buyer needs a record that can answer where an asset came from, what happened to it, which permissions apply, and how it entered a release.

The record can exist across linked tables, manifests, consent systems, processing logs, and release documents. It does not need to be one giant file. It does need stable identifiers and relationships that survive export.

02

What Provenance Covers in Robotics Data

Source-backed context[2] MLCommons Croissant metadata specification[3] Datasheets for Datasets

Real-world robotics data may pass through capture devices, contributor workflows, upload systems, privacy review, annotation, synchronization, calibration, segmentation, quality review, packaging, and delivery. Each stage can change the asset or its eligibility for use.

Provenance connects those stages. It can identify the contributor permission tied to a recording, the redaction model and parameters applied, the calibration used for derived pose, the reviewer decision, and the manifest that included the final file. This supports reproducibility, audit, removal, and controlled updates.

03

Rights and Technical QA Must Stay Separate

Source-backed context[1] W3C PROV overview[4] Project Aria data formats and metadata

A technically valid clip may lack permission for model training. A rights-approved asset may fail visibility or synchronization checks. Provenance should record these decisions independently so that one status never implies the other.

Permitted use also needs precision. Internal evaluation, model training, commercial deployment, redistribution, and publication can have different terms. The release should expose the decision used for inclusion, not force the buyer to infer rights from a folder name.

04

Provenance Failure Modes

Source-backed context[1] W3C PROV overview[2] MLCommons Croissant metadata specification[3] Datasheets for Datasets

Provenance breaks when files rely on names or directory position instead of stable IDs, transformations overwrite originals, consent cannot be linked at asset level, or release versions are assembled manually without manifests. Copying data between tools can also strip source relationships.

Acceptance should trace a sample of delivered assets backward to collection and forward through every material transformation. It should also test removal, correction, and release reconstruction. Hashes, manifests, immutable source records, and versioned processing specifications make those tests possible.

• Source and derived assets share no stable relationship
• Rights status exists only in email or contract prose
• Processing overwrites raw data or loses software version
• QA decisions have no reason or reviewer role
• Release contents cannot be reconstructed from a manifest

05

What a Buyer Should Specify

Implementation guidanceEGXO guidance for translating the definition into a project specification.

Define the questions the provenance system must answer and test them during the pilot. Require stable identifiers, permission linkage, transformation history, QA decisions, release manifests, and a correction or removal process. Inspect real records, not only policy descriptions.

• Asset, collection, contributor, task, and release identifiers
• Consent, permitted use, restrictions, and approval state
• Raw-to-derived transformations and software versions
• Technical QA, privacy review, and rights review as separate gates
• Manifest, hashes, release notes, and correction process
• Backward and forward traceability test on pilot assets