Deformable-Object Manipulation
Bimanual alignment and folding of flexible material with continuously changing geometry.
- Deformable objects
- Bimanual coordination
- Changing geometry
Data perspective / 01
Egocentric Data Puts the Model Where the Action Happens.
First-person capture records what a person sees while performing a task. For physical AI, that viewpoint can preserve useful relationships among intent, hands, tools, objects, and changing scene state.
Plain-English definition
Egocentric Means Actor-Centered.
An egocentric camera moves with the person or system performing the task. A head-, glasses-, chest-, wrist-, or robot-mounted view tends to preserve the evidence available to the actor at decision time.
An exocentric camera observes from outside. It may better show body pose, workspace layout, other agents, and events hidden from the actor. The two views answer different questions; synchronized capture is useful when a model must connect them.
Compare the Viewpoints ↗From Pixels to Usable Supervision
Capture configuration should follow the model objective, not the camera you happen to own.
Capture
Potential configurations include head-, chest-, wrist-, glasses-, or robot-mounted RGB video. Depth, audio, IMU, gaze, and synchronized metadata require program-specific validation.
Structure
Task boundaries, step segmentation, object interactions, action language, timestamps, scene attributes, and acceptance metadata.
Validate
Framing, visibility, motion, lighting, task completion, privacy, labeling consistency, versioning, and reproducible delivery checks.
Modalities to Validate in a Pilot
Selection depends on the model objective, hardware, environment, privacy constraints, and evidence required.
RGB
Video
Resolution, lens, frame rate, exposure, motion, and field-of-view tuned to the action.
Geometry
Depth and Pose
Optional depth, hand pose, body pose, object pose, calibration, and coordinate conventions.
Motion
IMU
Timestamped accelerometer and gyroscope data with alignment and drift requirements.
Attention
Gaze
Eye-gaze streams, calibration quality, validity flags, and privacy-aware use.
Language
Audio and Narration
Task instructions or narration with consent, privacy, timing, and transcription policy.
Mounts
Viewpoint
Head, glasses, chest, wrist, or robot placement selected through a visibility pilot.
Public vs. custom
Use Public Data for Shared Questions. Collect Custom Data for Deployment Gaps.
Public datasets support benchmarking, prototyping, research comparison, and early feasibility. Custom programs are justified when tasks, environments, sensors, labels, licenses, or failure cases do not match deployment.
A hybrid strategy can use public data for baselines and comparability, then targeted collection for domain fit, proprietary conditions, failure cases, and buyer-controlled rights.
Read the Procurement Framework ↗Collection Planning
Define the task and release before choosing volume.
- 01
Task and Environment
Goals, steps, objects, tools, settings, variations, failures, and terminal conditions.
- 02
People and Instructions
Eligibility, consent, compensation, safety, natural variation, stop rules, and protocol version.
- 03
Hardware and Observability
Mount, modality, resolution, frame rate, critical visibility moments, and occlusion tolerance.
- 04
Annotation and Metadata
Task segments, actions, objects, interactions, language, pose, timestamps, QA, and lineage.
- 05
Acceptance and Release
Integrity, coverage, privacy, license, schema, dataset card, version, and buyer ingest proof.
Manipulation Across Task Families
View the Full Sample Library ↗
Bimanual Food Assembly
- Bimanual manipulation
- Tool use
- Sequential actions
Object Cleaning and Dishwashing
- Object rotation
- Contact-rich actions
- Continuous handling
Packing and Spatial Arrangement
- Packing
- Container interaction
- Object placement
Sorting and Categorization
- Categorization
- Fine manipulation
- Repeated actions
Fine Manipulation and Tool Use
- Tool use
- Fine motor control
- Food preparation
Where Egocentric Data Fits
Use it when the model must learn from the actor’s available information rather than an observer’s convenient angle.
- Robot learning
Human demonstrations for manipulation, navigation, and task planning.
- VLA systems
Visual sequences aligned with language, action boundaries, and task intent.
- World models
Long-horizon state transitions grounded in real environments.
- Evaluation
Controlled task variation, failure cases, and robustness testing.
Quality, privacy, and provenance
Technical Quality Is Necessary. Release Eligibility Is Separate.
A usable program checks media integrity, task visibility, metadata, annotations, coverage, and buyer ingest. A releasable program also connects each asset to consent, privacy review, licensing scope, lineage, retention, and distribution tier.
First-person recording creates specific risk around bystanders, screens, documents, voices, reflections, and private environments. Minimize before capture, screen before release, and reapprove when the distribution channel changes.
Review the Control Framework ↗Questions to Resolve Before a Pilot
The right answer depends on the learning objective, capture conditions, rights, and target loader.
What is egocentric data?
Data captured from the perspective of the actor performing a task, usually through a wearable or actor-mounted camera and optional synchronized sensors.
Is egocentric data the same as first-person video?
First-person video is the most common form, but an egocentric dataset may also include audio, depth, IMU, gaze, pose, language, task structure, and metadata.
When is exocentric data better?
When whole-body motion, workspace geometry, other agents, stable world coordinates, or events outside the actor’s field of view are central to the objective.
Can human demonstrations train robots directly?
Sometimes they provide useful representation or task supervision, but many systems still need embodiment mapping, robot-native actions, teleoperation, simulation, or robot rollouts.
Which delivery formats are supported?
Formats must be validated against the actual collection and buyer loader. Native media plus structured metadata, RLDS, LeRobot, or custom exports may be scoped, but none is promised by this page.
Primary research context
Public Datasets That Define the Category
Custom collection
Start With the Model Failure, Then Design the Data.
Share the task, environment, viewpoint, metadata, privacy rules, and acceptance criteria you need.
Scope a Collection