Robot-free data collection for physical AI: how robotics teams build training data without running a robot
Explore robot-free data collection for physical AI and how robotics teams build training data using simulation, synthetic data, and teleoperation.
Most teams building vision-language-action models still collect training data the same way. They put a robot on a task, run it over and over, and record every attempt. If you are trying to scale that process past a handful of demonstrations, you already know where it breaks down. That is why more robotics teams are looking at robot-free data collection for physical AI training, an approach that captures human manipulation demonstrations directly, without a robot in the loop at all.
This article is for the engineer who has heard of VLA models, understands imitation learning at a basic level, and is now stuck on the less glamorous problem. Where does the training data actually come from, and how do you collect enough of it without burning your robot budget on data logging instead of development?
Why robot-only data collection is a bottleneck
Recording footage from a physical robot running a task seems like the obvious way to generate training data. It is also the slowest and most expensive way.
A few problems show up quickly once you try to scale this approach.
The robot has to already exist and be functional. If you are prototyping a new manipulation policy or a new gripper design, you cannot collect data on hardware that is not finished yet. That forces data collection to wait behind hardware development, which is backwards if data is actually your bottleneck.
Robot time is expensive and slow. Every demonstration ties up a physical robot, a workspace, and usually an operator. Teleoperating a robot arm through hundreds or thousands of pick, place, and assembly variations takes far longer than performing the same tasks with your own hands. Calibration drift, motor wear, and safety stops all eat into throughput further.
Single-embodiment data does not generalize well. Data collected on one robot arm, with one gripper geometry and one camera mount, teaches a policy the quirks of that specific setup. Scaling to new embodiments or new tasks often means starting the data collection process over.
Coverage is hard to scale. Building a dataset large enough to train a robust VLA model means capturing wide variation in objects, backgrounds, lighting, and task execution. Doing that on a robot means physically reconfiguring the workspace and re-running the robot for every variation, which is a slow way to build breadth into a dataset.
None of this is a criticism of robot teleoperation as a data source. It still has a place, particularly for fine-tuning on a specific embodiment late in development. The problem is treating it as the only viable input to the pipeline. As physical AI develops alongside broader business automation, teams are also separating physical-world learning from software-based automation workflows, including robotic process automation for digital business. This distinction helps organizations choose the right automation approach for each process instead of relying on physical robots where software automation or human demonstrations may be more efficient.
What robot-free data collection actually means
Robot-free data collection replaces the robot with wearable or handheld sensors that capture a human performing the same manipulation task the model is meant to learn. A person wears a first-person camera or holds a handheld device shaped like an end effector and performs the pick, place, assembly, or sorting task directly. The sensors record synchronized RGB, depth, and motion data exactly as if a robot had been doing the work.
This matters for a few concrete reasons that go beyond simply being faster than teleoperation.
Human hands are fast and flexible data collectors. A person can move between task variations, objects, and environments far faster than reconfiguring a robot cell. That translates directly into more demonstrations per hour and broader coverage per dataset.
You are not constrained by robot embodiment while collecting. Because the data captures human hand and viewpoint motion rather than a specific robot’s joint trajectories, the same demonstration set can inform policies across different robot platforms, provided your training pipeline handles the embodiment gap appropriately.
Data collection can start before the robot does. Teams can begin building a demonstration dataset while hardware is still in development, rather than waiting for a finished robot to become available.
In practice, robot-free collection usually combines three capture perspectives, each contributing a different layer of information to the training set.
Egocentric, first-person vision
An egocentric or “ego” view captures what a person sees while performing a task, including scene context, object layout, and the sequence of actions taken to complete it. This gives a model a sense of task flow and environmental context that a fixed third-person camera does not capture as naturally.
Handheld manipulation capture
A handheld device, often built around a gripper-shaped mount, captures how a task is actually completed. This is the layer that records grasping, moving, placing, assembling, and organizing objects, which is the core data a manipulation policy needs to learn from.
Wrist-level, near-field interaction capture
A wrist-mounted camera fills in the close-range detail that egocentric and handheld views can miss, particularly during occlusion-heavy moments like the instant a hand closes around an object. This detail matters most for fine-grained, contact-rich manipulation tasks where hand-object contact geometry is part of what the model needs to learn.
Orbbec’s hardware for robot-free capture
Related walkthrough
China Just Built a Factory Where Robots Build Robots | 10,000 Humanoid Robots a Year
Orbbec’s robot-free data collection platform is built around three standardized device categories that map to the three capture perspectives above, EGO, UMI, and WristCam, with contract manufacturing and joint design manufacturing options available for teams that need a customized form factor.
EGO for egocentric capture
EGO devices are head-mounted or body-worn units built for first-person data collection. Orbbec offers both RGB-only EGO configurations, including dual-camera and quad-camera variants, and an RGB-D EGO configuration that pairs an RGB camera with a Gemini 330 series stereo depth camera for synchronized color and depth capture.
The Dual-EGO configuration uses two 2-megapixel fisheye cameras on a 120mm baseline, with a combined field of view of 165 degrees horizontal by 130 degrees vertical, a recommended capture distance of 0.2 to 2 meters, and a resolution of 1600 x 1200 at up to 60 fps. Synchronization accuracy is rated at 1 millisecond or better, with a 6-axis IMU sampling at up to 1000 Hz. The unit weighs 200g. Quad-EGO extends coverage to four fisheye cameras for a combined horizontal field of view past 270 degrees, at a weight under 400g including battery.
The RGB-D EGO variant is where depth enters the pipeline directly. It integrates a Gemini 330 series camera, which runs on Orbbec’s MX6800 depth-engine ASIC for in-camera depth processing, so depth computation happens on the sensor rather than consuming host compute during capture.
UMI for handheld manipulation capture
UMI devices are handheld units that mimic a gripper’s field of view and motion while a person performs a manipulation task by hand. They capture images, IMU data, and gripper opening angle data simultaneously. Orbbec’s UMI configurations include a Finger variant with a 130-degree opening angle and a Gripper variant with a 60-degree opening angle, both combining a fisheye RGB camera (165° H by 130° V) with a monochrome global-shutter SLAM camera (130° H by 95° V) for tracking. Frame rates run at 30 or 60 fps with sub-millisecond synchronization across sensors.
WristCam for near-field detail
WristCam is the smallest device in the lineup, a single 2-megapixel fisheye camera under 75g, built to sit at the wrist and capture close-range hand-object interaction that the ego or handheld views may occlude. It shares the same 165° by 130° field of view and 0.2 to 2 meter recommended capture distance as the other modules, keeping the perspectives consistent when devices are combined.
Why synchronization matters more than sensor count
Adding more sensors to a capture rig is the easy part. Keeping them aligned in time and space is the part that actually determines dataset quality.
Multi-sensor fusion across ego, wrist, and handheld devices only works if every stream shares a common time reference. A misaligned frame between the RGB, depth, and IMU streams introduces noise into exactly the kind of fine-grained manipulation data a VLA model needs to learn from. Orbbec’s platform targets synchronization accuracy of 1 millisecond or better across its capture modules, using a unified timestamp reference across RGB, depth, IMU, and gripper data.
Occlusion during close-range manipulation is a separate problem. The moment a hand closes around an object is often the moment a first-person or handheld view loses visibility of the exact contact point, which is why the wrist-level module exists as a distinct capture layer rather than an afterthought.
Consistency at scale is the third variable that matters once a dataset moves past a pilot batch. If capture devices behave inconsistently between collection sessions or across a large batch of units, dataset quality varies in ways that are hard to detect until training results come back worse than expected. Manufacturing consistency across devices, not just per-unit specs, is part of what determines whether a dataset scales cleanly.
Fitting robot-free capture into an existing pipeline
Hardware only solves half the problem. The other half is getting synchronized multimodal data out of the capture devices and into a training pipeline without building a custom integration from scratch for every project.
Orbbec’s SDK provides API-level access to the depth, RGB, and IMU streams from its cameras, along with camera-IMU frame synchronization and stream processing support. It is open source and cross-platform, with support for Android, iOS, Windows, and Linux, so integration work can fit into whatever data pipeline a team already runs rather than requiring a platform switch. For teams with project-specific requirements around device control, calibration data access, or custom data export formats, that support extends beyond the standard SDK feature set as well.
FAQs About Robot Free Data Collection
Not necessarily. Most teams use it to build the bulk of a demonstration dataset quickly and cheaply, then use robot teleoperation later for embodiment-specific fine-tuning once the hardware is finalized. The two approaches address different stages of the same pipeline rather than competing for the same role.
Each module works independently, and the right starting point depends on what your policy needs to learn. If task context and scene understanding matter most, start with an ego view. If the priority is manipulation mechanics like grasping and placing, a handheld device captures that directly. Combining modules gives a more complete picture of scene context, task execution, and hand-object interaction, which most teams eventually want once they move past an initial pilot dataset.
Depth comes from the capture device itself. Orbbec's RGB-D EGO configuration pairs an RGB camera with a Gemini 330 series stereo depth camera worn on the body, so depth data is captured alongside RGB and IMU data during the same human demonstration rather than reconstructed afterward.
Household manipulation, industrial assembly, warehouse picking and sorting, tool-use tasks, and bimanual operation are all common use cases, since all of them can be performed by a human wearing or holding the capture devices in a way that mirrors how a robot would eventually need to complete the same task.
Teams evaluating a robot-free data collection setup for their own VLA or physical AI pipeline can find development kit details, device specifications, and SDK documentation on Orbbec's robot-free data collection page.
