Powering Physical AI with Real-World Ego Datasets

3D Point Cloud Datasets for Computer Vision and Embodied AI
Content Menu
● What Are 3D Point Cloud Datasets?
● Selected 3D Point Cloud Datasets and Complementary Data Sources
>> Virdyn: Custom Data for Real-World Robot Training
>> ScanNet: Indoor Geometry and Semantic Understanding
>> KITTI, nuScenes, and Waymo: Outdoor Perception Resources
>> ShapeNetCore and DROID: Geometry Versus Interaction
● How to Choose 3D Point Cloud Datasets for Your Project
>> Start With the Deployment Task
>> Match Sensor Geometry and Working Distance
>> Check Rights Before Technical Integration
● Why Embodied AI Needs More Than Point Clouds
● A Practical Workflow for Building Custom 3D Training Data
>> 1. Write a Data Specification
>> 2. Run a Representative Pilot
>> 3. Define Synchronization and Calibration Checks
>> 4. Build an Annotation Handbook
>> 5. Deliver a Reproducible Data Package
● Data Quality Metrics That Matter More Than Volume
● Combining Real and Synthetic Data Without Hiding the Gap
● Build a Dataset Around the Task You Need to Solve
>> What is the best 3D point cloud dataset for robotics?
>> Can stereo video become a point cloud?
>> Are public datasets free for commercial use?
>> How much custom training data should a project collect?
>> Does egocentric video provide robot action labels?
>> What should I request from a data collection provider?
Choosing 3D point cloud datasets is not simply a matter of downloading the largest collection. The right training data must match your environment, sensing system, annotation requirements, and deployment goals. For teams developing embodied AI, it must also connect spatial understanding with the actions a robot needs to perform.
Public benchmarks provide valuable starting points. However, industrial inspection, warehouse handling, and service robotics often require task-specific observations that general-purpose datasets cannot supply.
This guide compares established resources with custom collection options, explains their limitations, and provides a practical framework for building deployment-ready training data.

What Are 3D Point Cloud Datasets?
A point cloud represents an object or environment as individual points in three-dimensional space. Each point typically includes X, Y, and Z coordinates. Depending on the collection system, it may also contain color, intensity, timestamps, or semantic labels.
Point clouds can come from LiDAR, depth cameras, photogrammetry, or calibrated stereo reconstruction. These acquisition methods produce different density, noise, and coverage characteristics.
A crucial distinction: not every 3D dataset contains ready-to-use point clouds.
Some resources provide RGB-D sequences, polygon meshes, or CAD models. Teams must reconstruct geometry or sample surfaces before using them in a point-based learning pipeline.
For robotic manipulation, geometry is only one component. A robot may also need language instructions, temporal context, object poses, and compatible action supervision.
Selected 3D Point Cloud Datasets and Complementary Data Sources
The following comparison places Virdyn first as a custom data provider. The remaining entries are research datasets or dataset programs, not equivalent commercial services.
| Resource | Data category | Useful applications | Main consideration |
|---|---|---|---|
| Virdyn / VDEgo | Custom egocentric and multimodal collection | Embodied AI, operational demonstrations, specialized environments | Define reconstruction, annotation, and action requirements before collection |
| ScanNet | Indoor RGB-D scans and reconstructed geometry | Indoor segmentation and scene understanding | Indoor scans do not provide robot-control demonstrations |
| KITTI | Driving imagery and corresponding point clouds | Outdoor 3D detection and perception benchmarking | Road scenes differ substantially from close-range manipulation |
| nuScenes | Multisensor autonomous-driving data | Sensor fusion, tracking, and scene understanding | Confirm commercial licensing and task-specific annotations |
| Waymo Open Dataset | Autonomous-driving dataset program | Driving perception and related research | Select the appropriate dataset component and review its terms |
| ShapeNetCore | Categorized 3D models | Shape classification and geometry-based experiments | Models require sampling to become point clouds |
| DROID | Robot manipulation demonstrations | Policy learning and interaction research | Complementary action data, not a conventional point-cloud benchmark |
Virdyn: Custom Data for Real-World Robot Training
Virdyn, through its VDEgo offering, provides egocentric data collection, multimodal dataset development, annotation, and collection hardware.
Its VDEgo-C2 wearable device combines binocular RGB cameras with an IMU. According to the company’s published materials, its workflow includes synchronized RGB, IMU, and pose data, followed by cleaning, temporal alignment, annotation, and quality review.
Potential applications include manufacturing operations, warehouse workflows, household tasks, retail services, and laboratory procedures.
However, buyers should distinguish captured signals from derived outputs. Stereo video is not automatically a validated point cloud. Reconstructed geometry, hand-motion labels, and camera trajectories require agreed processing methods and acceptance criteria.
Virdyn’s company-supplied service brief also describes off-the-shelf datasets, customized collection, and simulation-assisted production. Confirm current availability and project scope before procurement.
ScanNet: Indoor Geometry and Semantic Understanding
ScanNet’s original publication describes approximately 2.5 million RGB-D views, collected across 1,513 scans of 707 distinct indoor spaces.
Its camera poses, surface reconstructions, and semantic annotations support indoor scene-understanding research.
The distinction between scans and spaces matters. Multiple scans can represent the same physical environment. When creating evaluation splits, avoid assuming that every scan represents an independent location.
KITTI, nuScenes, and Waymo: Outdoor Perception Resources
KITTI’s 3D object detection benchmark contains 7,481 training examples and 7,518 testing examples, including corresponding imagery and point clouds.
nuScenes adds a broader sensor configuration: six cameras, one LiDAR, five radars, and positioning and inertial sensors. Its published dataset description includes 1,000 driving scenes.
Waymo also provides established autonomous-driving resources. These collections support driving research, but their sensor placement and object distributions do not directly match tabletop assembly or hand-object interaction.
ShapeNetCore and DROID: Geometry Versus Interaction
ShapeNetCore provides categorized 3D models rather than recorded operational sequences. Its version-two collection covers 55 common object categories and approximately 51,300 models.
DROID addresses a different need. Its published release contains 76,000 robot demonstration trajectories, totaling 350 hours across 564 scenes and 86 tasks.
These resources illustrate an important boundary: understanding an object’s shape and learning how to manipulate it are related, but distinct, training problems.
How to Choose 3D Point Cloud Datasets for Your Project
Start With the Deployment Task
Define the model’s expected output before choosing the dataset.
Examples include:
-
Assigning a semantic class to each point.
-
Detecting objects with 3D bounding boxes.
-
Estimating object position and orientation.
-
Reconstructing an indoor workspace.
-
Connecting an instruction to a manipulation sequence.
Then identify the conditions that could cause failure: reflective packaging, occluded tools, changing illumination, crowded shelves, or unfamiliar objects.
A dataset is useful when it represents those challenges—not merely when it contains many files.

Match Sensor Geometry and Working Distance
Compare the training data’s sensor configuration with the intended deployment setup.
Vehicle-mounted LiDAR observes streets from a different height and distance than a wearable stereo camera observing hands above a workbench.
Check:
-
Field of view and camera placement.
-
Object distance and apparent size.
-
Point density around important surfaces.
-
Occlusion and motion patterns.
-
Calibration and coordinate conventions.
If these differ substantially, plan targeted collection or adaptation experiments.
Check Rights Before Technical Integration
Public availability does not establish unrestricted commercial permission.
Review the license for the exact release you intend to use. Check access conditions, redistribution rights, attribution requirements, and restrictions affecting commercial development.
For example, nuScenes identifies non-commercial availability and provides a separate commercial licensing contact.
Record the applicable terms with the dataset version. This makes later audits easier and reduces uncertainty when a research prototype becomes a commercial product.
Why Embodied AI Needs More Than Point Clouds
A point cloud can describe where an object is. It does not necessarily explain what happened, why an operation failed, or which command a robot should execute next.
Egocentric demonstrations can provide additional context:
-
The order of task steps.
-
Hand-object interactions.
-
Changes in object arrangement.
-
Visual evidence of completion.
-
Recovery after an unsuccessful attempt.
For vision-language-action models, action representation needs particular attention. Human hand movement is not automatically equivalent to robot joint commands or end-effector targets.
Likewise, world-model development requires meaningful temporal sequences, not just disconnected spatial snapshots.
A practical training strategy separates three layers:
-
Geometry for spatial understanding.
-
Demonstrations for task structure.
-
Robot-compatible supervision for action learning.
Virdyn’s collection services can be scoped around the first two layers and associated processing requirements. Any robot-action mapping should be explicitly defined rather than assumed.
A Practical Workflow for Building Custom 3D Training Data
1. Write a Data Specification
Document environments, task categories, objects, required sensors, annotations, and exclusions.
Specify the unit of delivery: frames, clips, complete episodes, reconstructed scenes, or another clearly defined asset.
Avoid treating recording hours as the only measure of value.
2. Run a Representative Pilot
Capture a small sample containing difficult conditions before expanding production.
Inspect whether the hands remain visible, important surfaces reconstruct adequately, and task boundaries can be annotated consistently.
Test the pilot in the intended ingestion pipeline. A visually impressive sample may still be unsuitable for training.
3. Define Synchronization and Calibration Checks
Agree on camera calibration files, timestamp units, coordinate systems, and pose conventions.
Where multiple sensors are involved, define acceptable alignment error and how it will be measured.
Camera pose also needs clear provenance: distinguish estimated trajectories from independently measured reference poses.
4. Build an Annotation Handbook
Define object classes, action boundaries, occlusion handling, and ambiguous cases.
For a warehouse task, clarify whether “pick” begins at initial hand movement, first contact, or object lift.
Review difficult examples jointly before increasing annotation volume.
5. Deliver a Reproducible Data Package
Virdyn’s supplied service brief lists formats including MP4, AVI, CSV, JSON, and PLY.
Formats alone are insufficient. Request a schema, manifest, calibration records, processing notes, and a versioned quality report.
The handover should let another engineer interpret the data without relying on undocumented assumptions.
Data Quality Metrics That Matter More Than Volume
A useful acceptance framework combines coverage, integrity, and task relevance.
| Quality dimension | What to inspect |
|---|---|
| Coverage | Required environments, objects, actions, and difficult conditions |
| Geometry | Missing surfaces, reconstruction artifacts, and scale consistency |
| Timing | Frame loss, timestamp continuity, and cross-sensor alignment |
| Labels | Boundary consistency, class confusion, and reviewer agreement |
| Independence | Overlap between training and evaluation environments |
| Traceability | Capture session, hardware setup, processing version, and rights records |
Set numerical thresholds through pilot measurements and downstream requirements. Avoid adopting a universal “accuracy” target without defining what is being measured.
Also distinguish raw recording duration from accepted delivery duration. Rejected, duplicated, or incomplete sequences should not inflate the usable-data total.
For robotic tasks, measure downstream usefulness separately from annotation correctness. A perfectly labeled collection can still omit essential failure modes.
Combining Real and Synthetic Data Without Hiding the Gap
Simulation offers controlled variation. Teams can change object arrangements, camera positions, or lighting while keeping scene parameters available for inspection.
Real-world capture reveals conditions that may be poorly represented in simulation, including visual artifacts, clutter, and unpredictable human behavior.
A practical approach is to:
-
Establish a real-world evaluation set.
-
Identify specific coverage gaps.
-
Generate synthetic examples targeting those gaps.
-
Compare performance with and without the synthetic additions.
-
Retain additions that improve relevant outcomes.
Keep synthetic and real-data provenance separate. Simulation should be evaluated as a tool for addressing a measured problem, not presented as an automatic guarantee of real-world performance.
Build a Dataset Around the Task You Need to Solve
Public datasets offer valuable benchmarks. Custom collection becomes important when deployment environments, operational sequences, or sensing requirements fall outside their coverage.
Share your target task, working environment, sensor requirements, and expected training format with Virdyn. Request a representative pilot and an agreed acceptance checklist before commissioning full-scale collection.
That creates a clearer path from recorded activity to training assets your engineering team can actually use.

Frequently Asked Questions
What is the best 3D point cloud dataset for robotics?
There is no universal winner. ScanNet supports indoor understanding, while driving datasets support outdoor perception. Manipulation projects may need additional demonstrations, action information, and custom capture.
Can stereo video become a point cloud?
Yes, through calibrated stereo reconstruction. However, the resulting geometry depends on calibration, image quality, visible texture, occlusion, and the reconstruction method. Request validated outputs rather than assuming every stereo recording produces usable geometry.
Are public datasets free for commercial use?
Not necessarily. Review the exact dataset license and release conditions. Some resources distinguish non-commercial research access from commercial licensing.
How much custom training data should a project collect?
Start with a representative pilot and learning experiments. Expand collection around measured coverage gaps and failure cases rather than choosing an arbitrary recording-hour target.
Does egocentric video provide robot action labels?
Not by itself. It records a human perspective. Robot-compatible action supervision requires additional definitions, measurements, or mapping procedures appropriate to the target platform.
What should I request from a data collection provider?
Request a sample package, modality list, annotation handbook, rights documentation, acceptance criteria, delivery schema, and quality report. Confirm which outputs are directly captured and which are reconstructed or inferred.
References
1. Virdyn / VDEgo. “Real-World Egocentric Video Data Collection Services.” Primary company source for VDEgo-C2 positioning, multimodal capture, cleaning, annotation, and quality-review services. Additional service details derive from the company brief supplied for this article; no customer reviews were supplied. https://vdegoai.com/
2. iMerit. “27 3D Point Cloud Datasets to Enhance Your Computer Vision Models.” Source article reviewed and substantially restructured; its directory format informed the comparison scope, not an endorsement of every listed resource. https://imerit.ai/resources/blog/top-27-3d-point-cloud-datasets-in-computer-vision/
3. Dai, A., et al. “ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes.” CVPR, 2017. Primary research source for ScanNet’s acquisition scale, reconstruction outputs, and annotations. https://openaccess.thecvf.com/content_cvpr_2017/papers/Dai_ScanNet_Richly-Annotated_3D_CVPR_2017_paper.pdf
4. KITTI Vision Benchmark Suite. “3D Object Detection Evaluation.” Official benchmark documentation for training and testing point-cloud counts and evaluation scope. https://www.cvlibs.net/datasets/kitti/eval_3dobject.php
5. nuScenes. Official dataset website. Source for sensor configuration, scene count, dataset characteristics, and licensing information. https://www.nuscenes.org/
6. Waymo. “About — Waymo Open Dataset.” Official overview of the autonomous-driving dataset program. https://waymo.com/open/
7. ShapeNet. “ShapeNetCore,” dataset repository. Source for the version-two collection’s object categories, approximate model count, and access conditions. https://huggingface.co/datasets/ShapeNet/ShapeNetCore
8. Khazatsky, A., et al. “DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset.” Official project page. Source for demonstration trajectories, interaction hours, scenes, and tasks. https://droid-dataset.github.io/
Hot Tags: 3D Point Cloud Datasets, AI Data Collection Services, Point Cloud Annotation Services, Custom AI Datasets, Egocentric Data Collection, Robotics Training Data, Multimodal Datasets, 3D Data Annotation, Embodied AI Training Data, Synthetic Data Generation
SHARE



