> ## Documentation Index > Fetch the complete documentation index at: https://kensou.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # 3dgs as simulator comparison # 3DGS-as-Simulator: Manipulation vs Drone vs AD **Status:** v1 — opinionated draft. Compute / latency / asset-size numbers marked `UNVERIFIED` need rig-side validation. **Wedge tier:** W1 **TL;DR:** Three communities are calling the same idea "3DGS-as-simulator" while building three very different things. Splat-Sim (manipulation) trades hours-per-scene fit for grasp-loop fidelity. Aerial Gym-lineage drone sim2real trades fidelity for *millions of parallel rollouts*. NVIDIA Cosmos / AD scene reconstruction trades both for *editable counterfactuals*. **The fidelity-vs-controllability trade-off resolves in opposite directions per embodiment — and the first product-ready 3DGS-sim will not be the most photorealistic one.** *** ## 1 · Why this comparison is overdue In the last 18 months, every embodiment community discovered that **3DGS reconstructs a scene faithfully enough to render synthetic training data**. Three independent waves landed: * **Manipulation** — Splat-Sim (Qureshi et al. 2024), RoboGS, GaussianGrasper — fit a scene from a few minutes of phone capture, drop in articulated robot URDFs, train policies via domain randomization on top. * **Drone / aerial** — Aerial Gym lineage (ETH Zürich / NVIDIA), Splat-Nav (Chen et al. 2024), GS-Splatter sim — fit an outdoor environment, fly a quadrotor through it in massively parallel sims, transfer policies to real flight. * **AD** — NVIDIA Cosmos, Waymo's "Block-NeRF → 3DGS" lineage, UniSim, MARS — reconstruct logged driving scenes, *edit* them (add traffic, change weather), use as a closed-loop validation environment. Looked at separately, each looks like the same paper rewritten with different keywords. Looked at together, **the architectures, compute budgets, fidelity targets, and product paths are different enough that they will not converge in 2026**. That's the crossing question. *** ## 2 · The three teams side by side | | Manipulation (Splat-Sim lineage) | Drone (Aerial Gym lineage) | AD (Cosmos / UniSim) | | ------------------------------------- | -------------------------------- | ----------------------------------------- | ----------------------------------------- | | Scene scale | 1–3 m³ tabletop / cabinet | 100–10000 m² outdoor | 100m–km driving log | | Capture | 2–5 min phone video | drone pre-flight or aerial photogrammetry | sensor-rig log (camera + LiDAR + GNSS) | | Fit time `UNVERIFIED` | 5–30 min per scene | 30 min–4 hr | hours–days per log | | Render rate target | 30 Hz for grasp servo | **1k+ Hz across parallel envs** | 10 Hz for closed-loop eval | | Fidelity target | photometric + contact geometry | flyable obstacles, sky, no perfect ground | drivable + dynamic agents editable | | Domain randomization style | textures, lighting, object poses | wind, lighting, sensor noise | traffic, weather, agent behavior | | Asset count per scene | 1 scene + N objects (URDFs) | 1 scene + 1 ego model | 1 scene + 10s of dynamic agents | | Real-robot in the loop? | yes (grasp validation) | yes (zero-shot transfer) | rarely (validation env, not training env) | | Today's primary use | policy training | RL training + sim2real | safety case + regression testing | | Compute per training run `UNVERIFIED` | 1 GPU × hours | 8 GPUs × days (massive parallel) | 64+ GPUs × days | The shape of this table is what `crossing/` exists to show. *Same representation, three orthogonal optimization targets.* *** ## 3 · Why fidelity-vs-controllability resolves differently per embodiment Every simulator faces the same triangle: ``` fidelity /\ / \ / \ / \ / \ speed ─────── controllability (parallel) (editable) ``` You can pick two. The interesting part is *which two each embodiment picks*: * **Manipulation picks fidelity + controllability.** Why: contact dynamics is the bottleneck, and you need to spawn random object poses but render them photorealistically (because the policy is a VLM consuming RGB). Speed is sacrificed because one robot needs one rollout at a time, not 4096 of them. *Splat-Sim's 30 min fit is fine.* * **Drone picks speed + fidelity.** Why: drone RL training takes *millions* of episodes; sim must run on a GPU at 1000+ Hz. Editability is sacrificed because the agent flies the same wind/obstacle scene over and over. *Aerial Gym does not need to add traffic to your park.* * **AD picks controllability + fidelity.** Why: the value is in counterfactuals ("would the policy have hit this cyclist if we shifted them 1.5 m left?"). Speed is sacrificed because each "rerun" is one drive log, not 10000 parallel episodes — the validation question is sequential, not statistical. *Cosmos renders 10 fps and that's fine for the safety case.* **No two embodiments pick the same two corners.** This is why a "universal 3DGS simulator" is the wrong product. Whoever tries to build it ships something that's mediocre at all three. *** ## 4 · The compute budget reality The hidden axis is per-embodiment economics: * **Manipulation** can afford 30 minutes of scene fit because a teleop demo costs an operator a similar amount of human time. Fit-once-train-many amortizes well. * **Drone** *cannot* afford 4 hours of scene fit per environment because the training pipeline wants 1000s of distinct environments per RL curriculum. Aerial Gym's answer is parametric / procedural environments with 3DGS-style appearance baked on top — not full per-scene fits. * **AD** can afford days of fit per log because logs are precious (millions of miles of fleet data, the company already owns them). Per-log fit is a one-time cost amortized over years of regression testing. Number to anchor: a 3DGS scene of moderate size (\~500k Gaussians) fits in 5–10 min on an RTX 4090 `UNVERIFIED`. Same scene at AD-grade (10M+ Gaussians, dynamic agents, weather conditioning) is hours–days on an 8×H100. **That gap is two orders of magnitude.** No single rendering stack survives it. *** ## 5 · When 3DGS-as-sim beats traditional sim — and when it doesn't Traditional simulators (Isaac Sim, AirSim, CARLA, MuJoCo) have a 10-year head start on contact dynamics, parallelism, and editability. 3DGS-as-sim wins specifically when: * **Photometric realism is the bottleneck** — VLM-based manipulation policies, end-to-end driving from camera. Splat-Sim and Cosmos win here because mesh-based PBR is still uncanny at close range. * **Real-scene transfer is the bottleneck** — "I want to test my policy on *this exact warehouse*". 3DGS fits the warehouse in 30 min; building a CAD model of the warehouse takes weeks. * **Counterfactuals on logged data** — "what if the pedestrian had been wearing a yellow jacket?". Editing a 3DGS scene is easier than editing a CAD scene because the representation is point-cloud-native. It *loses* when: * **Contact dynamics matter** — 3DGS gives you appearance, not friction or compliance. Splat-Sim still uses MuJoCo / PhysX under the hood; the 3DGS only renders. Pure 3DGS does not simulate grasping. * **Massive parallelism matters** — Isaac Sim runs 4096 envs on one GPU. 3DGS at 1000 Hz across 4096 envs is research-stage; the Aerial Gym work has shown one piece of this but not the full stack. * **Generalization across scenes matters** — A policy trained in one fitted scene overfits. You still need procedural / parametric environments for breadth. The pattern: **3DGS-as-sim is the appearance backend; the physics, parallelism, and editability backends are still being filled in per embodiment**. *** ## 6 · The product paths in 2026 ``` Manipulation (Splat-Sim) Drone (Aerial Gym) AD (Cosmos / UniSim) ───────────────────────── ────────────────── ──────────────────── capture warehouse cell fly recon over field fleet log (camera+LiDAR) │ │ │ ▼ ▼ ▼ fit 3DGS (30 min) fit 3DGS (hours) fit 3DGS + agents (days) │ │ │ ▼ ▼ ▼ spawn URDF robot, load into Aerial Gym, edit scene (add agents, randomize objects + texture run 10M RL rollouts change weather) │ │ │ ▼ ▼ ▼ train VLM policy transfer policy to real closed-loop policy eval │ │ │ ▼ ▼ ▼ deploy on real robot flight test safety case + ship update ``` The three pipelines look superficially identical and architecturally diverge at every stage past "fit 3DGS". *** ## 7 · 2-year outlook + falsifiable prediction What's actually shipping by 2027: 1. **Manipulation** — Splat-Sim-style pipelines integrated into Isaac Sim / NVIDIA GR00T workflow. Photoreal cell scanning at consumer phone capture quality. *Most product-ready first.* 2. **Drone** — Aerial Gym + 3DGS appearance for racing / inspection RL. Sim2real gap narrows but doesn't close; outdoor wind and lighting still bite. 3. **AD** — Cosmos and equivalents (Waymo / Tesla internal) become safety-case standard. Public-facing 3DGS-as-AD-sim ships as a benchmark / leaderboard, not a training environment. **Falsifiable prediction:** before 2027-12, a manipulation-policy stack will ship a commercial product (likely Physical Intelligence π0-class or Skild AI) that uses Splat-Sim-style 3DGS rendering as a *training-time* simulator and ships a successful real-world VLA policy on top. **No equivalent product will ship in drone or AD by the same date** — both stay in research / internal-validation use through 2027. Bet against any AD vendor claiming "trained in 3DGS sim" before 2028. *** ## For the reader * **Manipulation engineer** — Splat-Sim is your near-term win. Don't expect it to fix your contact dynamics; pair it with MuJoCo / PhysX. Watch the GR00T / Cosmos integration story. * **Drone engineer** — Aerial Gym + 3DGS is the cheapest sim2real path for outdoor inspection. Don't expect 1000-Hz photoreal rendering on a single GPU yet. Watch the ETH / NVIDIA collab. * **AD engineer** — Cosmos-class tooling is a validation environment, not a training environment, for at least 2 more years. Use it to *score* policies, not train them. * **Researcher** — the open lane is *editable + parallel + photoreal*. Nobody owns all three. Whoever closes that triangle wins the next two NeurIPS cycles. *** ## References * Splat-Sim — Qureshi et al. 2024. [https://arxiv.org/abs/2409.10161](https://arxiv.org/abs/2409.10161) `UNVERIFIED ID` * Aerial Gym — Kulkarni et al. *IROS 2023* / NVIDIA. [https://arxiv.org/abs/2305.16510](https://arxiv.org/abs/2305.16510) * Splat-Nav — Chen et al. 2024. [https://arxiv.org/abs/2403.02751](https://arxiv.org/abs/2403.02751) `UNVERIFIED ID` * NVIDIA Cosmos — [https://www.nvidia.com/en-us/ai/cosmos/](https://www.nvidia.com/en-us/ai/cosmos/) (product page; technical report TBD) * UniSim — Yang et al. *CVPR 2023*. [https://arxiv.org/abs/2308.01898](https://arxiv.org/abs/2308.01898) `UNVERIFIED ID` * Block-NeRF — Tancik et al. *CVPR 2022*. [https://arxiv.org/abs/2202.05263](https://arxiv.org/abs/2202.05263) * 3DGS — Kerbl et al. *SIGGRAPH 2023*. [https://arxiv.org/abs/2308.04079](https://arxiv.org/abs/2308.04079) * Isaac Sim — NVIDIA. [https://developer.nvidia.com/isaac-sim](https://developer.nvidia.com/isaac-sim) * CARLA — Dosovitskiy et al. *CoRL 2017*. [https://arxiv.org/abs/1711.03938](https://arxiv.org/abs/1711.03938) ## Boundary This doc compares *use-cases* of 3DGS-as-simulator across embodiments. It does **not** dissect 3DGS internals (that's `foundations/3dgs/3dgs_siggraph2023_dissection.md`), nor Splat-Sim implementation (`embodiments/manipulation/sim/`), nor Cosmos data engineering (`embodiments/driving/sim/`). The per-embodiment scene-scale and sensor-stack details live in `crossing/scale-comparison/` and `crossing/sensor-stack-matrix/`. *** *Last opinion update: 2026-05-21. §7 prediction will be scored at 2027-12.*