Introduction

Physical AI is transitioning from theoretical lab demonstrations to real-world applications, where machines must perceive, analyze, and act safely. At SIGGRAPH 2025 in Vancouver (August 10-14), NVIDIA Research showcased groundbreaking advancements in graphics and simulation that facilitate this shift. They unveiled new tools and papers that connect neural rendering, 3D world generation, and physics-based simulation to robotics, autonomous vehicles, and digital content creation.

---

What is Physical AI and Why Now?

Physical AI refers to the technology stack that allows AI systems to comprehend and interact within the physical environment. It integrates neural graphics, synthetic data generation, physics-based simulation, reinforcement learning, and reasoning models. NVIDIA emphasizes the feedback loop between simulation and AI: better simulation leads to improved model training, and smarter models enhance simulation realism. This interconnection is pivotal to the company’s research agenda and its announcements at SIGGRAPH 2025.

---

Key Takeaways at a Glance

• Omniverse NuRec Libraries: NVIDIA introduced libraries to reconstruct expansive 3D worlds from real sensor data, utilizing RTX for enhanced rendering, thus strengthening the transition from real capture to simulation.
• Cosmos Expansion: The Cosmos family now includes Cosmos Reason, a customizable vision-language reasoning model tailored for physical AI tasks.
• Metropolis Platform Updates: Enhancements to the Metropolis platform facilitate the development of video analytics agents and smart infrastructure integration.
• Research Papers: NVIDIA researchers presented findings on topics including inverse and forward neural rendering, differentiable visibility, physics-aware geometry, and motion synthesis grounded in physical realities.

---

How Physical AI Unites Graphics, Simulation, and Robotics

To effectively train robots or autonomous systems in simulations so that their skills transfer to the real world, developers require virtual environments that mimic reality. This involves photorealistic rendering, realistic materials, accurate lighting, and reliable physics, along with consistent 3D geometry reconstructed from images and videos. NVIDIA’s research groups are advancing in these areas, emphasizing the reciprocal relationship between forward rendering (3D to 2D) and inverse rendering (2D to 3D) through AI.

---

Omniverse NuRec: Reconstructing the Real World for Simulation

NuRec consists of models and services that process camera and lidar data to reconstruct 3D scenes and render them in Omniverse and Isaac Sim using neurally encoded representations, like 3D Gaussian splats. The new NuRec libraries feature RTX-accelerated, ray-traced 3D Gaussian splatting, allowing for large-scale world reconstruction with seamless transitions from capture to simulation stages.

Key Features:

• What it Does: Converts real-world sequences into sim-ready 3D scenes, formatted as USDZ for use in Omniverse and Isaac Sim.
• Why It Matters: Realistic replay of actual environments narrows the simulation-to-real gap and generates high-quality synthetic data.
• Where to Start: Refer to NuRec documentation, explore the Isaac Sim 5.0 neural volume rendering, and utilize Omniverse neural rendering guidance.

These advancements were highlighted alongside updates that improve interoperability between MuJoCo (MJCF) and OpenUSD, along with enhancements to Isaac Sim and Isaac Lab for robot learning.

---

From Forward to Inverse: Neural Rendering Progress

NVIDIA’s DiffusionRenderer merges inverse and forward rendering into a cohesive neural framework. It estimates geometry buffers from real video for editing and training, synthesizing photorealistic frames from these buffers without requiring full light-transport simulation. This enables creators to relight scenes, edit materials, or insert objects while simultaneously producing synthetic training data that resembles real environments.

On the representation front, the 3D Gaussian Unscented Transform (3DGUT) extends Gaussian splatting to accommodate distorted cameras and secondary rays, aligning rasterization-friendly splats with physics-based ray tracing. This advancement helps eliminate trade-offs between speed and fidelity in rendering.

---

Cosmos World Foundation Models and Cosmos Reason

Cosmos represents a family of world foundation models tailored for physical AI. Cosmos Transfer accelerates photorealistic synthetic data generation from simulation scenes or spatial controls, while a distilled variant enhances throughput by compressing multi-step sampling into a single step. Cosmos Reason focuses on reasoning with images and video via a chain-of-thought approach, facilitating tasks like data curation, video analytics, and high-level planning for robots. Research on Cosmos-Reason1 reveals supervised fine-tuning and reinforcement learning techniques that enhance performance on benchmarks for physical common sense and embodied decision-making.

Why It Matters:

If large language models have transformed how software reasons about text, world foundation models aim to do the same for physical perception and action by encapsulating world dynamics and semantics into reusable components. Cosmos models are accessible for post-training on specific robotics and AV tasks, seamlessly integrating with Omniverse and Isaac pipelines.

---

Metropolis and Blueprint Updates for Smart Infrastructure

For scalable urban and industrial applications, NVIDIA is enhancing the Metropolis platform and its blueprints, which blend perception, retrieval, and reasoning. The Video Search and Summarization blueprint integrates models like Cosmos Reason, enabling operators to traverse extensive video archives, detect events, and generate concise summaries. Hardware support spans from edge GPUs to data center servers, connecting operations from factory floors to the cloud.

---

Research Highlights from SIGGRAPH 2025

Physics-Aware 3D Geometry from Images and Video

One major challenge in converting 2D footage into 3D assets is ensuring structural stability. Shapes may appear correct but fail simulations if they lack physically plausible geometry. NVIDIA researchers revealed techniques that embed physics-aware constraints into reconstruction, allowing virtual objects to behave like their real counterparts. This fosters the creation of robust training environments for robots and AVs.

Physically Grounded Motion: From Parkour to Agile Robots

Another area of research combines learned motion generation with physics-based tracking controllers to create lifelike movements for complex maneuvers, such as parkour. This generates synthetic data for challenging behaviors and enhances training for agile humanoids and virtual characters. Literature showcases iterative loops that produce motions and rectify simulation artifacts, expanding what controllers can learn.

Differentiable Visibility for Inverse Rendering

Differentiable rendering poses challenges in estimating how geometry changes affect visibility. A SIGGRAPH 2025 paper suggests a robust, efficient estimator through fixed-step walks on spherical caps and silhouette queries, leading to quicker, more stable geometry reconstruction from images and video. This bolsters the inverse rendering pipelines critical to physical AI.

Generative Detail Enhancement for Materials

Artists often spend hours adding subtle effects like weathering or aging to materials. NVIDIA’s research introduces a tool that employs a diffusion model and a differentiable physically-based renderer to create coherent multi-view material edits from text prompts, efficiently integrating these edits into PBR texture maps.

ViPE: Video Pose Engine for 3D Geometric Perception

Accurate camera intrinsics, poses, and dense depth from in-the-wild video are essential for creating spatial AI datasets. ViPE reliably estimates these across various video types, running at 3-5 FPS on a single GPU, and has already annotated around 96 million frames. NVIDIA has released the technical report and datasets to expedite research and product development.

---

From Research to Products: The Emerging Physical AI Stack

Hardware and Platforms

Physical AI workloads are facilitated across edge devices and data centers. New RTX Pro GPUs and Jetson Thor cater to developers requiring on-device reasoning and high-throughput perception, while Blackwell-based workstations manage training, world generation, and large-scale simulations. These platforms are purpose-built to cohesively run Omniverse, Isaac, Metropolis, and Cosmos models.

Developer Entry Points

• Cosmos: World foundation and reasoning models for video and images, with post-training functionalities available in NeMo and DGX Cloud.
• Omniverse + NuRec: Neural reconstruction and rendering transforming real video into USD scenes for simulation.
• Isaac Sim and Isaac Lab: Open-source simulation and robot-learning frameworks integrating NuRec support and new sensor and robot schemas.
• Metropolis + Blueprints: Building vision AI agents that incorporate reasoning and retrieval capabilities.

Practical Examples

• Robotics: Utilize NuRec to reconstruct factory aisles, synthesize diverse conditions with Cosmos Transfer, and post-train a robot policy using Isaac Lab.
• Autonomous Vehicles: Generate world-consistent video data with DiffusionRenderer, apply varying lighting and weather conditions, and evaluate planners against edge cases.
• Content Creation: Enhance materials with diffusion-driven inverse rendering and refine scene lighting with neural rendering to accelerate creative processes.

---

How This Shifts the State of the Art

Neural rendering has made the creation of lifelike 3D visuals from 2D inputs feasible. World foundation models package spatial and temporal structures into reusable building blocks. Differentiable rendering strengthens this ecosystem by enabling learning from images how environments should look and behave. The updates showcased at 2025 illustrate how these components are merging into a comprehensive stack that converts everyday video into simulatable environments and elevates simulators into data engines for robots and AVs.

---

Getting Started: Resources

• SIGGRAPH 2025: Overview and schedule for watching talks and accessing papers.
• NVIDIA’s SIGGRAPH Hub: Contains sessions, certifications, and demos.
• Press Materials: Summaries highlighting updates on Omniverse, Cosmos, and blueprints revealed during the event.

---

Conclusion

The journey toward physical AI hinges on better reconstructions, smarter simulations, and models capable of reasoning about space, time, and causality. NVIDIA Research’s contributions at SIGGRAPH 2025 advance these pillars through NuRec for world capture, Cosmos for world understanding and generation, and a series of research papers that link inverse and forward rendering into a practical toolchain. For teams developing robotics, autonomous vehicles, and spatial applications, the message is clear: the processes to transition from pixels to policies – and back – are finally merging.

---

FAQs

What is Physical AI?

Physical AI encompasses AI systems designed to perceive, reason, and act in the real world. It ties together neural rendering, physics simulation, synthetic data, and reasoning models, enabling trained policies to transfer effectively from simulation to reality.

How Does NuRec Differ from Classic Photogrammetry or NeRF Pipelines?

NuRec specializes in end-to-end ingestion-to-simulation for Omniverse and Isaac Sim, utilizing 3D Gaussian splats coupled with RTX ray tracing. It exports to USDZ and directly integrates into simulation workflows, unlike standalone reconstruction tools.

What is Cosmos Reason Used For?

Cosmos Reason is a vision-language reasoning model specifically fine-tuned for physical AI applications. It assists with data curation and annotation, serves video analytics agents, and facilitates high-level planning in robotic systems, with the ability for post-training on targeted tasks.

Why Are Differentiable Rendering Papers Relevant to Robotics?

Techniques in inverse rendering that extract geometry, materials, and lighting from images enhance dataset integrity and simulation accuracy. Improved reconstructions result in better synthetic data and more reliable transitions from simulation to real-world applications.

Where Can I Explore These Models and Tools?

Begin with NVIDIA’s Cosmos page for model families, review the NuRec and Omniverse documentation for reconstruction and rendering, and utilize Isaac Sim for simulation and robotic learning.