NVIDIA uses AI agents to train Microduck
NVIDIA has published a workflow in which AI agents plan, build and test a simulated obstacle course for the small Microduck robot, showing how its robotics stack can automate much of sim-to-real preparation.
NVIDIA detailed on its technical blog a tutorial-style workflow in which AI agents handle planning, asset creation and simulation setup for training Microduck, a small reference robot, to navigate an obstacle course.
The pipeline starts with a natural-language prompt describing the course and what the robot must learn. Running locally on NVIDIA RTX Spark, the Hermes agent and NVIDIA Nemotron 3.5 Bolt turn that prompt into a project plan. They review Microduck's repository and documentation, generate a photorealistic concept image of the course with a local image model, and build an inventory of reusable assets from it. Each asset gets an identifier, a description and a cropped reference image, and the inventory notes where objects are cut off or hidden.
Hermes then packages the plan, research, concept and asset inventory and hands the work to a second agent, Astra. Astra uses cloud image editing to sharpen the low-resolution crops, then builds base geometry in FreeCAD and adds detail and materials in Blender, with both tools run programmatically. It exports the results as USD assets and records dimensions it cannot infer from the images as modeling assumptions. Developers can also draw on SimReady Central Library, which NVIDIA describes as a free library of more than 2,000 simulation-ready OpenUSD assets.
Policy training runs in the cloud with Isaac Lab, with Newton handling physics and Omniverse libraries rendering visual observations. Trained policies come back to RTX Spark for evaluation in Isaac Sim.
The post reports no training metrics or success rates. NVIDIA says real-world deployment will need further work to close the gap between simulated and physical behavior.