
Lessons from Jim Fan
Jim Fan co-leads NVIDIA’s GEAR research group, which studies agents that act in virtual environments and physical robots. He describes robotics as a data-and-systems challenge requiring internet-scale examples, simulation, and real-robot experience. — Sequoia Training Data Interview.
Part 1: Start With Open-Ended Worlds
- On choosing Minecraft: MineDojo treated Minecraft as a serious research environment because it contains long-horizon tasks, diverse objects, open-ended goals, and a giant external knowledge base. — MineDojo Paper
- On internet-scale embodied data: MineDojo's core bet was that an agent could learn from Minecraft videos, wiki pages, and community discussions instead of relying only on handcrafted reward functions. — NVIDIA MineDojo Technical Blog
- On broad task coverage: The MineDojo benchmark was designed around thousands of open-ended tasks rather than a small set of narrow game objectives. — MineDojo Project
- On research prizes: MineDojo won a NeurIPS 2022 Outstanding Datasets and Benchmarks Paper Award, which helped validate Minecraft as more than a toy domain. — NVIDIA NeurIPS Award Announcement
- Use simulation to scale exploration before hardware: Fan describes simulation as a high-throughput data source that complements expensive real-robot data, while acknowledging a sim-to-real gap. The original cheapest-place superlative is not established. — Sequoia Training Data Interview.
- Release reusable research infrastructure: The MineDojo team released its simulation suite, knowledge bases, learning implementation and pretrained models to enable follow-on embodied-agent research. — MineDojo Paper.
Part 2: Turn Language Models Into Skill Builders
- On Voyager's claim: Voyager was framed as the first LLM-powered embodied lifelong learning agent in Minecraft, continuously exploring, acquiring skills, and making discoveries without human intervention. — Voyager Paper
- On code as action: Voyager's key move was using GPT-4 to write executable code, turning language reasoning into actions inside the game. — Voyager Project
- On the skill library: Voyager stores successful programs in a persistent skill library, so progress becomes accumulated code rather than a temporary chat transcript. — Voyager Project
- On self-debugging: The agent uses environment feedback and execution errors to revise its own code, making debugging part of the learning loop. — Voyager Project
- On curriculum design: Voyager's automatic curriculum pushes it toward novelty, so it keeps searching for new items and capabilities instead of repeating solved tasks. — Voyager Project
- Store skills as reusable programs: In Voyager, successful executable programs enter a retrievable skill library, allowing later tasks to reuse and combine learned behaviors. The paper does not establish world models or prompt length as the general alternative. — Voyager Project.
Part 3: Make Robotics Promptable
- On VIMA's premise: VIMA showed that robot manipulation tasks could be expressed through multimodal prompts that interleave text and visual tokens. — VIMA Paper
- On general robot manipulation: VIMA treated manipulation as a sequence-modeling problem, using a transformer agent that reads prompts and autoregressively outputs motor actions. — VIMA Project
- Measure prompt-based manipulation at scale: In simulated tabletop tasks, the VIMA paper reports improved zero-shot generalization relative to comparison models and data-efficient training. This is a measured benchmark result, not proof that robotics follows every language-model pattern. — VIMA Paper.
- On early foundation models: Fan describes VIMA as one of the early multimodal foundation models for robot manipulation, placing it in the lineage before today's humanoid models. — Jim Fan Personal Site
- Condition robot actions on multimodal prompts: VIMA encodes text-and-visual task prompts and uses a transformer controller to predict motor actions in simulated tabletop manipulation. It does not establish one deployed policy for all vision, language and motor-control products. — VIMA Project.
- Test across diverse prompted tasks: VIMA-Bench includes varied task templates and procedurally generated instances to evaluate how a prompted manipulation model generalizes. The paper does not test transfer to real homes or factories. — VIMA Project.
Part 4: Automate the Bottlenecks
- On Eureka's target: Eureka attacked reward design, one of the most stubborn human bottlenecks in reinforcement learning. — Eureka Paper
- On coding rewards: Eureka used coding LLMs to generate and improve reward functions, turning reward design into an iterative program-search problem. — Eureka Paper
- On simulation loops: Eureka used GPU-accelerated simulation in Isaac Gym to evaluate batches of candidate reward programs quickly. — Eureka Project
- On robot dexterity: Fan highlights Eureka because it taught a five-finger robot hand difficult skills like pen spinning, showing that language models can help with physical control indirectly. — Jim Fan Personal Site
- Automate part of reward design: Eureka tests whether a coding language model can generate and refine reward functions that experts previously hand-designed. It does not show that the full robotics research process can be automated. — Eureka Paper.
Part 5: Build Foundation Models for Humanoids
- On Project GR00T: NVIDIA announced Project GR00T as a general-purpose foundation model initiative for humanoid robots. — NVIDIA Project GR00T Announcement
- On humanoid learning: GR00T is meant to help robots understand natural language and learn coordination, dexterity, and other skills from human actions. — NVIDIA Project GR00T Announcement
- On platform thinking: The GR00T work sits inside a broader Isaac robotics stack that includes simulation, data generation, and workflow infrastructure. — NVIDIA Project GR00T Announcement
- On GR00T N1: GR00T N1 was released as an open foundation model for generalist humanoid robots. — NVIDIA GR00T N1 Research
- On dual systems: GR00T N1 uses a vision-language module for interpretation and a diffusion transformer module for generating fluid motor actions. — GR00T N1 Paper
- On data mixture: The GR00T N1 recipe uses real robot trajectories, human videos, and synthetic data rather than betting on one data source. — GR00T N1 Paper
- On deployment: The paper reports deployment on a Fourier GR-1 humanoid for language-conditioned bimanual manipulation. — GR00T N1 Paper
- Build an adaptable humanoid platform: NVIDIA says its GR00T and Isaac effort includes model, simulation and compute infrastructure for multiple humanoid-robot partners. This is a stated platform strategy, not proof of adoption by every partner. — NVIDIA Project GR00T Announcement.
- On synthetic data: GR00T-Gen and related workflows use simulation-ready environments and synthetic trajectories to expand robot training coverage. — NVIDIA GR00T Technical Blog
Part 6: Treat Human Video as Robot Fuel
- On EgoScale: EgoScale reframed dexterous robot learning as a scaling problem over egocentric human video. — EgoScale Project
- On data scale: EgoScale trained on more than 20,854 hours of action-labeled egocentric human video, over 20 times larger than prior efforts cited by the project page. — EgoScale Project
- On the scaling law: The EgoScale team reported a near-perfect log-linear relationship between human data scale and validation loss. — EgoScale Project
- On robot transfer: EgoScale combined large-scale human pretraining with a small amount of aligned human-robot mid-training to improve downstream dexterous manipulation. — EgoScale Paper
- On high-DoF hands: The policy improved average success rate over a no-pretraining baseline using a 22-degree-of-freedom robotic hand. — EgoScale Paper
- Use human video to scale dexterity training: EgoScale shows that large human-video pretraining can improve dexterous robot learning when paired with a smaller aligned human-robot stage. The experiments do not establish that teleoperation data is broadly obsolete. — EgoScale Paper.
- On embodiment transfer: EgoScale's transfer across higher- and lower-DoF hands supports the idea that human motion can become an embodiment-agnostic training signal. — EgoScale Paper
Part 7: Move From Simulation to World Models
- On DreamGen: DreamGen uses video world models to generate neural trajectories that help robot policies generalize across behaviors and environments. — DreamGen Project
- On pseudo-actions: Because video models generate videos rather than robot commands, DreamGen recovers pseudo-action sequences through latent action or inverse-dynamics models. — DreamGen Paper
- On zero-shot generalization: DreamGen reports zero-shot behavior and environment generalization after using teleoperation data from only a single pick-and-place task in one environment. — DreamGen Project
- On DreamDojo: DreamDojo is a generalist robot world model trained from large-scale human videos, then post-trained on small target-robot data. — DreamDojo Paper
- Release DreamDojo research artifacts: NVIDIA published a DreamDojo model card and links to model files, its paper, project page and code. This shows a concrete artifact release without claiming every part of the research is open. — NVIDIA DreamDojo Model Card.
- On DreamZero: DreamZero introduced a World Action Model built on a pretrained video diffusion backbone. — DreamZero Paper
- Model future video and actions together: The DreamZero authors argue that jointly predicting future video and actions can address some physical-motion generalization limits of vision-language-action models. This is their research claim, not proof that VLAs have been replaced. — DreamZero Paper.
Part 8: The Research Operating System
- On GEAR's mission: GEAR's stated mission is to build foundation models for embodied agents in both virtual and physical worlds. — NVIDIA GEAR Lab
- On the research portfolio: GEAR organizes its work around multimodal foundation models, general-purpose robots, foundation agents, and simulation or synthetic data. — NVIDIA GEAR Lab
- On career positioning: Fan's own biography connects Stanford vision research, early OpenAI experience, NVIDIA robotics, and a consistent interest in agents that act in worlds. — Jim Fan Personal Site
- On OpenAI roots: Fan notes that he was OpenAI's first intern in 2016 and worked on World of Bits, an early agent project for browser control from pixels. — Jim Fan Personal Site
- On public thought leadership: Fan's recent AI Ascent appearance placed his robotics argument in front of founders and researchers focused on the next phase of AI infrastructure. — Sequoia AI Ascent 2026
- Combine complementary robot-data sources: In a direct interview, Fan argues that robot foundation models need internet-scale examples, simulation and real-robot data because each supplies something the others lack. The old exact 2026 staged sequence was not verified from the talk recording. — Sequoia Training Data Interview.
- Build a virtual-and-physical research stack: GEAR’s published research agenda combines multimodal and world models, general-purpose robots, agents in simulations, and synthetic-data infrastructure. This is an agenda, not a forecast with an established timeline. — NVIDIA GEAR Lab.