# Lessons from Jitendra Malik
Jitendra Malik is a UC Berkeley computer vision researcher whose group contributed to image segmentation, recognition and robotics. He argues that perception and action in the physical world are important for developing broader AI. — Berkeley Faculty Research Lecture.

Part 1: The Sensorimotor Road to AI
- The Goal of AI: Malik endorses an embodiment hypothesis drawn from Smith and Gasser: interaction, perception, physical-world models and grounded language all matter for broader AI, even though AI need not copy infant development exactly. — Malik's Foundation Models Response.
- Action and Perception: Malik uses Gibson's see-to-move and move-to-see formulation to explain why perception and action evolved together. — Berkeley Faculty Research Lecture.
- Limits of Text: Malik regards language models as useful but says text alone omits the sensorimotor grounding involved in broader intelligence. — Malik's Foundation Models Response.
- Physical Grounding: Malik identifies interaction with a physical world and models of commonsense physics as important components of broad AI. — Malik's Foundation Models Response.
- Biological Precedents: Malik argues that perception and movement preceded symbolic language over evolutionary history, motivating his sensorimotor research program. — Berkeley Faculty Research Lecture.
- Vision's Purpose: Malik describes guidance of action as a fundamental role of vision, while acknowledging that people also use vision for purposes such as aesthetic judgment. — Lex Fridman Podcast #110.
- Physical Competence as a Test: Malik contrasts success on language-heavy exams with continuing difficulty in locomotion, driving and manipulation, and emphasizes robustness and adaptation in the physical world. — Berkeley Faculty Research Lecture.
Part 2: Lessons from Child Development
- Children's Sensorimotor Learning: Malik points to young children's sensorimotor abilities as a challenge for AI systems, particularly robots that still struggle with everyday physical tasks. — Berkeley Faculty Research Lecture.
- Turing's Child-Mind Proposal: Malik cites Alan Turing's proposal to study an educated child-like machine rather than only an adult mind, and treats development as inspiration for AI research. — Berkeley Faculty Research Lecture.
- The Scientist in the Crib: Crediting Alison Gopnik's phrase, Malik describes infants learning through physical exploration and multiple senses. — Berkeley Faculty Research Lecture.
- Learning Beyond Labels: Malik notes that children can connect touch and sight while exploring an object without needing an adult to supply its label at every step. — Lex Fridman Podcast #110.
- Bridging Development and AI: Malik credits child-development researchers Smith and Gasser for six learning principles and argues that computer vision can draw more from that field. — Lex Fridman Podcast #110.
- Play as Active Learning: Malik favors learning settings where an agent can select actions and learn from their consequences, as children do in play. — Lex Fridman Podcast #110.
- Learning to Walk by Trial and Error: Malik uses infants' repeated attempts at walking to illustrate why his robot-control work trains through many simulated trials. — Berkeley Faculty Research Lecture.
- Multimodal Learning: Malik emphasizes that children combine vision, touch and hearing as they interact with the world, and uses that as inspiration for AI learning. — Berkeley Faculty Research Lecture.
- Active Learning: Malik argues that agents should be able to choose experiences through real interaction or simulation, instead of only passively consuming a fixed dataset. — Lex Fridman Podcast #110.
Part 3: The Three R's of Computer Vision
- The Three Rs: Malik and coauthors organize computer vision around recognition, reconstruction and reorganization, emphasizing the useful information flow among them. — The Three R's of Computer Vision.
- Recognition: Malik defines recognition as assigning categories to an image or object; his three-R framework treats this as only one part of visual understanding. — Lex Fridman Podcast #110.
- Reconstruction: Malik describes reconstruction as inferring the objects, shapes and viewpoint that could have produced an observed image. — Lex Fridman Podcast #110.
- Reorganization: Malik and coauthors show how object recognition can help segment a scene when bottom-up boundaries alone are ambiguous. — The Three R's of Computer Vision.
- Connecting the Three Rs: Rather than treating recognition, reconstruction and grouping as isolated benchmark tasks, Malik and coauthors study how their outputs can improve one another. — The Three R's of Computer Vision.
- Normalized Cuts: Shi and Malik formulated image segmentation as graph partitioning and introduced the normalized-cut criterion to balance separation between groups with coherence within them. — Normalized Cuts Paper.
- Edges and Regions: Perona and Malik's anisotropic-diffusion method smooths within regions while preserving boundaries, showing why region structure and edges must be considered together. — Perona–Malik Diffusion Paper.
- Three-Dimensional Context: Malik and coauthors give examples in which recognition, segmentation and 3D reconstruction inform one another when interpreting cluttered scenes. — The Three R's of Computer Vision.
- Video and Motion: Malik and coauthors use spatial and motion cues together for action recognition and explore reconstruction of moving objects from video. — The Three R's of Computer Vision.
Part 4: Real-World Robotics and Adaptation
- Real-World Robotics: Malik's lecture emphasizes that visual understanding for robots must help guide manipulation and locomotion, not merely produce plausible-looking demonstrations. — TTIC Distinguished Lecture.
- Anticipating Change: In Malik's RMA work, adaptation is designed into the policy from the outset because the robot may encounter conditions it has not seen before. — Berkeley RMA Interview.
- Adaptation by Design: Malik describes environmental adaptation as a forethought rather than an afterthought in the RMA policy. — Berkeley RMA Interview.
- Simulation and Reality: Malik's team trains control policies across varied conditions in simulation, then uses recent movement history to infer conditions that cannot be directly read out in the real world. — Berkeley Faculty Research Lecture.
- Hardware Constraints: Malik explains that extensive real-world trial and error can damage robot hardware, which is one reason his team trains many control-policy trials in simulation. — Berkeley Faculty Research Lecture.
- Proprioception: Malik's robot uses joint-angle and movement history to estimate its environment, and his later work uses proprioceptive signals to help train vision. — Berkeley Faculty Research Lecture.
- Legged Adaptation: The RMA system adjusts a quadruped's control policy rapidly as terrain and payload conditions change, without fine-tuning the deployed robot for each setting. — RMA: Rapid Motor Adaptation.
- Touch in Manipulation: In Malik's coauthored NeuralFeels system, tactile sensing improves estimation of an in-hand object's shape and pose, especially when vision is occluded. — NeuralFeels Paper.
- Action and Feedback: Malik explains rapid motor adaptation as acting, sensing the consequences and using them to revise an estimate of the current environment. — Berkeley Faculty Research Lecture.
Part 5: Deep Learning's Impact and Limits
- R-CNN and Pretraining: Malik identifies his group's R-CNN work as an early use of ImageNet pretraining followed by fine-tuning for object detection. — Malik's Foundation Models Response.
- The Deep-Learning Shift: The coauthored R-CNN work combines learned convolutional features with region proposals, illustrating a shift from purely hand-designed detection features. — R-CNN Paper.
- Biology and Network Design: Malik contrasts deep feedforward vision networks with biological visual pathways that use feedback, and regards the relationship between them as an open research topic. — Lex Fridman Podcast #110.
- Limits of Scaling: Malik welcomes useful large pretrained language models while questioning whether they can serve as a template for all AI without sensorimotor grounding. — Malik's Foundation Models Response.
- SlowFast Pathways: The coauthored SlowFast architecture processes video with a slower pathway for spatial semantics and a faster pathway for temporal motion. — SlowFast Networks Paper.
- Interpreting Learned Models: Asked about opaque neural representations of 3D understanding, Malik says acceptability depends on the setting rather than requiring one universal level of interpretability. — Lex Fridman Podcast #110.
- A Surprise in Nonconvex Learning: Malik says the robustness of deep networks across varied initializations surprised him despite earlier concern about nonconvex optimization; he does not claim local minima are solved in general. — Lex Fridman Podcast #110.
- Shared Visual Representations: Malik and coauthors show how learned representations can support several related visual tasks, including detection, segmentation and reconstruction. — The Three R's of Computer Vision.
Part 6: Rethinking Data and "Labels"
- Relying on Labels: Malik questions training protocols that depend on large supplies of explicit object-boundary labels and looks to weaker supervision and child-like learning instead. — Lex Fridman Podcast #110.
- Self-Supervision: Malik describes touch and vision of the same object as a cross-calibration signal that lets an agent learn something about 3D structure without an assigned name. — Lex Fridman Podcast #110.
- Beyond Static Images: Malik identifies long-form video understanding, including agents' goals and likely next actions, as a problem not captured by static image classification. — Lex Fridman Podcast #110.
- Annotation Requirements: Malik argues that training segmentation systems from many manually specified object boundaries uses too much top-down knowledge. — Lex Fridman Podcast #110.
- Motion as Information: In Malik's coauthored action-recognition work, combining appearance and motion cues improves the recognition of actions in video. — The Three R's of Computer Vision.
- Cross-Modal Calibration: Malik's example of a child seeing and touching a ball illustrates how different senses can calibrate one another without an external object label. — Lex Fridman Podcast #110.
- Unseen Situations: Malik uses unexpected road and terrain conditions to show why agents need to generalize beyond the specific cases seen during training. — Berkeley Faculty Research Lecture.
Part 7: Historical Context and Paradigms
- Earlier Vision Methods: Malik and coauthors describe how earlier visual pipelines handled grouping and recognition separately before learned features made more joint systems practical. — The Three R's of Computer Vision.
- Perona-Malik Diffusion: Perona and Malik introduced a spatially varying diffusion process that smooths within regions while keeping region boundaries relatively sharp. — Perona–Malik Diffusion Paper.
- Shape Contexts: Belongie, Malik and Puzicha describe shape contexts that encode the spatial distribution of other points relative to a reference point, helping match corresponding shapes. — Shape Contexts Paper.
- Gibson's Influence on the Lecture: Malik explicitly invokes Gibson's perception-for-action formulation to frame his sensorimotor path toward AI. — Berkeley Faculty Research Lecture.
- The Next AlexNet Moment: Malik points to AlexNet as an example of a paradigm-shifting advance and argues academia should leave room for neglected ideas, without claiming the breakthrough was only hardware and data. — Malik's Foundation Models Response.
- Scientific Progress: Drawing on four decades in AI, Malik warns that researchers can over-invest in current paradigms and miss a promising idea elsewhere. — Malik's Foundation Models Response.
Part 8: The Future of Intelligence
- Foundation Models: Malik regards large language models as useful but questions whether they are a foundation for broad AI while lacking sensorimotor grounding. — Malik's Foundation Models Response.
- AGI Timelines: Malik says human-level AI may be possible in principle, but unknown unknowns make confident near-term timelines unreliable. — Lex Fridman Podcast #110.
- Hands and Tools: Malik uses the evolution of hands, tool use and manipulation to illustrate an important strand of intelligence that predates symbolic language. — Berkeley Faculty Research Lecture.
- Autonomous Driving: Malik contrasts AI's language-test results with the continuing difficulty of robust autonomous driving in unpredictable physical situations. — Berkeley Faculty Research Lecture.
- Moravec's Paradox: Malik explicitly credits Hans Moravec for the observation that child-level perception and mobility have proved harder to engineer than some adult-level test performances. — Berkeley Faculty Research Lecture.
- The Kitchen-Task Gap: Malik lists ordinary kitchen actions a young person can perform but robots cannot, using them to illustrate the difficulty of embodied intelligence. — Berkeley Faculty Research Lecture.