Learning Structured World Models From and For Physical Interactions
- Dr. Yunzhu Li, Columbia University
- Time: 2025-01-06 10:00
- Host: Dr. Hao Dong
- Venue: Room 204, Courtyard No.5, Jingyuan
Abstract
Humans possess a strong intuitive understanding of the physical world. Through observations and interactions with our environment, we build mental models that predict how the world changes when we apply specific actions (i.e., intuitive physics). My research builds on these insights to develop model-based reinforcement learning (RL) agents that, through interaction, construct neural-network-based predictive models capable of generalizing across a range of objects made from diverse materials. The core idea behind my work is to introduce novel representations and integrate structural priors into learning systems to model dynamics at various levels of abstraction. I will discuss how such structures enhance model-based planning algorithms, enabling robots to accomplish complex manipulation tasks (e.g., manipulating object piles, shaping deformable foam to match target configurations, and crafting dumplings from dough using various tools). Furthermore, I will present our recent progress in developing purely learning-based, 3D interactable neural digital twins and how we combine neural dynamics models with a GPU-accelerated branch-and-bound framework to facilitate more effective long-horizon trajectory optimization in challenging, contact-rich manipulation tasks (e.g., non-prehensile planar pushing with obstacles, object sorting, and rope routing).
Biography
Yunzhu Li is an Assistant Professor of Computer Science at Columbia University. Before joining Columbia, he was an Assistant Professor at UIUC CS and spent time as a Postdoc at Stanford, collaborating with Fei-Fei Li and Jiajun Wu. Yunzhu earned his PhD from MIT under the guidance of Antonio Torralba and Russ Tedrake. His work lies at the intersection of robotics, computer vision, and machine learning, with the goal of helping robots perceive and interact with the physical world as dexterously and effectively as humans do. Yunzhu’s work has been recognized with the Best Paper Award at ICRA, the Best Systems Paper Award at CoRL, and the Best Paper Awards at multiple workshops. Yunzhu is also the recipient of the Sony Faculty Innovation Award, the Adobe Research Fellowship, and was selected as the First Place Recipient of the Ernst A. Guillemin Master’s Thesis Award in AI and Decision Making at MIT. His research has been published in top journals and conferences, including Nature, Science, RSS, NeurIPS, and CVPR, and featured by major media outlets, including CNN, BBC, and The Wall Street Journal.