Importance of Physical Understanding in AI Development

Robotic AI’s reliance on vision is limiting its ability to interact with the physical world accurately. Vision systems dominate robotic AI because they’re cost-effective and can collect massive datasets. But this overemphasis on vision overlooks the critical role of force sensing—providing tactile data that vision simply can’t replicate. Without it, robots are limited to estimating force feedback from visuals, leading to inefficiencies in delicate tasks like assembly, gripping, or threading. As Edward Adelson, professor at Massachusetts Institute of Technology, explained in his TED Talk, “Force feedback allows robots to perform tactile tasks that vision alone cannot achieve—like folding a towel or threading a cable—by feeling their way through interactions, just as humans do.” Adelson’s work on GelSight technology highlights how tactile sensing can unlock superhuman precision for robots, enabling them to understand their environment through touch. The challenge? Force sensors are an added cost, generate less data, and are harder to integrate. But they offer essential benefits: • Reliability and Safety: For tasks where mistakes aren’t an option, force feedback provides the assurance vision alone cannot. • Deeper Learning: Force sensing enriches AI by adding layers of contact-based data for more robust decision-making. • Expanding Applications: From industrial automation to medical robotics, tactile data opens doors to tasks beyond vision’s reach. ATI Industrial Automation supports robotics through robust, precise robotic force sensors—helping to bring accuracy to robotic AI data collection. Edward Adelson’s TED Talk: https://lnkd.in/epeCvwqj #robotics

In my last post, we explored Soft-body Dexterity and how robots touch the world with nuance. Today, we will explore how they might understand it: World Models Grounded in Human Narrative: From Physics to Semantics. To thrive in human spaces, robots need more than physics. They need to understand why things matter, from how an object falls to why it matters to you. Embodied AI Agents will need two layers of understanding: 🌍 Physical World Model: Simulates physics, motion, gravity, and materials...enabling robots to interact with the physical world. 🗣️ Semantic and Narrative World Model: Interprets meaning, intention, and emotion. These are some examples: 🤖 A Humanoid Robot in an Office: It sees more than a desk, laptop, and spilled coffee; it understands the urgency. It lifts the laptop and grabs towels, not from a script, but by inferring consequences from context. 🤖 A Domestic Robot at Home: It knows slippers by the door mean someone’s home. A breeze could scatter papers. It navigates not just with geometry but with semantic awareness. 🤖 An Elder Care Robot: It detects tremors, slower gait, and a shift in tone, not as data points, but signs of risk. It clears a path and offers help because it sees the story behind the signal. Recent research: 🔬 NVIDIA Cosmos A platform for training world models that simulate rich physical environments, enabling autonomous systems to reason about space, dynamics, and interactions. https://lnkd.in/g3zJwDmb 🔬 World Labs (Fei-Fei Li) Building "Large World Models" that convert 2D inputs into 3D environments with semantic layers. https://lnkd.in/gwQ2FwzV 🔬 Dreamer Algorithm Equips AI agents with an internal model of the world, allowing them to imagine futures and plan actions without trial-and-error. https://lnkd.in/gnPZeRy5 🔬 WHAM (World and Human Action Model) A generative model that simulates human behavior and physical environments simultaneously, enabling realistic, ethical AI interaction. https://lnkd.in/gt5NJ8az These are some relevant startups, leading the way: 🚀 Figure AI (Helix): Multimodal robot reasoning across vision, language, and control. Grounded in real-time world modeling for dynamic, human-aligned decision-making. https://lnkd.in/gj6_N3MN 🚀 World Labs: Converts 2D images into fully explorable 3D spaces, allowing AI agents to “step inside” a visual world and reason spatially and semantically. https://lnkd.in/grMS9sjs What's the time horizon? 2–4 years: Context-aware agents in homes, apps, and services; reasoning spatially and emotionally. 5–7 years: Robots in real-world settings, guided by meaning, story, and human context. World models transform a robot from a tool into a cognitive partner. Robots that understand space are helpful. Robots that understand stories — are transformative. It’s the difference between executing commands... and aligning with purpose. Next up: Silent Voice — Subvocal Agents & Bone-Conduction Interfaces.

A recent study by Harvard and MIT concluded that today’s AI models can predict physical phenomena but can’t yet explain them. This shouldn't be surprising to anyone, because “AI for Science” today is still largely being driven by AI researchers... for other AI researchers. Even AI for Science departments at AI Research Labs are lead by people with a CS background. I say this as a physicist who experienced intense discrimination in AI research circles for years. I have even been told I couldn't "understand researche" because my PhD was in Particle Physics, not in CS... 😵💫 And now, some of those same people want to solve all of science 🤯 Just to be clear, I believe AI deserves a central spot in scientific circles as a tool for all scientists to accelerate their work; it can even be a game changer. But this is also the right time for us all to call for cross-disciplinary partnerships. If AI wants to contribute to scientific discovery, it needs to become more scientific by: 👉 Encouraging collaboration with domain experts (and treat them as peers) 👉 Build architectures inspired by physical priors, not just data trends 👉 Value explanations and not just predictions Until then, AI for Science will keep predicting planetary motion while missing the laws that govern it. #AIforScience #AIResearch #Research #Physics

I recently spoke to Gartner about what is next in #AI. Here are my thoughts: We have seen impressive progress in #llm by scaling data and compute. Will this continue to hold? Yes, I believe so, but most of those gains will be in reasoning tasks where we have precise metrics to measure uplift, as well as the ability to have synthetic data to train further, and also the freedom to trade off computation for accuracy at test time. This is seen in the recent o1 model. For reasoning tasks, we will also be able to remove hallucination when we can construct accurate verifiers that can certify every statement that #llm makes. We have been doing this in our Leandojo project for mathematical theorem proving. However, there is one area of reasoning where #llm will never be good enough: understanding the physical world. This is because language is only high-level knowledge, and cannot simulate the complex physical phenomena needed in many applications. For instance, LLMs can talk about playing tennis or look up a weather app, but they cannot internally simulate any of these processes. While images and videos can help improve their knowledge of the physical world, models like Sora learn physics by accident, and hence, still produce physically wrong outputs. How can we overcome this? By teaching AI physics from the ground up. We are building AI models that are trained in a physics-informed manner at multiple scales. They are several orders of magnitude faster than traditional simulations, and can also generate novel designs that are physically valid. You can watch some of those examples in my recent TED talk.