Your Baby's Rattle Is Smarter Than GPT-4V

Pick up a baby rattle. Shake it.

Notice that you can't actually experience that in separate streams. There's no "first I'll process the sound; now I'll cross-reference the visual" — it arrives as a single, unified event. The vibration in your palm is the sound is the motion of the beads is the object. That seamless fusion is called multisensory integration, and developmental psychologists have known for decades that infants master it from the earliest weeks of life. AI researchers are still working on it.

This gap — between an infant's effortless sensory binding and what our most sophisticated multimodal models actually do — is one of the most revealing fault lines in the whole field. After spending a weekend debugging a wheeled robot with my eight-year-old nephew, watching him instinctively poke and tilt and shake it to figure out what was wrong, I keep coming back to the same question: what would it even take to build a machine that integrates its senses the way a six-month-old does?

Let's dig in.

The McGurk Moment

In 1976, British psychologists Harry McGurk and John MacDonald published a short paper in Nature that still makes my jaw drop. They showed that when you dub an auditory "ba" sound onto a video of someone clearly mouthing "ga," most viewers don't hear "ba." They hear "da" — a compromise percept that the brain synthesizes by averaging the conflicting signals from two modalities. The auditory cortex doesn't win. The visual cortex doesn't win. The brain manufactures a third percept that never existed in either stream.

This is multisensory integration at its most dramatic: the brain isn't just combining inputs, it's generating perception from the intersection of multiple channels, weighting each one by its reliability. The technical name for this is Bayesian optimal integration. Vision usually dominates when signals conflict (hence the McGurk effect), but in the dark, touch takes over. The weights shift dynamically, automatically, without conscious effort.

And babies do this from very early on. By four to six months, infants robustly match faces to voices. Andrew Meltzoff and Richard Borton demonstrated in 1979 that newborns could recognize by sight an object they had only ever touched with their mouths — evidence that cross-modal binding precedes language, precedes explicit memory, and is built into the architecture of the developing brain from the start.

How Infants Get There: The Embodied Curriculum

Here's what strikes me as a builder: the infant's sensorimotor curriculum is extraordinarily well-designed. They don't learn vision first, then hearing, then touch. They learn them together, through action.

A baby reaching for a rattle is simultaneously:

• Watching the hand approach (vision + proprioception)
• Feeling the grasp tighten (tactile + motor)
• Hearing the sound change as they shake it (auditory + action consequence)
• Updating their model of what the object is from all three channels at once

This action-contingent learning is crucial. Sensory integration doesn't happen in passive observation — it happens through the temporal correlation between what the body does and what the world returns. When I move my hand toward a thing and feel resistance at the exact moment I see contact, I learn that those two signals refer to the same event. That's how binding happens: through doing.

The social dimension matters too, and in ways that are still striking. Endevelt-Shapira et al. (2024) found something remarkable in a longitudinal study of mother-infant pairs: affectionate maternal touch at just three months of age was significantly correlated with vocabulary outcomes at 18–30 months. Touch — a modality with zero direct linguistic content — predicting language development almost two years later. Cross-modal at the deepest level: the social body and the developing mind are learning together in ways that don't respect the boundaries of individual senses.

What Multimodal AI Actually Does

Now let's open the hood on what a "multimodal" AI model is actually doing, because I think the term obscures more than it reveals.

A system like GPT-4V (or Gemini, or similar vision-language models) works roughly like this: your image gets encoded by a visual backbone — often a vision transformer or CLIP-style encoder — your text gets tokenized and embedded by the language model, and these two representations are combined, typically through cross-attention mechanisms, before the model generates a response.

This is genuinely impressive. These systems can describe images, answer visual questions, and catch subtle inconsistencies between images and text. But look at what's actually happening: two separate encoders, each pre-trained on its own modality, are talking to each other through an attention layer. The integration happens late, at the representation level, and it's fundamentally different from the kind of fusion the infant brain performs.

The infant's visual cortex, auditory cortex, and somatosensory cortex aren't communicating through a bottleneck layer. They're wired together through the superior temporal sulcus and posterior parietal cortex, which develop multisensory response properties through the first years of life — and more importantly, they're connected through the motor system. The body is the medium through which sensory integration occurs.

Gopnik, Farrell, Shalizi, and Evans (2025) make the essential point plainly: large AI models are a new kind of cultural and social technology, powerful for distilling and recombining the accumulated record of human knowledge, but fundamentally different from biological minds that develop through embodied experience. The difference isn't just philosophical — it's architectural. Statistical models trained on text and images inherit the record of human sensory experience without ever having had sensory experience. They've learned about rattles from descriptions of rattles and pictures of rattles. They have never shaken one.

The Grounding Gap

Here's the practical consequence. Vong et al. (2024) published a remarkable result in Science: a neural network trained on just 61 hours of head-mounted camera footage from a single child between ages 6 and 25 months could learn to map dozens of words to their visual referents. The model learned "cup" by seeing cups from the egocentric, first-person perspective of a child who was reaching for, drinking from, and playing with cups. Not from a million labeled images — from 61 hours of embodied, action-contingent visual experience.

This is grounding — the connection between a symbol and the physical events it refers to — and it's one of the hardest problems in AI. The developmental insight is that grounding happens through action. You don't learn what "cup" means by observing cups from all angles simultaneously. You learn it by picking one up, feeling its weight, hearing it clink, watching liquid pour into it. The word becomes meaningful because it's connected to a whole sensorimotor ensemble, not just a visual category.

Current multimodal AI systems are getting better at cross-modal association — they can link a visual representation of a cup to the word "cup" with impressive reliability. What they can't do is extend that understanding through action. They don't know what it feels like to pick up a cup heavier than expected. They don't have the motor prediction error that fires when a cup is empty and you over-compensate.

Taniguchi et al. (2024), publishing in Science Robotics, show one of the most promising paths forward: a developmental robotics system that learns compositional language-action mappings through interactive, grounded experience rather than large-scale pretraining. The robot builds up its understanding incrementally through scaffolded social interaction — much like a child learning to pair language with physical action under a caregiver's guidance. The result is more efficient and more generalizable than data-intensive approaches, and the key ingredient is exactly what's missing from standard multimodal AI: an embodied agent that acts on the world and uses the sensory consequences of its own actions as training signal.

The Rubber Hand and the Robot

Let me run a thought experiment. In the rubber hand illusion, you lay a rubber hand on the table in front of someone, hide their real hand from view, and stroke both hands simultaneously with a brush. Within minutes, the person feels the strokes on the rubber hand as if it were their own. The brain, faced with synchronous visual and tactile signals, incorporates the rubber object into the body schema.

Could you run this experiment on a robot? In a sense, developmental robotics researchers do exactly this — they give robots the experience of receiving simultaneous visual and proprioceptive feedback when they touch objects, and the systems learn to build representations that fuse those channels. It works, up to a point. But the sophistication of the binding, the speed of its development, the flexibility of its application — the gap between robot and infant is still enormous.

Here's my honest take after spending a weekend debugging a wheeled robot with my nephew: the kid's debugging strategy was entirely multisensory. He listened to the motor sounds to diagnose wheel slippage. He felt the vibration through the chassis to detect surface friction. He watched the drift pattern to infer which motor was underpowered. At no point did he consult separate "vision data" and "audio data" — his brain served up an integrated diagnosis. The robot had no such luxury. Each sensor returned a number. The synthesis was entirely our job.

What This Means for the Field

A few concrete takeaways for anyone building systems that need to interact with the physical world:

Late fusion isn't enough. The dominant paradigm in multimodal AI — encode each modality separately, fuse late — produces impressive cross-modal association but misses the deep sensorimotor integration that makes biological perception so robust. Early fusion, and especially grounding through action, is where the real gains seem to be.

Bodies matter. The infant's sensory integration is inseparable from the motor system. If you want AI that truly binds the senses, you probably need AI that acts. This is why developmental robotics — systems like the one Taniguchi et al. (2024) describe, where robots learn through social interaction and physical exploration — represents a more ecologically valid path than training on static datasets.

Temporal correlation is the binding glue. In biological multisensory integration, signals from different modalities get bound together by their temporal correlation — they happen at the same time as a result of the same event. Building AI systems that exploit this principle explicitly, rather than through implicit statistical co-occurrence in training data, might be a key design move.

Touch is underrated. Endevelt-Shapira et al.'s (2024) finding that tactile social experience at three months predicts language two years later is a reminder that cross-modal effects span timescales we don't usually think about. If we're serious about grounded AI, we need richer sensory channels — including haptic feedback — not just vision and language pipelines talking to each other.

---

The six-month-old with the rattle is doing something computationally extraordinary: constructing a unified model of a physical object from simultaneous, action-contingent streams of visual, auditory, and tactile data, all calibrated against each other in real time. Our best multimodal models can recognize the rattle in an image and describe it accurately. But they've never held one.

That gap is worth taking seriously — not as a reason for pessimism, because the progress in embodied AI over the past few years has been genuinely exciting, but as a reminder of exactly what problem we're trying to solve. Sensory integration isn't a feature to bolt on. It's the foundation. And every time I watch a kid shake something to figure out what's inside it, I'm reminded that biology got to that solution a very long time ago.