August 23rd, 2022
Written by: Catrina Hacker
We might not give it much thought, but the ability to see in three dimensions is a large part of what makes our visual experience so rich. This is why production companies have spent so much time developing methods to film and produce 3D movies. For a long time, neuroscientists thought that this ability to see in 3D was special and could only be accomplished by the complicated neural circuits they observed in the primate brain. Recently, this view has been challenged by work demonstrating that many other animals, even insects like the praying mantid, see in 3D1–3. While understanding 3D vision in humans can help correct abnormal vision, studying how praying mantids accomplish this impressive feat offers hints for how we might engineer efficient machines that can do the same.
Our brains use both eyes to see in 3D
Our ability to see in 3D, known as depth perception, relies on the fact that we have two eyes simultaneously viewing the world in front of us from slightly different angles. To see this in action, simply hold an index finger up in front of another object (for example a water bottle on your desk). While focusing on your index finger, switch back and forth between viewing it with only your left or right eye. You should notice that the relative locations of your finger and the object jump back and forth as you switch eyes even though nothing in the world itself is moving. This mismatch between what our left and right eyes see, known as binocular disparity, might sound like a problem, but this redundancy is what our visual system exploits to help us see in 3D1.
Binocular disparity is helpful because how different an object looks to each eye differs depending on how far the object is from where you’re looking. In the example above, you’re looking at your index finger. When an object is far from your finger, the image in each eye will be more different (it will bounce back and forth more as you switch between your left and right eye). The closer that object gets to your finger, the more similar the images will be in each eye. Our brain uses this information to help us see in depth in a process called stereopsis1,3.
To gain a better intuition for the signals our eyes combine to see in depth, read through this tutorial with several demos you can do yourself at home.
Studying praying mantid stereopsis with 3D movies
Like humans, praying mantids have two front-facing eyes that could help them see in 3D. Unlike humans, they can’t explicitly tell experimenters what they see. To study stereopsis in mantids, a group of neuroscientists led by Dr. Jenny Read at Newcastle University had to come up with a clever way to help the praying mantids tell them what they were seeing. Their solution? Take the mantids to the movies.
You may remember watching old 3D movies that required you to wear glasses with one red and one blue lens (here’s an example if that doesn’t sound familiar)4. These movies take advantage of what we just learned about the relationship between the differences in the images of objects in each eye and their perceived depth, so that we see the movie in 3D. They do this by combining two images on each frame of the movie: one in red and the other in blue (Figure 1).
When you put on your red/blue 3D glasses, the color of the lenses selectively blocks parts of the image displayed in that color so that each eye receives a different image. Because the position of the image in each eye is different, this creates a binocular disparity that should cause you to perceive a single image either popping out of the screen or pushed behind the screen (whether the image pops out of the screen or is pushed behind depends on which way you’re wearing your 3D glasses). The bigger the difference between the red and blue images, the more the image will appear to pop out of the screen in 3D (Figure 1). The group of neuroscientists who wanted to study praying mantid stereopsis used this same trick on the mantids.
The neuroscientists fitted the praying mantids with their own custom pairs of 3D glasses and put them in front of a screen (Figure 2)2 where they watched a 3D movie of a dot that should appear to move closer and further from the mantid. To get the animals to report what they were seeing, the experimenters took advantage of the fact that praying mantids will naturally reach out to grab prey at a fixed distance in front of them. They found that the mantids would reach out to grab the dot when the 3D image looked like it was at the distance they usually grab their prey, even though there wasn’t anything in front of them. If the mantids didn’t have stereopsis, then they wouldn’t reach out to grab the dot because it wouldn’t look like it was in front of them. In other words, they showed that just like humans, praying mantids combine the images from their eyes to see in 3D.
You can watch the praying mantids participate in the experiments and hear from the researchers themselves in the video below. You can also see and learn more in any of these other videos: here, here, and here.
Humans and praying mantids offer two different solutions to the problem of building 3D vision
How humans fuse the images from our eyes to produce a sense of depth still isn’t fully understood by vision scientists, but what we do know is that it requires the complicated neural circuitry of our large and complex primate visual systems3. Because of this complexity, for a long time it was assumed that only primates had stereopsis. The praying mantid experiment described above challenges that view by showing that an insect, the praying mantid, also has this capability. With simpler nervous systems it stands to reason that praying mantids also have simpler neural solutions for how to see in 3D. The same group of neuroscientists at Newcastle University are recording from the neurons in the praying mantid brain to better understand how their much simpler nervous system accomplishes the difficult problem of seeing in 3D (Figure 3)5.
Stereopsis is a powerful ability that plays an important role in how we interact with and understand the world around us. This power makes it a desirable ability for machines and robots that also need to interact with and navigate the world. Deciding how far away a pedestrian is on the road could be a life-or-death calculation made by a self-driving car. Similarly, knowing which surfaces are near and far is essential information for a drone finding a place to land. While there are many cues that can help with the perception of depth (for example, things that are closer are bigger than things that are further away), stereopsis is an essential part of this process, especially for things that are closer to us.
What we’ve learned so far about how the brain produces stereopsis in humans provides important building blocks to understand and treat problems with human vision. However, given how complicated this neural activity is, mimicking what we know about the human visual system would be an inefficient way to give man-made machines and robots stereopsis. “Insects are amazing because they achieve such complex behavior—including stereoscopic 3D vision—with their tiny brains,” says Dr. Read, “Learning how they implement their stereo vision could shed new light on our own vision, and/or could inspire new approaches in machine vision.” Through studying human and praying mantid stereopsis, we can take advantage of the wisdom of nature in evolving at least two distinct solutions to the same problem.
Thank you to Dr. Jenny Read of Newcastle University for sharing images from her praying mantid research, and to Dr. Ben Chin and David White at the University of Pennsylvania for their helpful comments on drafts of this post.
References
1. Nityananda, V. & Read, J. C. A. Stereopsis in animals: evolution, function and mechanisms. J. Exp. Biol. 220, 2502–2512 (2017).
2. Nityananda, V. et al. Insect stereopsis demonstrated using a 3D insect cinema. Sci. Rep. 6, 18718 (2016).
3. Read, J. C. A. Binocular Vision and Stereopsis Across the Animal Kingdom. Annu. Rev. Vis. Sci. 7, 389–415 (2021).
4. How Do 3-D Glasses Work? https://www.brainfacts.org/thinking-sensing-and-behaving/vision/2020/how-do-3d-glasses-work-072820.
5. Rosner, R., von Hadeln, J., Tarawneh, G. & Read, J. C. A. A neuronal correlate of insect stereopsis. Nat. Commun. 10, 2845 (2019).
Image credits
Cover photo: Published with permission of Dr. Jenny Read. Photo credit: Newcastle University.
Figure 1: Figure made with illustrator.
Figure 2: Published with permission of Dr. Jenny Read. Photo credit: Newcastle University.
Figure 3 Published with permission of Dr. Jenny Read. Photo credit: Newcastle University.
PennNeuroKnow 
Leave a Reply