The Walt Disney Co. DIS is known for its box office hits: Beauty and the Beast , Rogue One: A Star Wars Story , and Captain America: Civil War , just to name a few. As one of the biggest media conglomerates in the world, Disney is looking to better understand its moviegoing audience so that its upcoming movie line-up can continue to be moneymakers and crowd pleasers.

SwissCognitiveDisney hopes to do this through artificial intelligence (AI) and facial recognition technology, using deep learning techniques to track the facial expressions of an audience watching a movie in order to gauge any emotional reaction to it.

Face recognition as a rating predictor

Speaking at the IEEE Computer Visio and Pattern Recognition (CVPR) conference last week, the media company detailed its new AI project between Disney Research, Disney’s science and tech arm, and Caltech, short for The California Institute of Technology. Called “factorized variational autoencoders,” or FVAEs, the researchers said the technology works so well that after observing an audience member’s face for just 10 minutes, it can predict how the person will react to the rest of the movie. The FVAEs go on to then recognize many facial expressions from movie viewers on their own, like smiles and laughter, and can make connections between different viewers to see if a particular movie is getting a wanted reaction at the right place and time.

Blockbuster formula

“The FVAEs were able to learn concepts such as smiling and laughing on their own,” Zhiwei Deng, a Ph.D. student at Simon Fraser University (who served as a lab associate at Disney Research) told . “What’s more, they were able to show how these facial expressions correlated with humorous scenes.” Disney’s research team tested its technology at a 400-seat theater equipped with four infrared cameras to film the audience during 150 showings of nine movies, including Big Hero 6 , The Jungle Book , Star Wars: The Force Awakens , and Zootopia . The result? A dataset encompassing 16 million facial landmarks by 3,179 audience members that were fed to the FVAEs. […]