The National Comedy Center in Jamestown, New York, is dedicated to telling the story of comedy in America. The new museum opened in early August with a variety of advanced interactive experiences that allow visitors to dive deep into the genre’s great minds and unique voices. From George Carlin’s methodical stage notes to original scripts from “Seinfeld” and “The Mary Tyler Moore Show,” the Center combines a fascinating collection of historical archives with the latest in interactive exhibit technology.
Individual personalization is at the heart of the new Museum’s visitor experience with an intelligent technology system developed by Cortina Productions. Visitors begin their tour at registration stations by selecting their favorite comedic works, ranging from movies and TV shows to podcasts and cartoons. The visitor’s selections are used to establish their own personal sense of humor profile, which works behind the scenes to tailor the content of the interactive exhibits to the visitor’s taste. Activated by a simple tap of the wrist, exhibits recognize each visitor through an RFID-enabled wristband worn throughout their visit.
One great example of how this personalization system is used is the Laugh Battle game. The game pits one visitor against another in a battle of “I bet you can’t make me laugh.” In this game, each player sits behind an interactive touch screen station and players take turns telling provided jokes, while the other does their best not to lose it and crack up. What they don’t realize is that behind the scenes, the system is actively working against them by analyzing their personal sense of humor profile and feeding the opponent jokes that are very likely to make them laugh! If the player gives in and laughs, the joke teller scores a point!
In addition to developing the back-end of the sense of humor profile system, Cortina’s programming team was presented with the challenge of developing a way for the game to reliably detect when a player laughs. Bryan Heisey, Cortina’s director of software development said, “Our initial thought was to take a photograph of the visitor when they first sat down and have the software record their facial landmarks. That way, our software would be able to detect a change in a player’s expression, like a smile or laugh, and award the point.”
However, the team determined this would require an initial photo calibration step for each player at the beginning of every game, thereby adding an unwanted delay to the start of gameplay. Through additional research, Bryan found an Artificial Intelligence (AI) program from Microsoft within their Azure Cognitive Services that looked promising, without the need for the initial calibration. This powerful component would analyze the player’s face to detect specific emotions, not just a change in expression, and do it on the fly. He was able to incorporate it into Cortina’s software and quickly had a working prototype to test on some would-be office comedians. “I was completely blown away by how accurate and responsive it was. “
After the successful prototype testing, Cortina’s development team worked to fine-tune the software’s configurations to accurately detect a laugh.
Microsoft’s data determines emotions on a percentage value. For example, it may analyze a person’s face and say there is 65% likelihood that they’re sad or a 90% chance that they’re surprised. For our game, the Microsoft system gave us a percentage likelihood that a player is happy, but not specifically laughing. So, our team took those percentage values and did some iterations and extensive testing. With the system analyzing frames from the camera every half second, we were able to refine and pinpoint the configuration that was consistently the most accurate in detecting a laugh.
Laugh Battle is forgiving enough to allow the flash of a smile or a quick smirk to come and go, but if a visitor is actively smiling or laughing, the interactive nabs it and awards a point to the opponent. The Microsoft team was excited to partner with the National Comedy Center and Cortina Productions on the development of this interactive exhibit because it shows how AI can be used in a positive way to create a fun and memorable experience.
Cortina Productions is currently leveraging AI and facial recognition technology in other upcoming projects scheduled to open in 2018. When asked about how he anticipates AI being used in the future, Bryan envisions a museum with next-generation personalization:
We’re taking the guesswork out of what visitors want in their museum experience. I see the potential for truly adapting the exhibit space based on what people are interested in and what subjects they’re drawn to. It’s not something curators will need to analyze. The system is constantly learning and evolving. For example, the system may learn that on Tuesdays we get a lot of middle school groups. I should probably feature this story in the gallery. We’re entering the era of museums automatically tailoring experiences to the visitor.
Laugh Battle will be featured at Microsoft’s annual Envision Conference in Orlando, Florida, from September 24-26.