Movie Maven: How AI Is Helping Divine Viewer Behavior from Video Data

Type 'teenage angst' into whatismymovie.com, and the search engine will recommend titles such as Heathers, Rebel Without a Cause and The Basketball Diaries.

Change the search to 'elephants' and the results will include titles ranging from Dumbo and The Elephant Man to Smokey and the Bandit II (in which the transporting of an elephant figures prominently).

Intended as a tech demo, whatismymovie.com isn't your typical movie search engine. It's powered by an AI-infused algorithm that combines natural language understanding as well as text and pattern recognition to understand the contents of video files. This makes it possible to match queries and results in new ways, and expose data that hadn't been searchable previously.

The technology behind the site is the work of AI video startup Valossa, a spinoff from the University of Oulu in Finland. While the company's aspirations are much more ambitious than helping people find movies, CEO and co-founder Mika Rautiainen gets a kick out of the impact whatismymovie.com is having on those who find it.

'People are finding their long-lost movies they couldn't remember the name of by describing them,' said Rautiainen.

Beyond matching fans with movies, Valossa is focused on applying its technology to help media companies understand their media assets better, and ultimately gain insight into how viewers are affected by what they watch. For instance, how their emotions change when a certain actor is on screen.

Eventually, this could lead to content being adjusted on the fly as viewer attention wanes, more entertaining (and effective) advertising and better monetization of content in online distribution.

'Broadcasters and other video content companies can extract data from what's happening at every second of playback and correlate it with behavior,' said Rautiainen. 'They're looking for every possible piece of information that helps them make a bigger impact with their content.'

More Than Movie Search

Rautiainen first started working on 'semantic video understanding' while on a research exchange program at the University of Maryland. There, he worked on algorithms for detecting objects and activities, such as explosions, in video content, and making search engines that were benchmarked against standardized video search problems. In 2010, he and his research team started collaborating with Finland's national broadcaster, Yle, on indexing broadcast video and making it discoverable.

When the next generation of machine learning and AI were first building steam in the scientific community, Rautiainen started looking at how he could combine his work with the new breed of AI, and he began assembling the team that would become Valossa. He opted to locate the company in Oulu, a former Nokia town, because of the abundant availability of highly skilled video engineers and computer vision researchers.

Valossa's initial product, Valossa AI, is a cloud service accelerated by a GPU cluster running in the Amazon Web Services cloud. It enables media companies to gain an understanding of things such as who are the most prominent actors in a video, when and where they're on screen, and visual context such as the presence of foliage or urban structures.

It also analyzes and indexes what's being said as well as background sounds. They recently released video insight tools that enable inspection of video content through visual reports, scene-level search and overview dashboards. Combining these insights with behavioral data on users can inform content decisions.

Internally, the company is using NVIDIA GPUs, both on premise and in the cloud, to speed the training of its deep learning algorithms - boosting performance by at least 30x compared to CPUs.

Speeding Up Time

This vast speed-up allows the Valossa AI engine to index and annotate everything it sees in a 60-minute video in just 10 minutes - a task that could take CPUs over two hours, at much greater cost. Anyone can see the technology in action by visiting the Valossa site and signing up to run videos through the company's deep learning-infused cluster for free.

Companies can pay for the service based on processing time for the minutes and hours of content consumed, or as a subscription model for customers with ongoing high-volume video needs. Interactive video insight tools are available as a preview for people signing in to the portal. Valossa also offers on-premise installations for enterprise use.

Down the line, Rautiainen anticipates Valossa's technology will be of interest to far more than media companies, as more businesses look to extract value using video intelligence. They'll be able to understand how presentations are affecting viewers and make real-time adjustments.

Said Rautiainen: 'We're allowing people to understand how their content data is structured and, moreover, find relationships between these structures and actual impact.'