Ongoing Project

The research in this project was carried out at Fraunhonfer Institute for Integrated Circuits, Erlangen, Germany during 2016-17.

Automatic Video Tagging Using Deep Learning

Video tagging refers to extracting the key information from a video and summarizing it into a few key tags, representing the overall theme of the video. The aim is not only to understand what is present in each frame of the video, but also to identify the few key topics that best describe what the video is about. This work is different than other related work in this domain in the following ways:

1. We intend to work at a much higher level by understanding the overall context in the individual frames of a video. The context represents the interaction of the objects in a scene and their overall meaning. The examples of context include romance, fight, violence, nudity, action, etc.
2. This work is different than typical event or scene recognition tasks, where each item belongs to a single event or scene.
3. It is also different than most object recognition tasks, where the goal is to label everything visible in an image. This would produce thousands of labels on each video but without answering what the video is really about.
4. This work does include, but it is not limited to, genre classification of movies. A movie typically has 2-3 genres which do not reveal other information in the movie (e.g., violence, nudity, sex, etc).

Applications

• Video classification
• Detecting categories not allowed for movie classification (e.g. violence in kids movies)
• Context based video search
• Efficient archiving

Methodology

We have used deep learning to train a Convolution Neural Network (CNN) to classify a number of scenes in a given video, understanding the context in each video frame and summarizing the results in a few key tags. Our tagging method is summarized in the following diagram: