Improving image to video matching to support entity resolution with motion detection and feature extraction

Abstract

The need for image based video search is increasing rapidly as today with the expansion of big data and the increasing power of hardware. But there are only a few highly successful implementations in existence. In this project I have developed a search method combining motion detection and Different Feature Detection algorithms, then evaluated the method’s effectiveness and compared the two approaches of Real-time Video Search and searching against a Database of feature data taken from videos. Key frames of videos are extracted using motion detection, by the difference of consecutive key frames and the Otsu’s threshold. Speeded Up Robust Features (SURF), Harris-Stephens corners with Fast Retina Keypoint (FREAK) descriptor and color features are the feature detection/description methods used for extracting features. The features extracted from key frames are matched with those of the given image and M-estimator SAmple and Consensus (MSAC) algorithm is used to find ‘Affine transformations’ from the matching points. Different thresholds are taken by combining the feature extraction methods for filtering the results. Two prototypes were produced for comparing searching normally and searching against a database of features. Images of cats are being used to search videos where, some of which have cats throughout, some which have intermediate intervals of cats and while others have no cats. After evaluating against sets of images of incrementing size, the search method produced an intermediate level accuracy (48.89%) of search results. Furthermore, comparing the two prototypes for 5 images and 3 sets of videos, the stored summary prototype is seen slower than the real-time video search, and a trivial difference in result statistics is found.