View-invariant Alignment and Matching of Video Sequences 2003

Abstract:  In this paper, we propose a novel method to establish temporal correspondence between the frames of two videos. The 3D epipolar geometry is used to eliminate the distortion generated by the projection from 3D to 2D. Although the fundamental matrix contains the information of the extrinsic property of projective geometry between views, it is sensitive to noise and fails for non-rigid human body movement. Therefore, we propose using a rank constraint of the corresponding points in two views to measure the similarity between trajectories. This rank constraint shows more robustness and is easier to compute than the fundamental matrix. Furthermore, a dynamic programming approach using the similarity measurement is proposed to find the non-linear time-warping function for videos containing human activities. In this way, videos of different individuals taken at different times and from distinct viewpoints can be synchronized. Moreover, a temporal pyramid of trajectories is applied to improve the accuracy of the view-invariant dynamic time warping approach. We show various applications of this approach, such as video synthesis, human action recognition and computer aided training. Compared to the state-of-the-art techniques, our method shows a great improvement.