Structured and sequential representations for human action recognition

Abstract

Human action recognition problem is one of the most challenging problems in the computer vision domain, and plays an emerging role in various elds of study. In this thesis, we investigate structured and sequential representations of spatio-temporal data for recognizing human actions and for measuring action performance quality. In video sequences, we characterize each action with a graphical structure of its spatio-temporal interest points and each such interest point is quali ed by its cuboid descriptors. In the case of depth data, an action is represented by the sequence of skeleton joints. Given such descriptors, we solve the human action recognition problem through a hyper-graph matching formulation. As is known, hyper-graph matching problem is NP-complete. We simplify the problem in two stages to enable a fast solution: In the rst stage, we take into consideration the physical constraints such as time sequentiality and time irreversibility for the actions; in the second stage we approximate the problem using a sparse subset of spatio-temporal interest points. The reduced problem is then elegantly solved with the dynamic programming technique. Our approach results in competitive performance gures vis- a-vis the state-of-the-art action recognition algorithms. The proposed hyper-graph matching formulation has also been applied to the problem of the quality of action rendition. Finally, we present an alternative formulation of the action recognition problem via Hidden Markov Models (HMMs). To learn HMM parameters, contrary to the conventional approach, Expectation-Maximization algorithm, we demonstrate the practical employment of a spectral algorithm. Given the large variations in action sequences, we resort to a clustering scheme for exploring the subgroups in the training data and for learning multiple HMMs per action category.