Isolated sign language classification using hand descriptors and trajectory based methods - Özdemir, Oğulcan. (Thesis (M.S.) - Bogazici University. Institute for Graduate Studies in Science and Engineering, 2018., 2018)
In this thesis, we propose a sign language recognition system in which we have adapted and simpliﬁed the Improved Dense Trajectory (IDT) approach which was originally proposed for large-scale human action recognition problem. Since the sign language recognition problem mostly focuses on hand gestures, body posture and facial expressions, we have extracted IDT features and ﬁltered the trajectories around the hand region by matching the trajectory coordinates with hand coordinates obtained by pose extraction. In addition to trajectory ﬁltering, we also propose Hand Descriptors, a spatio-temporal feature extraction method, for sign language recognition. In our proposed method, we extract spatio-temporal descriptors around left and right hands. After descriptor extraction, we encoded each sign video as Fisher Vectors which were derived from a Gaussian Mixture Model which was estimated from the training de scriptors. Then, we have trained Support Vector Machines to perform sign language classiﬁcation using the Fisher Vectors as its inputs. We have conducted experiments on two subsets of the BosphorusSign dataset and evaluated the performance of the system in terms of feature extraction speed, computational complexity and memory require ment. In our experiments, the combination of all descriptors yields the best recognition performance on both subsets for both features. We have found that trajectory ﬁlter ing approach yields a similar recognition performance to the baseline approach while the number of trajectories are drastically reduced. Moreover, we have analysed the eﬀects of using diﬀerent parameters and video resolutions on the performance of the Hand Descriptors. Our experiments have shown that hand region produces the most important features in our sign language classiﬁcation system.
Keywords: Sign language.; Trajectory optimization -- Mathematical methods.