ABSTRACT
Human action recognition has a widespread application in medical health, human-machine interaction, sports competition and similar. In this study, the authors used the literature review method to introduce the standard sensors and corresponding methods used for motion data collection and briefly described the general techniques for motion data preprocessing. Also, introduced the process for extracting features of motion data and pointed out the research status of standard feature recognition methods as well as briefly described their application status. Finally, the current problems related to the recognition features were discussed as well as those related to future development. The future development may focus on hardware system optimization, model optimization and personalized recognition.
PDF Abstract XML References Citation
How to cite this article
DOI: 10.3923/itj.2022.8.13
URL: https://scialert.net/abstract/?doi=itj.2022.8.13
INTRODUCTION
Human action recognition is used to classify and recognize specific motion types and motion modes of the human body. It has a high research value and can be applied in many fields. For example, it can be used for disease diagnosis and health monitoring in medical health1,2, for analyzing limb movement skills of athletes in sports3,4 and for identification in security certification. Typical targets for activity recognition include various daily motions of the human body3, such as standing, walking, running, sitting and similar5. At the same time, they also may involve some fine movements, such as finger joint bending, etc.
The methods of human motion behaviour feature recognition can be roughly divided into 2 categories: Machine vision-based and sensor-based. The recognition based on machine vision takes image processing technology as the core to describe and recognize human behaviour in video sequence frame by frame. The processing process is complex, the amount of computation is large and it is easy to be affected by environmental factors5. The sensor-based recognition has the advantages of low cost, small size and not easily affected by the environment. In this paper, the research status of motion behaviour recognition based on sensors is introduced in detail. Firstly, the mainstream methods of motion data acquisition and processing are listed. Secondly, the research status of motion behaviour feature extraction and feature recognition is introduced. Then, the application status is briefly described. Finally, the existing problems and future research directions are pointed out.
This study aimed to discuss trends in the current status of sensor-based activity recognition list mainstream methods for acquiring and processing data as well as introducing the research status and prospects extraction and recognition of activity features.
Acquisition and processing of motion data: The intuitive manifestation of human action is the purposeful movement of the limbs and trunk that can be characterized by the observable movement parameters, such as human body displacement, limb speed and acceleration, force distribution on joints and surface electromyography, etc. Acceleration sensors, pressure sensors and surface electromyography devices are the 3 most commonly used sensors for acquiring human movement data. In addition to those sensors, other devices, such as a magnetometer, GPS, infrared sensors, cardio tachometer, sphygmomanometer and baroreceptor2,6,7 can also be used to acquire human movement data.
Acquisition of motion data
Motion data acquired by the acceleration sensor: The acceleration sensor is an inertial sensor, which is the most commonly used in the study of human action feature recognition. With the continuous progress of MEMS technology, multi-axis acceleration sensors are often used in current research to collect acceleration data of human movement. They can be installed or worn on the chest2, waist8, wrist3,9, ankle joint3,10 and other positions of the human body to measure the changes in the acceleration and angular velocity of the corresponding parts. Also, acceleration data of human body movement can be collected through smartphones, smartwatches and other devices with built-in acceleration sensors1,9,11.
Motion data acquired by the pressure sensor: The walking gait is one of the aspects that attract much concern in recognition of action features. The gait parameters include stride length, stride frequency, oscillation duration, support duration and similar12-14. In the study of gait, pressure sensors are used to collect data on plantar pressure during walking. Also, a pressure sensor can be installed in the plantar pedis of an exoskeleton robot to measure the plantar reaction force. Compared with a single-point pressure sensor, a pressure sensor array can collect pressure data from larger areas to obtain the pressure distribution on the stressing surface15-17.
Motion data acquired by surface electromyography: The EMG is generated by superimposing the action potential of multiple motor units in the muscles during limb movement, which can reflect the activity of the muscles. The sEMG can be acquired by pasting the electrode pads on the muscles and using the corresponding EMG acquisition equipment18. The sEMG acquisition device can be used to measure the muscle movements of the upper and lower limbs19, which can then be used to analyze the sequence of muscle movements and the principle of limb movements. Besides, sEMG can also be used to measure fine hand movements, which then can be used to develop EMG-controlled prostheses20,21.
Motion data processing
Filtering: The filtering process is used to remove the noise from the collected motion data, thus making the data more realistic. For sensors that output analogue signals, filtering can be directly achieved using hardware circuits. Analogic acceleration sensors are susceptible to high-frequency noise, for which hardware filtering can be applied using filtering circuits, such as RC low-pass filter18 and Butterworth low-pass filter, etc. Median filter 16 and moving average filter3 are 2 commonly used digital filters that can smooth the original data. Besides, the Kalman filter and digital Butterworth filter2,22 are also used in sensor filtering.
Data partition: Typically, only part of the large amount of motion data collected by the sensors is of interest and needs to be identified. A more straightforward data segmentation method involves segmenting the data by sliding a fixed-length data window along the motion data sequence according to a certain overlap rate8,18,23-25. In actual research, data can also be manually segmented according to specific needs. Meanwhile, a more reasonable method is to segment data using varying-length windows. The minimum amount of data required to identify different activities is 1st estimated, which is then used as the window length for data segmentation. Furthermore, a threshold can be set according to the wavelet energy2 and acceleration amplitude3 exhibited by the activities, to determine the start and end of the activities and then to segment valid motion data.
Feature recognition methods
Abstraction of motion features: Generally, activity recognition is performed after data features are extracted and eigenvectors are constructed rather than directly using the collected motion data. This can reduce the complexity of the recognition model while improving the accuracy of recognition.
Extraction of time-domain features: Common time-domain features include meaning, standard deviation, root mean square, extreme value, mean absolute value, zero-crossing point and change of slope sign3-9,18,20,21. For a 3 axis acceleration sensor, the signal vector magnitude is also a commonly used time-domain feature2,11,26, which is applicable when the sensor direction is not of essential importance. regarding pressure data, the plantar pressure is commonly used, from which the stride length, stride width, single stride time and other gait features can be extracted13,14.
Extraction of frequency domain and other features: Frequency domain features, which can be obtained by calculating the frequency spectrum of motion data are also commonly used for feature extraction. Their extraction effect is generally better than time-domain feature extraction. The common frequency domain features are average frequency, median frequency, peak spectral power and average spectral power3,16,18.
Feature selection and dimensionality reduction: After extracting various activity features, some features may not be suitable for characterizing a particular motion and the activity recognition process may become too complicated if the number of features is very large. To solve this issue, Purushothaman and Vikas24 have suggested Particle Swarm Optimization (PSO) and Ant Colony Optimization (ACO) for feature selection. Meanwhile, feature dimensionality reduction is used to map eigenvectors from a higher dimension to a lower dimension. Hsu et al.3, adopted a feature dimensionality reduction method, which combined Non-Parametric Weighted Feature Extraction (NWFE) and Principal Component Analysis (PCA) to reduce 252 normalized features to 18.
Action feature recognition: In the early stages, most of the action studies only measured the macroscopic quantities exhibited by the human body, such as energy consumption and physical exercise amount. After that, greater attention was paid to modes of identifying specific activities that led to new methods of judging activities based on the magnitude of changes in motion data. Then, using the features extracted from the motion data, methods based on Naïve Bayes (NB), k Nearest Neighbors (k-NN), Decision Tree (DT), random forest and Support Vector Machine (SVM) have achieved excellent results in the application of activity classification, which greatly promoted the development of sensor activity recognition.
Fixed threshold: The classification method based on a fixed threshold can quickly identify some activities. The threshold is mostly determined through a large number of pre-experiments but its setting limits the versatility of the method. Mathie et al.27, used a 3 axis acceleration sensor to collect waist acceleration data and compared the amplitude of the acceleration signal in the sliding data window with a preset threshold to identify whether the human body is still or in motion in the window. Karantonis et al.28, also adopted a similar method, except that they calculated the acceleration signal amplitude area (signal magnitude area).
Naïve Bayes (NB): The NB is the most commonly used method in the bayesian classification algorithm, which is characterized by easy implementation and a small amount of calculation. However, NB assumes that the input features are independent of each other, which is almost impossible in practical application. Hence, unsatisfactory classification results may be obtained when there are large correlations between input features.
k-nearest neighbors (k-NN): The K-NN is a simple and effective non-parametric statistical classification method that measures the distance between the test sample and the training sample set. Esakia et al.9, proposed a classification model combining DTW (Dynamic Time Warping) and k-NN, using DTW as the distance measurement between samples, which can be used to identify motion and gestures. Nguyen et al.29, extracted the features of plantar pressure data from 5 different walking exercises and classified different walking types using k-NN with satisfactory results. Rabin et al.21, used multiple classifiers to classify time-domain features of hand sEMG, among which k-NN showed good classification performance and successfully realized classification of 6 gestures.
Decision tree (DT): The DT is an efficient classification algorithm. Its complexity is only related to the number of tree layers, which is suitable for applications with demanding classification execution time. Common DT algorithms include ID3 and C4.5, among which ID3 determines the features to be split when building trees according to the information entropy gain. In contrast, C4.5 builds trees according to the information gain rate.
Random forest: RF is an integrated learning method based on the Bagging algorithm, which can overcome the problem of poor generalization of DT. Besides, RF has good anti-noise ability and at the same time, is not prone to over-fitting. Zhou et al.30, collected acceleration data from users’ waists using a 3 axis acceleration sensor and classified 5 daily activities and 1 abnormal activity by using the bee colony optimized RF algorithm. Pancholi and Joshi18, developed an sEMG acquisition module for upper limb prostheses, generated a feature matrix of 16 features extracted from sEMG and identified various hand motions using the RF algorithm.
Support vector machine: SVM is a supervised learning model that utilizes nonlinear mapping to transform training data into a high-dimensional space and classifies the data using a hyperplane, which can solve the binary classification problem3. To enables SVM to solve multi-classification problems, one-against-one3 or one-against-all strategy1 is often adopted. Hsu et al.3, collected acceleration data at the wrist and ankle and then used one-against-one based LS-SVM to classify a variety of daily activities and sports. Purushothaman and Vikas24 successively used QSV (Quadratic SVM) and CSVM (Cubic SVM) to identify finger movements using sEMG data. The difference between them is that one uses a quadratic hyperplane and the other uses a cubic hyperplane in the feature space.
Deep learning: When deep learning is used for activity recognition, motion data can be directly used as input, without the need to construct eigenvectors in advance. In practical applications, when processing time-series data, like motion data, Long Short Term Memory (LSTM), which was developed from Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN), can achieve better performance.
Application of human motion behaviour feature recognition: At present, sensor-based human activity feature recognition research has achieved many results, which have been applied to many fields such as medical health, sports competition, safety certification, indoor positioning, human-machine interface and so on. In terms of medical health, action features can be used to assist in rehabilitation training and diagnosis of specific diseases, such as lower limb rehabilitation training. Regarding the human-machine interface, the recognition results of action features can be used to control virtual devices22.
Prospects: With the development of sensor technology and the large-scale application of machine learning technology, the following aspects should be translated into development trends.
Hardware system optimization: The available activity recognition systems can only perform relatively simple and fixed tasks due to various limitations, such as size, power consumption and processing performance, etc. Breaking through these limitations can make the activity recognition system highly integrated and can perform more intensive calculations, even achieve monolithic integrated sensors, high-performance processors and network-connected devices.
Model optimization: It includes the optimization of the model structure to reduce the space occupied by the model. Also, the flexible network parameters can be locally retrained and timely modified to maintain the validity of the model at any moment.
Individualized recognition: Focusing on specific motion data generated by users and hardware optimization and model optimization based on the recognition system can make the recognition system adapt to different individuals automatically.
CONCLUSION
The universality of the motion behaviour recognition model is poor, most of the existing studies are based on a single or a small number of experimental subjects. The irrationality of data segmentation window, in the preprocessing stage of motion data, a data segmentation window is often used to segment the data and lack a large number of data sources, in the current study, the sample data are mostly collected by researchers themselves. This study launched that future development may focus on hardware system optimization, model optimization and personalized recognition.
SIGNIFICANCE STATEMENT
This research will help researchers discover the development of sensor technology and the great potential of machine learning technology. By listing the application fields and progress of various types of sensors, it provides guidance for many researchers to explore key areas.
ACKNOWLEDGMENT
This work was supported by the Jiangsu Province Key Research and Development Plan (Social Development) General Project (grant number BE2017766), “elderly easy to fall gait characteristics and its early warning and protection technology research”.
REFERENCES
- Bisio, I., A. Delfino, F. Lavagetto and A. Sciarrone, 2017. Enabling IoT for in-home rehabilitation: Accelerometer signals classification methods for activity and movement recognition. IEEE. Internet Things J., 4: 135-146.
CrossRefDirect Link - Ejupi, A., M. Brodie, S.R. Lord, J. Annegarn, S.J. Redmond and K. Delbaere, 2017. Wavelet-based sit-to-stand detection and assessment of fall risk in older people using a wearable pendant device. IEEE Trans. Biomed. Eng., 64: 1602-1607.
CrossRefDirect Link - Hsu, Y.L., S.C. Yang, H.C. Chang and H.C. Lai, 2018. Human daily and sport activity recognition using a wearable inertial sensor network. IEEE Access, 6: 31715-31728.
CrossRefDirect Link - Ya-Wei, S., Y. Chuan-Bao and Y. Wen-Bing, 2016. Establishment and application of finite element model of foot structure. China Tissue Eng. Res., 20: 4376-4382.
Direct Link - Iakovakis, D.E., F.A. Papadopoulou and L.J. Hadjileontiadis, 2016. Fuzzy logic‐based risk of fall estimation using smartwatch data as a means to form an assistive feedback mechanism in everyday living activities. Healthcare Technol. Lett., 3: 263-268.
CrossRefDirect Link - Venuto, D.D., V.F. Annese, G. Mezzina and G. Defazio, 2018. FPGA-based embedded cyber-physical platform to assess gait and postural stability in parkinson’s disease. IEEE Trans. Components, Packaging Manuf. Technol., 8: 1167-1179.
CrossRefDirect Link - Sok, P., T. Xiao, Y. Azeze, A. Jayaraman and M.V. Albert, 2018. Activity recognition for incomplete spinal cord injury subjects using hidden markov models. IEEE Sens. J., 18: 6369-6374.
CrossRefDirect Link - Esakia, A., S. McCrickard, S. Harden, M. Horning and N. Ramalingam, 2020. Using smartwatches to facilitate a group dynamics-based statewide physical activity intervention. Int. J. Hum. Comput. Stud., Vol. 142.
CrossRefDirect Link - Jing, Y. and Y. Song, 2019. Smart shoe design based on multi-sensor. Sens. Microsyst., 38: 100-103.
CrossRefDirect Link - Fridriksdottir, E. and A.G. Bonomi, 2020. Accelerometer-based human activity recognition for patient monitoring using a deep neural network. Sensors, Vol. 20.
CrossRefDirect Link - Hegde, N., T. Zhang, G. Uswatte, E. Taub and J. Barman et al., 2018. The pediatric smartshoe: Wearable sensor system for ambulatory monitoring of physical activity and gait. IEEE Trans. Neural Syst. Rehabilitation Eng., 26: 477-486.
CrossRefDirect Link - Agrawal, T.K., S. Thomassey, C. Cochrane, G. Lemort and V. Koncar, 2017. Low-cost intelligent carpet system for footstep detection. IEEE Sens. J., 17: 4239-4247.
CrossRefDirect Link - Choi, H.S., C.H. Lee, M. Shim, J.I. Han and Y.S. Baek, 2018. Design of an artificial neural network algorithm for a low-cost insole sensor to estimate the ground reaction force (GRF) and calibrate the center of pressure (CoP). Sensors, Vol. 18.
CrossRefDirect Link - Cheng, J., M. Sundholm, B. Zhou, M. Hirsch and P. Lukowicz, 2016. Smart-surface: Large scale textile pressure sensors arrays for activity recognition. Pervasive Mob. Comput., 30: 97-112.
CrossRefDirect Link - Song, Y. and Y. Hua, 2011. Research on the difference of plantar pressure of different basketball shoes. J. Capital Inst. Phys. Edu., 23: 465-468.
CrossRefDirect Link - Pancholi, S. and A.M. Joshi, 2018. Portable emg data acquisition module for upper limb prosthesis application. IEEE Sens. J., 18: 3436-3443.
CrossRefDirect Link - Yawei, S., Y. Da and S. Wen, 2011. Analysis of surface electromyography characteristics of trampoline athletes in take-off stage after landing on net. J. Beijing Sport Uni., 34: 65-67.
CrossRefDirect Link - Mastinu, E., J. Ahlberg, E. Lendaro, L. Hermansson, B. Hakansson and M. Ortiz-Catalan, 2018. An alternative myoelectric pattern recognition approach for the control of hand prostheses: A case study of use in daily life by a dysmelia subject. IEEE. J. Transl. Eng. Health Med., 6: 1-12.
CrossRefDirect Link - Rabin, N., M. Kahlon, S. Malayev and A. Ratnovsky, 2020. Classification of human hand movements based on EMG signals using nonlinear dimensionality reduction and data fusion techniques. Expert Syst. Appl., Vol. 149.
CrossRefDirect Link - Du, Y., W. Jin, W. Wei, Y. Hu and W. Geng, 2017. Surface EMG-based inter-session gesture recognition enhanced by deep domain adaptation. Sensors, Vol. 17.
CrossRefDirect Link - Jia, H., G. Wei and D. Wang, 2018. Human posture detection and recognition based on plantar pressure behavior analysis method. Meas. Control Technol., Vol. 37.
CrossRefDirect Link - Purushothaman, G. and R. Vikas, 2018. Identification of a feature selection based pattern recognition scheme for finger movement recognition from multichannel emg signals. Australasian Phys. Eng. Sci. Med., 41: 549-559.
CrossRefDirect Link - Khushaba, R.N., A.H. Al-Timemy, A. Al-Ani and A. Al-Jumaily, 2017. A framework of temporal-spatial descriptors-based feature extraction for improved myoelectric pattern recognition. IEEE Trans. Neural Syst. Rehabil. Eng., 25: 1821-1831.
CrossRefDirect Link - Wang, C., W. Lu, M.R. Narayanan, D.C.W. Chang, S.R. Lord, S.J. Redmond and N.H. Lovell, 2016. Low-power fall detector using triaxial accelerometry and barometric pressure sensing. IEEE Trans. Ind. Inf., 12: 2302-2311.
CrossRefDirect Link - Mathie, M.J., A.C.F. Coster, N.H. Lovell and B.G. Celler, 2003. Detection of daily physical activities using a triaxial accelerometer. Med. Bio. Eng. Comput., 41: 296-301.
CrossRefDirect Link - [8] Karantonis, D.M., M.R. Narayanan, M. Mathie, N.H. Lovell and B.G. Celler, 2006. Implementation of a real-time human movement classifier using a triaxial accelerometer for ambulatory monitoring. IEEE Trans. Inf. Technol. Biomed., 10: 156-167.
CrossRef - Nguyen, N.D., D.T. Bui, P.H. Truong and G.M. Jeong, 2018. Classification of five ambulatory activities regarding stair and incline walking using smart shoes. IEEE Sens. J., 18: 5422-5428.
CrossRefDirect Link - Zhou B., P. Li and L. Li, 2015. Improved random forest algorithm and its application in human posture patterns recognition. Comput. Eng. Appl., 51: 86-92.
CrossRefDirect Link