北京生物医学工程

基于脑电信号深度学习的情绪识别研究现状

A review of EEG emotion recognition based on deep learning

作者：李颖洁李玉玲杨帮华

单位：上海大学通信与信息工程学院生物医学工程研究所（上海 200444）

关键词：脑电信号; 深度学习; 情绪; 情绪识别; 特征提取

分类号：R318.04; B841.1; B842.6

出版年·卷·期（页码）：2020·39·6（634-642）

摘要：

基于脑电信号的情绪识别方法与传统的人脸识别、语音识别等方法相比，表现出更高的可靠性。然而，由于脑电信号具有低信噪比、非平稳性以及被试间差异大等特点，传统机器学习方法很难进一步提高情绪分类准确性。近年来，随着深度学习在图像分类、语音识别等领域的成功应用，许多研究者将其应用于脑电情绪识别。本文在Web of Science和Google Scholar中利用deep learning、EEG及emotion recognition等关键词检索到154篇相关文献，并基于PRISMA准则筛选出了31篇近几年内将深度学习应用于脑电情绪分类的文献。文中从脑电信号的预处理、特征提取和深度网络输入形式、深度网络架构选择及参数设置等方面，介绍了基于脑电深度学习的情绪识别研究进展。同时，以某情绪脑电公共数据库（a Dataset for Emotion Analysis using EEG, Physiological, and video signals, DEAP）相关研究为例进行各种深度网络架构的比较。本文进一步将文献分析结果提炼，为有兴趣将深度学习技术应用于情绪脑电数据的研究人员，提供一些处理过程中方法选择与参数设置的建议。

Emotion recognition based on EEG signals exhibits higher reliability than traditional approaches such as behavioral, facial and voice. However, due to the characteristics of EEG signals with low signal-to-noise ratio, non-stationary, and large individual differences, it is difficult to improve the classification accuracy by using traditional machine learning methods. In recent years, deep learning has successfully achieved impressive performance in tasks such as image classification and speech recognition, it also proposed to achieved competitive accuracy in EEG-based recognition task. In this paper, we formalized keywords such as deep learning, EEG and emotion recognition search on Web of Science and Google Scholar. The PRISMA criteria was used to identify studies and narrow down the collection, which led this review from an original count of 154 studies to a final count of 31 studies. This paper introduced the research progress on EEG-based emotion recognition using deep learning, from the aspects of EEG signal preprocessing, feature extraction, the choice of input form, architecture and parameter setting of deep networks. Meanwhile, the studies using a public database DEAP (a Dataset for Emotion Analysis using EEG, Physiological, and video signals) are summarized to compare the performance of different deep network architecture. This paper further refines the results of literature analysis which can provide some suggestions for researchers interested in applying deep learning in EEG-based emotion recognition in the selection of EEG signal preprocessing methods and parameters setting.

参考文献：

[1] Picard RW. Affective computing: challenges[J]. International Journal of Human-Computer Studies, 2003, 59(1-2): 55-64.
[2] Bal E, Harden E, Lamb D, et al. Emotion recognition in children with autism spectrum disorders: Relations to eye gaze and autonomic state[J]. Journal of Autism and Developmental Disorders, 2010, 40: 358-370.
[3] Harms MB, Martin A, Wallace GL. Facial emotion recognition in autism spectrum disorders: a review of behavioral and neuroimaging studies[J]. Neuropsychology Review, 2010, 20(3): 290-322.
[4] Kessous L, Castellano G, Caridakis G. Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis[J]. Journal on Multimodal User Interfaces, 2010, 3: 33-48.
[5] Busso C, Deng Z, Yildirim S, et al. Analysis of emotion recognition using facial expressions, speech and multimodal information[C]// the 6th International Conference on Multimodal Interfaces. PA, USA: Association for Computing Machinery, 2004: 205-211.
[6] Kim KH, Bang SW, Kim SR. Emotion recognition system using short-term monitoring of physiological signals[J]. Medical and Biological Engineering and Computing, 2004, 42: 419-427.
[7] Black MH, Chen NTM, Iyer KK, et al. Mechanisms of facial emotion recognition in autism spectrum disorders: Insights from eye tracking and electroencephalography[J]. Neuroscience & Biobehavioral Reviews, 2017, 80: 488-515.
[8] Alarc?o SM, Fonseca MJ. Emotions recognition using EEG signals: a survey[J]. IEEE Transactions on Affective Computing, 2019, 10(3): 374-393.
[9] Zheng WL, Zhu JY, Lu BL. Identifying stable patterns over time for emotion recognition from EEG[J]. IEEE Transactions on Affective Computing, 2019, 10(3): 417-429.
[10] Cacioppo JT, Berntson GG, Larsen JT, et al. The psychophysiology of emotion[M]//The handbook of emotions. 2nd ed. New York, USA: Guilford Press, 2000: 173-191.
[11] 聂聃, 王晓韡, 段若男, 等. 基于脑电的情绪识别研究综述[J]. 中国生物医学工程学报, 2012, 31(4): 595-606.  Nie D, Wang XW, Duan RN, et al. A survey on EEG based emotion recognition[J]. Chinese Journal of Biomedical Engineering, 2012, 31 (4): 595-606.
[12] Moher D, Shamseer L, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement[J]. Systematic Reviews, 2015, 4: 1.
[13] Koelstra S, Mühl C, Soleymani M, et al. DEAP: a database for emotion analysis; using physiological signals[J]. IEEE Transactions on Affective Computing, 2011, 3: 18-31.
[14] Wang X, Zhang T, Xu X, et al. EEG emotion recognition using dynamical graph convolutional neural networks and broad learning system[C]//2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Madrid, Spain: IEEE Press, 2018,1: 1240-1244.
[15] Qiao R, Qing C, Zhang T, et al. A novel deep-learning based framework for multi-subject emotion recognition[C]//2017 4th International Conference on Information, Cybernetics and Computational Social Systems (ICCSS). Dalian, China: IEEE Press, 2017: 181-185.
[16] Li Y, Huang J, Zhou H, et al. Human emotion recognition with electroencephalographic multidimensional features by hybrid deep neural networks[J]. Applied Sciences, 2017, 7(10): 1060.
[17] Xing X, Li Z, Xu T, et al. SAE+ LSTM: A New Framework for Emotion Recognition from Multi-Channel EEG[J]. Frontiers in Neurorobotics, 2019, 13: 37.
[18] Chen T, Ju S, Yuan X, et al. Emotion recognition using empirical mode decomposition and approximation entropy[J]. Computers & Electrical Engineering, 2018, 72: 383-392.
[19] Salama ES, El-Khoribi RA, Shoman ME, et al. EEG-based emotion recognition using 3D convolutional neural networks[J]. International Journal of Advanced Computer Science and Applications, 2018, 9(8): 329-337.
[20] Jung TP, Makeig S, Humphries C, et al. Removing electroencephalographic artifacts by blind source separation[J]. Psychophysiology, 2010, 37: 163-178.
[21] Nathan K, Contreras-Vidal JL. Negligible motion artifacts in scalp electroencephalography (EEG) during treadmill walking[J]. Frontiers in Human Neuroscience, 2015, 9: 708.
[22] Delorme A, Sejnowski T, Makeig S. Enhanced detection of artifacts in EEG data using higher-order statistics and independent component analysis[J]. Neuroimage, 2007, 34(4): 1443-1449.
[23] Akhtar MT, Mitsuhashi W, James CJ. Employing spatially constrained ICA and wavelet denoising, for automatic removal of artifacts from multichannel EEG data[J]. Signal Processing, 2012, 92(2): 401-416.
[24] Islam MK, Rastegarnia A, Yang Z. Methods for artifact detection and removal from scalp EEG: A review[J]. Neurophysiologie Clinique, 2016, 46(4-5): 287-305.
[25] Daly I, Nicolaou N, Nasuto SJ, et al. Automated artifact removal from the electroencephalogram: a comparative study[J]. Clinical EEG and Neuroscience, 2013, 44(4): 291-306.
[26] Mannan MMN, Kamran MA, Jeong MY. Identification and removal of physiological artifacts from electroencephalogram signals: a review[J]. IEEE Access, 2018, 6: 30630-30652.
[27] Zheng WL, Lu BL. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks[J]. IEEE Transactions on Autonomous Mental Development, 2020, 7(3): 162-175.
[28] Yanagimoto M, Sugimoto C. Recognition of persisting emotional valence from EEG using convolutional neural networks[C]//2016 IEEE 9th International Workshop on Computational Intelligence and Applications (IWCIA). Hiroshima, Japan: IEEE Press, 2016: 27-32.
[29] Alhagry S, Fahmy AA, El-Khoribi RA. Emotion recognition based on EEG using LSTM recurrent neural network[J]. International Journal of Advanced Computer Science and Applications, 2017, 8(10): 355-358.
[30] Yang Y, Wu Q, Qiu M, et al. Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network[C]//2018 International Joint Conference on Neural Networks (IJCNN). Rio de Janeiro, Brazil: IEEE Press, 2018: 1-7.
[31] Das BB, Kumar P, Kar D, et al. A spatio-temporal model for EEG-based person identification[J]. Multimedia Tools and Applications, 2019, 78(19): 28157-28177.
[32] Kandel ER, Schwartz JH, Jessell TM, et al. Principles of neural science[M]. 5th ed. New York, USA: McGraw-hill, 2000, 4: 1227-1246.
[33] Zheng WL, Zhu JY, Peng Y, et al. EEG-based emotion classification using deep belief networks[C]//2014 IEEE International Conference on Multimedia and Expo (ICME). Chengdu, China: IEEE Press, 2014: 1-6.
[34] Li H, Qing C, Xu X, et al. A novel DE-PCCM feature for EEG-based emotion recognition[C]//2017 International Conference on Security, Pattern Analysis, and Cybernetics (SPAC). Shenzhen, China: IEEE Press, 2017: 389-393.
[35] Li J, Zhang Z, He H. Hierarchical convolutional neural networks for EEG-based emotion recognition[J]. Cognitive Computation, 2018, 10(2): 368-380.
[36] Huang J, Xu X, Zhang T. Emotion classification using deep neural networks and emotional patches[C]//2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Kansas City, MO, USA: IEEE Press, 2017: 958-962.
[37] Wang F, Zhong S, Peng J, et al. Data augmentation for EEG-based emotion recognition with deep convolutional neural networks[C]//International Conference on Multimedia Modeling. Bangkok, Thailand: Springer, Cham, 2018: 82-93.
[38] Zhang T, Zheng W, Cui Z, et al. Spatial–temporal recurrent neural network for emotion recognition[J]. IEEE Transactions on Cybernetics, 2019, 49(3): 839-847.
[39] Luo Y. EEG data augmentation for emotion recognition using a conditional wasserstein GAN[C]//2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Honolulu, HI, USA: IEEE Press, 2018: 2535-2538.
[40] Xu H, Plataniotis KN. Affective states classification using EEG and semi-supervised deep learning approaches[C]//2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP). Montreal, QC, Canada: IEEE Press: 2016: 1-6.
[41] Al-Nafjan A, Hosny M, Al-Wabil A, et al. Classification of human emotions from electroencephalogram (EEG) signal using deep neural network[J]. International Journal of Advanced Computer Science and Applications (IJACSA), 2017, 8(9): 419-425.
[42] Zhang W, Liu Q. Using the center loss function to improve deep learning performance for EEG signal classification[C]//2018 Tenth International Conference on Advanced Computational Intelligence (ICACI). Xiamen, China: IEEE Press, 2018: 578-582.
[43] Rayatdoost S, Soleymani M. Cross-corpus EEG-based emotion recognition[C]//2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP). Aalborg, Denmark: IEEE Press, 2018: 1-6.
[44] Moon SE, Jang S, Lee JS. Convolutional neural network approach for EEG-based emotion recognition using brain connectivity and its spatial information[C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Seoul, South Korea: IEEE Press, 2018: 2556-2560.
[45] Chen JX, Zhang PW, Mao ZJ, et al. Accurate EEG-based emotion recognition on combined features using deep convolutional neural networks[J]. IEEE Access, 2019, 7: 44317-44328.
[46] Chao H, Dong L, Liu Y, et al. Emotion recognition from multiband EEG signals using CapsNet[J]. Sensors, 2019, 19(9): 2212-2246.
[47] Chao H, Zhi H, Dong L, et al. Recognition of emotions using multichannel EEG data and DBN-GC-based ensemble deep learning framework[J]. Computational Intelligence and Neuroscience, 2018, 2018(2): 1-11.
[48] Lu Y, Wang M, Wu W, et al. Identification of emotional valences via memory-informed deep neural network with entropy features[C]// 2019 4th International Conference on Multimedia Systems and Signal Processing. Guangzhou, China: ACM, 2019: 31-35.
[49] Liu N, Fang Y, Li L, et al. Multiple feature fusion for automatic emotion recognition using EEG signals[C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Seoul, Korea: IEEE Press, 2018: 896-900.
[50] Li X, Song D, Zhang P, et al. Emotion recognition from multi-channel EEG data through convolutional recurrent neural network[C]//2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Shenzhen, China: IEEE Press, 2016: 352-359.
[51] Li X, Song D, Zhang P, et al. Deep fusion of multi-channel neurophysiological signal for emotion recognition and monitoring[J]. International Journal of Data Mining and Bioinformatics, 2017, 18(1): 1-27.

服务与反馈：

【文章下载】【加入收藏】

提示：您还未登录，请登录！点此登录