Keywords:
emoton recognition
deep learning
machine learning
human-computer interaction
databases
applications

  • Abstract

    In recent years, the real-time facial expression recognition system based on artificial intelligence technology has garnered significant attention from academia and industry. This paper presents a systematic literature review and bibliometric analysis to examine the latest publications in this field, summarizing the development and research significance of facial expression recognition technology and emphasizing its vital role in human-computer interaction and affective computing. The study used PRISMA to review 386 articles published from January 2019 to December 2023 in Web of Science, Scopus, IEEE Xplore, and ACM Digital Library. It encompasses covering various research methodologies, datasets, and application areas, as well as artificial intelligence technology, algorithms, and models. This review highlights advancements in Facial Expression Recognition, particularly the predominant use of databases such as FER2013 and CK+ while identifying Convolutional Neural Networks as the primary technique for real-time emotion classification. A quantitative analysis of research trends over the past five years indicates a shift toward keywords like transfer learning and applications in domains such as healthcare and the Internet of Things. Contemporary deep learning models, including CNNs, ResNet, and VGG, demonstrate impressive accuracy in classifying seven basic emotions, facilitating real-time applications across multiple fields. However, challenges such as overfitting, sensitivity to environmental factors, and the necessity for high-performance computing resources impede the broader deployment of these systems. These findings underscore the urgent need for further research to address these limitations and enhance the ethical application of FER technologies. Finally, based on the review and analysis results, this paper outlines future research directions for this technology, including multimodal information fusion, computational modelling, personalized emotion recognition, and interdisciplinary cooperation, thereby providing valuable references and inspiration for future works.

  • References

    1. Adhikary, A., & Bhandari, N. (2021). PosEmotion—Combining real-time 2D body pose estimation and facial emotion recognition to analyze human behavior. 2021 26th International Conference on Automation and Computing (ICAC) (pp. 1–6). https://doi.org/10.23919/ICAC50006.2021.9594155
    2. Ahmed, T. U., Hossain, S., Hossain, M. S., Islam, R. U., & Andersson, K. (2022). A deep learning approach with data augmentation to recognize facial expressions in real time. (pp. 487-500). Springer. https://doi.org/10.1007/978-981-16-7597-3_40
    3. Aifanti, N., Papachristou, C., & Delopoulos, A. (2010). The MUG facial expression database. 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10 (pp. 1-4).
    4. Aiswarya, P., Manish, & Mangalraj, P. (2020). Emotion recognition by inclusion of age and gender parameters with a novel hierarchical approach using deep learning. 2020 Advanced Communication Technologies and Signal Processing (ACTS) (pp. 1-6). https://doi.org/10.1109/ACTS49415.2020.9350479
    5. Alhajlah, M. (2023). A novel efficient patient monitoring FER system using optimal DL-features. Computers, Materials and Continua, 74(3), 6161–6175. https://doi.org/10.32604/cmc.2023.032505
    6. Alisawi, M., & Yalçın, N. (2023). Real-time emotion recognition using deep learning methods: Systematic review. Intelligent Methods in Engineering Sciences, 2(1), 5–21. https://doi.org/10.58190/imiens.2023.7
    7. Alreshidi, A., & Ullah, M. (2020). Facial emotion recognition using hybrid features. Informatics, 7(1). https://doi.org/10.3390/informatics7010006
    8. Arabian, H., Wagner-Hartl, V., & Moeller, K. (2021). Traditional versus neural network classification methods for facial emotion recognition. Current Directions in Biomedical Engineering, 7(2), 203-206. https://doi.org/10.1515/cdbme-2021-2052
    9. Ashraf, A., Gunawan, T. S., Arifin, F., Kartiwi, M., Sophian, A., & Habaebi, M. H. (2023). Enhanced emotion recognition in videos: A convolutional neural network strategy for human facial expression detection and classification. Indonesian Journal of Electrical Engineering and Informatics, 11(1), 286–299. https://doi.org/10.52549/ijeei.v11i1.4449
    10. Awais, M., Raza, M., Singh, N., Bashir, K., Manzoor, U., Islam, S. U., & Rodrigues, J. J. P. C. (2021). LSTM-based emotion detection using physiological signals: IoT framework for healthcare and distance learning in COVID-19. IEEE Internet of Things Journal, 8(23), 16863–16871. https://doi.org/10.1109/JIOT.2020.3044031
    11. Bakariya, B., Singh, A., Singh, H., Raju, P., Rajpoot, R., & Mohbey, K. K. (2023). Facial emotion recognition and music recommendation system using CNN-based deep learning techniques. Evolving Systems. https://doi.org/10.1007/s12530-023-09506-z
    12. Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019). Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423–443. https://doi.org/10.1109/TPAMI.2018.2798607
    13. Barron-Estrada, M. L., Zatarain-Cabada, R., & Oramas-Bustillos, R. (2019). Emotion recognition for education using sentiment analysis. Research in Computing Science, 148(5), 71–80. https://doi.org/10.13053/rcs-148-5-8
    14. Bhardwaj, P., Gupta, P. K., Panwar, H., Siddiqui, M. K., Morales-Menendez, R., & Bhaik, A. (2021). Application of deep learning on student engagement in e-learning environments. Computers and Electrical Engineering, 93, 107277. https://doi.org/10.1016/j.compeleceng.2021.107277
    15. Bhargavi, Y., D, B., & Prince, S. (2023). AI-based emotion therapy bot for children with autism spectrum disorder (ASD). 2023 9th International Conference on Advanced Computing and Communication Systems (ICACCS) (Vol. 1, pp. 1895-1899). https://doi.org/10.1109/ICACCS57279.2023.10112868
    16. Brintha, N. C., Narayana, J. A., Jaswanth, G. L. V. S., Chandrapal, G. J., & Venkat, D. (2022). Realtime facial emotion detection using machine learning. 2022 International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems (ICSES) (pp. 1-5). https://doi.org/10.1109/ICSES55317.2022.9914318
    17. C, V., & Palaniswamy, S. (2022). Emotion recognition at real-time applications using meta-learning. 2022 IEEE 3rd Global Conference for Advancement in Technology (GCAT) (pp. 1-6). https://doi.org/10.1109/GCAT55367.2022.9971841
    18. Calvo, M. G., & Lundqvist, D. (2008). Facial expressions of emotion (KDEF): Identification under different display-duration conditions. Behavior Research Methods, 40(1), 109–115. https://doi.org/10.3758/BRM.40.1.109
    19. Canedo, D., & Neves, A. J. R. (2019). Facial expression recognition using computer vision: A systematic review. Applied Sciences, 9(21), 4678. https://doi.org/10.3390/app9214678
    20. Chaiyarak, S., Nilsook, P., & Wannapiroon, P. (2021). An empirical study of intelligent virtual universal learning platforms. 2021 Research, Invention, and Innovation Congress: Innovation Electricals and Electronics (RI2C) (pp. 66-73). https://doi.org/10.1109/RI2C51727.2021.9559785
    21. Chandra, M. A., & Bedi, S. S. (2021). Survey on SVM and their application in image classification. International Journal of Information Technology, 13(5), 1-11. https://doi.org/10.1007/s41870-017-0080-1
    22. Chauhan, V. K., Dahiya, K., & Sharma, A. (2019). Problem formulations and solvers in linear SVM: A review. Artificial Intelligence Review, 52(2), 803-855. https://doi.org/10.1007/s10462-018-9614-6
    23. Chen, C. (2017). Science mapping: A systematic review of the literature. Journal of Data and Information Science, 2(2), 1-40. https://doi.org/10.1515/jdis-2017-0006
    24. Cherkassky, V., & Ma, Y. (2004). Practical selection of SVM parameters and noise estimation for SVM regression. Neural Networks, 17(1), 113-126. https://doi.org/10.1016/S0893-6080(03)00169-2
    25. Cotter, S. (2020). Low complexity deep learning for mobile face expression recognition. Proceedings of the 3rd International Conference on Vision, Image and Signal Processing. https://doi.org/10.1145/3387168.3387175
    26. Das, P. J., Talukdar, A. K., & Sarma, K. K. (2019). A framework for human behaviour detection using combined analysis of facial expression and eye gaze. 2019 International Conference on Image and Signal Processing for Communication (IESPC) (pp. 154-160). https://doi.org/10.1109/IESPC.2019.8902367
    27. De Ocampo, A. L. P. (2023). Haar-CNN cascade for facial expression recognition. 2023 International Electrical Engineering Congress (iEECON) (pp. 89-92). https://doi.org/10.1109/iEECON56657.2023.10126902
    28. Dhall, A. (2019). EmotiW 2019: Automatic emotion, engagement and cohesion prediction tasks. 2019 International Conference on Multimodal Interaction (pp. 546-550). https://doi.org/10.1145/3340555.3355710
    29. Dhope, P., & Neelagar, M. B. (2022). Real-time emotion recognition from facial expressions using artificial intelligence. 2022 2nd International Conference on Artificial Intelligence and Signal Processing (AISP) (pp. 1-6). https://doi.org/10.1109/AISP53593.2022.9760654
    30. Dudekula, U., & Purnachand, N. (2023). Analysis of facial emotion recognition rate for real-time application using NVIDIA Jetson Nano in deep learning models. Indonesian Journal of Electrical Engineering and Computer Science, 30(1), 598-605. https://doi.org/10.11591/ijeecs.v30.i1.pp598-605
    31. Dwijayanti, S., Iqbal, M., & Suprapto, B. Y. (2022). Real-time implementation of face recognition and emotion recognition in a humanoid robot using a convolutional neural network. IEEE Access, 10, 89876-89886. https://doi.org/10.1109/ACCESS.2022.3200762
    32. Eck, N. J. V., & Waltman, L. (2009). How to normalize cooccurrence data? An analysis of some well-known similarity measures. Journal of the American Society for Information Science and Technology, 60(8), 1635-1651. https://doi.org/10.1002/asi.21075
    33. Ekman, P. (1971). Universals and cultural differences in facial expressions of emotion. Nebraska Symposium on Motivation, 19, 207-283.
    34. Fakhar, S., Baber, J., Bazai, S. U., Marjan, S., Jasinski, M., Jasinska, E., Chaudhry, M. U., Leonowicz, Z., & Hussain, S. (2022). Smart classroom monitoring using novel real-time facial expression recognition system. Applied Sciences, 12(23), 12134. https://doi.org/10.3390/app122312134
    35. Feak, C., & Swales, J. (2009). Telling a research story: Writing a literature review. University of Michigan Press/ELT. https://doi.org/10.3998/mpub.309338
    36. Fernandes, J. V. M. R., Alexandria, A. R. D., Marques, J. A. L., Assis, D. F. D., Motta, P. C., & Silva, B. R. D. S. (2024). Emotion detection from EEG signals using machine deep learning models. Bioengineering, 11(8), 782. https://doi.org/10.3390/bioengineering11080782
    37. Filipovic, F., Despotovic-Zrakic, M., Radenkovic, B., Jovanic, B., & Živojinovic, L. (2019). An application of artificial intelligence for detecting emotions in neuromarketing. 2019 International Conference on Artificial Intelligence: Applications and Innovations (IC-AIAI) (pp. 49-494). https://doi.org/10.1109/IC-AIAI48757.2019.00016
    38. Gao, J., & Zhao, Y. (2021). TFE: A transformer architecture for occlusion aware facial expression recognition. Frontiers in Neurorobotics, 15, 763100. https://doi.org/10.3389/fnbot.2021.763100
    39. Goel, A. K., Jain, A., Saini, C., Ashutosh, Das, R., & Deep, A. (2022). Implementation of AI/ML for human emotion detection using facial recognition. 2022 IEEE 4th International Conference on Cybernetics, Cognition and Machine Learning Applications (ICCCMLA) (pp. 511–515). https://doi.org/10.1109/ICCCMLA56841.2022.9989091
    40. Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., ... & Bengio, Y. (2015). Challenges in representation learning: A report on three machine learning contests. Neural Networks, 64, 59-63. https://doi.org/10.1016/j.neunet.2014.09.005
    41. Gupta, S., Kumar, P., & Tekchandani, R. K. (2023). Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models. Multimedia Tools and Applications, 82(8), 11365-11394. https://doi.org/10.1007/s11042-022-13558-9
    42. H. Zhu, P. Hu, X. Tang, & D. Xia. (2022). NAGNet: A Convolutional Neural Network for Real-Time Sentiment Analysis of Students. 2022 12th International Conference on Information Technology in Medicine and Education (ITME), 75–80. https://doi.org/10.1109/ITME56794.2022.00027
    43. Haddaway, N. R., Page, M. J., Pritchard, C. C., & McGuinness, L. A. (2022). PRISMA2020: An R package and Shiny app for producing PRISMA 2020‐compliant flow diagrams, with interactivity for optimised digital transparency and Open Synthesis. Campbell Systematic Reviews, 18(2), e1230. https://doi.org/10.1002/cl2.1230
    44. Han, B., Yun, W.-H., Yoo, J.-H., & Kim, W. H. (2020). Toward Unbiased Facial Expression Recognition in the Wild via Cross-Dataset Adaptation. IEEE Access, 8, 159172–159181. https://doi.org/10.1109/ACCESS.2020.3018738
    45. Hart, C. (2011). Doing a literature review: Releasing the social science research imagination (Repr.). Sage.
    46. Hashan, A. M., Adnan Adhab, K. A.-S., Islam, R. M. R. U., Avinash, K., & Dey, S. (2023). Automated human facial emotion recognition system using depthwise separable convolutional neural network. 2023 IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) (pp. 113–117). https://doi.org/10.1109/IAICT59002.2023.10205785
    47. Hashan, A. M., Adnan Adhab, K. A.-S., Islam, R. M. R. U., Avinash, K., & Dey, S. (2023). Automated human facial emotion recognition system using depthwise separable convolutional neural network. 2023 IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) (pp. 113-117). https://doi.org/10.1109/IAICT59002.2023.10205785
    48. Hatem, A. S., & Al-Bakry, A. M. (2022). Real-time human emotion recognition using transfer learning. 2022 International Conference on Data Science and Intelligent Computing (ICDSIC) (pp. 224–227). https://doi.org/10.1109/ICDSIC56987.2022.10076032
    49. He, Z., & Qin, X. (2022). Analysis of facial expressions in class based on lightweight convolutional neural network. 2022 International Conference on Industrial Automation, Robotics and Control Engineering (IARCE) (pp. 68-74). https://doi.org/10.1109/IARCE57187.2022.00023
    50. Hu, H., Zhu, Y., Zhang, Y., Zhou, Q., Feng, Y., & Tan, G. (2019). Comprehensive driver state recognition based on deep learning and PERCLOS criterion. 2019 IEEE 19th International Conference on Communication Technology (ICCT) (pp. 1678-1682). https://doi.org/10.1109/ICCT46805.2019.8947282
    51. Huang, M., Zhang, X., Lan, X., Wang, H., & Tang, Y. (2022). Convolution by multiplication: Accelerated two-stream Fourier domain convolutional neural network for facial expression recognition. IEEE Transactions on Circuits and Systems for Video Technology, 32(3), 1431-1442. https://doi.org/10.1109/TCSVT.2021.3073558
    52. Hussain, S. A., & Salim Abdallah Al Balushi, A. (2020). A real time face emotion classification and recognition using deep learning model. Journal of Physics: Conference Series, 1432(1), 012087. https://doi.org/10.1088/1742-6596/1432/1/012087
    53. Ilyas, B. R., Abderrazak, T. A., Sofiane, B. M., Bahidja, B., Imane, H., & Miloud, K. (2023). A robust-facial expressions recognition system using deep learning architectures. 2023 International Conference on Data Analytics for Business and Industry (DASA) (pp. 541–546). https://doi.org/10.1109/DASA59624.2023.10286798
    54. J. Kaur, J. Saxena, J. Shah, Fahad, & S. P. Yadav. (2022). Facial Emotion Recognition. 2022 International Conference on Computational Intelligence and Sustainable Engineering Solutions (CISES), 528–533. https://doi.org/10.1109/CISES54857.2022.9844366
    55. J. L. Silva, I. Oliveira, Z. Topolniak, & A. B. Alvarez. (2021). A CNN Approach Implemented to Emotional Facial Expression Recognition for the Prevention of Autistic Meltdowns. 2021 2nd Sustainable Cities Latin America Conference (SCLA), 1–6. https://doi.org/10.1109/SCLA53004.2021.9540183
    56. J. R. Hou Lee & A. Wong. (2020). TimeConvNets: A Deep Time Windowed Convolution Neural Network Design for Real-time Video Facial Expression Recognition. 2020 17th Conference on Computer and Robot Vision (CRV), 9–16. https://doi.org/10.1109/CRV50864.2020.00010
    57. J. Vice, M. M. Khan, & S. Yanushkevich. (2019). Multimodal Models for Contextual Affect Assessment in Real-Time. 2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI), 87–92. https://doi.org/10.1109/CogMI48466.2019.00020
    58. J. Yang, T. Qian, F. Zhang, & S. U. Khan. (2021). Real-Time Facial Expression Recognition Based on Edge Computing. IEEE Access, 9, 76178–76190. https://doi.org/10.1109/ACCESS.2021.3082641
    59. J. Zhu, Y. Wang, R. La, J. Zhan, J. Niu, S. Zeng, & X. Hu. (2019). Multimodal Mild Depression Recognition Based on EEG-EM Synchronization Acquisition Network. IEEE Access, 7, 28196–28210. https://doi.org/10.1109/ACCESS.2019.2901950
    60. Jaffar, S. S., & Abdulbaqi, H. A. (2022). Facial expression recognition in static images for autism children using CNN approaches. 2022 Fifth College of Science International Conference of Recent Trends in Information Technology (CSCTIT) (pp. 202-207). https://doi.org/10.1109/CSCTIT56299.2022.10145689
    61. Jain, A., Verma, R., Khokhar, G. S., & Bhadauria, M. (2022). Akshi: An assistance system for visually challenged using machine learning. 2022 4th International Conference on Artificial Intelligence and Speech Technology (AIST) (pp. 1–6). https://doi.org/10.1109/AIST55798.2022.10064996
    62. Jaiswal, S., & Nandi, G. C. (2020). Robust real-time emotion detection system using CNN architecture. Neural Computing and Applications, 32(15), 11253–11262. Scopus. https://doi.org/10.1007/s00521-019-04564-4
    63. Jeong, M., Nam, J., & Ko, B. C. (2020). Lightweight multilayer random forests for monitoring driver emotional status. IEEE Access, 8, 60344-60354. https://doi.org/10.1109/ACCESS.2020.2983202
    64. Jesson, J., Matheson, L., & Lacey, F. M. (2011). Doing Your Literature Review: Traditional and Systematic Techniques. SAGE Publications.
    65. K, S. V., & Thripurala, S. (2023). Real-time facial emotion detection system using multimodal fusion deep learning architecture. 2023 International Conference on Electrical, Electronics, Communication and Computers (ELEXCOM) (pp. 1-6). https://doi.org/10.1109/ELEXCOM58812.2023.10370457
    66. K. Mayuri, N. V. Krishna Rao, N. Jayanthi, T. AlakanandaKasam, G. Nalla, & S. Jaggavarapu. (2021). Understanding Customer Reviews using Facial Expression Recognition System. 2021 5th International Conference on Electronics, Communication and Aerospace Technology (ICECA), 1482–1486. https://doi.org/10.1109/ICECA52323.2021.9676126
    67. Kabakus, A. T. (2020). PyFER: A facial expression recognizer based on convolutional neural networks. IEEE Access, 8, 142243–142249. https://doi.org/10.1109/ACCESS.2020.3012703
    68. Kanade, T., Cohn, J. F., & Yingli Tian. (2000). Comprehensive database for facial expression analysis. Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), 46–53. https://doi.org/10.1109/AFGR.2000.840611
    69. Karnati, M., Seal, A., Yazidi, A., & Krejcar, O. (2022). FLEPNet: Feature level ensemble parallel network for facial expression recognition. IEEE Transactions on Affective Computing, 13(4), 2058-2070. https://doi.org/10.1109/TAFFC.2022.3208309
    70. Kartheek, M. N., Prasad, M. V. N. K., & Bhukya, R. (2022). DRCP: Dimensionality Reduced Chess Pattern for Person Independent Facial Expression Recognition. In INTERNATIONAL JOURNAL OF PATTERN RECOGNITION AND ARTIFICIAL IN℡LIGENCE (Vol. 36, Issue 11). WORLD SCIENTIFIC PUBL CO PTE LTD. https://doi.org/10.1142/S021800142256016X
    71. Kaur, S., & Kulkarni, N. (2023). FERFM: An Enhanced Facial Emotion Recognition System Using Fine-tuned MobileNetV2 Architecture. IETE Journal of Research. Scopus. https://doi.org/10.1080/03772063.2023.2202158
    72. Khan, M., Hariharasitaraman, S., Joshi, S., Jain, V., Ramanan, M., SampathKumar, A., & Elngar, A. A. (2022). A deep learning approach for facial emotions recognition using principal component analysis and neural network techniques. Photogrammetric Record, 37(180), 435–452. Scopus. https://doi.org/10.1111/phor.12426
    73. Khine, W. S. S., Siritanawan, P., & Kotani, K. (2021). Automatic peak frame selection from dynamic facial expressions. 2021 60th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE) (pp. 1088-1093).
    74. Kim, Y., Soyata, T., & Behnagh, R. F. (2018). Towards Emotionally Aware AI Smart Classroom: Current Issues and Directions for Engineering and Education. IEEE Access, 6, 5308–5331. https://doi.org/10.1109/ACCESS.2018.2791861
    75. Ko, B. (2018). A Brief Review of Facial Emotion Recognition Based on Visual Information. Sensors, 18(2), 401. https://doi.org/10.3390/s18020401
    76. Kong, Y., Ren, Z., Zhang, K., Zhang, S., Ni, Q., & Han, J. (2021). Lightweight facial expression recognition method based on attention mechanism and key region fusion. Journal of Electronic Imaging, 30(6). Scopus. https://doi.org/10.1117/1.JEI.30.6.063002
    77. Kossaifi, J., Tzimiropoulos, G., Todorovic, S., & Pantic, M. (2017). AFEW-VA database for valence and arousal estimation in-the-wild. Image and Vision Computing, 65, 23–36. https://doi.org/10.1016/j.imavis.2017.02.001
    78. L. Liakopoulos, N. Stagakis, E. I. Zacharaki, & K. Moustakas. (2021). CNN-based stress and emotion recognition in ambulatory settings. 2021 12th International Conference on Information, Intelligence, Systems & Applications (IISA), 1–8. https://doi.org/10.1109/IISA52424.2021.9555508
    79. L. Xie, J. Zhao, H. Wei, K. Zhang, & G. Pang. (2019). Online Kernel-Based Structured Output SVM for Early Expression Detection. IEEE Signal Processing Letters, 26(9), 1305–1309. https://doi.org/10.1109/LSP.2019.2929713
    80. L. Zahara, P. Musa, E. Prasetyo Wibowo, I. Karim, & S. Bahri Musa. (2020). The Facial Emotion Recognition (FER-2013) Dataset for Prediction System of Micro-Expressions Face Using the Convolutional Neural Network (CNN) Algorithm based Raspberry Pi. 2020 Fifth International Conference on Informatics and Computing (ICIC), 1–9. https://doi.org/10.1109/ICIC50835.2020.9288560
    81. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324. https://doi.org/10.1109/5.726791
    82. Lee, J. R., Wang, L., & Wong, A. (2021). EmotionNet Nano: An efficient deep convolutional neural network design for real-time facial expression recognition. Frontiers in Artificial Intelligence, 3, 609673. https://doi.org/10.3389/frai.2020.609673
    83. Li, S., Deng, W., & Du, J. (2017). Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2584-2593). https://doi.org/10.1109/CVPR.2017.277
    84. Lievrouw, L. A. (1989). The invisible college reconsidered: Bibliometrics and the development of scientific communication theory. Communication Research, 16(5), 615-628. https://doi.org/10.1177/009365089016005004
    85. Lin, K.-Y., Gamboa-Gonzalez, A., & Wehner, M. (2021). Soft robotic sensing, proprioception via cable and microfluidic transmission. Electronics, 10(24), 3166. https://doi.org/10.3390/electronics10243166
    86. Livingstone, S. R., & Russo, F. A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLOS ONE, 13(5), e0196391. https://doi.org/10.1371/journal.pone.0196391
    87. LokeshNaik, S. K., Punitha, A., Vijayakarthik, P., Kiran, A., Dhangar, A. N., Reddy, B. J., & Sudheeksha, M. (2023). Real time facial emotion recognition using deep learning and CNN. 2023 International Conference on Computer Communication and Informatics (ICCCI) (pp. 1-5). https://doi.org/10.1109/ICCCI56745.2023.10128259
    88. Lopes, A. T., de Aguiar, E., de Souza, A. F., & Oliveira-Santos, T. (2017). Facial expression recognition with convolutional neural networks: Coping with few data and the training sample order. Pattern Recognition, 61, 610-628. https://doi.org/10.1016/j.patcog.2016.07.026
    89. Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010). The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops (pp. 94-101). https://doi.org/10.1109/CVPRW.2010.5543262
    90. Lyons, M., Kamachi, M., & Gyoba, J. (1998). The Japanese Female Facial Expression (JAFFE) Dataset [Dataset]. Zenodo. https://doi.org/10.5281/zenodo.3451523
    91. Mahrab, N., Salim, S. A., Ali, A. I., Mim, I. J., & Khan, R. (2021). Facial expression based automated restaurant food review system using CNN. 2021 IEEE International Conference on Artificial Intelligence in Engineering and Technology (IICAIET) (pp. 1-4). https://doi.org/10.1109/IICAIET51634.2021.9573899
    92. Majib, M. S., Rahman, M. M., Sazzad, T. M. S., Khan, N. I., & Dey, S. K. (2021). VGG-SCNet: A VGG net-based deep learning framework for brain tumor detection on MRI images. IEEE Access, 9, 116942-116952. https://doi.org/10.1109/ACCESS.2021.3105874
    93. Marques, J. A. L., Neto, A. C., Silva, S. C., & Bigne, E. (2024). Predicting consumer ad preferences: Leveraging a machine learning approach for EDA and FEA neurophysiological metrics. Psychology & Marketing, 41(3). https://doi.org/10.1002/mar.22118
    94. Martínez, J., & Vega, J. (2023). ROS system facial emotion detection using machine learning for a low-cost robot based on Raspberry Pi. Electronics, 12(1), 90. https://doi.org/10.3390/electronics12010090
    95. Melinte, D. O., & Vladareanu, L. (2020). Facial expressions recognition for human-robot interaction using deep convolutional neural networks with rectified Adam optimizer. Sensors, 20(8), 2393. https://doi.org/10.3390/s20082393
    96. Metgud, P., Naik, N. D., S, S. M., & Prasad, A. S. (2022). Real-time student emotion and performance analysis. 2022 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) (pp. 1-5). https://doi.org/10.1109/CONECCT55679.2022.9865114
    97. Miao, Y., Dong, H., Jaam, J. M. A., & Saddik, A. E. (2019). A deep learning system for recognizing facial expression in real-time. ACM Transactions on Multimedia Computing, Communications, and Applications, 15(2). https://doi.org/10.1145/3311747
    98. Mikhaylevskiy, S., Chernyavskiy, V., Pavlishen, V., Romanova, I., & Solovyev, R. (2021). Fast emotion recognition neural network for IoT devices. 2021 International Seminar on Electron Devices Design and Production (SED) (pp. 1-6). https://doi.org/10.1109/SED51197.2021.9444517
    99. Mokadam, P., Kulkarni, T., & Mulla, N. (2020). Customer reaction analysis using convolutional neural network and aspect based LSTM model. 2020 2nd International Conference on Innovative Mechanisms for Industry Applications (ICIMIA) (pp. 696-700). https://doi.org/10.1109/ICIMIA48430.2020.9074939
    100. Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019). AffectNet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1), 18-31. https://doi.org/10.1109/TAFFC.2017.2740923
    101. Muhammad, G., & Hossain, M. S. (2021). Emotion recognition for cognitive edge computing using deep learning. IEEE Internet of Things Journal, 8(23), 16894-16901. https://doi.org/10.1109/JIOT.2021.3058587
    102. Muhammad, S., Ahmed, S., & Naik, D. (2021). Real time emotion based music player using CNN architectures. 2021 6th International Conference for Convergence in Technology (I2CT) (pp. 1-5). https://doi.org/10.1109/I2CT51068.2021.9417949
    103. Mukhopadhyay, M., Pal, S., Nayyar, A., Pramanik, P. K. D., Dasgupta, N., & Choudhury, P. (2020). Facial emotion detection to assess learner's state of mind in an online learning system. Proceedings of the 2020 5th International Conference on Intelligent Information Technology (pp. 107-115). https://doi.org/10.1145/3385209.3385231
    104. Nandi, A., Xhafa, F., Subirats, L., & Fort, S. (2020). A survey on multimodal data stream mining for e-learner’s emotion recognition. 2020 International Conference on Omni-Layer Intelligent Systems (COINS) (pp. 1–6). https://doi.org/10.1109/COINS49042.2020.9191370
    105. Nandi, A., Xhafa, F., Subirats, L., & Fort, S. (2020). A survey on multimodal data stream mining for e-learner's emotion recognition. 2020 International Conference on Omni-Layer Intelligent Systems (COINS) (pp. 1-6). https://doi.org/10.1109/COINS49042.2020.9191370
    106. Ozdemir, M. A., Elagoz, B., Alaybeyoglu, A., Sadighzadeh, R., & Akan, A. (2019). Real time emotion recognition from facial expressions using CNN architecture. 2019 Medical Technologies Congress (TIPTEKNO) (pp. 1-4). https://doi.org/10.1109/TIPTEKNO.2019.8895215
    107. Afsar, P., Malayil, S., Verghese, A., Sithara, A., Pakarath, A., & Ashmin, K. T. (2023, July). Real-Time Student Emotion and Drowsiness Detection Using YOLOv5 and CNN for Enhanced Learning. In 2023 International Conference on Innovations in Engineering and Technology (ICIET) (pp. 1-8). IEEE.https://doi.org/10.1109/ICIET57285.2023.10220929
    108. Pantic, M., Valstar, M., Rademaker, R., & Maat, L. (2005). Web-based database for facial expression analysis. 2005 IEEE International Conference on Multimedia and Expo (pp. 317-321). https://doi.org/10.1109/ICME.2005.1521424
    109. Pathak, R., & Singh, Y. (2020). Real time baby facial expression recognition using deep learning and IoT edge computing. 2020 5th International Conference on Computing, Communication and Security (ICCCS) (pp. 1-6). https://doi.org/10.1109/ICCCS49678.2020.9277428
    110. Patil, V. K., Pawar, V. R., Kulkarni, S. P., Mehta, T. A., & Kharea, N. R. (2023). Real time emotion recognition with AD8232 ECG sensor for classwise performance evaluation of machine learning methods. International Journal of Engineering, Transactions B: Applications, 36(6), 1040-1047. https://doi.org/10.5829/ije.2023.36.06c.02
    111. Picard, R. W. (1997). Affective computing. MIT Press.
    112. Pinto, L. V. L., Alves, A. V. N., Medeiros, A. M., Costa, S. W. D. S., Pires, Y. P., Costa, F. A. R., & Seruffo, M. C. D. R. (2023). A systematic review of facial expression detection methods. IEEE Access, 11, 61881-61891. https://doi.org/10.1109/ACCESS.2023.3287090
    113. Pranav, E., Kamal, S., Chandran, C. S., & Supriya, M. H. (2020). Facial emotion recognition using deep convolutional neural network. 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS) (pp. 317-320). https://doi.org/10.1109/ICACCS48705.2020.9074302
    114. Pu, L., & Zhu, L. (2021). Differential residual learning for facial expression recognition. Proceedings of the 2021 5th International Conference on Machine Learning and Soft Computing (pp. 103-108). https://doi.org/10.1145/3453800.3453819
    115. Rajan, S., Chenniappan, P., Devaraj, S., & Madian, N. (2020). Novel deep learning model for facial expression recognition based on maximum boosted CNN and LSTM. IET Image Processing, 14(7), 1227-1232. https://doi.org/10.1049/iet-ipr.2019.1188
    116. Ramirez Rios, F. E., & Reyes Duke, A. M. (2023). Building of a convolutional neuronal network for the prediction of mood states through face recognition based on object detection with YOLOV8 and Python. 2023 IEEE International Conference on Machine Learning and Applied Network Technologies (ICMLANT) (pp. 1-6). https://doi.org/10.1109/ICMLANT59547.2023.10372862
    117. Rastgoo, M. N., Nakisa, B., Maire, F., Rakotonirainy, A., & Chandran, V. (2019). Automatic driver stress level classification using multimodal deep learning. Expert Systems with Applications, 138, 112793. https://doi.org/10.1016/j.eswa.2019.07.010
    118. Rathour, N., Khanam, Z., Gehlot, A., Singh, R., Rashid, M., Alghamdi, A. S., & Alshamrani, S. S. (2021). Real-time facial emotion recognition framework for employees of organizations using Raspberry-Pi. Applied Sciences, 11(22), 10540. https://doi.org/10.3390/app112210540
    119. Riaz, M. N., Shen, Y., Sohail, M., & Guo, M. (2020). eXnet: An efficient approach for emotion recognition in the wild. Sensors, 20(4), 1087. https://doi.org/10.3390/s20041087
    120. Ruan, X., Palansuriya, C., & Constantin, A. (2022). Real-time feedback based on emotion recognition for improving children's metacognitive monitoring skill. Proceedings of the 21st Annual ACM Interaction Design and Children Conference (pp. 672-675). https://doi.org/10.1145/3501712.3538831
    121. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533-536. https://doi.org/10.1038/323533a0
    122. Russell, J. A. (1991). Culture and the categorization of emotions. Psychological Bulletin, 110(3), 426-450. https://doi.org/10.1037/0033-2909.110.3.426
    123. Salman, A., & Busso, C. (2022). Privacy preserving personalization for video facial expression recognition using federated learning. Proceedings of the 2022 International Conference on Multimodal Interaction (pp. 495-503). https://doi.org/10.1145/3536221.3556614
    124. Samadiani, N., Huang, G., Cai, B., Luo, J., Chi, C.-H., Xiang, Y., & He, J. (2019). A review on automatic facial expression recognition systems assisted by multimodal sensor data. Sensors, 19(8), 1863. https://doi.org/10.3390/s19081863
    125. Samal, A., & Iyengar, P. A. (1992). Automatic recognition and analysis of human faces and facial expressions: A survey. Pattern Recognition, 25(1), 65-77. https://doi.org/10.1016/0031-3203(92)90007-6
    126. Sathya, R., Manivannan, R., & Vaidehi, K. (2022). Vision-based personal face emotional recognition approach using machine learning and tree-based classifier. In [Book Title] (pp. 561-573). Springer. https://doi.org/10.1007/978-981-16-6723-7_42
    127. Savchenko, A. V. (2022). Video-based frame-level facial analysis of affective behavior on mobile devices using EfficientNets. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 2358–2365). https://doi.org/10.1109/CVPRW56347.2022.00263
    128. Savchenko, A. V., Savchenko, L. V., & Makarov, I. (2022). Classifying emotions and engagement in online learning based on a single facial expression recognition neural network. IEEE Transactions on Affective Computing, 13(4), 2132–2143. https://doi.org/10.1109/TAFFC.2022.3188390
    129. Sengupta, A., Ye, Y., Wang, R., Liu, C., & Roy, K. (2019). Going deeper in spiking neural networks: VGG and residual architectures. Frontiers in Neuroscience, 13, 95. https://doi.org/10.3389/fnins.2019.00095
    130. Shetty, C., Khan, A., Singh, T., & Kharatmol, K. (2021). Movie review prediction system by real time analysis of facial expression. 2021 6th International Conference on Communication and Electronics Systems (ICCES) (pp. 1109–1113). https://doi.org/10.1109/ICCES51350.2021.9489171
    131. Shi, M., Xu, L., & Chen, X. (2020). A novel facial expression intelligent recognition method using improved convolutional neural network. IEEE Access, 8, 57606-57614. https://doi.org/10.1109/ACCESS.2020.2982286
    132. Shit, S., Rana, A., Das, D. K., & Ray, D. N. (2023). Real-time emotion recognition using end-to-end attention-based fusion network. Journal of Electronic Imaging, 32(1), 013050. https://doi.org/10.1117/1.JEI.32.1.013050
    133. Siam, A. I., Soliman, N. F., Algarni, A. D., Abd El-Samie, F. E., & Sedik, A. (2022). Deploying machine learning techniques for human emotion detection. Computational Intelligence and Neuroscience, 2022, 8032673. https://doi.org/10.1155/2022/8032673
    134. Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annual Review of Psychology, 70(1), 747-770. https://doi.org/10.1146/annurev-psych-010418-102803
    135. Siddiqui, N., Reither, T., Dave, R., Black, D., Bauer, T., & Hanson, M. (2022). A robust framework for deep learning approaches to facial emotion recognition and evaluation. 2022 International Conference on Advanced Computing and Machine Learning (CACML) (pp. 68-73). https://doi.org/10.1109/CACML55074.2022.00020
    136. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv. https://doi.org/10.48550/arXiv.1409.1556
    137. Singh, R., Saurav, S., Kumar, T., Saini, R., Vohra, A., & Singh, S. (2023). Facial expression recognition in videos using hybrid CNN & ConvLSTM. International Journal of Information Technology, 15(4), 1819-1830. https://doi.org/10.1007/s41870-023-01183-0
    138. Singh, R., Singh, V., Verma, P., Rao, G. V. E., & Bakthula, R. (2023). Real-time mood-based music auto-play system from facial expressions. In [Book Title] (pp. 363-373). Springer. https://doi.org/10.1007/978-981-99-3734-9_30
    139. Srinivas, P., Khamar, S. N., Borusu, N., Vuyyuru, H., & Raghavendra, K. M. G. (2023). Identification of facial emotions in hitech modern era. 2023 2nd International Conference on Edge Computing and Applications (ICECAA) (pp. 1202-1208). https://doi.org/10.1109/ICECAA58104.2023.10212285
    140. Taini, M., Zhao, G., Li, S. Z., & Pietikainen, M. (2008). Facial expression recognition from near-infrared video sequences. 2008 19th International Conference on Pattern Recognition (pp. 1-4). https://doi.org/10.1109/ICPR.2008.4761697
    141. Talaat, F. M. (2023). Real-time facial emotion recognition system among children with autism based on deep learning and IoT. Neural Computing and Applications, 35(17), 12717-12728. https://doi.org/10.1007/s00521-023-08372-9
    142. Thomas, M., & S., S. H. (2022). ANN based facial emotion detection and music selection. 2022 International Interdisciplinary Humanitarian Conference for Sustainability (IIHC) (pp. 927-931). https://doi.org/10.1109/IIHC55949.2022.10060593
    143. Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001). Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(2), 97-115. https://doi.org/10.1109/34.908962
    144. Turcian, D., & Stoicu-Tivadar, V. (2023). Real-time detection of emotions based on facial expression for mental health. Studies in Health Technology and Informatics, 309, 272-276. https://doi.org/10.3233/SHTI230795
    145. Van Eck, N. J., & Waltman, L. (2010). Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics, 84(2), 523-538. https://doi.org/10.1007/s11192-009-0146-3
    146. Vedaldi, A., & Zisserman, A. (2016). VGG convolutional neural networks practical. Department of Engineering Science, University of Oxford.
    147. Vijayeeta, P., & Pattnayak, P. (2022). A deep learning approach for emotion based music player. 2022 OITS International Conference on Information Technology (OCIT) (pp. 278-282). https://doi.org/10.1109/OCIT56763.2022.00060
    148. Villaroya, S. M., Gamboa-Montero, J. J., Bernardino, A., Maroto-Gomez, M., Castillo, J. C., & Salichs, M. A. (2022). Real-time engagement detection from facial features. 2022 IEEE International Conference on Development and Learning (ICDL) (pp. 231-237). https://doi.org/10.1109/ICDL53763.2022.9962228
    149. Wahab, M. N. A., Nazir, A., Ren, A. T. Z., Noor, M. H. M., Akbar, M. F., & Mohamed, A. S. A. (2021). EfficientNet-Lite and hybrid CNN-KNN implementation for facial expression recognition on Raspberry Pi. IEEE Access, 9, 134065–134080. https://doi.org/10.1109/ACCESS.2021.3113337
    150. Wahab, M. N. A., Nazir, A., Ren, A. T. Z., Noor, M. H. M., Akbar, M. F., & Mohamed, A. S. A. (2021). EfficientNet-Lite and hybrid CNN-KNN implementation for facial expression recognition on Raspberry Pi. IEEE Access, 9, 134065-134080. https://doi.org/10.1109/ACCESS.2021.3113337
    151. Waltman, L., Van Eck, N. J., & Noyons, E. C. M. (2010). A unified approach to mapping and clustering of bibliometric networks. Journal of Informetrics, 4(4), 629-635. https://doi.org/10.1016/j.joi.2010.07.002
    152. Wang, H., Tobón V., D. P., Hossain, M. S., & Saddik, A. E. (2021). Deep learning (DL)-enabled system for emotional big data. IEEE Access, 9, 116073-116082. https://doi.org/10.1109/ACCESS.2021.3103501
    153. Webb, N., Ruiz-Garcia, A., Elshaw, M., & Palade, V. (2020). Emotion recognition from face images in an unconstrained environment for usage on social robots. 2020 International Joint Conference on Neural Networks (IJCNN) (pp. 1-8). https://doi.org/10.1109/IJCNN48605.2020.9207494
    154. Werner, P., Lopez-Martinez, D., Walter, S., Al-Hamadi, A., Gruss, S., & Picard, R. W. (2022). Automatic recognition methods supporting pain assessment: A survey. IEEE Transactions on Affective Computing, 13(1), 530-552. https://doi.org/10.1109/TAFFC.2019.2946774
    155. Winyangkun, T., Vanitchanant, N., Chouvatut, V., & Panyangam, B. (2023). Real-time detection and classification of facial emotions. 2023 15th International Conference on Knowledge and Smart Technology (KST) (pp. 1-6). https://doi.org/10.1109/KST57286.2023.10086866
    156. Woodward, K., Kanjo, E., & Tsanas, A. (2023). Combining deep learning with signal-image encoding for multi-modal mental wellbeing classification. ACM Transactions on Computing for Healthcare, 4(3). https://doi.org/10.1145/3631618
    157. Xu, Y., Wei, L., & Wang, A. (2022). A study on the evaluation of English classroom learning status based on expression recognition technology. 2022 International Symposium on Advances in Informatics, Electronics and Education (ISAIEE) (pp. 439-442). https://doi.org/10.1109/ISAIEE57420.2022.00097
    158. Yang, B., Zhang, Q., & Liu, Z. (2022). ICANet: A method of short video emotion recognition driven by multimodal data. 2022 2nd International Conference on Networking Systems of AI (INSAI) (pp. 22–25). https://doi.org/10.1109/INSAI56792.2022.00014
    159. Zarif, N. E., Montazeri, L., Leduc-Primeau, F., & Sawan, M. (2021). Mobile-optimized facial expression recognition techniques. IEEE Access, 9, 101172-101185. https://doi.org/10.1109/ACCESS.2021.3095844
    160. Zhang, S., Liu, R., & Sawada, H. (2023). Deep learning-based system for real-time face tracking and expression recognition. 2023 IEEE International Conference on Mechatronics and Automation (ICMA) (pp. 1598-1603). https://doi.org/10.1109/ICMA57826.2023.10215584
    161. Zhang, Z., Fort, J. M., & Giménez Mateu, L. (2023). Facial expression recognition in virtual reality environments: Challenges and opportunities. Frontiers in Psychology, 14, 1280136. https://doi.org/10.3389/fpsyg.2023.1280136
    162. Zhang, Z., Lin, P., Ma, S., & Xu, T. (2022). An improved Yolov5s algorithm for emotion detection. 2022 5th International Conference on Pattern Recognition and Artificial Intelligence (PRAI) (pp. 1002-1006). https://doi.org/10.1109/PRAI55851.2022.9904113
    163. Zhao, G., Yang, H., & Yu, M. (2020). Expression recognition method based on a lightweight convolutional neural network. IEEE Access, 8, 38528-38537. https://doi.org/10.1109/ACCESS.2020.2964752
    164. Zhao, Y., & Zeng, J. (2020). Library intelligent book recommendation system using facial expression recognition. 2020 9th International Congress on Advanced Applied Informatics (IIAI-AAI) (pp. 55-58). https://doi.org/10.1109/IIAI-AAI50415.2020.00021
    165. Zhou, N., Liang, R., & Shi, W. (2021). A lightweight convolutional neural network for real-time facial expression detection. IEEE Access, 9, 5573-5584. https://doi.org/10.1109/ACCESS.2020.3046715
Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Copyright (c) 2025 The Authors

How to cite

Qian, C., Marques, J. A. L., & Alexandria, A. R. de. (2025). Real-time emotion recognition based on facial expressions using Artificial Intelligence techniques: A review and future directions. Multidisciplinary Reviews, 8(10), 2025328. https://doi.org/10.31893/multirev.2025328
Crossref
Scopus
Google Scholar
Europe PMC
  • Article viewed - 198
  • PDF downloaded - 54