Teaching language models to read faces and voices

Large language models read words, but people communicate with their faces and voices too. Since 2019 we have built ways for language models to take in nonverbal behavior, from adding vision and sound inside BERT to describing expressions in plain text, and more recently foundation models and alignment methods that make AI understand and behave more like people in conversation.

TextMI turns facial expressions and tone of voice into words, so an ordinary pretrained language model can use them.
1,181citations to MAG-BERT (ACL 2020), our most-cited paper (Google Scholar, November 2026)
Human-levelmultimodal sentiment analysis on CMU-MOSI, a first, with MAG-XLNet
7Bparameters in OmniSapiens, a foundation model for social behavior (ICML 2026)

2019 to 2021

Adding vision and sound to BERT

Pretrained language models such as BERT and XLNet had no way to take in a speaker's face or voice. Our Multimodal Adaptation Gate (MAG) adds them during fine-tuning by shifting the model's internal representation of each word according to the visual and acoustic cues around it. MAG-XLNet reached human-level performance on multimodal sentiment analysis for the first time. The same team built UR-FUNNY and a humor-aware transformer to teach these models to get jokes.

  • Integrating Multimodal Information in Large Pretrained Transformers W. Rahman, M. K. Hasan, S. Lee, A. Zadeh, C. Mao, L.-P. Morency, and E. Hoque Association for Computational Linguistics (ACL 2020), 2020 Lets BERT-style language models take in voice and facial cues, not only words, improving sentiment and behavior understanding.
  • UR-FUNNY: A Multimodal Language Dataset for Understanding Humor M. K. Hasan, W. Rahman, A. Zadeh, J. Zhong, M. I. Tanveer, L.-P. Morency, and E. Hoque Empirical Methods in Natural Language Processing (EMNLP 2019), Hong Kong, 2019 UR-FUNNY: a multimodal dataset of words, voice and gestures from TED talks for teaching AI to understand humor.
  • Humor Knowledge Enriched Transformer for Understanding Multimodal Humor M. K. Hasan, S. Lee, W. Rahman, A. Zadeh, R. Mihalcea, L.-P. Morency, and E. Hoque AAAI Conference on Artificial Intelligence (AAAI 2021), 2021 Teaching AI to get jokes: a transformer that combines words, voice and facial cues with external humor knowledge to recognize humor in video.

2023

Describing expressions in words

TextMI takes a simpler route: it converts facial expressions and tone of voice into short text descriptions and appends them to what was said, so an unmodified pretrained language model can use them. It beat existing methods on multimodal sarcasm detection and came close to the state of the art on sentiment and humor, a strong, simple baseline for settings with little data.

2025 to 2026

Foundation models and human-like AI

OmniSapiens is a 7-billion-parameter foundation model for understanding social behavior across many tasks and modalities. HAL turns human-likeness, a quality that is hard to define, into an interpretable score built from conversational traits and uses it to align language models; in Chatbot Arena-style evaluations, people more often judged the aligned model human-like. VisualActBench asks whether vision-language models can work out what to do from a scene alone, the way people do.

  • OmniSapiens: A Foundation Model for Social Behavior Processing via Heterogeneity-Aware Relative Policy Optimization K. Ong, S. Boughorbel, L. Xiao, C. Ekbote, W. Dai, A. Qu, J. Wu, R. Mao, E. Hoque, E. Cambria, G. Mengaldo, and P. P. Liang International Conference on Machine Learning (ICML 2026), Seoul, 2026 A 7B-parameter foundation model for understanding human social behavior across many tasks and modalities, trained with a new reinforcement-learning method that balances uneven behavioral data.
  • HAL: Inducing Human-likeness in LLMs with Alignment M. Hasan, J. Zhao, and E. Hoque arXiv:2601.02813, 2026 Human-likeness is hard to define, measure or train for. HAL makes it measurable and uses it as an alignment target, so language models can become more human-like in conversation directly rather than only through scale.
  • VisualActBench: Can VLMs See and Act like a Human? D. Zhang, P. Liu, X. Zhou, Y. Ge, G. Lan, J. Bi, C. Brinton, E. Hoque, and J. Luo arXiv:2512.09907, 2025 A benchmark that tests whether vision-language models can work out what to do from a scene alone, without being told in words, the way people do.

People

This line of work was led by PhD students Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee and Masum Hasan, with Amir Zadeh and Louis-Philippe Morency (Carnegie Mellon), Rada Mihalcea (Michigan) and, on OmniSapiens, Paul Pu Liang (MIT).

More

← All research