Multimodal humor analysis

Teaching AI to get jokes. We built UR-FUNNY, the first humor dataset with text, vision and audio together, and models that use context and outside knowledge to recognize humor.

Humor lives in words, voice and face together.
1,866TED videos in the UR-FUNNY dataset
8,257punchlines with their context, plus as many non-humorous excerpts
AAAI 2021Humor Knowledge Enriched Transformer

UR-FUNNY and humor-aware transformers

UR-FUNNY pairs each punchline with the setup before it. Building on it, M-BERT injected multimodal information into pretrained language models (ACL 2020), and the Humor Knowledge Enriched Transformer added external humor knowledge (AAAI 2021).

  • UR-FUNNY: A Multimodal Language Dataset for Understanding Humor M. K. Hasan, W. Rahman, A. Zadeh, J. Zhong, M. I. Tanveer, L.-P. Morency, and E. Hoque Empirical Methods in Natural Language Processing (EMNLP 2019), Hong Kong, 2019 UR-FUNNY: a multimodal dataset of words, voice and gestures from TED talks for teaching AI to understand humor.
  • Integrating Multimodal Information in Large Pretrained Transformers W. Rahman, M. K. Hasan, S. Lee, A. Zadeh, C. Mao, L.-P. Morency, and E. Hoque Association for Computational Linguistics (ACL 2020), 2020 Lets BERT-style language models take in voice and facial cues, not only words, improving sentiment and behavior understanding.
  • Humor Knowledge Enriched Transformer for Understanding Multimodal Humor M. K. Hasan, S. Lee, W. Rahman, A. Zadeh, R. Mihalcea, L.-P. Morency, and E. Hoque AAAI Conference on Artificial Intelligence (AAAI 2021), 2021 Teaching AI to get jokes: a transformer that combines words, voice and facial cues with external humor knowledge to recognize humor in video.

More

← All research