Publications
160 papers, patents and theses, 2005 to 2026. Students are typically first authors. Also on Google Scholar, and in full in the CV.
2026
-
HAL: Inducing Human-likeness in LLMs with AlignmentPaper (arXiv)PDF
In brief
Human-likeness is hard to define, measure or train for. HAL makes it measurable and uses it as an alignment target, so language models can become more human-like in conversation directly rather than only through scale.
-
Replicating Human Motivated Reasoning Studies with LLMsPaper (arXiv)PDF
In brief
People often reason toward the conclusion they want. We replicated classic motivated-reasoning experiments with base language models to test whether they show the same bias, which matters for using LLMs to simulate survey respondents.
-
Can LLMs Emulate Human Belief Dynamics?Paper (arXiv)PDF
In brief
We replicated an established study of how beliefs form and change in social networks with 12 language models. The answer is a clear no: the models fail in systematic ways, a caution for using LLMs as stand-ins for people.
-
NeuroSift for Task-Aware Quality Assurance of Multimedia Data in Remote Parkinson Disease Assessment: Machine Learning Model Development and Validation StudyArticle (open access)
In brief
Videos recorded at home are often too dark, off-frame or incomplete for AI to analyze. NeuroSift checks each recording for the problems that matter for that specific task, so unreliable data can be caught before it skews Parkinson's screening.
-
Digital Harf: A Clinically Integrated Multimodal AI System for Pervasive Arabic Speech and Language TherapyPaper (arXiv)
In brief
Autistic children in Arabic-speaking countries face a shortage of speech therapists and almost no culturally grounded digital materials. Digital Harf is a multimodal AI system for Arabic speech and language therapy, built to work alongside clinicians.
-
Scalable AI-Based Clinical Communication Training and Automated AssessmentPaper (arXiv)PDF
In brief
Building on our randomized trial with SOPHIE, this work extends AI-patient practice and automated feedback to a broader range of clinical conversations, so clinicians can practice repeatedly at scale.
-
The Effects of Generative AI Virtual Patient in Serious Illness Communication Skills: Randomized Controlled TrialPDFDOI
In brief
A randomized controlled trial of SOPHIE, our generative-AI virtual patient for practicing serious-illness conversations. Learners who trained with SOPHIE improved on the core skills these conversations demand: empathizing, being explicit and empowering the patient.
-
Validation of Remote Multimodal AI Screening for Parkinson Disease Across Diverse Settings
In brief
Can a few minutes in front of a webcam screen for Parkinson's? This study validates our multimodal PARK screening (speech, facial expression, finger tapping) across diverse settings, a step from lab prototype toward real-world use.
-
AI Can Enhance Creativity in Social Networks
In brief
We built SocialMuse, an AI that recommends whom to draw inspiration from in an online idea-generation network. Networks guided by the AI outperformed AI-agnostic networks on several creativity measures and grew more decentralized.
-
PULSAR: Graph-Based Positive Unlabeled Learning with Multi-Stream Adaptive Convolutions for Parkinson's Disease RecognitionPDF (preprint)DOI
In brief
PULSAR screens for Parkinson's from webcam videos of a finger-tapping task, using graph neural networks over hand keypoints. It is aimed at places with little access to neurologists, trained on 382 participants.
-
AI Virtual Standardized Patient Training for Perinatal Dental Communication: A Pilot StudyDOI
In brief
A pilot of AI virtual standardized patients for dental students learning to counsel pregnant patients. It extends our clinical-communication training approach beyond medicine.
-
Harf-Speech: A Clinically Aligned Framework for Arabic Phoneme-Level Speech Assessment
In brief
Validated tools for scoring Arabic pronunciation barely exist. Harf-Speech scores pronunciation phoneme by phoneme on a clinical scale and agrees with certified speech-language pathologists (Pearson r = 0.79), a building block for scalable Arabic speech therapy.
-
Design and Evaluation of a Culturally Adapted Multimodal Virtual Agent for PTSD ScreeningPDF (preprint)
In brief
PTSD is widely underreported among military personnel. Molhim is a culturally adapted, LLM-driven virtual avatar that conducts structured screening conversations and administers the PCL-5 checklist in a military healthcare setting.
-
Multimodal Remote Digital Phenotyping for Detecting and Tracking Early Parkinsonian Change in LRRK2 CarriersPDFPMCDOI
In brief
People who carry LRRK2 gene variants are at high risk of Parkinson's. From remote webcam tasks by 829 participants, including 158 LRRK2 carriers, our model told carriers without symptoms apart from controls with 92.9% accuracy, and a continuous score tracked decline in two carriers who later developed Parkinson's.
-
Benchmarking Video Foundation Models for Remote Parkinson's Disease ScreeningPDF (preprint)arXiv
In brief
The largest study yet of video foundation models for remote Parkinson's screening: 1,888 participants, 32,847 videos, 16 clinical tasks, seven models. Which model works best depends strongly on the task, which matters for anyone building screening tools.
-
OmniSapiens: A Foundation Model for Social Behavior Processing via Heterogeneity-Aware Relative Policy Optimization
In brief
A 7B-parameter foundation model for understanding human social behavior across many tasks and modalities, trained with a new reinforcement-learning method that balances uneven behavioral data.
2025
-
AI Standardized Patient Improves Human Conversations in Advanced Cancer CarePaper (arXiv)PDF
In brief
Practicing with SOPHIE, an AI standardized patient, improved how clinicians handled serious-illness conversations in advanced cancer care, offering a cheaper and more scalable alternative to actors.
-
VisualActBench: Can VLMs See and Act like a Human?Paper (arXiv)PDF
In brief
A benchmark that tests whether vision-language models can work out what to do from a scene alone, without being told in words, the way people do.
-
How LLMs Fail to Support Fact-CheckingPaper (arXiv)PDF
In brief
We tested ChatGPT, Gemini and Claude at countering political misinformation and found consistent gaps, especially in grounding their answers in reliable sources.
-
Personalized Large Language Models Can Increase the Belief Accuracy of Social NetworksPaper (arXiv)PDF
In brief
In networked experiments, personalized LLM responses made not only individuals but whole social networks hold more accurate beliefs on contested issues.
-
New Developments in Artificial Intelligence to Enhance Communication Research and Skills Training
In brief
How AI can move healthcare communication research beyond patient surveys and slow human coding of recorded visits, and how it can help clinicians practice and improve.
-
Designing Nudges, Not Shortcuts, in the Age of AIArticle (IEEE)
In brief
An essay on AI in education: rather than tools that hand students the answers, pervasive computing should design AI that nudges people to think and learn for themselves.
-
Exploring the Role of Randomization on Belief Rigidity in Online Social NetworksPDF (preprint)
In brief
Social media can harden beliefs through echo chambers. Online experiments test whether adding randomness to who people are connected with loosens belief rigidity, a simple network-level intervention.
-
'Poker with Play Money': Exploring Psychotherapist Training with Virtual PatientsPDFDOI
In brief
What happens when psychotherapists train with virtual patients? This CSCW study examines how practicing with AI patients fits into therapist training.
-
Estimating Autism Prevalence Among Children in the Kingdom of Saudi Arabia: A Comprehensive Multisource Benchmarking Approach
In brief
Combines multiple data sources to estimate how common autism is among children in Saudi Arabia, filling a gap in national planning data for diagnosis and services.
-
AI-Enabled Parkinson's Disease Screening Using Smile VideosPreprint (arXiv)DOI
In brief
A smile recorded on a webcam can flag Parkinson's. Using facial-expression videos from 1,059 people recorded at home and in clinics, the model detects reduced facial expressivity (hypomimia) and was tested on external videos it never saw in training. Published in NEJM AI.
-
A Novel Fusion Architecture for PD Detection Using Semi-Supervised Speech EmbeddingsPDFarXiv
In brief
Screening for Parkinson's from one spoken sentence recorded at home. A fusion of modern speech embeddings, trained on 1,306 people (392 with PD), beats traditional acoustic features and holds up across demographics.
-
Don't Follow the Leader: Independent Thinkers Create Scientific InnovationIn brief
In science, following the most visible leaders tends to produce less novel work. Independent thinkers are more likely to create scientific innovation.
-
Hi5: Synthetic Data for Inclusive, Robust, Hand Pose EstimationPDF (preprint)
In brief
Hand-pose models underrepresent many skin tones and natural gestures. Hi5 is a synthetic dataset of 583,000 diverse, annotated hand images; models trained only on it match those trained on human-labeled data and are more robust.
-
Beyond Accuracy: Enhancing Parkinson's Diagnosis with Uncertainty Quantification of Machine Learning ModelsFull text (PMC)
In brief
A Parkinson's prediction is only useful if you know when to trust it. This work adds uncertainty quantification to PD screening models so low-confidence cases can be flagged for clinicians. Best Paper nominee.
-
Personalizing LLM Responses to Combat Political Misinformation
In brief
Can a chatbot correct political misinformation better if it tailors its answer to the reader? This study tests personalized LLM responses as a counter to misinformation.
-
Accessible, At-Home Detection of Parkinson's Disease via Multi-task Video AnalysisPDFarXiv
In brief
Parkinson's affects movement, face and voice, so we combine all three. UFNet fuses webcam videos of finger tapping, facial expression and speech from 845 people and weighs each task by its uncertainty, improving at-home detection (AAAI 2025).
2024
-
SOPHIE: Testing a Virtual, Interactive, AI-Augmented End-of-Life Communication Training ToolAbstract
In brief
Conference presentation of SOPHIE as an alternative to traditional communication workshops for end-of-life conversations, and the design of the study testing it.
-
Managing Emotional Dialogue for a Virtual Cancer Patient: A Schema-Guided ApproachEarlier version (arXiv)
In brief
How do you make a virtual cancer patient respond with realistic emotion? A schema-guided dialogue manager lets SOPHIE handle open-ended, emotionally charged conversations with doctors in training.
-
A User-Centered Framework to Empower People with Parkinson's Disease
In brief
Designed with people with Parkinson's, this framework turns at-home video screening into feedback and resources users can act on.
-
A Survey on Open Information Extraction from Rule-based Model to Large Language Model
In brief
A survey of open information extraction, from rule-based systems to large language models.
-
Getting on the Right Foot: Using Observational and Quantitative Methods to Evaluate Movement Disorders
In brief
Combines clinical observation and quantitative video analysis to evaluate gait in movement disorders.
2023
-
TextMI: Textualize Multimodal Information for Integrating Non-Verbal Cues in Pre-Trained Language ModelsPaper (arXiv)PDF
In brief
Instead of changing a language model's architecture, TextMI describes facial expressions and tone of voice in words, so an ordinary pretrained language model can use them to understand sentiment and humor.
-
Using AI to Measure Parkinson's Disease Severity at HomePDFArticle
In brief
Tap your fingers in front of a webcam and AI rates Parkinson's severity. On 250 people worldwide, the model outperformed a certified rater (error 0.59 vs 0.79) and approached expert neurologists.
-
SEER: Sustainable E-commerce with Environmental-impact RatingPDF (preprint)Article
In brief
Shoppers say they want sustainable products but rarely buy them. SEER adds explainable environmental-impact ratings to online stores; in a 98-person experiment, users with SEER made more sustainable choices.
-
Auto-Gait: Automatic Ataxia Risk Assessment with Computer Vision from Gait Task Videos
In brief
Webcam gait videos analyzed with computer vision to estimate risk of spinocerebellar ataxia, a step toward remote screening for rare movement disorders.
-
Validating a Virtual Human and Automated Feedback System for Training Doctor-Patient Communication Skills
In brief
SOPHIE, a virtual patient with automated feedback, validated with 30 participants: those who practiced with it outperformed controls in overall communication, empathy and empowering the patient.
2022
-
Novel Idea Generation in Social Networks Is Optimized by Exposure to a 'Goldilocks' Level of Idea-VariabilityPDFArticle
In brief
Exposure to too little or too much variety in others' ideas both hurt creativity. A 'Goldilocks' level of idea variability in your social network produces the most novel ideas.
-
A Mental Trespass? Unveiling Truth, Exposing Thoughts and Threatening Civil Liberties with Non-Invasive AI Lie DetectionPDFArticle
In brief
AI lie detection from face, body and voice is coming fast. We argue it threatens civil liberties, and a survey of 129 people shows consent and accuracy drive whether the public accepts it.
-
Discourse Behavior of Older Adults Interacting with a Dialogue Agent Competent in Multiple TopicsPDF (preprint)Article
In brief
How older adults talk with a conversational agent that can discuss many topics, informing companion technology for healthy aging.
-
KnowUREnvironment: An Automated Knowledge Graph for Climate Change and Environmental Issues
In brief
An automated knowledge graph that organizes research and information on climate change and environmental issues.
-
NADBenchmarks: A Compilation of Benchmark Datasets for Machine Learning Tasks Related to Natural Disasters
In brief
A curated catalog of benchmark datasets for machine learning on natural disasters, organized by the disaster-management cycle.
-
Assistive Video Filters for People with Parkinson's Disease to Remove Tremors and Adjust Voice
In brief
Real-time video filters that smooth visible tremors and adjust voice, so people with Parkinson's can feel more comfortable on video calls. Best Paper nominee led by undergraduates.
-
Systematic Feature Isolation for Bias Research Using Deepfakes
In brief
Uses deepfakes to change one feature of a person at a time, such as appearance, so researchers can isolate the sources of bias in how people are judged. Best Paper nominee.
-
Suggestive Motivational Interviewing Agent for Remote Group Discussion
In brief
An AI agent that nudges remote group discussions toward more balanced, motivating conversation.
-
Auto-Gait: Automatic Ataxia Risk Assessment with Computer Vision on Gait Task Videos
In brief
Webcam gait videos analyzed with computer vision to estimate risk of spinocerebellar ataxia, a step toward remote screening for rare movement disorders.
-
A Week in the Life with Parkinson's Disease: A Holistic Overview from Four Digital Technologies -
Analyzing Gait Videos to Identify and Evaluate Spinocerebellar Ataxia Types 1 and 3 -
Longitudinal Evaluation of Hypomimia in Individuals with Parkinson's Disease Using a Video Analytics Tool
2021
-
Could You Become More Credible by Being White? Assessing Impact of Race on Credibility with DeepfakesPaper (arXiv)PDF
In brief
We used deepfakes to change a speaker's apparent race on video while keeping everything else the same, to test how race alone changes how credible people seem in video calls.
-
Exploring the Use of Real-Time Camera Filters on Embodiment and CreativityPaper (ACM)
In brief
Real-time camera filters change how we look on video calls. This study with the MIT Media Lab explored whether they also change how people feel and think, including how creative they are.
-
Detecting Parkinson's Disease from an Online Speech-Task: Observational StudyPDFArticle
In brief
Anyone, anywhere can record one sentence online to screen for Parkinson's. With 726 participants across the US and beyond, speech features alone reached 0.75 AUC.
-
DBATES: Distinguishing Benefits of Audio, Textual, and Facial Expression Features in Competitive Debate Speeches
In brief
A dataset and analysis of competitive debate speeches showing how much audio, text and facial expression each contribute to judging a speaker.
-
Cues to Gender and Racial Identity Reduce Creativity in Diverse Social NetworksPDFArticle
In brief
In diverse online groups, cues to participants' gender and racial identity reduced creativity, a finding with direct implications for how collaborative platforms present people.
-
Analyzing Head Pose in Remotely-Collected Videos of People with Parkinson's Disease
In brief
Head movement in home-recorded videos carries signal about Parkinson's, supporting remote assessment without special hardware.
-
Video-Based Analyses of Parkinson's Disease Severity: A ReviewFull text (PMC)Article
In brief
A review of how video analysis is being used to measure Parkinson's severity, and what is needed for clinical use.
-
A Patient-Centered Digital Scribe for Automatic Medical DocumentationPDFArticle
In brief
A digital scribe that drafts medical documentation from doctor-patient conversations, so clinicians can look at patients instead of screens.
-
Novel Computational Linguistic Measures, Dialogue System and the Development of SOPHIE: Standardized Online Patient for Healthcare Interaction Education
In brief
Introduces SOPHIE, a virtual standardized patient for practicing end-of-life conversations, with computational measures of how well clinicians communicate.
-
Hitting Your MARQ: Multimodal ARgument Quality Assessment in Long Debate Video
In brief
Assessing the quality of arguments in long debate videos by combining what speakers say with how they say it.
-
Humor Knowledge Enriched Transformer for Understanding Multimodal Humor
In brief
Teaching AI to get jokes: a transformer that combines words, voice and facial cues with external humor knowledge to recognize humor in video.
2020
-
Immediate or Reflective? Effects of Real-Time Feedback on Group Discussions over VideochatPaper (arXiv)PDF
In brief
A video-chat system that analyzes a group discussion automatically and gives feedback either during the conversation or afterward. We compared how each kind of feedback changes discussions between people who disagree.
-
Upskilling the Future Workforce Using AI and Affective ComputingAbstract (ACM)
In brief
A keynote on using AI not to replace people but to amplify human abilities, especially the social and communication skills that set us apart.
-
Aging and Engaging: A Pilot Randomized Controlled Trial of an Online Conversational Skills Coach for Older Adults
In brief
A pilot randomized trial of an online conversation coach that helps older adults practice social engagement, aimed at reducing isolation.
-
Technology-Driven Alteration of Nonverbal Cues and Its Effects on NegotiationPDF (preprint)arXiv
In brief
Video tools and deepfakes can now change how we look and sound in real time. We examine what that means for negotiation, and the ethical questions it raises.
-
The Relationship Between Deteriorating Mental Health Conditions and Longitudinal Behavioral Changes in Google and YouTube Usage Among College Students in the United States During COVID-19: Observational StudyFull text (PMC)Article
In brief
During COVID-19, changes in college students' Google and YouTube use tracked worsening mental health, pointing to passive, privacy-aware early warning signs.
-
Creativity in Temporal Social Networks: How Divergent Thinking Is Impacted by One's Choice of Peers
In brief
Who you choose to draw ideas from shapes how creative you become. Online experiments trace how peer choices affect divergent thinking over time.
-
Leveraging Free-Hand Sketches for Potential Screening of PTSD
In brief
Simple free-hand drawings may help screen for PTSD, offering a low-cost tool where clinicians are scarce.
-
Are You Really Looking at Me? A Feature-Extraction Framework for Estimating Interpersonal Eye Gaze from Conventional Video
In brief
Estimates whether people are making eye contact using ordinary video, with no eye tracker needed.
-
Deep Phenotyping of Parkinson's DiseaseFull text (PMC)Article
In brief
A multi-site effort to characterize Parkinson's in depth with digital and clinical measures.
-
Using Artificial Intelligence to Analyze and Teach Communication in HealthcarePDFArticle
In brief
How AI can analyze and help teach communication between clinicians and patients.
-
Discovering Intimate Partner Violence from Web Search History
In brief
Web search histories can reveal signs of intimate partner violence, suggesting new ways to reach people who may not seek help.
-
A Virtual Conversational Agent for Teens with Autism Spectrum Disorder: Experimental Results and Design Lessons
In brief
A virtual agent that lets autistic teens practice conversation and get feedback, with experimental results and design lessons.
-
Integrating Multimodal Information in Large Pretrained Transformers
In brief
Lets BERT-style language models take in voice and facial cues, not only words, improving sentiment and behavior understanding.
-
Spatio-Temporal Attention and Magnification for Classification of Parkinson's Disease from Videos Collected via the Internet
In brief
Magnifying subtle motion in videos collected online helps AI classify Parkinson's.
-
Leveraging Shared and Divergent Facial Expression Behavior Between Genders in Deception Detection
In brief
Men and women show different facial cues when lying. Modeling those differences improves deception detection.
2019
-
Predicting TED Talk Ratings from Language and ProsodyPaper (arXiv)PDF
In brief
Using more than 2,200 TED talks and 5.5 million viewer ratings, we predicted how audiences rate a talk (inspiring, funny, persuasive) from what speakers say and how they say it.
-
A Causality-Guided Prediction of the TED Talk Ratings from the Speech-Transcripts Using Neural NetworksPaper (arXiv)PDF
In brief
Neural networks that predict TED talk ratings from transcripts, designed with a causal model so that predictions are not skewed by confounds such as a speaker's popularity.
-
The PARK Framework for Automated Analysis of Parkinson's Disease Characteristics
In brief
PARK: a web tool that guides people through standard Parkinson's tasks at home and analyzes the recorded videos, the foundation of our remote screening work.
-
UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
In brief
UR-FUNNY: a multimodal dataset of words, voice and gestures from TED talks for teaching AI to understand humor.
-
Facial Expression Based Imagination Index and a Transfer Learning Approach to Detect Deception
In brief
Facial expressions reveal when people are imagining rather than recalling, a signal useful for detecting deception.
-
What Computers Can Teach Us About Doctor-Patient Communication: Leveraging Gender Differences in Cancer Care
In brief
Computational analysis of cancer-care conversations reveals gender differences in doctor-patient communication.
-
Upskilling Together: How Peer-Interaction Influences Speaking-Skills Development Online
In brief
Speakers practicing in an online community improved faster when they interacted with peers.
-
Visual Cues for Disrespectful Conversation Analysis
In brief
Visual cues that signal disrespect in conversations, a step toward tools for healthier discussions.
-
LIWC Into the Eyes: Using Facial Features to Contextualize Linguistic Analysis in Multimodal Communication
In brief
Combines facial expressions with word-level language analysis to better interpret what people mean.
2018
-
It's Time to Do Something: Mitigating the Negative Impacts of Computing Through a Change to the Peer Review ProcessPaper (arXiv)PDF
In brief
A proposal, from the ACM Future of Computing Academy, that peer reviewers require computing papers to discuss the plausible negative impacts of their work, from privacy to employment, rather than assuming research is a net good.
-
Development of an Electromyographically Controlled Virtual Arm for Poststroke Motor RehabilitationAbstract
In brief
A virtual arm, controlled by muscle signals from the patient's own arm, for early rehabilitation after stroke when the arm is too weak to move much.
-
The Holographic Electromyographic Automated Rehabilitation Technology (HEART) ProjectAbstract
In brief
A mixed-reality version of the muscle-controlled virtual arm, shown as a hologram, for upper-limb rehabilitation in people with moderate to severe weakness after stroke.
-
CoCo: Collaboration Coach for Understanding Team Dynamics During Video Conferencing
In brief
CoCo coaches teams during video calls, showing how participation and dynamics affect group performance.
-
Automated Dyadic Data Recorder (ADDR) Framework and Analysis of Facial Cues in Deceptive Communication
In brief
A framework for recording and analyzing honest and deceptive conversations online, revealing the facial cues of lying. Covered by Newsweek and The Times.
-
Buildup of Speaking Skills in an Online Learning Community: A Network-Analytic ExplorationPDFArticle
In brief
Network analysis of an online public-speaking community shows how skills spread among peers.
-
Automated Analysis and Prediction of Job Interview Performance
In brief
AI that predicts how interviewers will rate a candidate from what they say and how they say it, and explains which behaviors matter.
-
The What, When, and Why of Facial Expressions: An Objective Analysis of Conversational Skills in Speed-Dating Videos
In brief
Objective analysis of facial expressions in speed-dating videos reveals what, when and why expressions matter in conversation.
-
Say CHEESE: Common Human Emotional Expression Set Encoder Analysis of Smiles in Honest and Deceptive Communication
In brief
Smiles differ subtly when people are lying. A smile-encoding model finds those differences in honest and deceptive conversations.
-
How Emotional Trajectories Affect Audience Perception in Public Speaking
In brief
Analyzing TED talks shows how the emotional arc of a talk shapes how audiences rate it.
-
Aging and Engaging: A Social Conversational Skills Training Program for Older Adults
In brief
A conversation-skills training program for older adults, with an AI coach that gives feedback on social engagement.
-
Understanding Social Interpersonal Interaction via Synchronization Templates of Facial Events
In brief
Shows how synchronized facial expressions between two people predict social outcomes such as negotiation results.
-
Analyzing the Impact of Gender on the Automation of Feedback for Public Speaking
2017
-
Social Skills Training with Virtual Assistant and Real-Time FeedbackPaper (ACM)
In brief
A virtual assistant that lets people practice conversations privately and gives real-time feedback on nonverbal cues such as eye contact and smiling.
-
Automatic Identification of Non-Meaningful Body-Movements and What It Reveals About HumansPaper (arXiv)PDF
In brief
From 84 public-speaking videos we extracted 314 body-movement patterns and learned to tell meaningful gestures from mannerisms such as pacing or shifting weight.
-
Aging and Engaging: The Development of an Automated Tool to Teach Social Engagement SkillsArticle
In brief
Early development of an automated tool that teaches social engagement skills to older adults.
-
A Semi-Automated, Collaborative, and Online Training Module for Communication Skills
In brief
Combines automated analysis with crowd feedback to help people practice communication skills online.
-
Automated Video Judgement on a Large-Sized Corpus Collected Online
In brief
Automated scoring of a large corpus of videos recorded online, with results comparable to human judges. Best Paper nominee.
-
Modeling Doctor-Patient Communication with Affective Text Analysis
In brief
Analyzing the emotional content of doctor-patient conversations to model communication in cancer care.
-
Managing Casual Spoken Dialogue Using Flexible Schemas, Pattern Transduction Trees, and Gist Clauses
In brief
A dialogue manager that keeps casual conversations on track, used in LISSA and later SOPHIE.
2016
-
Methods and Apparatus for Conversation CoachPatent
In brief
The US patent from my PhD on an automated conversation coach; Microsoft later built on the idea in PowerPoint's Speaker Coach.
-
The LISSA Virtual Human and ASD Teens: An Overview of Initial Experiments
In brief
Initial experiments with LISSA, a virtual human that helps autistic teens practice conversation.
-
ROC Comment: Automated Descriptive and Subjective Captioning of Behavioral Videos
In brief
Automatically generates descriptive and subjective comments on videos of people's behavior.
-
AutoManner: An Automated Interface for Making Public Speakers Aware of Their Mannerisms
In brief
Shows public speakers their own repetitive mannerisms, automatically found in their videos.
2015
-
Unsupervised Extraction of Human-Interpretable Nonverbal Behavioral Cues in a Public Speaking Scenario
In brief
Finds interpretable nonverbal cues in public-speaking videos without labels.
-
ROC Speak: Semi-Automated Personalized Feedback on Nonverbal Behavior from Recorded Videos
In brief
ROC Speak: record a talk in your browser and get automated plus crowd feedback on your nonverbal behavior. Highlighted by Microsoft Research.
-
Vowel Shapes: An Open-Source, Interactive Tool to Assist Singers with Learning Vowels
In brief
An open-source tool that visualizes vowel sounds to help singers learn them.
-
LISSA: Live Interactive Social Skill Assistance
In brief
LISSA, a live virtual assistant that gives real-time feedback on social skills during conversation.
-
Automated Prediction of Job Interview Performance: The Role of What You Say and How You Say It
In brief
Predicts job-interview ratings from both what candidates say and how they say it.
-
Role of Synchronized Facial Expressions to Predict Negotiation Outcome
In brief
Facial synchrony between negotiators predicts negotiation outcomes.
-
Rhema: A Real-Time In-Situ Intelligent Interface to Help People with Public Speaking
In brief
Rhema gives speakers real-time feedback through Google Glass while they talk, without distracting them.
2014
-
Rich Nonverbal Sensing to Enable New Possibilities in Social Skills Training
In brief
How rich sensing of nonverbal behavior opens new ways to train social skills.
-
A Google Glass App to Help the Blind with Small Talk -
M.I.D.A.S. Touch: Magnetic Interactive Device for Alternative Sight through Touch -
Processing Nonverbal Behavior in the Cloud
2013
-
Computers to Help with Conversations: Affective Framework to Enhance Human Nonverbal Skills
In brief
My PhD thesis: an affective computing framework that helps people improve their nonverbal conversation skills.
-
MACH: My Automated Conversation coacHPDFMIT News
In brief
MACH, a 3D virtual coach that reads your face and voice to help you practice job interviews. UbiComp Best Paper (2013) and 10-Year Impact Award (2023).
2012
-
Mood Meter: Counting Smiles in the Wild
In brief
MIT's Mood Meter counted smiles across campus with cameras, measuring a community's mood in real time.
-
Exploring Temporal Patterns towards Classifying Frustrated and Delighted SmilesPDFMIT News
In brief
People often smile when frustrated. By looking at how a smile unfolds over time, AI can tell frustrated from delighted smiles better than people can.
2011
-
Are You Friendly or Just Polite? Analysis of Smiles in Spontaneous Face-to-Face Interactions
In brief
Polite and amused smiles look similar but unfold differently. Timing analysis tells them apart.
-
Acted vs. Natural Frustration and Delight: Many People Smile in Natural Frustration
In brief
Many people smile when genuinely frustrated, which acted datasets miss. Best Paper nominee at IEEE FG.
-
Machine Learning for Affective Computing
2010
-
I See You: Towards Robust Recognition of Facial Expressions and Speech Prosody in Real Time
-
Design and Evaluation of Interactive, Customizable and Extensible Speech Enabled Games as Speech Therapy for Kids with Autism
2009
-
When Human Coders (and Machines) Disagree on the Meaning of Facial Affect in Spontaneous Videos
In brief
When human coders and machines disagree about facial expressions in spontaneous video, and what that means for emotion recognition.
-
Exploring Speech Therapy Games with Children on the Autism Spectrum
In brief
Speech-therapy games designed with children on the autism spectrum.
-
User-Centered Design of Technology for Just-In-Time, In-Situ Exploration of Facial Affect on the Autism Spectrum
2008
2007
2006
-
Robust Recognition of Emotion from Speech
In brief
Recognizing emotion from speech robustly. Best Paper nominee at IVA 2006.
-
Robust Recognition of Emotion in e-Learning Environment
2005
-
Embodied Conversational Agents: Multimodal Communication and Embodiment

















































































































































