Penn State Erie, The Behrend College · Spring 2004 · CENBD 481 Senior Design

VisionPSU

A human-interactive robot that recognized people by their face and voice, using a neural network.

  • Best Senior Design Award
  • IEEE Region 1 Student Design Contest, RIT
  • IEEE Region 2 Student Activities Conference
The VisionPSU robot: a metal frame with two camera eyes on a lit control box

In 2004, as a computer engineering senior at Penn State, I worked with four classmates to build a robot that could look at you, find your face, listen to your voice and tell you who you were. We trained a neural network on our own faces and voices, ran it on a 2.2 GHz Pentium 4, and paid for the most expensive part, an LCD screen that displayed your name, out of our own pockets. Two decades later I still work on the same question: how machines can understand people. This page keeps what we made, redrawn from our original website but in the same black, teal and red we picked back then.

Ehsan Hoque, 2026

[About]

VisionPSU Capstone Project title slide, with the five team members' faces detected in boxes

VisionPSU was a human-interactive robot with video and audio recognition. It was designed to tell apart a specific set of people, the five team members and our project mentors, from what it saw through its camera and heard through its microphone. When it recognized the person in front of it, it showed that person's name on an LCD mounted on the robot and responded with physical movement, following the face it had found.

The system was built from six modules that had to work together in real time: the robotics, a communication layer, a real-time control system, audio processing, image processing and a neural network. The five of us came from three majors: computer engineering, computer science and electrical engineering.

“The next evolutionary step would be to expand the interaction between humans and machines to a social interface.”From our final presentation, 2004
~90%
face-based identification accuracy
~80%
voice-based identification accuracy
~2 s
from seeing a person to naming them
10 fps
640 × 480 camera capture
$722
total build cost
2 wires
into the robot: one USB, one power

[Accomplishments]

  1. Best Senior Design Award, Penn State Erie, given by the local IEEE chapter and judged by engineers from local industry.
  2. Represented Penn State at the IEEE Region 2 Student Activities Conference 2004, Cleveland State University.
  3. Competed at the 4th Annual IEEE Region 1 Student Design Contest, hosted by the IEEE student chapter at the Rochester Institute of Technology, against teams from schools including Carnegie Mellon.

On the engineering side, we had face detection and recognition working in real time, voice recognition working in real time, a robot that moved in response to the person in front of it, and everything running through a single USB connection to one computer.

[How it worked]

Seeing

The camera captured 640 × 480 frames at 10 frames per second. We found faces with Intel's open-source computer vision library, OpenCV, then still new and experimental. Inside each face we located the eyes and nose by template matching: sliding small pictures of an eye, a nose and a mouth across the face and keeping the best match. The brightness profile across the eyes and nose became a short numeric signature for each person.

Hearing

Half a second of speech, about 10,000 samples, was windowed around the loudest part of the waveform, padded to 1,024 points and passed through a Fast Fourier Transform. The first 400 frequency values became the voice signature.

Deciding

Two feed-forward neural networks trained with back-propagation in MATLAB, one on face signatures and one on voice signatures, each returned how similar the input was to every known person. The whole loop, from camera to a name on the screen, took a little over two seconds.

Moving

The body was built mostly from parts we already had. Servos driven by pulse-width signals turned the head and camera eyes, a cold-cathode light showed the robot was powered, and only two pieces needed the machine shop: the plate that holds the eyes and the cover box.

[Results]

From a camera frame to a signature

  1. Face found in a camera frame1. Find the face
  2. Eye region located inside the face2. Locate the eyes
  3. Correlation map highlighting facial features3. Match feature templates
  4. Chart of normalized eye profiles from four images of one person4. Profile across the eyes

Four photos of the same person produced nearly the same eye profile, and different people produced different ones. Where two people's eye profiles looked alike, their nose profiles told them apart. Those patterns were what the neural network learned from.

[Members]

  • Mohammed E. HoqueComputer EngineeringFace detection and recognition; led image processing
  • Troy TancraitorElectrical EngineeringRobotics and hardware
  • Jonathan LobaughComputer Science and EngineeringNeural network, real-time control, integration
  • Joseph MallozziComputer ScienceCommunication layer
  • Lee SteenIn memoriam, 1982–2011Computer Science and MathematicsAudio processingAfter Penn State, Lee earned a Doctor of Science in information systems from Robert Morris University, served with the Department of Defense and taught at several universities.
  • Dr. Ralph FordProject advisorAdvisors: Ralph Ford and Ron McCarty

Every portrait above is a capture from the robot's own camera, box and all. Dr. Ford was the only one who beat it.

From the original 2004 Members page, about me: “He can be very emotionless at times (ironically, one of his research interests is ‘emotion detection’).”

[Photos: IEEE Student Design Contest, RIT]

[Reports]

The VisionPSU poster: abstract, applications, algorithms, technologies, schedule and team

[back to About]   Best viewed at 800 × 600. (Just kidding: any screen works now.)