HCI Project - Using gesture and speech with Kinect for education
This fall term, I took a course in Human-Computer Interaction (HCI) CSC2524 with Prof. Daniel Wigdor at the University of Toronto. The topics in interactive computing focused on user experiences with next-generation input & output technologies. For my course project I decided to work with novel input modalities of gesture and speech with Microsoft Kinect as new hardware and developed a proof of concept for use in education.
I was inspired by our class topic on case studies in Multi-Modal Input: Gesture + Speech where we discussed two papers "Put that There" by Richard Bolt and Multimodal multiplayer tabletop gaming by Tse et al. The papers discuss the use of deictic references which are speech terms such as "this" and "that" which can be used with spatial gestures like pointing to an object or location. I got the idea that deixis would be an efficient mode of interaction with the computer for younger children in a learning environment. Gestures and speech used as multimodal input could simplify complex learning and make classroom education interactive and fun. So I decided to do my project on this topic.
As proof of concept, I developed few prototypes using Microsoft Visual C# and Kinect SDK beta 2, which demonstrate pointing to objects on screen and using voice commands to select them. The Coding4Fun Kinect Toolkit provides a hover button control. I referred to the tutorial by Michael Crump. Another project posted on MSDN channel9 made use of the hover button implementing it as a "hand" cursor that has a timer based ring used to press the buttons. In my implementation I have used voice command recognition to fire the hovering event inside the hover button control, when it is found to be on top of other buttons on the canvas.
The first activity asks the child to choose a particular color from a mixed group of colored tiles. This could be adapted to a variety of activities that teach identification. The second activity uses specific commands to point out jungle or farm animals. Again this activity could be used to enhance the vocabulary of the child and could be adapted in several different ways to address topics in math, language, geography and so on. The third activity helps develop creative thinking where the child can select objects and manipulate them by moving them around, making them bigger or smaller, changing their colors and so on. This can help teach positions, directions, opposites, sorting, grouping and much more.
My daughters, aged 4 and 7 tried out my system and as a mother involved in their early years learning development, I definitely see the potential use of this technology in education. I had the opportunity to study the prototypes in my daughter's Junior Kindergarten classroom with her classmates. It was very encouraging and I find myself motivated to continue exploring this space even beyond the course.