Monday, February 22, 2010

Sixthsense Device---Integrating information with the real world

'SixthSense' is a wearable gestural interface that augments the physical world around us with digital information and lets us use natural hand gestures to interact with that information. By using a camera and a tiny projector mounted in a pendant like wearable device, 'SixthSense' sees what you see and visually augments any surfaces or objects we are interacting with. It projects information onto surfaces, walls, and physical objects around us, and lets us interact with the projected information through natural hand gestures, arm movements, or our interaction with the object itself. 'SixthSense' attempts to free information from its confines by seamlessly integrating it with reality, and thus making the entire world your computer.

Ever wanted to be able to manipulate images on a computer the way Tom Cruise did in Minority Report? A new Media Lab invention, sixthsense, lets you do just that as it allows users to manipulate digital information with hand gestures.

By wearing just a hat with a tiny projector and a camera, a sixthsense user can make any flat surface a connection to the world to check email, map out a location, or draw with fingers.

Designed by Pranav K. Mistry G, a PhD student in the Fluid Interfaces Group of the Lab, sixthsense has the ability to track colors, hand movements, and gestures. It connects with its owner’s digital devices.

Many natural hand gestures are possible with sixthsense. Snapping your fingers as if you were taking a photo on an actual camera or tapping your wrist with a circular gesture maps to the physical actions of taking a picture and checking the time.

“You can take a photo of a random book, and check its prices on Amazon. You can compare prices between goods in the supermarket” and check which ones are green products, said Mistry.

“There is a lot of information on the Internet, but humans do not have access to it at all times. Sixthsense gives you the ability to receive information about anything and anyone you encounter, anywhere, and at all times,” added Mistry.

Discussing the motivations behind his work, Mistry said “the digital world has brought many devices to human life, yet it has diluted human interactions. People have started using social networks as their major path for socializing. You would see people sitting individually in cafes, each busy with his laptop or phone. My task is to use digital work to integrate digital work into human’s lives.”

The idea for the sixthsense project came to Mistry about six months ago. “It came as a crazy idea of thinking of the term head mountain projector! I just started thinking of actually making real head mountain projectors that would truly connect to people’s physical world!”

Mistry initially implemented his inspiration as a projector helmet where the camera tracked what the wearer did with his or her hand. Further modifications resulted in a cap with a smaller projector, and, finally, into a small device containing a projector and a camera.

Mistry initially called the device “WUW” as in “wear ur world.” But when it was introduced, sixthsense was judged to be a better title.

Mistry also incorporated his Indian background into his invention. Bringing your hands together in the Indian gesture of welcome, “Namaste”, causes the main menu to open up.

Mistry foresees several improvements to ‘sixthsense’, one of which is incorporating the use of computer-vision based techniques that do not require the user to wear color markers. “I have a lot of applications in mind to make sixthsense more practical for use.”

“I believe that we should use systems to learn about users rather than have users learn about systems.”

Remarkably, Mistry developed SixthSense in less than five months, and it costs under $350 to build (not including the phone). Users must currently wear colored "marker­s" on their fingers so that the system can track their hand gestures, but he is designing algorithms that will enable the phone to recognize them directly. --Brittany Sauser

1. Camera: A webcam captures an object in view and tracks the user's hand gestures. It sends the data to the smart phone.

2. Colored Markers: Marking the user's fingers with red, yellow, green, and blue tape helps the webcam recognize gestures. Mistry is working on gesture-recognition algorithms that could eliminate the need for the markers.

3. Projector: A tiny LED projector displays data sent from the smart phone on any surface in view--object, wall, or person. Mistry hopes to start using laser projectors to increase the brightness.

4. Smart Phone: A Web-enabled smart phone in the user's pocket processes the video data, using vision algorithms to identify the object. Other software searches the Web and interprets the hand gestures.

The SixthSense prototype implements several applications that demonstrate the usefulness, viability and flexibility of the system. The map application lets the user navigate a map displayed on a nearby surface using hand gestures, similar to gestures supported by Multi-Touch based systems, letting the user zoom in, zoom out or pan using intuitive hand movements. The drawing application lets the user draw on any surface by tracking the fingertip movements of the user’s index finger. SixthSense also recognizes user’s freehand gestures (postures). For example, the SixthSense system implements a gestural camera that takes photos of the scene the user is looking at by detecting the ‘framing’ gesture. The user can stop by any surface or wall and flick through the photos he/she has taken. SixthSense also lets the user draw icons or symbols in the air using the movement of the index finger and recognizes those symbols as interaction instructions. For example, drawing a magnifying glass symbol takes the user to the map application or drawing an ‘@’ symbol lets the user check his mail. The SixthSense system also augments physical objects the user is interacting with by projecting more information about these objects projected on them. For example, a newspaper can show live video news or dynamic information can be provided on a regular piece of paper. The gesture of drawing a circle on the user’s wrist projects an analog watch.

The current prototype system costs approximate $350 to build.