Duy-Nguyen Ta

SURFTrac3D

Sat, 15 Aug 2009 15:07:58 -0400

This is the result of my second summer internship with Nokia at NRC Hollywood lab. I experimented applying SURFTrac for 3D outdoor Augmented Reality.

I first built a database of SURF features extracted from a set of images and used Bundler to construct their 3D coordinates. Then, in the first image captured from the video stream, I extracted SURF features and matched with those in the database. After that, I use traditional RANSAC with Homography model to reject wrong matches. The correct matches are then used to initialize the camera pose.

SURFTrac is used to track SURF features in subsequent camera frames. New features in the frames are detected and matched with the database. From those matches, I compute 6 dof camera poses for each frame and augment all the 3D points for visualization. All the red dots in the video below are those 3D points in the database augmented onto the real scene. They are NOT 2D detected features. Please note how stable their projections are.

I also render a panel of Nokia logo onto the building to show that it’s a real 3D scene tracking. You might notice that the panel is not properly aligned. However, the main error doesn’t come from wrong pose estimation, but rather from my bad modeling skill. The position of the plane panel in 3D virtual world is not properly setup. You can see how “stable” the misalignment is. :-)

Video:

Art of Defense

Thu, 12 Mar 2009 01:01:31 -0400

Art of Defense explores novel game design and interaction techniques for mobile AR on commodity phone (Nokia N95). The design leverages the phone’s screen size limitations as part of the game challenge. Its dynamic multi-marker building technique enables the game’s mobility and portability, whereas the sketch-based interaction enhances the tangibility and content relationship between the physical and virtual worlds. These techniques encourage both physical movement and tangible interaction with the game pieces -- the two key differentiators between AR and other phone-based games. Art of Defense also demonstrates a new type of AR exploration game, where the players control which parts of the space they can explore. Our first version is shown at ISMAR’08.

Our second version is a 2-player collaborative game. We redesigned the game significantly to enforce collaboration between players. We replaced sketches with tokens to simplify the interactions and make it really tangible. The game was also fine-tuned carefully to become really interesting. Our user study reveals many interesting lessons about table-top AR collaborative games. For more details about the game, please see our paper below in Sandbox SIGGRAPH 2009.

Paper:

D.N. Ta, K. Raveendran, Y. Xu, K. Spreen, and B. MacIntyre, “Art of defense: a collaborative handheld augmented reality board game,” Proceedings of the 2009 ACM SIGGRAPH Symposium on Video Games, 2009, pp. 135–142. (pdf)

Video version 1: single player game with sketches:

Video version 2: multiplayer collaborative AR game with tangible tokens:

Update: An interesting video of our very first version on vimeo, captured by Dylski (?). I believe this was my demo in ISMAR’08 at Cambridge, UK.

SURFTrac

Sat, 6 Dec 2008 00:50:55 -0500

This is the result of my 3-month internship at Nokia Research Center, Palo Alto. We present a light-weight tracking method for outdoor augmented reality using pure computer vision technique with a database of geotagged and labeled images.

After extracting and matching SURF features of the first video frame with the database, our method reliably tracks those features locally in the subsequent frames. This method is fast enough to be implemented on mobile devices. The technique is demonstrated in the context of augmenting a video stream with labels of buildings visible in the video.

This project was demonstrated on N95 in ISMAR’08 and CVPR’09. Also, see an application of SURFTrac for 3D tracking and camera pose estimation here.

Paper:

D.N. Ta, W.C. Chen, N. Gelfand, and K. Pulli, “SURFTrac: Efficient tracking and continuous object recognition using local feature descriptors,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009. (pdf)

Video:

Lego Edge Tracking

Tue, 6 May 2008 15:08:17 -0400

This is a joint project with Yong-Dian Jian during my second year in Georgia Tech, to experiment with using Lego cube for Augmented Reality applications. We utilize the Lego’s 3D model, which could be built easily, and develop a model-based edge tracking to track the 3D Lego cube in real-time.

The Lego cube exposes interesting challenges for tracking algorithm. Its lack of texture fails most methods relying on point features. We use Canny method to extract edges from the capture image, however, the result is very unstable with many noisy edges from the background and shadow, and missing edges in the Lego model due to color similarity between some faces. Inspired by a paper of Georg Klein, we experimented with particle filter for edge tracking. The performance was improved but still unstable. The final stable result as you see in the video below was done with traditional Gauss-Newton non-linear least square optimization combine with many small tricks. :-)

We would love to continue this project and make it more robust in the future.

Video:

Ultimate Commodity

Sun, 15 Jul 2007 22:03:40 -0400

Link to the project page at NTU.
This is one of a few traditional theater performances that use Augmented Reality technology. Ultimate Comodity is a play by Daniel Jernigan, basing on a dystopic short story of Gopal Barathaman, a Singaporean author. The story is about a dream of a young female scientist, who wants to turn Singapore into an international organ-donor country. From her mom’s DNA, she invented a chemical element that can make human’s organ ready for implanting. She then decided to put that chemical element secretly into drinking water for all Singapore citizens. However, her mistake in DNA x makes faces of all people who drink that water look very alike. The interesting chaos happen when her father mistook her as her mother and she thought her father as her boyfriend because they all look alike to her mother...

Augmented Reality comes into play to replace the actor and actress’ real head by a model of the mother’s head.

The first demo version was showcased at The Esplanade Theater On the Bay in Singapore on September 2006. In that version, we used marker tracking and overlaid 2D videos of the mother’s face to demonstrate the concept. However, although we tried our best to mitigate the story and play actions so that the performance could be merged well with the technology, the use of markers caused a lot of troubles for the actresses.

We developed the second version in 3 months to show at the Fringe Festival 2007 in Toronto. In that version, we used stereo tracking with 2 Unibrain Fire-i cameras mounted on the ceiling, tracking a 5x5 color-LED boards on top of each actor/actresses’ heads. The rotation is tracked by MTx inertial sensors and transferred to the computer using Bluetooth.

The graphics is also improved significantly with realistic 3D models of the mother’s head. The static model of the head is constructed by Dr. Quah Chee Kwang from 5 images and a generic 3D head model. After that, 3D artists refined the model and generated different models for lip animation for different visemes. My program creates the morphing animation among those viseme models to synchronize with the prerecorded voice.

The Fringe Festival is famous for exact and limited timing for every show. We have only 2 hours to set up and calibrate the system before the first show. For every subsequent shows, we only have 15 minutes to prepare and start the system.

One of the important lessons I have learnt is the whole iterative design process -- writing >> technology >> directing >> acting >> and then come back to writing/technology or directing. We tried to integrate the technology difficulties as part of the story itself and made them invisible as much as possible in the play. We ended up describe the LED head-mounted markers as important components of the technology in the play. I was also playing a role in the play as the scientist’s colleague. (I didn’t do anything but sat on the stage to ... control the computer and hand over necessary equipments for the actor/actresses at the correct time!)

Our team includes: Prof. Russell Pensyl (director and producer), Prof. Daniel Jernigan (writer), Gerald Chew (co-director, actor), Debra Teng (actress, the mother), Sara Yang (actress, the young scientist), Anand (stage manager), Kiat (sound and lighting manager), Dr. Quah Chee Kwang (technical advisor), Lee Shang Ping (hardware developer), and me (software developer and tracking technology). Original ideas for the first version was also came from Diego Diaz and Clara Boj. This projects created many interesting memories with a lot of hard work and collaborations among all team members.