Orientation Aware Scene Understanding for Mobile Camera



(Left: Original image with horizon line, detected by camera. Middle: Annotated Ground Truth Image (marked using touch screen), Right: Result of our approach)

In 14th ACM International Conference on Ubiquitous Computing (Ubicomp 2012), Pittsburgh, PA, USA (September 5-8, 2012).


Jing Wang

Georgia Institute of Technology

Grant Schindler

Georgia Institute of Technology

Irfan Essa

Georgia Institute of Technology


We present a novel approach that allows anyone to quickly teach their smartphone how to understand the visual world around them. We achieve this visual scene understanding by leveraging a camera-phone’s inertial sensors to lead to both a faster and more accurate automatic labeling of the regions of an image into semantic classes (e.g. sky, tree, building). We focus on letting a user train our system from scratch while out in the real world by annotating image re- gions in situ as training images are captured on a mobile de- vice, making it possible to recognize new environments and new semantic classes on the fly. We show that our approach outperforms existing methods, while at the same time per- forming data collection, annotation, feature extraction, and image segment classification all on the same mobile device.


Jing Wang, Grant Schindler, Irfan Essa (2012), "Orientation Aware Scene Understanding for Mobile Camera", In Proceedings of ACM International Conference on Ubiquitous Computing (Ubicomp), Pittsburgh, PA, Sep 2012.
       Author = {Jing Wang and Grant Schindler and Essa},
       Booktitle = {Proceedings of ACM International Conference on Ubiquitous Computing (Ubicomp)},
       Month = {September},
       Title = {Orientation Aware Scene Understanding for Mobile Camera},
       Year = {2012},


Demo / Source / Dataset
Other Relevant Information

The documents contained in these directories are included by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without explicit permission of the copyright holder.