Highlights and News

Fall 2018:

I received the Early Career Award for Scientists and Engineers (ECASE-Army).

Fall 2018:

Our team from FAIR won the VizWiz Challenge at ECCV 2018 on Answering Visual Questions from Blind People.

Summer 18:

Our team from FAIR won the Visual Question Answering Challenge at CVPR 2018. Our winning entry is publicly available via Pythia.

Fall 2017:

Our work on language emergence, studying the properties of languages invented by AI agents won a best paper award at EMNLP 2017.

Spring 17:

I received the Office of Naval Research (ONR) Young Investigator Program (YIP) award.

Fall 2016:

I am moving to Georgia Tech (School of Interactive Computing).

03/16/16:

Interview with WVTF Radio IQ for our work on Visual Question Answering.

Opening:

My group has an opening for a post-doc. Please email me for details.

About Me

I am an Assistant Professor in the School of Interactive Computing at Georgia Tech and a Research Scientist at Facebook AI Research (FAIR).

From Jan 2013 - Aug 2016, I was an Assistant Professor in the Bradley Department of Electrical and Computer Engineering at Virginia Tech, where I led the VT Machine Learning & Perception group. From Sep 2010 - Dec 2012, I was a Research Assistant Professor at Toyota Technological Institute at Chicago (TTIC), a philanthropically endowed academic computer science institute located in the campus of University of Chicago.

I received my Ph.D. in 2010 from Carnegie Mellon University, advised by Tsuhan Chen. I had the pleasure of spending the last 1.5 years of my grad school life at Cornell University, following my advisor's move.

Research

I am interested in artificial intelligence (AI). More specifically, my research lies at the intersection of machine learning and computer vision, with forays into natural language processing.

• see (or more generally perceive their environment through vision, audition, or other senses),
• talk (i.e. hold a natural language dialog grounded in their environment),
• act (e.g. navigate their environment and interact with it to accomplish goals), and
• reason (i.e., consider the long-term consequences of their actions).

Here are some representative projects from my lab:

• Summarizing beliefs of AI agents via diverse plausible predictions/hypotheses:
  Diverse Beam Search, Multiple Choice Learning, Tutorial on Diversity at CVPR '13 and CVPR '16,
• Vision-and-language:
  Image Captioning, Visual Question Answering, Visual Dialog, Human-AI GuessWhich games,
• Embodied AI:
  Embodied Question Answering, hierarchical reinforcement learning for embodied navigation,
• Explainable, Unbiased, Trustworthy AI:
  Grad-CAM (Visual Explanations), Human-vs-machine attention,
• CloudCV (building platforms for reproducible AI research):
  EvalAI, CloudCV demos for VQA, Visual Dialog, Grad-CAM, Diverse Beam Search.

Teaching

• Fall 2018: CS 4803/7643: Deep Learning.
• Fall 2017: CS 7643: Deep Learning.
• Spring 2016: ECE 2574: Data Structures and Algorithms.
• Fall 2015: ECE 6504: Deep Learning for Perception. Video lectures available!
• Spring 2015: ECE 5984: Introduction to Machine Learning. All video lectures available!
• Fall 2014: ECE 2574: Data Structures and Algorithms.
• Spring 2014: ECE 6504: Advanced Topics in Machine Learning: Probabilistic Graphical Models.
• Fall 2013: ECE 4984/5984: Introduction to Machine Learning & Perception.
• Spring 2013: ECE 4984/5984: Introduction to Machine Learning & Perception.
• Spring 2012: Co-Instructor for TTIC 31070: Convex Optimization. Primary Instructor: Nati Srebro.

Group

I lead the Machine Learning & Perception Group. Here are the current and past members:

Research Scientists

• Stefan Lee
• Peter Anderson

Postdoc

• Zhile Ren

Students

• Aishwarya Agrawal, PhD student (2014 — Present)
• Yash Goyal, PhD student (2014 — Present)
• Michael Cogswell, PhD student (2015 — Present)
• Abhishek Das, PhD student (2016 — Present)
• Ashwin Kalyan, PhD student (2016 — Present)
• Nirbhay Modhe, PhD student (2017 — Present)
• Erik Wijmans, PhD student (2017 — Present; Co-advised with Irfan Essa)
• Harsh Agrawal, PhD student (2018 — Present)

• Deshraj Yadav, MS student

Visiting Students

• Rishabh Jain, Intern
• Sanyam Agrawal, Intern

MLP Group Alumni (Graduated Students & Former Interns)

• Qing Sun, PhD 2017 (next position: Facebook AI)
• Akrit Mohapatra, MS 2017 (next position: Ebay)
• Aroma Mahendru, MS 2017 (next position: Bloomberg)
• Latha Pemula, MS 2016 (next position: Amazon)
• Gordon Christie, PhD 2016 (next position: Army Research Lab)
• Harsh Agrawal, MS 2016 (next position: Snapchat Research)
• Prakriti Banik, MS 2015 (next position: Bloomberg)
• Neelima Chavali, MS 2015 (next position: Many Trees Inc)
• Clint Solomon, MS 2015 (next position: PhotoKharma)


• Deshraj Yadav, Intern (BTech student from JSS Noida; next position: MS student at GT)
• Tejas Khot, Intern (BTech student from University of Mumbai; next position: MS student at RI, CMU)
• Viraj Prabhu, Intern (BTech student from BITS Pilani; next position: MS student at GT)
• Ashwin Kalyan, Intern (BTech student from NIT Surathkal; next position: PhD student at GT)
• Abhishek Das, Intern (BTech student from IIT-Roorkee; next position: PhD student at GT)
• Khushi Gupta, Intern (BTech student from IIT-Gowahati; next position: MS student at RI, CMU)
• Avi Singh, Intern (BTech student from IIT-Kampur; next position: PhD student at UC Berkeley)
• Ahmed Osman, Intern (MS student from UManchester; next position: PhD student at VT)
• Faruk Ahmed, Intern (BTech student from IEM; next position: PhD student at UMontreal)
• Senthil Purushwalkam, Intern (BTech student from IIT-G; MS student at RI, CMU)
• Aroma Mahendru, Intern (BTech student from IIT-BHU; next position: MS student at VT)
• Ratnesh Kumar, Intern (PhD student from INRIA; next position: MERL)
• Qi Lou, Intern (MS student from OSU; next position: PhD student at UC Irvine)
• Ankit Laddha, Intern (BTech student from IIT-D; next position: MS student at RI, CMU)
• Adarsh Prasad, Intern (BTech student from IIT-D; next position: PhD student at UT-Austin)
• Harsh Agrawal, Intern (BTech student from DTU; next position: MS student at VT)
• Abhimanyu Dubey, Intern (BTech student from IIT-D)

Former Student Collaborators

• Payman Yadollahpour (PhD student from TTIC)
• Kun Duan (PhD student from Indiana U)
• Abner Guzman-Rivera (PhD student from UIUC)
• Daozheng Chen (PhD student from UMD)
• Jason Lew (MEng student at Cornell)
• Tanmay Verma (BTech student from IIIT-D)
• Charvi Puri (BTech student from IIIT-D)
• Kevin Tang (BS student at Cornell, now grad student at Stanford)

Press Coverage

The research performed by my group (and our collaborators) has been covered here:

• Four Indian American Researchers Named Young Investigator Awardees for 2017 by Office of Naval Research
  
• Is Artificial Intelligence Permanently Inscrutable?
  
• Deep learning is creating computer systems we don't fully understand.
  
• Robot eyes and humans fix on different things to decode a scene.
  
• Teaching computers to describe images as people would. Project Lead: Meg Mitchell.
  
• A Giant Leap for Machine Kind; When Robots Can See.
  
• Artificial Intelligence Algorithm Taught to Recognise Humor.
  
• AI Algorithm Identifies Humorous Pictures.
  
• What’s in This Picture? AI Becomes as Smart as a Toddler.
  
• How to automatically detect the most important people in a photograph.
  
• Dhruv Batra receives NSF CAREER Award for machine perception research.
  

  ECE News	DAC News

• He aims to mimic the human brain’s capabilities.
  
• Researcher seeks to remove ambiguity in computer visual recognition systems.
  

  COE News	VT News	Phys Org News

• Stocking the visual toolbox.
  

  ECE News

Quoted in coverage not directly about my work:

• The Curious Case of Apple’s Supposed Self-Driving Car.
  
  • WIRED, Feb 2015.
  • Business Insider, Feb 2015.
  • Realty Today, Feb 2015.
  • Opp Trends, Feb 2015.
  • Rt.com, Feb 2015.

Funding

My group is supported by the following:

• Army Research Lab (ARL) and Army Research Office (ARO)
• Defense Advanced Research Projects Agency (DARPA)
• National Science Foundation (NSF)
• Office of Naval Research (ONR)
• Amazon
• Facebook
• Google
• Samsung
• Siemens

Talks

On the web

• Visual Question Answering
  Keynote talk, Multimodal Machine Learning Workshop at NIPS, Dec 2015.
  
  
• Visual Question Answering
  Data Science Summit & Dato Conference, 2015.
  
  
• Visual Question Answering
  Deep Learning Summit, 2015.
  
  
• Hedging Against Uncertainty via Multiple Diverse Predictions
  Microsoft Research Talk, 2014.
  
  
• Submodular meets Structured: Finding Diverse Subsets in Exponentially-Large Structured Item Sets
  NIPS Workshop on Discrete and Combinatorial Problems in Machine Learning (DISCML), 2014.
  
  
• Hedging Against Uncertainty via Multiple Diverse Predictions
  CESCA Seminar, 2014.
  
  
• Beyond MAP: Making Multiple Predictions from Probabilistic Models: Diversity, DPPs and more
  Tutorial at the Conference on Computer Vision and Pattern Recognition (CVPR) 2013.
  
  
• Diverse M-Best Solutions in Markov Random Fields
  European Conference on Computer Vision (ECCV) 2012.
  
  
• Structured-Output Models for Computer Vision
  Microsoft Research Talk, 2012.
  
  
• Beyond Trees: MRF Inference via Outer-Planar Decomposition
  Microsoft Research Talk, 2009.
  
  

Slides from talks

• A tale of two negative results [slides (pptx)]
• Towards AI agents that can see, talk, and act [slides (pptx)]
• Visual Dialog [slides (pptx)]
• Towards Transparent VQA Models [slides (pptx)]
• Visual Question Answering [slides (pptx)]
• Submodular meets Structured: Finding Diverse Subsets in Exponentially-Large Structured Item Sets.
  [ slides (pptx) ]
• M-Best and Diverse M-Best MAP Inference in Graphical Models [ slides ]
• CloudCV: Large-Scale Distributed Computer Vision as a Cloud Service [ slides ]
• Hedging Against Uncertainty via Multiple Diverse Predictions [ slides (pptx) ]
• Structured-Output Models for Computer Vision
• The M-Best Mode Problem [ slides (pptx) ]
• Focused Inference and the M-Best Mode Problem [ slides (pptx) ]
• Focused Inference with Local Primal-Dual Gaps [ slides (pptx) ]
• On Graph-Structured Discrete Labelling Problems in Computer Vision [ slides (pptx) ]
• Beyond Trees: MRF Inference via Outer-Planar Decomposition [ slides (pptx) ]