Highlights and News

Opening:

My group has an opening for a post-doc. Please email me for details.

Spring 19:

I received the Presidential Early Career Award for Scientists and Engineers (PECASE).

Fall 2018:

I received the Early Career Award for Scientists and Engineers (ECASE-Army).

Fall 2018:

Our team from FAIR won the VizWiz Challenge at ECCV 2018 on Answering Visual Questions from Blind People.

Summer 18:

Our team from FAIR won the Visual Question Answering Challenge at CVPR 2018. Our winning entry is publicly available via Pythia.

Fall 2017:

Our work on language emergence, studying the properties of languages invented by AI agents won a best paper award at EMNLP 2017.

Spring 17:

I received the Office of Naval Research (ONR) Young Investigator Program (YIP) award.

03/16/16:

Interview with WVTF Radio IQ for our work on Visual Question Answering.

About Me

I am an Associate Professor in the School of Interactive Computing at Georgia Tech and a Research Scientist at Facebook AI Research (FAIR).

From Jan 2013 - Aug 2016, I was an Assistant Professor in the Bradley Department of Electrical and Computer Engineering at Virginia Tech, where I led the VT Machine Learning & Perception group. From Sep 2010 - Dec 2012, I was a Research Assistant Professor at Toyota Technological Institute at Chicago (TTIC), a philanthropically endowed academic computer science institute located in the campus of University of Chicago.

I received my Ph.D. in 2010 from Carnegie Mellon University, advised by Tsuhan Chen. I had the pleasure of spending the last 1.5 years of my grad school life at Cornell University, following my advisor's move.

Research

I am interested in artificial intelligence (AI). More specifically, my research lies at the intersection of machine learning and computer vision, with forays into natural language processing.

• see (or more generally perceive their environment through vision, audition, or other senses),
• talk (i.e. hold a natural language dialog grounded in their environment),
• act (e.g. navigate their environment and interact with it to accomplish goals), and
• reason (i.e., consider the long-term consequences of their actions).

Here are some representative projects from my lab:

• Summarizing beliefs of AI agents via diverse plausible predictions/hypotheses:
  Diverse Beam Search, Multiple Choice Learning, Tutorial on Diversity at CVPR '13 and CVPR '16,
• Vision-and-language:
  Image Captioning, Visual Question Answering, Visual Dialog, Human-AI GuessWhich games,
• Embodied AI:
  Habitat: A Platform for Embodied AI, Decentralized Distributed PPO, Embodied Question Answering,
• Explainable, Unbiased, Trustworthy AI:
  Grad-CAM (Visual Explanations), Human-vs-machine attention,
• CloudCV (building platforms for reproducible AI research):
  EvalAI, CloudCV demos for VQA, Visual Dialog, Grad-CAM, Diverse Beam Search.

Teaching

• Fall 2020: CS 4803/7643: Deep Learning.
• Fall 2019: CS 4803/7643: Deep Learning.
• Fall 2018: CS 4803/7643: Deep Learning.
• Fall 2017: CS 7643: Deep Learning.
• Spring 2016: ECE 2574: Data Structures and Algorithms.
• Fall 2015: ECE 6504: Deep Learning for Perception. Video lectures available!
• Spring 2015: ECE 5984: Introduction to Machine Learning. All video lectures available!
• Fall 2014: ECE 2574: Data Structures and Algorithms.
• Spring 2014: ECE 6504: Advanced Topics in Machine Learning: Probabilistic Graphical Models.
• Fall 2013: ECE 4984/5984: Introduction to Machine Learning & Perception.
• Spring 2013: ECE 4984/5984: Introduction to Machine Learning & Perception.
• Spring 2012: Co-Instructor for TTIC 31070: Convex Optimization. Primary Instructor: Nati Srebro.

Group

Research Scientists, Postdocs

Open Position. Please email me for details.

Students

• Nirbhay Modhe, PhD student (2017 — Present)
• Erik Wijmans, PhD student (2017 — Present); Co-advised with Irfan Essa
  ARCS Scholar Award Fellowship
• Harsh Agrawal, PhD student (2018 — Present)
  Snap Inc. Research Fellowship
• Arjun Majumdar, PhD student (2019 — Present)
• Joanne Truong, PhD student (2019 — Present); Co-advised with Sonia Chernova
  NSF Graduate Research Fellowship
  Google Women Techmakers Scholarship
  Adobe Research Fellowship
• Andrew Szot, PhD student (2020 — Present); Co-advised with Zsolt Kira
• Naoki Yokoyama, PhD student (2020 — Present); Co-advised with Sehoon Ha

• Rishabh Jain, MS student
• Joel Ye, MS student
• Sameer Dharur, MS student

• Harish Kamath, BS student

Visiting Students / Interns

• Yash Kant
• Abhinav Moudgil
• Ram Ramrakhya

Alumni

Former Research Scientists, Postdocs

• Stefan Lee, Postdoc 2016-17, Research Scientist 2017-19. Next position: Assistant Professor, Oregon State University
  Outstanding Research Scientist Award, College of Computing, Georgia Tech
  DARPA Riser
• Peter Anderson, Research Scientist 2018-19. Next position: Research Scientist in Language and Vision at Google.
• Zhile Ren, Postdoc 2018-19. Next position: Applied Research Scientist, Apple.

Former PhD Students

• Ashwin Kalyan, PhD 2020. Next position: Allen Institute for Artificial Intelligence (AI2).
  JP Morgan PhD Fellowship
  PhD Thesis: Improved search techniques for structured prediction.
• Abhishek Das, PhD 2020. Next position: Facebook AI Research.
  Facebook Fellowship
  Adobe Research Fellowship
  Snap Inc. Research Fellowship
  Sigma Xi Best Ph.D. Thesis Award, 2021
  PhD Thesis: Building agents that can see, talk, and act.
• Michael Cogswell, PhD 2020. Next position: SRI.
  Bradley Fellowship
  PhD Thesis: Disentangling neural network representations for improved generalization.
• Yash Goyal, PhD 2020. Next position: Samsung AI Lab, Montreal.
• Aishwarya Agrawal, PhD 2019. Next position: Google DeepMind / University of Montreal (MILA).
  NVIDIA Graduate Fellowship
  Sigma Xi Best Ph.D. Thesis Award, 2020
  Georgia Tech College of Computing PhD Dissertation Award, 2020
  Joint AAAI/ACM SIGAI Doctoral Dissertation Award Runner-up, 2019
  PhD Thesis: Visual Question Answering and Beyond.
• Qing Sun, PhD 2017. Next position: Facebook AI.
  PhD Thesis: Greedy Inference Algorithms for Structured and Neural Models.
• Gordon Christie, PhD 2016. Co-Advised with Kevin Kochersberger. Next position: Army Research Lab
  PhD Thesis: Collaborative Unmanned Air and Ground Vehicle Perception for Scene Understanding, Planning and GPS-denied Localization.

Former MS students and Interns

• Deshraj Yadav, MS 2018 (next position: Tesla Autopilot)
Snap Inc. Research Scholarship
• Akrit Mohapatra, MS 2017 (next position: Ebay)
• Aroma Mahendru, MS 2017 (next position: Bloomberg)
• Latha Pemula, MS 2016 (next position: Amazon)
• Harsh Agrawal, MS 2016 (next position: Snapchat Research)
• Prakriti Banik, MS 2015 (next position: Bloomberg)
• Neelima Chavali, MS 2015 (next position: Many Trees Inc)
• Clint Solomon, MS 2015 (next position: PhotoKharma)


• Rishabh Jain, Intern (next position: MS student at GT)
• Sanyam Agrawal, Intern
• Deshraj Yadav, Intern (BTech student from JSS Noida; next position: MS student at GT)
• Tejas Khot, Intern (BTech student from University of Mumbai; next position: MS student at RI, CMU)
• Viraj Prabhu, Intern (BTech student from BITS Pilani; next position: MS student at GT)
• Ashwin Kalyan, Intern (BTech student from NIT Surathkal; next position: PhD student at GT)
• Abhishek Das, Intern (BTech student from IIT-Roorkee; next position: PhD student at GT)
• Khushi Gupta, Intern (BTech student from IIT-Gowahati; next position: MS student at RI, CMU)
• Avi Singh, Intern (BTech student from IIT-Kampur; next position: PhD student at UC Berkeley)
• Ahmed Osman, Intern (MS student from UManchester; next position: PhD student at VT)
• Faruk Ahmed, Intern (BTech student from IEM; next position: PhD student at UMontreal)
• Senthil Purushwalkam, Intern (BTech student from IIT-G; MS student at RI, CMU)
• Aroma Mahendru, Intern (BTech student from IIT-BHU; next position: MS student at VT)
• Ratnesh Kumar, Intern (PhD student from INRIA; next position: MERL)
• Qi Lou, Intern (MS student from OSU; next position: PhD student at UC Irvine)
• Ankit Laddha, Intern (BTech student from IIT-D; next position: MS student at RI, CMU)
• Adarsh Prasad, Intern (BTech student from IIT-D; next position: PhD student at UT-Austin)
• Harsh Agrawal, Intern (BTech student from DTU; next position: MS student at VT)
• Abhimanyu Dubey, Intern (BTech student from IIT-D)

Former Student Collaborators

• Payman Yadollahpour (PhD student from TTIC)
• Kun Duan (PhD student from Indiana U)
• Abner Guzman-Rivera (PhD student from UIUC)
• Daozheng Chen (PhD student from UMD)
• Jason Lew (MEng student at Cornell)
• Tanmay Verma (BTech student from IIIT-D)
• Charvi Puri (BTech student from IIIT-D)
• Kevin Tang (BS student at Cornell, now grad student at Stanford)

Press Coverage

The research performed by my group (and our collaborators) has been covered here:

• Four Indian American Researchers Named Young Investigator Awardees for 2017 by Office of Naval Research
  
• Is Artificial Intelligence Permanently Inscrutable?
  
• Deep learning is creating computer systems we don't fully understand.
  
• Robot eyes and humans fix on different things to decode a scene.
  
• Teaching computers to describe images as people would. Project Lead: Meg Mitchell.
  
• A Giant Leap for Machine Kind; When Robots Can See.
  
• Artificial Intelligence Algorithm Taught to Recognise Humor.
  
• AI Algorithm Identifies Humorous Pictures.
  
• What’s in This Picture? AI Becomes as Smart as a Toddler.
  
• How to automatically detect the most important people in a photograph.
  
• Dhruv Batra receives NSF CAREER Award for machine perception research.
  

  ECE News	DAC News

• He aims to mimic the human brain’s capabilities.
  
• Researcher seeks to remove ambiguity in computer visual recognition systems.
  

  COE News	VT News	Phys Org News

• Stocking the visual toolbox.
  

  ECE News

Quoted in coverage not directly about my work:

• The Curious Case of Apple’s Supposed Self-Driving Car.
  
  • WIRED, Feb 2015.
  • Business Insider, Feb 2015.
  • Realty Today, Feb 2015.
  • Opp Trends, Feb 2015.
  • Rt.com, Feb 2015.

Funding

My group is supported by the following:

• Army Research Lab (ARL) and Army Research Office (ARO)
• Defense Advanced Research Projects Agency (DARPA)
• National Science Foundation (NSF)
• Office of Naval Research (ONR)
• Amazon
• Facebook
• Google
• Samsung
• Siemens

Talks

On the web

• Visual Question Answering
  Keynote talk, Multimodal Machine Learning Workshop at NIPS, Dec 2015.
  
  
• Visual Question Answering
  Data Science Summit & Dato Conference, 2015.
  
  
• Visual Question Answering
  Deep Learning Summit, 2015.
  
  
• Hedging Against Uncertainty via Multiple Diverse Predictions
  Microsoft Research Talk, 2014.
  
  
• Submodular meets Structured: Finding Diverse Subsets in Exponentially-Large Structured Item Sets
  NIPS Workshop on Discrete and Combinatorial Problems in Machine Learning (DISCML), 2014.
  
  
• Hedging Against Uncertainty via Multiple Diverse Predictions
  CESCA Seminar, 2014.
  
  
• Beyond MAP: Making Multiple Predictions from Probabilistic Models: Diversity, DPPs and more
  Tutorial at the Conference on Computer Vision and Pattern Recognition (CVPR) 2013.
  
  
• Diverse M-Best Solutions in Markov Random Fields
  European Conference on Computer Vision (ECCV) 2012.
  
  
• Structured-Output Models for Computer Vision
  Microsoft Research Talk, 2012.
  
  
• Beyond Trees: MRF Inference via Outer-Planar Decomposition
  Microsoft Research Talk, 2009.
  
  

Slides from talks

• A tale of two negative results [slides (pptx)]
• Towards AI agents that can see, talk, and act [slides (pptx)]
• Visual Dialog [slides (pptx)]
• Towards Transparent VQA Models [slides (pptx)]
• Visual Question Answering [slides (pptx)]
• Submodular meets Structured: Finding Diverse Subsets in Exponentially-Large Structured Item Sets.
  [ slides (pptx) ]
• M-Best and Diverse M-Best MAP Inference in Graphical Models [ slides ]
• CloudCV: Large-Scale Distributed Computer Vision as a Cloud Service [ slides ]
• Hedging Against Uncertainty via Multiple Diverse Predictions [ slides (pptx) ]
• Structured-Output Models for Computer Vision
• The M-Best Mode Problem [ slides (pptx) ]
• Focused Inference and the M-Best Mode Problem [ slides (pptx) ]
• Focused Inference with Local Primal-Dual Gaps [ slides (pptx) ]
• On Graph-Structured Discrete Labelling Problems in Computer Vision [ slides (pptx) ]
• Beyond Trees: MRF Inference via Outer-Planar Decomposition [ slides (pptx) ]