Highlights and News

Spring 18:

Devi Parikh and I received an AWS ML Research Award.

2017-18:

I am serving as an Area Chair for CVPR 2018, ICLR 2018, ECCV 2018, NIPS 2018.

Aug 2017:

Our work on language emergence, studying the properties of languages invented by AI agents won a best paper award at EMNLP 2017!

Spring 17:

I received the Office of Naval Research (ONR) Young Investigator Program (YIP) award!

Spring 17:

I received an Amazon Academic Research Award (AARA) for Visual Dialog.

Fall 2016:

I am moving to Georgia Tech (School of Interactive Computing).
I will be spending a year at Facebook AI Research as a Visiting Researcher.

2016-17:

I am serving as an Area Chair for CVPR 2017, ICLR 2017, NIPS 2017, UAI 2017.

03/16/16:

Interview with WVTF Radio IQ for our work on Visual Question Answering!

Opening:

My group has an opening for a post-doc. Please email me for details.

About Me

I am an Assistant Professor in the School of Interactive Computing at Georgia Tech and a Research Scientist at Facebook AI Research (FAIR).

From Jan 2013 - Aug 2016, I was an Assistant Professor in the Bradley Department of Electrical and Computer Engineering at Virginia Tech, where I led the VT Machine Learning & Perception group. From Sep 2010 - Dec 2012, I was a Research Assistant Professor at Toyota Technological Institute at Chicago (TTIC), a philanthropically endowed academic computer science institute located in the campus of University of Chicago.

I received my Ph.D. in 2010 from Carnegie Mellon University, advised by Tsuhan Chen. I had the pleasure of spending the last 1.5 years of my grad school life at Cornell, following my advisor's move.

In past, I have held visiting positions at:

• Microsoft Research, Summer 2015, Visiting Researcher hosted by Larry Zitnick and Meg Mitchell.
• MLD, CMU, Summer 2012, Visiting Faculty collaborating with Carlos Guestrin.
• CSAIL, MIT, Spring 2011, Visiting Scientist collaborating with Bill Freeman.

• Microsoft Research Lab Cambridge, Summer 2010, Research Intern with Sebastian Nowzin
  and Pushmeet Kohli.
• Kodak Research Lab, Summer 2008, Research Intern with Jiebo Luo.
• Intel Research Pittsburgh, Summer 2007, Research Intern with Rahul Sukthankar.
• Intel Research Pittsburgh, Summer 2006, Research Intern with Bart Nabbe.

Research

My research interests lie at the intersection of machine learning and computer vision, with a focus on developing intelligent systems that are able to

• concisely summarize their beliefs about the world with diverse plausible predictions,
• integrate information and beliefs across different sub-components or `modules' of AI (vision, language, reasoning), and
• and explain why they believe what they believe.

Honors and Awards

• AWS Machine Learning Research Award, 2018
• Office of Naval Research (ONR) Young Investigator Program (YIP) award, 2017
• Amazon Academic Research Award, 2016
• Google Faculty Research Awards, 2013, 2015
• Outstanding Reviewer Award, Conference on Computer Vision and Pattern Recognition (CVPR), 2015
• Outstanding New Assistant Professor award College of Engineering, Virginia Tech, 2015
• National Science Foundation (NSF) CAREER award, 2014
• Army Research Office (ARO) Young Investigator Program (YIP) award, 2014
• Carnegie Mellon University College of Engineering Dean’s Fellowship, 2007

Press Coverage

The research performed by my group (and our collaborators) has been covered here:

• Four Indian American Researchers Named Young Investigator Awardees for 2017 by Office of Naval Research
  
• Is Artificial Intelligence Permanently Inscrutable?
  
• Deep learning is creating computer systems we don't fully understand.
  
• Robot eyes and humans fix on different things to decode a scene.
  
• Teaching computers to describe images as people would. Project Lead: Meg Mitchell.
  
• A Giant Leap for Machine Kind; When Robots Can See.
  
• Artificial Intelligence Algorithm Taught to Recognise Humor.
  
• AI Algorithm Identifies Humorous Pictures.
  
• What’s in This Picture? AI Becomes as Smart as a Toddler.
  
• How to automatically detect the most important people in a photograph.
  
• Dhruv Batra receives NSF CAREER Award for machine perception research.
  

  ECE News	DAC News

• He aims to mimic the human brain’s capabilities.
  
• Researcher seeks to remove ambiguity in computer visual recognition systems.
  

  COE News	VT News	Phys Org News

• Stocking the visual toolbox.
  

  ECE News

Quoted in coverage not directly about my work:

• The Curious Case of Apple’s Supposed Self-Driving Car.
  
  • WIRED, Feb 2015.
  • Business Insider, Feb 2015.
  • Realty Today, Feb 2015.
  • Opp Trends, Feb 2015.
  • Rt.com, Feb 2015.

Funding

My group is supported by the following:

Research Grants and Gifts

• Army Research Lab (ARL) Grant W911NF-15-2-0080, 2015-2018
• Defense Advanced Research Projects Agency (DARPA) Grant HR0011-16-1-0002, 2015-2016
• Google Faculty Research Award, 2015
• Office of Naval Research (ONR) Grant N00014-14-1-0679, 2014-2017
• Army Research Office (ARO) Young Investigator Program (YIP) Award W911NF-14-1-0180, 2014-2017
• National Science Foundation (NSF) CAREER Award IIS-1350553, 2014-2019
• Virginia Tech Institute for Critical Technology and Applied Science (ICTAS)
  Junior Faculty Award, 2014-2016
• NSF EAGER Grant IIS-1353694, 2013-2015

Equipment, Educational, Organizational Grants and Gifts

• NRT-DESE: UrbComp: Data Science for Modeling, PI: Naren Ramakrishnan (CS, VT), 2015-2020
• State Council of Higher Education for Virginia (SCHEV) Equipment Trust Grant, 2014
• Oak Ridge Associated University (ORAU) Event Organization Grant
  (Co-organizing the 1st Mid Atlantic Computer Vision workshop at Virginia Tech), 2014
• Amazon Web Services (AWS) in Education: Machine Learning Research Grant, 2014, 2015
• Windows Azure for Research Award, 2014
• NVIDIA Academic Hardware Gift, 2014

Teaching

• Fall 2017: CS 7643: Deep Learning.
• Spring 2016: ECE 2574: Data Structures and Algorithms.
• Fall 2015: ECE 6504: Deep Learning for Perception. Video lectures available!
• Spring 2015: ECE 5984: Introduction to Machine Learning. All video lectures available!
• Fall 2014: ECE 2574: Data Structures and Algorithms.
• Spring 2014: ECE 6504: Advanced Topics in Machine Learning: Probabilistic Graphical Models.
• Fall 2013: ECE 4984/5984: Introduction to Machine Learning & Perception.
• Spring 2013: ECE 4984/5984: Introduction to Machine Learning & Perception.
• Spring 2012: Co-Instructor for TTIC 31070: Convex Optimization. Primary Instructor: Nati Srebro.

Group

I lead the Machine Learning & Perception Group. Here are the current and past members:

Research Scientist

• Stefan Lee

Students

• Qing Sun, PhD student (2012 — Present)
• Aishwarya Agrawal, PhD student (2014 — Present)
• Yash Goyal, PhD student (2014 — Present)
• Michael Cogswell, PhD student (2015 — Present)
• Abhishek Das, PhD student (2016 — Present)
• Ashwin Kalyan, PhD student (2016 — Present)
• Nirbhay Modhe, PhD student (2017 — Present)

• Akrit Mohapatra, MS student
• Deshraj Yadav, MS student

Co-Advised:

• Ujwal Krothapalli, PhD student (Primary Advisor: Lynn Abott)

Visiting Students

MLP Group Alumni (Graduated Students & Former Interns)

• Aroma Mahendru, MS 2017 (next position: Bloomberg)
• Latha Pemula, MS 2016 (next position: Amazon)
• Gordon Christie, PhD 2016 (next position: Army Research Lab)
• Harsh Agrawal, MS 2016 (next position: Snapchat Research)
• Prakriti Banik, MS 2015 (next position: Bloomberg)
• Neelima Chavali, MS 2015 (next position: Many Trees Inc)
• Clint Solomon, MS 2015 (next position: PhotoKharma)


• Deshraj Yadav, Intern (BTech student from JSS Noida; next position: MS student at GT)
• Tejas Khot, Intern (BTech student from University of Mumbai; next position: MS student at RI, CMU)
• Viraj Prabhu, Intern (BTech student from BITS Pilani; next position: MS student at GT)
• Ashwin Kalyan, Intern (BTech student from NIT Surathkal; next position: PhD student at GT)
• Abhishek Das, Intern (BTech student from IIT-Roorkee; next position: PhD student at GT)
• Khushi Gupta, Intern (BTech student from IIT-Gowahati; next position: MS student at RI, CMU)
• Avi Singh, Intern (BTech student from IIT-Kampur; next position: PhD student at UC Berkeley)
• Ahmed Osman, Intern (MS student from UManchester; next position: PhD student at VT)
• Faruk Ahmed, Intern (BTech student from IEM; next position: PhD student at UMontreal)
• Senthil Purushwalkam, Intern (BTech student from IIT-G; MS student at RI, CMU)
• Aroma Mahendru, Intern (BTech student from IIT-BHU; next position: MS student at VT)
• Ratnesh Kumar, Intern (PhD student from INRIA; next position: MERL)
• Qi Lou, Intern (MS student from OSU; next position: PhD student at UC Irvine)
• Ankit Laddha, Intern (BTech student from IIT-D; next position: MS student at RI, CMU)
• Adarsh Prasad, Intern (BTech student from IIT-D; next position: PhD student at UT-Austin)
• Harsh Agrawal, Intern (BTech student from DTU; next position: MS student at VT)
• Abhimanyu Dubey, Intern (BTech student from IIT-D)

Former Student Collaborators

• Payman Yadollahpour (PhD student from TTIC)
• Kun Duan (PhD student from Indiana U)
• Abner Guzman-Rivera (PhD student from UIUC)
• Daozheng Chen (PhD student from UMD)
• Jason Lew (MEng student at Cornell)
• Tanmay Verma (BTech student from IIIT-D)
• Charvi Puri (BTech student from IIIT-D)
• Kevin Tang (BS student at Cornell, now grad student at Stanford)

Talks

On the web

• Visual Question Answering
  Keynote talk, Multimodal Machine Learning Workshop at NIPS, Dec 2015.
  
  
• Visual Question Answering
  Data Science Summit & Dato Conference, 2015.
  
  
• Visual Question Answering
  Deep Learning Summit, 2015.
  
  
• Hedging Against Uncertainty via Multiple Diverse Predictions
  Microsoft Research Talk, 2014.
  
  
• Submodular meets Structured: Finding Diverse Subsets in Exponentially-Large Structured Item Sets
  NIPS Workshop on Discrete and Combinatorial Problems in Machine Learning (DISCML), 2014.
  
  
• Hedging Against Uncertainty via Multiple Diverse Predictions
  CESCA Seminar, 2014.
  
  
• Beyond MAP: Making Multiple Predictions from Probabilistic Models: Diversity, DPPs and more
  Tutorial at the Conference on Computer Vision and Pattern Recognition (CVPR) 2013.
  
  
• Diverse M-Best Solutions in Markov Random Fields
  European Conference on Computer Vision (ECCV) 2012.
  
  
• Structured-Output Models for Computer Vision
  Microsoft Research Talk, 2012.
  
  
• Beyond Trees: MRF Inference via Outer-Planar Decomposition
  Microsoft Research Talk, 2009.
  
  

Slides from talks

• A tale of two negative results [slides (pptx)]
• Towards AI agents that can see, talk, and act [slides (pptx)]
• Visual Dialog [slides (pptx)]
• Towards Transparent VQA Models [slides (pptx)]
• Visual Question Answering [slides (pptx)]
• Submodular meets Structured: Finding Diverse Subsets in Exponentially-Large Structured Item Sets.
  [ slides (pptx) ]
• M-Best and Diverse M-Best MAP Inference in Graphical Models [ slides ]
• CloudCV: Large-Scale Distributed Computer Vision as a Cloud Service [ slides ]
• Hedging Against Uncertainty via Multiple Diverse Predictions [ slides (pptx) ]
• Structured-Output Models for Computer Vision
• The M-Best Mode Problem [ slides (pptx) ]
• Focused Inference and the M-Best Mode Problem [ slides (pptx) ]
• Focused Inference with Local Primal-Dual Gaps [ slides (pptx) ]
• On Graph-Structured Discrete Labelling Problems in Computer Vision [ slides (pptx) ]
• Beyond Trees: MRF Inference via Outer-Planar Decomposition [ slides (pptx) ]