Facebook Project Ideas

Project 1: Confident Machine Translation

Motivation:
 What problem did this project try to solve?

In real applications, we would like a system to abstain from making a prediction when its confidence is too low. Unfortunately, machine translation systems are not well calibrated when they are trained on small amounts of data, as it is typical in low resource language pairs. The aim of this project is to enable a machine translation system to give up translating whenever its generations have insufficient quality.


Who cares? If you are successful, what difference will it make?

Whoever deploys MT systems in the real world cares about this, as the cost of not showing a translation is lower than the cost of showing a poor translation.


Approaches:
 How is the problem approached today, and what are the limits of current practice?

There is not much literature on this. Usually people use a threshold on the model logprobabilities with a threshold set on the validation set. 
 - What are the baselines? Is there any potential approaches students can start with, without major change from the baselines?


What’s the availability of the baselines? Is there any standard implementation, and are they open-sourced?

Students can start by using model log-probabilties (average per token, min, median, max) and plot sort of precision/recall curves (fraction of sentences translated VS BLEU).

Students can also explore training methods that yield better calibrated models, e.g., by regularizing more or by training ensembles. 


Metrics:
 What are the common datasets and benchmarks?
 What’s the availability of the datasets? Are they open-sourced? Is there any restriction regarding how students should use them?
 What is the state-of-the-art approach, if applicable?

Students can use data and models from: https://github.com/facebookresearch/flores 


Scope:
 What are possible directions that students could explore? E.g. In terms of modeling, data, efficient computation.

After baseline are tested, I’d recommend testing fancier decoding approaches like noisy channel decoding, and ad-hoc quality predictors.


How much work in each direction would justify a good grade?


A comprehensive analysis of different decoding, quality prediction methods on the BLEU vs. Fraction of translated sentences would be a nice achievement.

If students figure out a way to beat the baselines, this could turn into a paper.

What is the ideal size for a team tackling this project?

A couple of motivated people would suffice. 


Resources:
 - Are there any open datasets the students can train with?


Yes, FLORES.

What are the computing resources required to compute a baseline? (CPU/GPU days)

Not much if we do not ask them to retrain models, but perhaps train small predictors.

Project 2: Habitat Embodied Navigation Challenge @ CVPR20

FB AI/FAIR has created a fast photo-realistic simulator called Habitat (https://aihabitat.org/). The team organized an autonomous navigation challenges last year at CVPR19, and is planning to organize two more challenges this year – (1) PointNav – navigate to a point and (2) ObjectNav – find an object (“find the chair”) – both in unseen environments without access to a map.

Starter code is provided. Task is well-defined. Timeline is fairly aligned (deadline around early May / end of semester). Good testbed to try techniques learned in class (CNNs for egocentric perception, RNNs for recurrent policy, RL for navigation).

Project 3: MRI analysis

####GT Proposal - Attentive Reconstruction

####Motivation

Magnetic resonance imaging (MRI) scans are one of the most powerful imaging modalities for medical image diagnosis due to their adaptability and unparalleled soft tissue contrast. However, in contrast to imaging techniques such as x-ray CT and ultrasound, MRI scans have long acquisition times, with protocols often taking as long as an hour. The long scan time of MRI is a primary driver of the large monetary cost for MRI examinations. As such, shortening MRI examinations - while maintaining its high image quality and soft tissue contrast - is a topic of significant interest to the medical community. Decreasing the length of scans could decrease the cost, which would broaden access and may even allow MRI’s use in new applications where other imaging modalities are the current standard. For MR imaging accelerations, deep neural networks (DNNs) are the current state-of-the-art. DNNs are typically trained with metrics such as mean-squared error calculated over the entire image, but in many medical applications (e.g., knee imaging) pathology is highly localized, with accuracy in one area being more valuable than accuracy in another.

####Approaches

The reason for MRI scan length is the need to sample spatial frequencies of the object. The object’s spatial frequency representation - called “k-space” - must be fully sampled into order to avoid classical Shannon-Nyquist aliasing. The field of MRI reconstruction has developed a number of techniques for avoiding the aliasing while reducing sampling. The most common technique (and only technique with current clinical adoption) is to use parallel receive channels [1, 2, 3]. In this setting, the linear inverse problem becomes overdetermined even with Fourier undersampling, and an image can be reconstructed. Parallel imaging can be further improved with regularized approaches like compressed sensing, but compressed sensing uses handcrafted mathematical features of images for optimization that don’t represent the features of the actual data. Due to this failing, more recently deep learning methods have supplanted compressed sensing [4, 5]. Deep learning methods learn more natural image features, typically encoding them as weights in deep neural networks (DNNs). The simplest deep learning models applied to image reconstruction include U-Nets and ResNets. However, these image-to-image models neglect a lot of other information that can be leveraged via MRI physics equations. Hybrid approaches involving parallel imaging data models with DNN-based priors are the current state-of-the-art methods for accelerated MR image reconstruction, and a number of such approaches are currently being used for adoption in the clinic.

####Metrics

The largest data set for MR image reconstruction is the fastMRI data set released by Facebook AI Research and NYU Langone Health (https://fastmri.med.nyu.edu/), which includes data for both knee and brain imaging. Other data sets include the Calgary-Campinas Public Brain MR Dataset (https://sites.google.com/view/calgary-campinas-dataset/home) and mridata.org (http://mridata.org/). An important aspect of these data sets over other data sets is their inclusion of raw k-space data. Many medical imaging data sets only release DICOM image-format data, which discards most of the information in the raw k-space and warps the image statistics. Deep learning models trained at least in part on raw k-space data are the only acceptable models for MR image reconstruction. Metrics for training MR reconstruction methods include standard reference distance metrics such as mean-squared error (MSE) and structural similarity (SSIM). However, these metrics don’t exactly correlate with clinical quality, so the final validation step typically involves a human reader study with certified radiologists.

####Scope

Although SSIM has been found to correlate with radiologist image quality impressions, computation of SSIM over the entire image may overweight regions of the image that are not diagnostically relevant. The goals of this project include the following:

1. Determine regions of interests in knee scans that have the most diagnostic content.
2. Given a region of interest, can DNNs be trained to prioritize this area over another?
3. If (2) is true, then what is the best DNN architecture to detect and maximize accuracy in a given region?

The included raw k-space data sets do not include annotations for regions of interest, but some other public data sets have diagnosis information that can be converted to heat maps, e.g., https://stanfordmlgroup.github.io/projects/mrnet/. Knee imaging might be a good starting point for region-of-interest detection, as it is often the central join region where pathology occurs. The simplest implementation for this project would be to take a model such as the U-Net from the fastMRI repository and train it while upweighting the MSE loss from the middle of a knee image. Extensions that students may be able to implement would be simple feature-based detection of the knee joint region and optimization of metrics within that region. It would also be expected for the students to perform a corresponding visual quality assessment and documentation. Extremely successful projects would accomplish all of these goals and follow-up with a radiologist reader study and possibly a publication.

####Resources

A basic primer on MRI can be found in the fastMRI arXiv paper: https://arxiv.org/abs/1811.08839.

For training, students can use the knee portion of the fastMRI data set, available at https://fastmri.med.nyu.edu/.

To make prototyping easier, the fastMRI data set includes single-coil as well as multicoil data. Another data set with annotations that might be useful, depending on approach, is at https://stanfordmlgroup.github.io/projects/mrnet/.

Baseline U-Net models for fastMRI data are implemented in the fastMRI repository at https://github.com/facebookresearch/fastMRI.

Training of both the baseline U-Net as well as state-of-the-art unrolled models typically requires 3 days of training on an Nvidia GPU with 16 GB of memory.

####References

1. Sodickson, Daniel K., and Warren J. Manning. “Simultaneous acquisition of spatial harmonics (SMASH): fast imaging with radiofrequency coil arrays.” Magnetic resonance in medicine 38.4 (1997): 591-603.

2. Pruessmann, Klaas P., et al. “SENSE: sensitivity encoding for fast MRI.” Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 42.5 (1999): 952-962.

3. Griswold, Mark A., et al. “Generalized autocalibrating partially parallel acquisitions (GRAPPA).” Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 47.6 (2002): 1202-1210.

4. Hammernik, Kerstin, et al. “Learning a variational network for reconstruction of accelerated MRI data.” Magnetic resonance in medicine 79.6 (2018): 3055-3071. 5. Schlemper, Jo, et al. “A deep cascade of convolutional neural networks for dynamic MR image reconstruction.” IEEE transactions on Medical Imaging 37.2 (2017): 491-503.

Project 4: Transfer learning for machine translation quality estimation (Lucia Specia and Paco Guzman)

See description document.

© 2020 Georgia Tech