Machine Learning Seminar Series | Story of 3 D’s: Deep Image Prior, Diffusion Models, and Deep Linear Networks

Abstract: In this talk, first, I will dive into a purely unsupervised approach for inverse problems called deep image prior (DIP) that recovers images from limited/corrupted measurements by learning neural networks at reconstruction time with no training data. Classical DIP suffers from severe overfitting and spectral bias effects. We present an analysis of when and how DIP recovers images from limited/corrupted data by studying its learning dynamics in the kernel regime. We then propose a self-guided DIP that significantly outperforms classical DIP and is competitive against supervised and diffusion models (DMs) trained with a lot of data. Second, DMs, when exhibiting strong image reconstruction performance, often rely on domain-matched pretraining and large training sets. We propose TRACE, a trajectory-constrained reconstruction framework that replaces the pretrained DM with an untrained network optimized like in DIP and organizes reconstruction as a sequence of intermediate states. At each step, TRACE enforces stochastic proximal consistency (SPC) by inferring the reconstruction from a stochastically perturbed predecessor under measurement consistency, while proximally constraining the resulting estimate to its predecessor. A local Jacobian analysis characterizes how SPC regularizes the reconstruction. TRACE demonstrates strong performance against both untrained and pretrained neural-priors with the trajectory constraint improving even pretrained DM-based inverse solvers. Lastly, deep neural networks trained using gradient descent with fixed learning rate often operate in the regime of edge of stability (EOS), where the largest Hessian eigenvalue equilibrates about a stability threshold. Here, we analyze the learning dynamics of deep linear networks (DLNs) in deep matrix factorization beyond EOS, and show that loss oscillations follow a period-doubling route to chaos. Oscillations occur within a small subspace, whose dimension depends on the learning rate. The results help explain two key phenomena in deep networks: shallow models and simple tasks do not always exhibit EOS; and oscillations occur within top features.

Bio: Saiprasad Ravishankar is currently an Associate Professor in the Departments of Computational Mathematics, Science and Engineering and Biomedical Engineering at Michigan State University (MSU). He directs the signals, learning, and imaging (SLIM) research group and the data-driven neuroscience of meditation (DANSOM) lab at MSU. He received the B.Tech. degree in Electrical Engineering from the Indian Institute of Technology Madras, India, in 2008, and the M.S. and Ph.D. degrees in Electrical and Computer Engineering in 2010 and 2014, respectively, from the University of Illinois at Urbana-Champaign, where he was then an Adjunct Lecturer and a Postdoctoral Research Associate. From August 2015 to 2018, he was a postdoc in the Department of Electrical Engineering and Computer Science at the University of Michigan, and then a Postdoc Research Associate in the Theoretical Division at Los Alamos National Laboratory from August 2018 to February 2019, before joining MSU. His research interests include machine learning, computational and biomedical imaging, signal processing, image processing, inverse problems, data science, neuroscience, and physics and astrophysics applications. His research has been recognized with numerous awards including an NSF CAREER Award, IEEE Signal Processing Society Young Author Best Paper Award, and best student paper awards or finalist at numerous conferences such as the IEEE International Symposium on Biomedical Imaging (ISBI) 2018, IEEE International Workshop on Machine Learning for Signal Processing (MLSP) 2017, ISBI 2020, and Optical Imaging Congress 2023.

For CODA guest access, please contact shatcher8@gatech.edu at least 2 business days prior to the event.

While we prefer you join in person, if you are unable to attend, join online at:
https://gatech.zoom.us/j/98109284552?pwd=CrXDIye86vmEtCywAV7wmjjz1LqEY8.1
Meeting ID: 981 0928 4552
Passcode: 947902