Research Projects

Texture Optimization for Example-based Synthesis
 ( Project page | PDF )
Vivek Kwatra, Irfan Essa, Aaron Bobick, and Nipun Kwatra 
To Appear in SIGGRAPH 2005

energy plot    Keyboard flowing like smoke

We present a novel technique for texture synthesis using optimization. We define a Markov Random Field (MRF)-based similarity metric for measuring the quality of synthesized texture with respect to a given input sample. This allows us to formulate the synthesis problem as minimization of an energy function, which is optimized using an Expectation Maximization (EM)-like algorithm. In contrast to most example-based techniques that do region-growing, ours is a joint optimization approach that progressively refines the entire texture. Additionally, our approach is ideally suited to allow for controllable synthesis of textures. Specifically, we demonstrate controllability by animating image textures using flow fields. We allow for general two-dimensional flow fields that may dynamically change over time. Applications of this technique include dynamic texturing of fluid animations and texture-based flow visualization.

Graphcut Textures: Image and Video Synthesis Using Graph Cuts
( Project page )
Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick
Proc. ACM Transactions on Graphics, SIGGRAPH 2003
In this paper we introduce a new algorithm for image and video texture synthesis. In our approach, patch regions from a sample image or video are transformed and copied to the output and then stitched together along optimal seams to generate a new (and typically larger) output. In contrast to other techniques, the size of the patch is not chosen a-priori, but instead a graph cut technique is used to determine the optimal patch region for any given offset between the input and output texture. Unlike dynamic programming, our graph cut technique for seam optimization is applicable in any dimension. We specifically explore it in 2D and 3D to perform video texture synthesis in addition to regular image synthesis. We present approximative offset search techniques that work well in conjunction with the presented patch size optimization. We show results for synthesizing regular, random, and natural images and videos. We also demonstrate how this method can be used to interactively merge different images to generate new scenes.

2D Cel Animation Compression
( Project page )

Space-Time Surface Simplification and Edgebreaker Compression for 2D Cel Animations

Vivek Kwatra and Jarek Rossignac
International Journal on Shape Modeling, Volume8, Number 2, December 2002

Surface Simplification and Edgebreaker Compression for 2D Cel Animations

Vivek Kwatra and Jarek Rossignac
Proc. International Conference on Shape Modeling and Applications (SMI 2002)

Digitized cel animations are typically composed of frames, which contain a small number of regions, which each contain pixels of the same color and exhibit a significant level of shape coherence through time. To exploit this coherence, we treat the stack of frames as a 3D volume and represent the evolution of each region by the bounding surface of the 3D volume V that it sweeps out. To reduce transmission costs, we triangulate and simplify the bounding surface and then encode it using the Edgebreaker compression scheme. To restore a close approximation of the original animation, the client player decompresses the surface and produces the successive frames by intersecting V with constant-time planes. The intersection is generated in real-time with standard graphics hardware through an improved capping (i.e. solid clipping) technique, which correctly handles overlapping facets. We have tested this approach on real and synthetic black&white animations and report compression ratios that improve upon those produced using the MPEG, MRLE, and GZIP compression standards for an equivalent quality result.

Temporal Integration of Multiple Silhouette-based Body-part Hypotheses
( Project page )
Vivek Kwatra, Aaron F. Bobick, and Amos Y. Johnson
Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2001)

A method for temporally integrating appearance-based body-part labelling is presented.  We begin by modifying the silhouette labelling method of Ghost; that system first determines which posture best describes the person currently and then uses posture-specific heuristics to generate labels for head, hands, and feet. Our approach is to assign a posture probability and then estimate body part locations for all possible postures. Next we temporally integrate these estimates by finding a best path through the posture-time lattice. A density-sampling propagation approach is used that allows us to model the multiple hypotheses resulting from consideration of different postures.  We show quantitative and qualitative results where the temporal integration solution improves the instantaneous estimates.  This method can be applied to any system that inherently has multiple methods of asserting instantaneous properties but from which a temporally coherent interpretation is desired.