Advisor: Prof. Chenliang Xu (Computer Science)

Committee: Prof. Jiebo Luo (Computer Science), Prof. Zhiyao Duan (Computer Science), Prof. Yapeng Tian (UT Dallas, Computer Science)

Chair: Prof. Lisha Chen (Electrical & Computer Engineering)

Recent advances in generative modeling have reshaped machine perception, enabling systems that not only recognize multimodal signals but also synthesize and manipulate them. For example, modern models can generate photorealistic images or clone voices with high fidelity. However, despite this rapid progress, current multimodal models remain fundamentally limited: they often lack spatial grounding, struggle to disentangle overlapping sound sources in complex scenes, and offer only coarse mechanisms for user control. These limitations represent a key bottleneck in moving from passive recognition to active, intelligent content creation.

To address these limitations, this thesis introduces a unified framework for controllable generative modeling, bridging the gap between multimodal perception and synthesis. Specifically, we aim to solve three fundamental challenges: 1) how to ground generative models in physical and spatial reality, 2) how to leverage generative priors for high-fidelity audio–visual separation, and 3) how to implement precise, task-agnostic control mechanisms.

First, I establish perceptual foundations through Egocentric Audio-Visual Localization, which mod- els spatial cues via egomotion, and Acoustic Primitives, a joint-anchored representation of human soundfields. These demonstrate that physical grounding can be learned directly from dynamic multimodal streams. Second, I shift from discriminative separation to generative modeling with DAVIS, a diffusion-based conditional framework, and ZeroSep, a training-free separator leveraging text-to-audio generative priors. Finally, I introduce task-agnostic control mechanisms: VisAH uti- lizes language and video for expressive audio highlighting, while FreSca employs frequency-space manipulation to enable training-free steering across image and video generation.

In summary, the contributions of this thesis chart a coherent progression from physically grounded perception to controllable generative synthesis. The resulting systems can localize, separate, and manipulate multimodal content, laying the foundation for future multimodal agents capable of reasoning over and creating long-form, open-world content.

View on site | Email this event

Abstract:

As formal methods improve in expressiveness and power, they create new opportunities for non-expert adoption. In principle, formal tools are now powerful enough to enable developers to scalably validate realistic systems artifacts without extensive formal training. However, realizing this potential for adoption requires attention to not only the technical but also the human side—which has received extraordinarily little attention from formal-methods research.

This talk presents some of our efforts to address this paucity. We apply ideas from cognitive science, human-factors research, and education theory to improve the usability of formal methods. Along the way, we find misconceptions suffered by users, how technically appealing designs that experts may value may fail to help, and how our tools may even mislead users.

Bio:

Shriram Krishnamurthi is a Professor of Computer Science at Brown University. With collaborators and students, he has created several influential systems like DrRacket, Margrave, Flapjax, LambdaJS, Flowlog, and Pyret. He has also written multiple widely-used books. He also co-directs the Bootstrap integrated computing outreach program. For his work he has received SIGPLAN's Robin Milner Young Researcher Award, SIGPLAN's Software Award (jointly), SIGSOFT's Influential Educator Award, SIGPLAN's Distinguished Educator Award (jointly), and Brown's Wriston and Philip J. Bray teaching awards. He has authored over twenty papers recognized for honors by program committees. He has an honorary doctorate from the Università della Svizzera Italiana.

View on site | Email this event

In addition to providing hands-on opportunities to create something novel, the design projects offer opportunities for students to connect with potential employers, engage in global experiences, and conduct interdisciplinary research with collaborators from places like the University of Rochester Medical Center. Clients who have presented the design capstone teams with problems have been providing input throughout the semester as the students work on solutions under the guidance of their faculty advisors.

View on site | Email this event