Resource-constrained image generation and visual understanding: An interview with Aniket Roy

In the latest installment of our series of interviews with AAAI/SIGAI PhD consortium participants, we spoke with Aniket Roy to learn more about his work on generative models for computer vision tasks.

Tell us a little about your PhD – where did you study and what was the topic of your research?

I recently completed my PhD in Computer Science at Johns Hopkins University, working under the supervision of Bloomberg Distinguished Professor Rama Chellappa. My research primarily focused on developing methods for resource-constrained image generation and visual understanding. In particular, we considered how modern generative models can be adapted to operate efficiently while maintaining strong performance.

During my PhD, I worked extensively at the intersection of generative AI, multimodal learning, and few-shot learning. Much of my work involved designing techniques that enable models to learn new concepts or perform complex visual tasks with limited data and computational resources. This includes research on diffusion models, personalized image generation, and multimodal representation learning. Overall, the goal of my work is to make advanced vision and generative AI systems more adaptable, efficient, and practical for real-world applications.

Could you give us an overview of the research you conducted during your doctoral program?

During my PhD, my research broadly focused on improving the adaptability, efficiency, and quality of modern generative models for computer vision tasks. Rapid advances in generative AI, particularly diffusion and visual language models, have created new opportunities to address long-standing challenges such as data scarcity, controllable generation, and personalized image synthesis. The goal of my research is to develop methods by which these large-scale models can be effectively adapted with limited data and computational resources while maintaining high visual fidelity.

One line of my research dealt with learning in data-constrained environments. For example, I proposed FeLMi, a few-shot learning framework that leverages an uncertainty-based hard mixup strategy to improve robustness and generalization when only a small number of labeled samples are available. Building on this idea of ​​improving the quality of training data, we also developed Cap2Aug, which introduces caption-guided multimodal augmentation. This approach uses text descriptions to guide the generation of synthetic images, increasing visual diversity while reducing the area gap between real and generated data.

Cap2Aug Overview

Another aspect of my research focused on improving the perceived quality of images produced by diffusion models. In this direction, I proposed DiffNat, a plug-and-play regularization method based on the kurtosis-density property observed in natural images. By incorporating this principle into the diffusion model through KC loss, the generated images exhibit more natural texture statistics and increased perceptual realism, which also benefits downstream visual tasks.

A large part of my work has been exploring personalization and efficient adaptation of large-scale generative models. We have introduced DuoLoRA. It is a parameter-efficient framework for configuring low-rank adapters, allowing fine-grained control over content and style without requiring a complete retraining of the base model. Additionally, we extended personalization to zero-shot settings using a training-free text inversion approach that allows customization of arbitrary objects directly during generation. Finally, I proposed MultiLFG, a frequency-guided multi-LoRA synthesis framework that enables accurate and training-free fusion of multiple concepts in diffusion models using wavelet domain representations and timestep-aware weighting.

Overview of DuoLoRA.

Overall, my research contributes to building more efficient, adaptive, and controllable generation systems that enable the generation and understanding of high-quality images even in data-limited and resource-constrained scenarios.

Were there any particularly interesting projects or aspects of your research?

One of the projects I found particularly interesting during my PhD was DiffNat, which was published at TMLR 2025. Diffusion models are the backbone of many modern generative AI systems and have achieved impressive results in generating and editing realistic images. However, improving the perceived quality and naturalness of the generated images remains an important challenge.

Overview of DiffNat.

In this study, we introduced a simple but effective regularization technique called kurtosis concentration (KC) loss. It can be integrated into standard diffusion model pipelines as a plug-and-play component. This idea was inspired by the statistical properties of natural images. That is, when an image is decomposed into different bandpass filtered versions, for example using a discrete wavelet transform, the kurtosis values ​​across these frequency bands tend to be relatively consistent. In contrast, the generated images often show large discrepancies across these bands. Our method reduces the gap between the highest and lowest kurtosis values ​​across frequency components, encouraging the generated images to follow more natural image statistics.

Furthermore, we introduced a condition-independent perceptual guidance strategy during inference that further improves image fidelity without the need for additional training signals. We evaluated our approach across several diverse tasks, including personalized few-shot fine-tuning with text guidance, unconditional image generation, image super-resolution, and blind-face recovery. Across these tasks, incorporating KC loss and perceptual guidance consistently improved perceptual quality as measured through metrics such as FID and MUSIQ and human ratings.

What I particularly liked about this project is that it combines classic image statistics with modern diffusion models. This shows that relatively simple statistical insights about natural images can still play a powerful role in improving large-scale generative models.

What plans do you have with your PhD? Where are you currently working and what are you planning to research next?

During my PhD, I realized that I really enjoyed the research process, especially the moments when a intuition or idea turned out to work. The process of exploring new ideas and pushing known boundaries is extremely motivating for me.

To continue pursuing this goal, I decided to join NEC Laboratories America as a researcher. In this role, I hope to build on my PhD research by developing new methods for generative models and exploring how these models can interact with broader multimodal systems. I am particularly interested in advancing research at the intersection of generative models, visual/verbal/behavioral models, and embodied AI. More broadly, my goal is to contribute to the development of intelligent systems that can more effectively understand, generate, and interact with the visual world, while continuing to advance the scientific understanding of these models.

I’m curious how you got into this field. What inspired you to study computer vision and machine learning?

My interest in computer vision and machine learning began during my undergraduate years when I took a course in signal processing and image processing. I found these subjects particularly appealing because they allow me to experiment with algorithms and immediately see their effects on images. Its visual and intuitive aspects made the field very appealing and helped me understand how mathematical concepts directly translate into meaningful visual results.

At the same time, I was also interested in how the human brain processes visual information: how we can recognize objects, understand scenes, and easily interpret complex visual signals. That curiosity led me to wonder if it was possible to design computational models that mimic aspects of human perception and allow machines to understand visual data in a similar way.

A big influence during this period was my professor, Dr. Kuntal Ghosh, who encouraged me to think more deeply about these issues and approach them with a scientific mindset. His guidance played an important role in shaping my research interests. Since then, my curiosity about visual recognition and intelligent systems has continued to drive my research in computer vision and machine learning.

How was your experience with AAAI’s PhD consortium?

Unfortunately, due to visa-related issues, I was unable to directly participate in the AAAI Doctoral Consortium. However, a colleague was kind enough to help present a poster on my behalf during the event. Although I couldn’t physically be there, I was very encouraged by the response my work received. Several researchers who saw the poster contacted me and we had some very insightful discussions about research ideas and potential future directions. In that sense, I think this experience was very rewarding. The Doctoral Consortium is a great platform to share early-stage ideas, receive feedback from the community, and connect with other researchers working on related problems. I appreciated the opportunity to engage with people interested in this work, and those interactions helped spark new perspectives and collaborations.

Can you tell us an interesting fact (unrelated to AI) about yourself?

Outside of research, I’m a big fan of music and stand-up comedy, and I really enjoy traveling whenever I get the chance. Exploring new places, cultures, and perspectives is refreshing for me and a great way to recharge and stay curious about the world beyond work. I also like to write poetic satire from time to time, and I sometimes show it off. This is a fun creative outlet that allows me to combine humor and storytelling, and is quite different from the analytical nature of the research work I usually do.

About Aniket Roy

Aniket is currently a researcher at NEC Labs America. He received his PhD from the Department of Computer Science at Johns Hopkins University under the direction of Professor Rama Chellappa, Bloomberg Distinguished Professor. Prior to that, he completed his master’s degree from Indian Institute of Technology, Kharagpur. He won the Best Paper Award at IWDW 2016 and the Marcos Tomas Memorial Award for the Best Research Paper at the Master’s level. During his PhD, he studied the areas of few-shot learning, multimodal learning, diffusion models, LLM, and LoRA combined with publications in key areas such as NeurIPS, ICCV, TMLR, WACV, and CVPR, and also filed three US patents. During my doctoral studies, I also gained industrial experience through multiple internships at Amazon, Qualcomm, MERL, and SRI International. He was honored as an Amazon Fellow (2023-24) at JHU and was selected to participate in the ICCV’25 and AAAI’26 doctoral consortiums.

---