Aligning Text, Images, and 3D Structure Token-by-Token
We present a unified LLM that aligns language, images, and structured 3D scenes, demonstrating applications in rendering, recognition, instruction following, and 3D QA.
Aadarsh Sahoo
Ph.D. student at Caltech, advised by Prof. Georgia Gkioxari and Prof. Pietro Perona.
About
I am a Ph.D. student at Caltech advised by Prof. Georgia Gkioxari and Prof. Pietro Perona.
Previously, I was a student researcher at the MIT–IBM Watson AI Lab, pretraining large language models and doing research on vision–language models with Dr. Rameswar Panda, Dr. Rogerio Feris, and Prof. Yoon Kim. I completed a dual degree (B.Tech + M.Tech) at IIT Kharagpur, where I worked in the Computer Vision and Intelligence Research Lab under Prof. Abir Das.
In Summer 2021, I worked with Prof. Kate Saenko (Boston University) and Prof. Trevor Darrell (UC Berkeley) as a research intern for the DARPA LwLL project.
Research
My research interests lie in understanding the principles of learning across multiple modalities and how knowledge transfers between them, with the goal of designing embodied multimodal agents that benefit society. Questions like “Do toddlers use similar principles to learn new languages as they do to learn to walk?” excite me.
News
-
Preprint “Aligning Text, Images, and 3D Structure Token-by-Token (Kyvo)” released on arXiv (project page).
-
“XPL: A Cross-Model framework for Semi-Supervised Prompt Learning in VLMs” accepted by TMLR.
-
Paper on Anytime Domain Adaptation accepted at ICLR 2023.
-
Select, Label, and Mix (SLM) received the Best Paper Honorable Mention at WACV 2023.
-
Paper on Partial Domain Adaptation accepted at WACV 2023.
-
Started as a student researcher at the MIT–IBM Watson AI Lab in Cambridge, MA.
-
Extended Abstract on Partial Domain Adaptation accepted at the NeurIPS DistShift Workshop 2021.
-
Paper on Domain Adaptation in Action Recognition accepted at NeurIPS 2021.
-
Volunteer at the CVPR 2021 workshop Dynamic Neural Networks Meets Computer Vision (DNetCV).
-
Started internship at UC Berkeley and Boston University.
-
Paper accepted at the ECCV 2020 Workshop on Imbalance Problems in Computer Vision (IPCV).
-
Joined CVIR, IIT Kharagpur as an undergraduate researcher in Computer Vision.
-
Selected for Computer Science and Engineering major (<1% acceptance).
-
Admitted to IIT Kharagpur via JEE Advanced 2017.
Publications
XPL: A Cross-Model framework for Semi-Supervised Prompt Learning in Vision–Language Models
A semi-supervised prompt learning framework leveraging unlabeled data to improve VLM adaptation via cross-model consistency.
AnyDA: Anytime Domain Adaptation
We propose a domain-alignment approach with switchable depth, width, and input resolution to realize accuracy–efficiency trade-offs under different constraints.
Mitigating Dataset Imbalance via Joint Generation and Classification
We introduce a dataset repair strategy combining a classifier with a GAN to augment minority-class examples.
Services
Contact
Email: aadarsh.sahoo.99@gmail.com
Profiles: Google Scholar · LinkedIn · GitHub · X/Twitter