What are we looking for in applicants:
The ideal candidates should be conversant in fundamental mathematical topics such as probability and statistics, linear algebra, multivariable calculus, convex optimization etc. They must be proficient in Python, NumPy/SciPy, Scikit-learn and should possess at least beginner’s knowledge in PyTorch or TensorFlow. Prior hands-on experience with machine learning and/or computer vision projects will be highly desirable.
Physics-Grounded and Geometry-Preserving Video Generation🔎 Details: Developing a physics-grounded video generation framework that lifts 2D scenes into 3D representations to model realistic object and agent interactions. The framework enforces physical constraints to ensure plausible dynamics, preserves object geometry, and maintains object and agent fidelity across generated frames.
📖 Suggested reading:
Perpetual Wonder: https://arxiv.org/abs/2602.04876
Code Video Model: https://code-video-model.github.io/tech_report.pdf
PHYSLDM: https://arxiv.org/pdf/2610.07609
Video Demonstrations from ‘Code Video Model’ and ‘Perpetual Wonder’ works.
🔗 Contact person for more details: Kunsh Taneja (Mail: kunshtaneja@iisc.ac.in) and Shyam Marjit (Mail: shyammarjit@iisc.ac.in)
Mixture-of-Experts Architectures for Universal Multimodal Embedding Models🔎 Details: Universal embedding models today are mostly dense VLMs fine-tuned with contrastive loss. They trade capacity across modalities and tasks, and gains on one task (e.g. visual document retrieval) often cost another (e.g. text-only retrieval). MoE offers a way to add capacity where modalities and tasks conflict without raising per-query inference cost.
📖 Suggested reading:
MoEMB: https://arxiv.org/pdf/2609.08663
MoE: https://arxiv.org/abs/2503.07137
🔗 Contact person for more details: Anish Chaudhary (Mail: anishc@iisc.ac.in)
Agentic Model Surgery for VLMs🔎 Details: Agentic model surgery treats a vision-language model’s (VLM) weights and training code as programs that a frontier agent can read, write and verify, using only a budget of forward passes and probe images it generates itself. Where prior work tunes a training run (Reactive Optimizers) or removes a failure mode from a language model (Weight-Space Programming), this area asks the agent to add missing visual skills, move skills between models, repair models during deployment, forget targeted concepts, and audit models from their weights alone. Its central claim is that for vision an agent can write new, human-readable capability into a VLM without gradient training.
📖 Suggested reading:
Sparks of Weight-Space Programming: https://rikiyat.github.io/blog/weight-space-programming/
Reactive Optimizers: https://arxiv.org/pdf/2609.08663
🔗 Contact person for more details: Shyam Marjit (Mail: shyammarjit@iisc.ac.in)
Gradient Inversion Attacks on Vision-Language Models under Federated Parameter Efficient Fine-Tuning Framework🔎 Details: Early gradient inversion attacks on Vision-Language Models (VLMs) focused primarily on simple classification tasks using fixed category templates. Modern frameworks like MMGIA expand to complex tasks such as image captioning and medical report generation, but they rely on alternating two-stage optimization loops that require hundreds to thousands of backpropagation iterations, creating a severe computational time bottleneck. Crucially, existing methods assume full-parameter fine-tuning, leaving gradient inversion under Parameter-Efficient Fine-Tuning (PEFT), such as LoRA, which dominates resource-constrained FL, entirely unexplored. We intend to investigate a compelling research direction: To what extent are multimodal Vision-Language Models (VLMs) vulnerable to gradient inversion attacks when fine-tuned using Parameter-Efficient Fine-Tuning (PEFT) on single or dual modalities in Federated Learning?
📖 Suggested reading:
MMGIA: https://ijcai.org/proceedings/2025/886
DAGER: https://arxiv.org/pdf/2405.15586
GIAS: https://arxiv.org/pdf/2110.14962
🔗 Contact person for more details: Dr. Siladittya Manna (Mail: siladittyam@iisc.ac.in) and Ritik Kumar Badiya (Mail: ritikbadiya@iisc.ac.in)
Federated Learning with Noisy Labels🔎 Details: Despite advances in federated learning with noisy labels, existing approaches often address label-noise estimation, sample correction, and client-aware aggregation without fully accounting for their interplay with severe non-IID data distributions. In particular, distinguishing client-specific annotation corruption from legitimate distributional heterogeneity remains challenging, potentially leading to erroneous label correction or the suppression of informative clients. This research will address this gap by jointly estimating label reliability and client heterogeneity and incorporating these estimates into noise-aware local optimization and global aggregation, without requiring clean server-side data or sharing raw data.
📖 Suggested reading:
Federated learning with noisy labels Survey: https://www.sciencedirect.com/science/article/pii/S0893608026003503
🔗 Contact person for more details: Dr. Siladittya Manna (Mail: siladittyam@iisc.ac.in)
Foley-Evaluation Metric🔎 Details: For video-to-audio generation, causality between visual events and their corresponding sounds should be preserved. This project aims to develop a metric that evaluates whether the generated sound is causally triggered by the event occurring in the video, rather than merely being associated with or conditioned on the presence of that event.
📖 Suggested reading:
AVBench: https://arxiv.org/pdf/2605.24652
Foley Bench: https://arxiv.org/pdf/2511.13219
🔗 Contact person for more details: Akanksha Sharma (Mail: akanksha1@iisc.ac.in) and Ishan Tiwari (Mail: ishan.tiwari.it.jsr@gmail.com)
Towards Efficient Text/Image-to-3D Generation and Editing🔎 Details: Today’s 3D generators, from diffusion models to code-writing language agents, produce objects and scenes that look plausible but are costly, hard to verify and prone to structural or physical errors. This work aims to generate 3D content that is affordable, physically valid and decomposed into meaningful semantic parts that can be moved or edited easily.
📖 Suggested reading:
ShapeCraft: https://arxiv.org/pdf/2510.17603
Articraft: https://arxiv.org/pdf/2605.15187
🔗 Contact person for more details: Anay Loya (Mail: anayloya@iisc.ac.in) and Soumya Panja (Mail: soumyapanja@iisc.ac.in)
When Does Memorization Get Cheaper? A Fisher-Geometry View🔎 Details: Neural networks can fit almost anything, even random labels. A recent paper, Fisher Information and the Geometry of Memorization in Neural Networks (Bernstein et al., NeurIPS 2026) shows that you can tell apart two things inside a trained network using Fisher information: the general patterns it learns, and the individual examples it just memorizes. These live in different directions of the weights. An older paper, Measuring the Intrinsic Dimension of Objective Landscapes (Li et al., ICLR 2018), noticed something strange: the more examples a network memorizes, the cheaper each new one becomes. Nobody has explained why yet. In this project we want to understand how a trained network makes room for new examples it can only memorize. Does each example need its own space in the weights, or can examples share it? And can we predict this cost just by looking at the Fisher information of the trained network, before training on the new examples?
So far this has only been studied for networks trained from scratch. We also want to look at fine-tuning and especially PEFT methods that only allow small, restricted updates, and see whether memorizing new data there comes at the cost of forgetting what the model already knew.
📖 Suggested reading:
D. T. Bernstein, C. Goddard, L. Di Carlo, V. Ngampruetikorn, D. J. Schwab. Fisher Information and the Geometry of Memorization in Neural Networks. NeurIPS 2026.
C. Li, H. Farkhoor, R. Liu, J. Yosinski. Measuring the Intrinsic Dimension of Objective Landscapes. ICLR 2018.
C. Zhang, S. Bengio, M. Hardt, B. Recht, O. Vinyals. Understanding Deep Learning Requires Rethinking Generalization. ICLR 2017.
J. Attri, F. Guth. Memorization in Image Classification: Test Set Behavior and Scaling Laws. arXiv 2026.
J. Martens. New Insights and Perspectives on the Natural Gradient Method. JMLR 2020. (Good background on Fisher information.)
🔗 Contact person for more details: Shraddha Chavan (Mail: shraddha99.iisc@gmail.com)