Nimit Kalra

Bootstrapping Supervision for LLM Self-Improvement — July 2025
High-quality reasoning supervision is expensive — not just in dollars, but in latency, throughput, and annotation consistency. Human-labeled traces are noisy, subjective, and particularly hard to scale for non-general domains, where instructions are idiosyncratic and objective ground truths are often elusive. As a result, many recent works attempt to bootstrap supervision directly from the pretrained model, leveraging it as a strong prior over reasoning behaviors.

Verdict: A Library for Compound LLM Judge Systems
Nimit Kalra, Leonard Tang
[arXiv]   [code]   [docs]

Open-source library for scaling test-time compute via graphs of chained prompted evaluators. We achieve SOTA/near-SOTA performance on a wide variety of challenging automated evaluation tasks without additional training or resorting to specification/prompt overfitting.

Constitutional Classifiers: Defending against Universal Jailbreaks
Anthropic Safeguards Research Team
[arXiv]   [blog]

Synthetic data recipe for training output classifiers with streaming prediction to flag harmful content according to an explicit constitution. Focus on adversarial data augmentation and red-teaming.

Domain Adaptation Through Task Distillation, ECCV 2020
Brady Zhou*, Nimit Kalra*, Philipp Krähenbühl
[arXiv]   [code]   [presentation]

We leverage dense vision labels (e.g., segmentation masks, which are freely available in simulators) to transfer navigation policies across visually-diverse domains (maze navigation → autonomous driving). By training a policy that operates on labels, we can obtain action supervision in a new domain and distill an end-to-end visuomotor policy.