Nimit Kalra

Hey there! I'm a researcher at Haize Labs, focusing on improving adversarial robustness and automated model-based evaluation for LLMs. I authored Verdict, a framework for specifying compound LLM judge systems. Before that, I spent three years at Citadel.

During my time at UT Austin, I was advised by Philipp Krähenbühl on domain adaptation in computer vision and robotics, with a particular emphasis on data efficiency. I graduated with a B.A. in Math, a B.S. in Computer Science, and a bunch of credits in Political Science and Economics.

Publications

Verdict: A Library for Compound LLM Judge Systems
Nimit Kalra, Leonard Tang
[arXiv]   [code]   [docs]

Open-source library for scaling inference-time compute of LLM-as-a-judge systems by constructing arbitrary reasoning trace shapes. We achieve SOTA or near-SOTA performance on a wide variety of challenging automated evaluation tasks with no additional training.

Constitutional Classifiers: Defending against Universal Jailbreaks...
Mrinank Sharma, … Nimit Kalra, … Anthropic Safeguards Research Team
[arXiv]   [blog]

We introduce Constitutional Classifiers, a framework that trains classifier safeguards using explicit constitutional rules. Our output classifiers support streaming prediction: they assess the potential harmfulness of the complete model output at each token without requiring the full output to be generated.

Domain Adaptation Through Task Distillation
ECCV 2020
Brady Zhou*, Nimit Kalra*, Philipp Krähenbühl
[arXiv]   [code]   [presentation]

Domain adaptation framework for transferring tasks between visually-diverse domains. We successfully transfer agents that navigate mazes and race karts to drive autonomously in a photorealistic simulator.

Writing

Bootstrapping Suprvision — June 2025

squeezing it all out

Projects

[report]   Statement of Purpose for Computer Science Ph.D. Programs

[report]   Domain Adaptation Through Multi-Task Distillation via Noisy-Labels

[report]   A Bayesian Network Model for Sampling Dockless Scooter Traffic

[report]   [code]   Fast Random Kernelized Features: High-Dimensional SVM Classification

[report]   [slides]   Composition of Real Flows

Adventures

I enjoy a good road trip.

People

People who have had a major impact on me — whether a sparring buddy, mentor, or friend.

JagathAlexSrujayLeonardPhilippBradyYuweiSaakethDylanWillPrateek

Contact

I love meeting new people. Reach me at nimit@utexas.edu or schedule a chat.