Hey there! I'm a researcher at Haize Labs, focusing on automated model-based evaluation. I authored Verdict, a framework for specifying compound LLM judge systems. Before that, I spent three years at Citadel.
During my time at UT Austin, I collaborated with Philipp Krähenbühl on domain adaptation in computer vision and robotics, with a particular emphasis on data efficiency. I graduated with a B.A. in Math and a B.S. in Computer Science, along with some extra credits in Political Science and Economics.
Verdict: A Library for Compound LLM Judge Systems Open-source library for scaling inference-time compute of LLM-as-a-judge systems by constructing arbitrary reasoning trace shapes. We achieve SOTA or near-SOTA performance on a wide variety of challenging automated evaluation tasks with no additional training. |
|
Constitutional Classifiers: Defending against Universal Jailbreaks... We introduce Constitutional Classifiers, a framework that trains classifier safeguards using explicit constitutional rules. Our output classifiers support streaming prediction: they assess the potential harmfulness of the complete model output at each token without requiring the full output to be generated. |
|
Domain Adaptation Through Task Distillation Domain adaptation framework for transferring tasks between visually-diverse domains. We successfully transfer agents that navigate mazes and race karts to drive autonomously in a photorealistic simulator. |
A Question for Cory Booker (D-NJ) — June 2022 I had the opportunity to ask — "Some of the most beautiful and compelling ideas that we cherish in our nation today were at one point quite controversial and unpopular. As you mentioned, the Civil Rights movement was not a result of Senators in suits deciding to grant rights — it was the hard work of passionate citizens who convinced a nation with their ideas. And yet, there was a time when a majority would’ve surely preferred to censor such ideas. What do you see as the future of social media regulation and how can we protect free debate online?" |
21st Century Guard Labor — April 2022 It is fitting that being a "webmaster" requires no special certification or inclusion in some privileged class. Any script-kiddie from around the world can give a blog post the appearance of credibility by spoofing a publish date or changing the look-and-feel to match an academic article. Perhaps this is why the establishment and digital world are always at odds with another. |
[report] Statement of Purpose for Computer Science Ph.D. Programs
[report] Domain Adaptation Through Multi-Task Distillation via Noisy-Labels
[report] A Bayesian Network Model for Sampling Dockless Scooter Traffic
[report] [code] Fast Random Kernelized Features: High-Dimensional SVM Classification
I love exploring and understanding new cities. My goal is to maximize the number of MSAs I spend more than 3 months in. Here's what I have so far — more to come!
• [2021 — now] New York-Newark-Jersey City, NY-NJ-PA (#1 in pop.)
• [2021] Chicago-Naperville-Elgin, IL-IN-WI (Loop, River North) (#3)
• [2018, 2019] Seattle-Tacoma-Bellevue, WA (UDistrict, Ballard) (#15)
• [2017 — 2021] Austin-Round Rock-Georgetown, TX (#28)
• [2000 — 2017] Dallas-Fort Worth-Arlington, TX (#4)
I also really enjoy road trips. Still a ton I want to see!
Roads I want to drive:
People who have had a major impact on me — whether a sparring buddy, mentor, or friend.
Jagath ∙ Alex ∙ Srujay ∙ Leonard ∙ Philipp ∙ Brady ∙ Yuwei ∙ Saaketh ∙ Dylan ∙ Will ∙ Prateek
I love meeting new people. Reach me at nimit@utexas.edu or schedule a chat.