Research

Same Failure, Different Cause: Ungrounded Tool-Call Provenance in Simulated Agent Evaluation

Abhimanyu Prasad

Workshop: Grounded User Simulation for Model Evaluation and Training: Diversity, Fidelity, and Validity (UserSim), NeurIPS 2026

In one line: Two open models fail the same customer-service agent tests, but for different reasons: one acts on IDs the user stated without checking them, the other makes IDs up. A simulated user that always says the right ID hides the first problem completely.

What I did: I ran Qwen2.5-7B and Llama-3.1-8B as agents on τ²-bench (retail), with the same 7B simulated user. For every tool call that used an ID, I checked whether that exact ID had appeared in an earlier successful tool result. If it hadn't, the call counts as ungrounded. Qwen mostly copied IDs from the user. Llama mostly invented them. I also found a second failure: Llama's tool arguments were malformed in every chance it had. Everything runs on two free T4 GPUs with open weights.

My role: Sole author: the question, the experiments, the code and the paper. Ran on free Kaggle GPUs.

What came out: Accepted as a poster. On the matched tasks, 14 of Qwen's 15 ID-using tool calls were ungrounded (93.3%), against 43 of Llama's 95 (45.3%). The sample is small and the analysis is exploratory, not pre-registered. The code and run records are public.

Led to: ATTRIB paper

Ablate-and-Measure Breaks When the Measurement Depends on the Ablation

Abhimanyu Prasad

Workshop: 3rd Workshop on Attributing Model Behavior at Scale: Data Attribution and Provenance (ATTRIB), NeurIPS 2026

In one line: If you remove parts of an AI agent's instructions to see what matters, the score you're measuring can stop existing for some runs, and that changes the answer without warning.

What I did: I tested a 7B model (Qwen2.5-7B) on τ²-bench under 6 versions of its context, from full instructions to stripped-down ones. That is 12 tasks × 2 seeds × 6 conditions = 144 cells, about 16 GPU-hours on free Kaggle T4s. The benchmark's own scorer only scores actions when a run ends cleanly, so the grounding score was defined for 50-92% of runs, depending on the condition.

My role: Sole author. I also reviewed two submissions for this workshop. Reviews are anonymous, so there is no public record of that.

What came out: A null result: no condition was significantly different. It also shows how ablation studies can mislead. The paper discloses two pre-registration changes.

Builds on: UserSim paper

Verify on Read, Update in Log Time: Per-Block Integrity for Mutable Edge-AI Model Files

Abhimanyu Prasad

Workshop: SustainableAIoT-Afrasia - Sustainable AIoT for AfroAsian Communities, at IEEE AIoT 2026 (Kobe, 14-16 Dec 2026)

In one line: A way for small devices to check on every read that an AI model file hasn't been tampered with, without re-hashing the whole file each time.

What I did: I split model files into 4 KB blocks under a Merkle tree, so a read checks only its own block and a write updates only one path up the tree. I benchmarked it on my own laptop (Intel i5-5200U) and phone (Snapdragon) with real model files from about 100 KB to about 100 MB.

My role: Sole author: the method, the experiments on my own hardware and the paper.

Design guidance: Prof. Subodh Kumar (IIT Delhi)

What came out: Per-block checks stayed at about 3 µs regardless of model size. A single flipped weight was caught in its block with no bad bytes served. A write costs about 4-5 µs, against 6-44 ms to re-hash the file.

Builds on: model-integrity-bench · Led to: Zombie IoT article

Ordered Disagreement in Open-Weight LLM Annotator Committees: Lessons from Building a Bhojpuri Sentiment Resource

Abhimanyu Prasad, Deepak Kumar Rakesh

Transactions on Machine Learning Research (TMLR), submission #13742, sent 26 Oct 2026

In one line: Labelling text in a low-resource language is slow and costly, so I let several free AI models label it first and send only the cases where they disagree to people.

What I did: I built a Bhojpuri corpus of 90,207 sentences from 11 news, encyclopedic, literary and blog sources. Five open-weight LLMs from five companies labelled every sentence, and I kept every model's vote. Then I studied how and where they disagree. The models disagree in a fixed order: almost every sentence the strictest model calls subjective, the loosest one also calls subjective (99.2%). I labelled a held-out set by hand as a native speaker. The released dataset is a by-product of that study.

My role: First author: the pipeline, data collection, experiments and writing. Prof. Deepak Kumar Rakesh (NIT Bhopal) mentored the work and co-authored.

What came out: The Bhojpuri sentiment model and dataset on Hugging Face (see Projects).

Builds on: Tokenizer cost study

The Zombie in the Living Room: Why Cloud-Dependent IoT Devices Fail and How Engineers Can Design Better Alternatives

Abhimanyu Prasad

IEEE Potentials (magazine)

In one line: Why smart-home devices die when the company behind them shuts down its servers, and how engineers can design them to keep working.

My role: Sole author.

Builds on: IEEE AIoT paper