Skip to content

AgentCase Lab Replays AI Agent Failures to Find Root Causes

A Python portfolio project on GitHub turns agent runs into replayable traces and compares metrics before and after a fix, using clearly labelled simulated demo data.

Draftship Editorial4 min read

AgentCase Lab, published on GitHub as TngSHoO/agentcase-lab, is a project for AI agent failure replay and root cause analysis. According to the repository's README, it was created on 2026-11-04 and had 351 stars and 59 forks by 2026-11-14, meaning it collected that attention in its first ten days. It is written in Python and released under the MIT licence, which the README says allows use, modification, distribution and commercial use as long as the original author and licence notice are kept.

The README is explicit about what this is: a portfolio MVP, built from scratch by an individual, with AI coding tools used during development. The bundled cases are marked as simulated data and are not presented as real rankings or production results for any model. That framing matters more than the star count, because it tells you what you can and cannot conclude from the demo.

What the tool actually does

The project targets AI product and application teams that need to work out why an agent run went wrong. It splits a single run into an execution trace that can be replayed, then helps a team point at one of several problem areas: the prompt, knowledge retrieval, tool calls, context, long-term memory, permissions, or model hallucination. After a fix, it compares the before and after on success rate, latency, token use and cost.

The interface is organised around five core pages. A dashboard shows success rate, fix rate, root cause distribution, latency and cost. A failure case list can be filtered by scenario, status and root cause. A trace replay view exposes the user input, the plan, the tools used, memory, the model step and the final result. A root cause view runs a rule-based demonstration mode, with an optional model-backed mode. A version comparison view checks task status and key metrics across a fix. The repository also ships product documents covering a PRD, user journeys, a metrics definition and product decisions.

How it runs

It is a Streamlit application backed by SQLite, requiring Python 3.11 or newer. You create a virtual environment, install the requirements, and start the app, which serves on localhost port 8501. On first launch it creates the database and inserts 30 demo cases. The codebase separates the five page files from an agentcase package that holds the data models, SQLite access, metrics and analysis logic, with a docs folder and a test suite. Tests run through pytest, with ruff used for linting.

The optional live analysis mode is configured through environment variables in a .env file. The README's example points at an OpenAI-compatible service, with a specific API base and a chat model named in the sample, and warns against committing the file or a real API key to GitHub. In other words, the model-backed root cause analysis is bring-your-own endpoint, and it is off by default.

What it changes for people building apps

The useful contribution here is the vocabulary as much as the code. Teams shipping AI features often have logs but no shared way to describe a failure, so a bug gets filed as a vague complaint instead of being sorted into one of the project's categories, such as knowledge retrieval, tool calls or permissions. A fixed list of root cause categories plus a before-and-after metric comparison is a reasonable skeleton for that conversation, and it is small enough to run locally.

If you are adding AI features to an app you are building, the takeaway is to record the inputs, tool calls, context and outputs of each AI run from the start, because a trace you did not capture cannot be replayed.

Limits and open questions

The source leaves several things unresolved. It does not describe ingesting traces from any agent framework, so how real runs get into the tool is unclear. It gives no accuracy figures for the rule-based analysis. It does not state whether a hosted or multi-user version exists; as described, this is a local Streamlit app with a local SQLite database. The README notes that product definition, scope decisions, metrics design and acceptance were the maintainer's responsibility, which is a reasonable statement for a portfolio piece but not evidence of production use. Whether the 351 stars reflect real adoption or general interest in agent observability is not something the repository data can answer.

What this means for building apps with AI

Draftship is an AI app builder where you describe an app and it writes, builds and hosts the code, with a built-in database, user sign-in and hosting, and paid plans adding private projects, model choice, source access, custom domains and GitHub push. Nothing in the AgentCase Lab README mentions Draftship and the two are not connected. The general point stands on its own: as more apps get AI features, knowing why a run failed becomes part of ordinary maintenance, and projects like this one are an early sketch of what that workflow might look like.

Primary sources

This article was drafted with AI assistance from the primary sources listed below and checked automatically against them. An editor hasn't reviewed it yet.