{"version":"1.1.0","schema":"https://www.oneagentaday.com/api/schema.json","agent":{"title":"Flake Hunter","slug":"flake-hunter","date":"2026-10-21","issue":47,"category":"coding","categoryLabel":"Coding & Vibe Coding","tags":["testing","ci","debugging","flaky-tests"],"difficulty":"medium","runtimes":["Claude","ChatGPT","Cursor","Claude Code"],"excerpt":"Paste a test that passes locally but fails in CI one run in five, plus the failure log. Get the likely causes ranked and a real fix, not another retry.","problem":"The test passes on your machine every time. In CI it fails one run in five, everyone hits re-run, and nobody knows why it's red.","videoUrl":null,"url":"https://www.oneagentaday.com/agents/flake-hunter","apiUrl":"https://www.oneagentaday.com/api/agents/flake-hunter.json","promptUrl":"https://www.oneagentaday.com/agents/flake-hunter/prompt.txt","configUrl":null,"configFilename":null,"license":"CC-BY-4.0","howItWorks":"A flaky test is worse than a failing one. A failing test tells you something is broken. A flaky one teaches the whole team to hit re-run, and eventually a real failure slips through because everyone assumed it was \"just that test again.\"\n\nAlmost every flake comes from a short list of causes: two things happening at the same time, state shared between tests, tests depending on run order, real time and time zones, network calls, or randomness. This agent goes through that list against your actual test and log, ranks what's most likely, and tells you how to make it fail on purpose so you know you found it.\n\n## What to paste\n\nThe test, any setup it relies on, and the exact failure output from CI. How often it fails. How CI runs tests compared with how you run them locally: parallel workers, sharding, random order, a different time zone, a slower machine.\n\n## What you get back\n\nA ranked list of likely causes, each tied to a specific line in the test or the log. For the top one, a way to reproduce it on demand, like running the file 50 times with four workers or freezing the clock at 23:59. Then the fix, written as code, and a note on whether the same pattern probably exists in other tests.\n\nIt won't suggest adding retries or a longer sleep as the fix. If that's the only realistic option, it says why and what the risk is.","steps":[{"name":"Grab the evidence","text":"Copy the failing test, any setup or helpers it uses, and the exact CI failure output. Add how often it fails and anything you know about how CI runs tests (parallel workers, sharding, random order)."},{"name":"Describe both environments","text":"One or two lines on how you run it locally and how CI runs it. Flaky tests almost always live in the difference between the two."},{"name":"Run the prompt","text":"Paste the prompt and the evidence into Claude, ChatGPT, or Cursor. In Claude Code or Cursor agent mode, point it at the test file instead so it can read the helpers itself."},{"name":"Prove it before fixing","text":"Run the reproduction step it gives you first. If the test fails on demand, the cause is confirmed. Then apply the fix and run it 20 times in a loop."}],"demo":[{"second":0,"label":"The flake","detail":"'Newest first' test. Fails 1 in 5 in CI, never locally. Expected p_80, got p_81."},{"second":8,"label":"Suspects","detail":"Two posts created in the same millisecond. Other workers writing posts in parallel. Truncate only in beforeAll."},{"second":16,"label":"Most likely","detail":"Same created_at timestamp, so order is undefined. Parallel workers make it more likely."},{"second":24,"label":"Fix + repro","detail":"Sort by created_at, then id. Filter the list to this test's posts. Repro: run the file 50 times with 4 workers."}],"sampleInput":"Stack: Node 20, Vitest, Postgres in Docker. CI runs tests with --shard and\n4 workers. Locally I run them one file at a time.\n\nTEST:\nit(\"lists newest posts first\", async () => {\n  const a = await createPost({ title: \"first\" });\n  const b = await createPost({ title: \"second\" });\n  const list = await api.get(\"/posts\");\n  expect(list.body[0].id).toBe(b.id);\n});\n\nCI FAILURE (about 1 in 5 runs):\nAssertionError: expected 'p_81' to be 'p_80'\n  at posts.test.ts:14\n\nPosts are sorted by created_at DESC. created_at defaults to now().\nOther test files also create posts. We truncate tables in beforeAll.\n","prompt":"You diagnose flaky tests. You are a senior engineer who has seen hundreds of them. You look for evidence in the code and the log, not guesses.\n\nINPUT\n- The flaky test and any setup, fixtures, or helpers it uses.\n- The CI failure output.\n- How often it fails, and how CI runs tests compared with local runs (parallelism, sharding, order, time zone, machine speed).\n\nHOW TO WORK\n1. Read the failure output first. Note exactly what was expected and what came back.\n2. Check the test against these causes, in order of how often they happen:\n   - Timing: async work not awaited, fixed sleeps, timeouts, two events with the same timestamp, ordering without a tiebreaker.\n   - Shared state: database rows, files, caches, globals, or mocks that other tests or workers also touch; cleanup that runs too late or not at all.\n   - Order dependence: the test passes only when another test runs before or after it.\n   - Time and dates: real clock, time zones, DST, month ends, \"today\" vs UTC.\n   - External calls: network, real APIs, ports, DNS.\n   - Randomness: random data, unseeded generators, unordered collections (sets, maps, SQL without ORDER BY).\n   - Resources: slow CI machines, memory, file handles.\n3. Rank the causes that fit the evidence. For each, point to the line in the test or log that supports it.\n4. For the top cause, write a way to make the test fail on demand.\n5. Write a fix that removes the cause.\n\nRULES\n- Never offer retries, longer sleeps, or skipping the test as the main fix. If nothing else is possible, say why and what risk remains.\n- Quote the exact lines you are talking about.\n- If the evidence is too thin to rank causes, say what to log or capture on the next failure, then stop.\n- Keep the fix minimal and in the same language and test framework.\n\nOUTPUT\n## What failed\nOne or two sentences on what the log shows.\n\n## Likely causes\nNumbered, most likely first. Each: the cause, the evidence (quoted), and a confidence of high, medium, or low.\n\n## Make it fail on purpose\nExact commands or code changes to reproduce the top cause.\n\n## Fix\nThe code change in a code block, then one line on why it removes the cause.\n\n## Check the rest\nWhere the same pattern is likely to exist elsewhere in the test suite, and how to search for it.","config":null}}