
The failure you are guarding against
Ask a general assistant how your payments are retried and it may say "three times with exponential backoff", because that is what most code does. If your consumer makes one attempt and marks the job failed, that answer is wrong in the worst way: it sounds exactly like a right one.
What a source changes
- You can check it. A file name and a line range take seconds to open.
- The room can check it. Others stop arguing from memory and start reading the same lines.
- A wrong answer shows itself. If the lines do not say what the answer says, you see that before you repeat it.
What to ask of a tool
- Does every claim about the code come with a file and line?
- What happens when it points at something that is not there?
- How long does it take to get from the answer to the line, in my editor?
- Has anyone measured how often it is right, and on what kind of project?
How we measure it
- 12 / 12 questions answered from the right file
- 0 made-up file references
- 4.7 s median time to an answer with its source
12 questions a team would ask, on a 533-file production codebase, with the default model. Run by us in October 2026. That is a small test on one codebase, run by the people who built the product. It tells you what we aim at and what we check, not what will happen on every project.
A source is not a reason to stop reading
Answers are written by AI models and can be wrong, even with the right file open. The source is there so that checking is cheap. Read the lines, then say it in your own words. If you did not have time to check, say that it comes from a quick look.