How we test Ask

What the test is

We keep one fictional team’s notes: 31 short notes about a product launch called Harbor, an office move, a podcast, hiring and a few colleagues. We put 111 questions about them to Ask, the way you would in the app, with a stored Anthropic API key. Then we put the same questions through the other ways ChatMD can connect to an AI.

Every question starts a new conversation and ends with the same request: “Answer in one or two sentences and cite the note.” Ask chooses which notes to send with each question; we do not choose them for it. Web search is off.

We wrote the questions in English. They come in six kinds.

• Lookup

  29 questions

  The answer is written in one note.

  “Who owns the Harbor launch, and when is it?”

• Sum

  17 questions

  The answer needs arithmetic on figures from the notes: $22,000 less $6,400 already spent.

  “How much of the Harbor ads budget is left?”

• Date

  16 questions

  The answer needs two dates compared or a date worked out.

  “Is Tom Okafor back in time for the Harbor beta?”

• Conflicting notes

  17 questions

  Two notes disagree, or a later one replaces an earlier one: $18,000, then $22,000. The right answer is the newer figure.

  “What is the current Harbor budget?”

• Several notes

  17 questions

  The answer is spread over more than one note.

  “Who is drafting the Harbor press kit, and is it due before launch?”

• Not in the notes

  15 questions

  The notes cannot answer. The right answer says so instead of inventing one.

  “What salary is offered for the designer role?”

How an answer is scored

A program scores every answer. No person and no AI judges it, so the same answer always gets the same points.

• Each question has a short list of facts a right answer must state. Every fact the answer states earns 1 point. The program looks for each fact in the answer’s text.
• Naming the right note earns 1 more point. The note’s title can be in the answer or in its list of sources, and where several notes would do, any one counts.
• When the notes cannot answer, the only point is for saying so plainly, such as “the notes don’t say”. Inventing an answer scores 0.
• An answer that fails with an error scores 0.

For example, “What is the current Harbor budget?” is worth 2 points: 1 for stating $22,000 and 1 for naming the note from the 26 September Harbor meeting. The 111 questions are worth 263 points in all.

Matching patterns has a cost. An unusual wording of a right answer can miss a point, and a wrong answer that happens to use the right words can earn one.

A citation is the part of your notes that an answer points to as its source. With a stored Anthropic key, the AI can point to the exact passage it used. We count how often it did, and how often that passage holds at least one fact the question needs.

What we found

Run on October 16, 2026, on ChatMD 2.2.0: Ask with a stored Anthropic key, 111 questions, citations on.

The cost is an estimate: the app’s usage meter multiplies the tokens each request used by the provider’s published prices. The cost per 1,000 questions is that total divided by 111, times 1,000. The time runs from the test starting a new conversation to the finished answer, so it also includes typing the question, the test’s own clicks and a one-second pause after each answer. The last row counts only answers that came with the AI’s own citations, and leaves out the questions the notes cannot answer.

What the AI costs

Other ways to connect an AI

We put the same 111 questions through the other ways ChatMD 2.2 connects to an AI, on October 16, 2026. Each used the model ChatMD starts on for it, and every answer was scored the same way.

Ask was set to Relevant notes for every one, as for the key above: ChatMD picks the notes and sends them. Claude Code’s own default is Whole folder, where it reads your notes itself, and we have not measured that. Ollama always looks through your notes itself.

Claude Code and Codex count against your plan’s allowance, so there is no price per question. Codex chooses its own model; on our test PC it is set to GPT-6.1 Sol at high reasoning effort. The OpenAI and Grok figures come from the app’s usage meter, as for the Claude key; for Grok it records the charge xAI reports, when it sends one.

We did not measure Claude Code’s time: a fault in our test added a fixed 15-second wait to each of its answers. Ollama ran on our test PC, which has no graphics card, so its time depends on your hardware.

Not measured yet: Antigravity (not part of this round).

Two rules, pre-registered

A rule is pre-registered when it is written down before the test runs, so it cannot be bent to fit the result. We wrote both of these first.

What this does not tell you

• It is one fictional folder of 31 short notes, in English. Your notes are longer and messier, and may be in another language, so Ask may do better or worse on them.
• It is our own test, scored by our own program. No outside group has checked it.
• Scores move from run to run. Across our runs from October 9 to 16, on different builds of the app, Claude Haiku 5.5 scored between 92.8% and 95.8% and Claude Sonnet 5.5 between 95.4% and 96.6%. Claude Haiku 5.5’s own score moved by 3.0 points across those runs; the gap between the two models in this run is 3.0 points. The other ways to connect were each run once, so a gap of a point or two between two of them can be chance.
• Cost depends on how much of your notes Ask sends. Our notes were short, and every answer was asked to be one or two sentences.
• Not measured yet: Antigravity (not part of this round). We will add it here when we have measured it.
• We run this test again before each release. This page shows the run of October 16, 2026, on ChatMD 2.2.0.