AI research agent methodology: how AR Labs evaluates sources

AR Labs uses AI research agents to answer questions before making engineering decisions, and this page sets out the method the reports in this section follow. An AI agent is a language model run in a loop with tools, so it can act, not only reply; a research agent's tools are web search, page fetching and file writing. The method exists because an agent can produce a fluent, confident report from weak sources. So each source is scored before it is trusted, every major claim needs independent confirmation, disagreements and gaps are written down rather than smoothed over, and every finding carries a confidence grade. The page is for anyone who reads our research and wants to know how far to trust it, or who wants to run research with agents the same way.

How a piece of research runs

The person asking chooses a depth, and the agent works in passes that deepen progressively.

flowchart LR
  Q["Question and depth"] --> P1["Breadth pass: map the field"]
  P1 --> S1["Score each source"]
  S1 --> P2["Pathway pass: two to four lines in depth"]
  P2 --> T["Triangulate major claims"]
  T --> A{"Critical gaps?"}
  A -->|"Person approves"| P3["Depth pass"]
  A -->|"No"| W["Synthesis"]
  P3 --> W
  W --> R["Dated report: grades, conflicts, gaps"]

There are three depths:

Depth	Passes	Limits	Used for
Focused	Breadth only	Up to 5 searches and 15 sources; one short document	A quick question, a concept, a first look
Standard (the default)	Breadth, then pathways	Up to 10 searches and 30 sources; one to three documents	Most questions, such as choosing a tool or an approach
Comprehensive	Breadth, pathways, then depth	Up to 15 searches and 50 sources; several documents with an index	High-stakes decisions

The breadth pass maps the field: the main themes, the authoritative sources and the views that run against them. The pathway pass follows the two to four most promising lines in depth. The depth pass at the comprehensive level runs only when the pathway pass found critical gaps, and only after the person who asked has approved the specific areas it will cover, because it is the most expensive part of the work.

Scoring a source before using it

Every source is scored with the CRAAP test, a checklist devised by Sarah Blakeslee, a librarian at California State University, Chico. It asks five questions, and our agents rate each from 0 to 5:

Criterion	The question	Weight
Currency	Is it recent enough for this topic?	15%
Relevance	Does it answer the research question directly?	15%
Authority	Who wrote it, with what expertise, published where?	40%
Accuracy	Is it corroborated and backed by evidence?	25%
Purpose	Is it written to inform, or to sell or persuade?	5%

A source scoring 4.0 or more may support a major claim. One scoring 3.0 to 3.9 may be used but needs verification from other sources. Below 3.0, a source is reported only as a fringe view.

The agent also reads laterally, the practice Sam Wineburg and Sarah McGrew observed in professional fact-checkers: it spends little time on the page itself and more on what independent sources say about who published it.

Triangulation

A major claim needs at least three independent sources. Independent means different authors and institutions, not citing one another, and ideally different dates and methods, so that one study repeated by two blogs counts once. A claim with only two sources is reported as a frontier finding, with its uncertainty stated.

Conflicting evidence and gaps

When sources disagree, the report gives both views equal prominence, looks for a reason (a different context, population or method), and marks the finding as disputed. A disagreement is never resolved by dropping the inconvenient source.

When the evidence runs out, the report says so in an evidence-gaps section. Each gap lists the searches that were tried and what kind of source would fill it, so a reader can tell "not found" from "not looked for". The agent also stops searching on rules rather than by feel: when about four in five sources agree, when five sources in a row add less than a tenth that is new, or when a run takes twice its expected time, in which case it returns partial results with the gaps marked.

Confidence wording

Every finding carries one of four grades, and the words are used only in these senses:

• High: three or more independent sources, consistent findings, sound methods, recent data.
• Moderate: two sources, or one high-quality source, with some limitations.
• Low: a single source, outdated evidence, conflicting evidence, or quality concerns.
• Insufficient evidence: the claim cannot be made from what was found.

The grades describe the evidence found. They are not a promise that a finding is true, and the CRAAP scores are the agent's structured judgement, not calibrated probabilities.

Dated findings

Research describes the field at a moment. The method requires each report to state its scope limits, including the period its sources cover, so each finding can be read as true as of that period rather than in general. The reports in this section are published with the date the research was done, and a reader should expect fast-moving subjects, such as model benchmarks, to have moved since.

The guard against runaway delegation

A research agent that can start other agents can start too many. One of our standard research runs once delegated recursively: researcher agents started further researchers, five levels deep, until 376 agents had run. The agents had the tool that starts other agents, and their instructions told them to fan out.

The fix is structural rather than a request in the prompt. Every research agent is now a leaf worker: its tool list, which Claude Code enforces, contains search, fetching, reading and writing, but not the tool that starts agents. Splitting a question happens in exactly two places, at the orchestrator that assigns topics to agents and inside each agent's own list of searches, so nothing can recurse. A permission setting alone would not have done this, because permissions govern whether a tool call needs approval, not whether the tool exists.

flowchart TD
  O["Orchestrating session"] --> A1["Research agent"]
  O --> A2["Research agent"]
  O --> A3["Research agent"]
  A1 --> W["Web search and fetch"]
  A2 --> W
  A3 --> W
  A1 -.-x X["Start another agent: tool absent"]

What exists today

The method runs as the ResearchBot skill in Claude Code, which carries the depth levels, scoring and grading described here. For quick, broad surveys we use multi-agent research instead, which splits a question across three, nine or 24 leaf agents at once and does not apply the full scoring. Not every report in this section was produced at the same depth or with the same tool.