How to build an AI agent: the components and the process

This guide from AR Labs sets out how to build an AI agent that does a real job and keeps doing it: the components the agent needs and the order in which to build them. An AI agent is a language model run in a loop with tools, so it can act, not only reply. The guide is for a founder, product lead or engineer who wants more than a demonstration: an agent with a defined job, limits a person sets, a memory that survives the end of a conversation, and checks that do not depend on the agent's word.

Every step draws on agents we have built and run, and each example names the agent it comes from. The marketing manager agent runs continuously on a server on Hermes, Nous Research's open-source agent runtime; it researches, recommends and prepares writing briefs, while a person writes and approves. The strategy agent runs the lab's planning reviews, from the year down to the day, as a skill started in a coding session. The coding agents each take one step of a code change in our coding pipeline. The research agents search and report with sources. Salesbot is a browser voice agent, open to the public, that runs a short sales discovery conversation. Where a design and the running agent differ, we say which runs.

The idea underneath

An agent combines two kinds of computation: ordinary software, which is exact and repeatable, and a language model, which handles ambiguous language, synthesis and judgement where fixed rules run out. We build so that the software bounds, checks and records the model: code holds state, identity, permissions, contracts, calculations and the record of what happened, and the model decides only where judgement is needed. The model should not do work that ordinary code can do more safely and exactly. Each step below applies this, from choosing a script over an agent in Step 1 to the hooks and evidence checks of Step 9.

The components of an agent

Most of an agent is not the model. The harness is everything around the model that shapes what an agent does: the agent program, its instructions, skills, hooks, tools and checks. An agent with a standing job also needs a role, an authority table, records that outlast a session and channels. The table lists the components in the order this guide builds them; the component reference compares how each of our agents implements them. The seven concerns of agentic engineering for an agent people rely on (system design, tool contracts, context and retrieval, reliability, security, evaluation and the person's experience of it) each have a home in the steps.

Component	What it is	What goes wrong without it	Where our version is described
Role and outcome	The job, and the result it is judged on	The agent optimises whatever is easiest to count	Strategy agent cadences
Authority	What it may do alone, with a person's yes, or never	It asks about everything, or decides what it should not	Act or ask
Runtime	The program that runs the loop, and where it runs	Work stops when a conversation closes, or runs with nobody watching	Resident AI agents
Instructions	Standing role, current work and reference, kept apart	Copies of one rule drift apart and the agent reconciles them on every turn	Logical levels
Skills and tools	Packaged methods, and the actions it can take	It improvises methods, or holds tools it should not	Agent harness anatomy
Memory and state	Working state, decisions and evidence in files	Each session starts from nothing, or from stale decisions	Agent memory for changing facts
Task store and loop	Durable tasks and a rule for the next one	Plans live in conversation and vanish	Planning and execution loop
Channels and interface	How people and other agents reach it, what a person sees, who writes what	Lost requests, two writers on one file, a user who cannot tell what it is doing	File-based inbox protocol
Gates, checks and security	Hooks that can refuse, approvals, evidence, review, least privilege, input treated as data	Rules hold only while the agent remembers them; a web page can give it orders	Human-in-the-loop gates
Tests and evaluation	Scenarios and replays judged against the outcome, re-run after each change	"It seemed fine" becomes the release test	Testing a browser voice agent
Observation and cost	What the logs keep, which model runs, what caps apply	Silent failures and unbounded spend	Model routing
Failure handling and operation	Timeouts, retries and fallbacks; how it is started, restarted, updated and handed over	One slow service stalls it; only its builder can keep it running	How AR Labs builds an agent

Step 1: Decide whether you need an agent at all

What to decide. Whether the job needs an agent, or whether a script, a fixed workflow or a single model call would do it. Write the job in one sentence: what starts it, what exists when it is done, and who relies on the result.

Why. Erik Schluntz and Barry Zhang's essay Building effective agents (Anthropic, 2024) separates workflows, where code fixes the path and the model fills in steps, from agents, which "dynamically direct their own processes and tool usage". It recommends "finding the simplest solution possible, and only increasing complexity when needed". An agent earns its higher cost and compounding errors when the steps cannot be known in advance and the work needs judgement.

What we did. Our coding pipeline's driver is a script, not an agent: it reads the task graph and starts an agent for each step, because choosing the next step needs no judgement. Salesbot's conversation stages are a state machine in code, and the model only talks, because a sales conversation that must reach an end cannot leave its progress to the model; we call this structured conversation. The marketing manager agent is an agent because choosing the next useful piece of marketing work is judgement over changing evidence.

The mistake to avoid. Building an agent for a job that is really a fixed sequence. The ladder of changes applies here too: a written method, a configured tool or a script comes before AI.

• The job written in one sentence: trigger, result, who relies on it
• A script, a workflow and one model call each considered and rejected with a reason
• The judgement the agent will exercise named

Step 2: Define the role, the outcome and the authority

What to decide. Three documents, each short. A role: the job, its standing responsibilities, and the outcome it is judged on. An authority table: what the agent may do alone, what needs a person's yes, and what it may never do. And the division of work with the person it serves.

Why. A list of duties does not say where to stop or what counts as success.

What we did. For the marketing manager agent we drew the duties from published descriptions of the job, such as the US O*NET occupational profile, before deciding what the agent should hold. The division of work is fixed: the person writes, and the agent researches, recommends, briefs, coordinates and measures. Its outcome is measured from exposure among relevant readers to responses, conversations and paid engagements; reach and clicks only diagnose. Its authority, in short:

Alone	With a person's yes	Never
Research, analysis, plans, writing briefs	Publishing, sending, spending	Editing its own credentials or runtime settings
Maintaining its own working records	Confirming the shared marketing context; rewriting its own skills	Writing into another agent's workspace
Drafting proposals for decision	Adopting a proposal as a decision	Changing the company goals it reads

The strategy agent's authority uses a five-level delegation scale: it settles routine prioritisation alone and may only recommend strategic direction.

The mistake to avoid. Leaving authority implicit in the role description. Each line of the table becomes a check or an approval later (Step 9), so it must exist as a list.

• Role with standing responsibilities and one outcome measure
• Authority table with three columns, signed off by the person
• Division of work with the person written down

Step 3: Choose the runtime and where it runs

What to decide. The runtime is the agent program that runs the loop: it calls the model, routes tool calls and loads instructions. Decide how the agent lives: session-bound (started by a person for one piece of work), resident (a background service that keeps a channel open), scheduled (woken at set times), or per request (one run per call or event).

Why. A resident agent can be reached at any time but must be operated like any service; a session-bound one is cheap and supervised but does nothing unless started.

What we did. The strategy agent is session-bound: a skill a person starts in Claude Code. We once ran it as an always-on process that woke every half hour and passed each result through a filter for routine messages; 47 of 48 wake-ups were classed as routine and nothing reached the person, so we stopped the service. The marketing manager agent is resident on Hermes, one of the self-hosted agent runtimes our research compared, and we use the runtime's own mechanisms before building equivalents. Salesbot runs one Pipecat pipeline per call, and each coding step runs in a fresh session, so no step inherits another's reasoning.

The mistake to avoid. Choosing resident or scheduled because it sounds more capable. Learn the runtime's loading rules before writing instructions: Hermes reads the agent's identity file and workspace guide only when a session starts, so every change to the instructions ends by starting a new session.

• Lifetime chosen: session-bound, resident, scheduled or per request
• The runtime's loading, restart and permission rules read from its documentation
• Delivery to a person tested before any schedule is switched on

Step 4: Write the instructions, with one home for each rule

What to decide. Where each kind of instruction lives. Keep three apart: the standing role (identity, authority, standards), which loads every session; current work (the working state, Step 6), which changes daily; and reference material, read only when a task needs it.

Why. Instructions accumulate and copies drift. Context engineering is choosing what goes into a model's context window at each step, and Anthropic's guidance on it treats context as a finite budget, to be spent on "the smallest possible set of high-signal tokens". A rule stated twice in different words costs twice and leaves the agent to decide which version holds.

What we did. The marketing manager agent's session prompt came to about 3,900 words before it said a word. The size was modest; the problem was copies. One rule appeared in five files, each worded differently, and its approval limits appeared in four. The design we drew up in response uses logical levels, after Gregory Bateson: each level states the choices, errors and change process for the level below, and each rule has one home. Its implementation is deferred. Our coding agents' instruction files are short maps that route to one document per concern (see progressive disclosure).

The mistake to avoid. Answering each new problem with another paragraph in whichever file is open. Find the rule's home, change it there, and delete the copies.

• Standing role, current work and reference in separate places
• Each rule stated once; a search for its key words finds one home
• Instruction changes followed by a fresh session that confirms they loaded

Step 5: Give it skills and tools, reusing before building

What to decide. Which methods the agent follows and which actions it can take. A skill is packaged instructions, with any scripts, that an agent loads only when a task needs it; what it packages is a reusable method. A tool is an action the runtime exposes to the model, described by a schema that is the tool's contract. Skills alone are not a capability, which is skills combined with the tools, permissions, data, interface, tests and controls that produce a business outcome; the rest of this guide supplies those.

Why. An agent without a method improvises one that reads well; an agent with spare tools can act outside its job. A model knows a tool only through its schema, and it fills a loose field with something that fits the form but not the meaning, such as a customer's name where an ID belongs.

What we did. The marketing manager agent works from Corey Haines's public marketingskills library, 53 skills when installed. Adapting a handful of its skills one at a time had left methods without the command-line tools they call and no planning method at all, so the agent planned with a software planning skill. We installed the whole library as its author intends and adapt only where it collides with the runtime. For tools, least privilege is structural: our research agents' tool lists omit the tool that launches agents, so a research fan-out cannot recurse; read-only helpers have no editing tools. In Salesbot each question stage gives the model one tool for recording answers, besides a polite exit, and the goodbye stage gives it none. That recording tool is generated from the same table as the stage's questions, so the two cannot drift apart, and code checks every call: it drops blanks, placeholders and fields the stage does not have. A refused call returns a note telling the model what to ask, not a bare error, because a tool that fails teaches the model to stop calling it.

The mistake to avoid. Trimming a method library to the parts that look relevant, so the methods lose their tools, or granting every tool the runtime offers.

• An existing skill library or published method searched before writing one
• Each tool justified by a line of the role; nothing granted "in case"
• One job per tool; strict types, enumerations and examples in each schema
• Invalid input refused in code, with a message that says what is valid

Step 6: Design its memory and state

What to decide. What must survive a session, where it lives and who may write it. Four kinds: a working-state record that says where the work stands now; decisions, with who made them, when, in what words and what they govern; evidence, each item labelled by how it was obtained; and the task store (Step 7). Decide also what not to store.

Why. Every session starts with an empty context window, so a stale or bloated record misleads every session that reads it.

What we did. The marketing manager agent's resume point is the current-work section of one working-state file, which names active work, the next action and what waits on whom, and points to records rather than copying them. Evidence is labelled measured, provided by the person, calculated, estimated or proxy. Its design keeps decisions in an append-only log, never edited, so a reversed decision stays visible as reversed. We switched off a memory plugin that extracted facts in the background after finding most of its writes failing; our research on agent memory found that at this scale, reading curated, dated records in full is more accurate than model-extracted memory. When a store outgrows reading in full, retrieval becomes its own discipline, and its quality needs measuring apart from the agent's, because an agent answers as confidently from the wrong document as from the right one.

The mistake to avoid. A state file that accretes, so every session rereads superseded decisions as if they held. A working-state record states current truth on about one page; history belongs in git and in the decision log.

• One working-state record, rewritten (not appended to) after each piece of work
• Decisions recorded with exact wording, maker, date and scope, never edited
• Evidence labelled by source; source documents cited, not copied

Step 7: Give it a task store and a planning-and-execution loop

What to decide. Where tasks live, how the agent picks the next one, how it resumes after a break, and how it knows a task is done.

Why. The loop that survives real work has three parts: durable tasks with a status each, a rule for picking one next task, and a write-back a fresh session can trust. Anthropic's account of harnesses for long-running agents keeps a feature list with a pass flag for each item, a progress file and git history, works on one feature per session, and reports that without this structure the agent "declares victory on the entire project too early".

flowchart LR
  S["Task store: open, blocked, done"] --> P["Pick one ready task"]
  P --> A["Act through tools"]
  A --> C{"Evidence it is done?"}
  C -->|"no"| A
  C -->|"yes"| W["Write back: task, state, evidence"]
  W --> S
  W --> N{"Needs a person?"}
  N -->|"yes"| Q["Ask once; carry on with other work"]

What we did. The marketing manager agent's built-in to-do list lasted only one session, so its tasks moved to Linear, an issue tracker: issues and sub-issues for the work, "blocked by" relations for order, comments for progress. Its acceptance test is that a fresh session resumes the plan at the right task from the records alone. The strategy agent's next run is chosen by its state file, which records when each review last ran. Our coding pipeline's driver is stateless: on every pass it re-reads the task graph, dispatches ready steps and ends in one of a fixed set of recorded states (see unattended agent runs).

The mistake to avoid. Holding the plan in the conversation, or letting the agent decide by itself that it has finished.

• Tasks in a durable store, with dependencies, and a written rule for the next one
• "Done" defined per task as evidence, not as the agent's statement
• A fresh session tested on resuming from the records alone

Step 8: Connect it to people and other agents

What to decide. How a person reaches the agent, how other agents reach it, which messages count as requests, and who may write each shared record. Decide also its interface: the surface through which people start and supervise its work, such as voice, chat, a web page or a document, and what that surface shows.

Why. Agents that share files fail like distributed systems, but quietly, and a request that exists only in chat is lost when the session ends. An agent can be right on one run and wrong on the next, so the person using it needs to see what it is doing and what it is unsure of, and to have a way to correct it or reach a person.

What we did. Our agents hand work to each other through a file-based inbox: the request is a dated file, the reply is a file, and chat is used only to say where to look. The receiving agent keeps a list of permitted senders. A connection counts as working only when the recipient answers with substance. No agent writes into another's workspace, and each shared document has one writer (see interface contracts). Salesbot's page shows a progress rail of the stages and a status window that says in plain words what the agent is finding out and which answers it has. Before closing it reads the answers back for confirmation; if one is missing, it names what went unanswered rather than filling the gap. The visitor can stop at any stage, and the call ends politely.

The mistake to avoid. Calling a channel connected because the service is up. Test the round trip and record which identity answered.

• Each channel named, with what it carries: conversation, requests, records
• Permitted senders listed; one writer per shared record
• A round trip tested with a substantive reply
• The interface shows progress and uncertainty, and offers a correction and a way out

Step 9: Put gates, checks and security limits around it

What to decide. Which rules are enforced and how. A gate is a point work cannot pass until a condition is met: an approval, a check, or answers collected. A hook is a script run automatically around an agent's action that can block it. Decide which lines of the authority table become hooks, which become a person's approval, what evidence each task must leave, and how the agent is protected from what it reads.

Why. An agent follows a rule through an instruction, a runtime setting or a script, and only the last two work whether or not the model complies. We think a rule that matters is enforced by a check.

What we did. Around the coding agents, hooks deny edits to a project's main checkout and deny an unattended session any edit to the pipeline's own scripts and settings; scripted checks refuse to advance a task without its evidence; a model from a different vendor reviews each change; and a person merges (see harness engineering). In Salesbot, code decides when a stage is complete and when the call ends. The strategy agent stops at a checkpoint after each phase of a review for a person's decision. The marketing manager agent's runtime prevents it editing its own credentials and settings, and requires approval before it rewrites its own skills; its approval rule for publishing, sending and spending is written in its instructions, because the runtime's approval mechanism covers risky system commands, not business actions. A check before those actions is an open design item, and it matters for security as well, because this agent reads the web.

Security. An agent that reads web pages, email or a visitor's words can be handed instructions by whoever wrote them. This is prompt injection, and it is indirect when the instructions hide in a page or document the agent retrieves. No wording in a prompt reliably prevents it, so the defences are structural: treat everything the agent reads as data, never as instructions, and label it so in the context; grant the fewest permissions the role needs (Step 5), so a successful injection has little to reach; keep secrets out of the model's context; validate input in code; put a check or a person before any outward or irreversible action; and log every action with its inputs. Salesbot's server admits only the two message types its page sends, and the visitor's words reach its records only through recording tools whose input code checks. Our research treats downloaded material, such as video transcripts, as untrusted data.

The mistake to avoid. Gates that delay work without anyone deciding to. A gate that holds back an outcome the person has already committed to needs that person's yes to exist.

• Each "never" in the authority table enforced by a hook or a runtime setting
• Each "with a person's yes" routed to an approval with a recommendation
• Evidence of completion defined per task; an independent reviewer for consequential work
• Everything the agent reads treated as data; secrets out of its context; actions logged

Step 10: Test and evaluate it

What to decide. What scenarios the agent must handle, how to replay real sessions, which checks a machine can make and which need a person, and how to judge results against the role's outcome rather than the transcript. The scenarios become an evaluation set, run again after every change to the instructions, the model or a tool.

Why. An agent's output is fluent whether it is right or wrong, and a change that fixes nine cases can break the tenth. Anthropic's team building a multi-agent research system started evaluation with about 20 queries drawn from real use and used a model as a judge to grade outputs at scale. When a case fails, trace it backward through retrieval, tool choice, schema, execution and reasoning before rewording the prompt, as in our worked example of debugging by trace, and record which kind of failure it was.

What we did. Salesbot has 747 automated test functions in 35 files, and every change to it goes through our coding pipeline, which runs them. Calls that went wrong are kept as fixtures, and a replay script feeds a recorded call's visitor lines to the real model. A scorer compares each stored answer with what the visitor said: verbatim matches are decided automatically, paraphrases by a person. A second model judging stage completion was tried and rejected, because in side-by-side tests it moved the conversation on over the visitor. Speech recognition, interruption and voice quality still have no repeatable automated test (see testing a browser voice agent). The marketing manager agent is judged on its assignment's done list, including a resume from records alone, and a development session checks its saved work against the sources in its workspace.

The mistake to avoid. Checks that test form rather than substance. A filled-in template does not validate a finding; read the sources, not the agent's summary of them.

• A scenario set built from real work, including past failures
• Recorded sessions replayed; each check marked automatic or for a person
• Results judged against the role's outcome; the set re-run after every change

Step 11: Observe it and pay for it

What to decide. What a person needs to see, what the logs keep, which model each kind of work uses, and what caps stop runaway cost.

Why. Agents fail quietly, and cost grows with autonomy: Anthropic reports that its agents use about four times the tokens of a chat, and multi-agent systems about fifteen times.

What we did. Salesbot writes a transcript and a one-row-per-turn trace that needs no model call to produce. The coding pipeline's reports are generated from the task records, not from any session's memory of what it did. The strategy agent's always-on experiment (Step 3) showed that a filter deciding what the person sees must itself be visible. On cost, our model routing rules are deterministic work to scripts, no delegation below about ten tool calls, a stated model on every agent launch, and a stated number of agents before any fan-out. Unattended coding runs stop at a budget, allow one review fix round, and park a task after three failed attempts. Both resident designs make an execution budget a condition of unattended scheduled work, and neither agent runs scheduled work today.

The mistake to avoid. Letting helpers inherit the strongest model by default, or waking an agent on a timer with no limit on what each wake-up may spend.

• A log a person can read, and a report generated from records
• Model and effort named for each kind of work
• Budgets on attempts, fix rounds and spend before anything runs unattended

Step 12: Plan for failure, then deploy and operate it

What to decide. What happens when each part fails: a timeout on every model call, tool call and outside service, retries with backoff and a cap, actions safe to repeat so a retry after a crash does not post or send twice, and a fallback for each part, whether a simpler path or a person. Then how the agent is onboarded, started, restarted and updated, who may change what, and how its workspace is versioned.

Why. Every model, tool and service an agent depends on is sometimes slow, down or wrong, and an agent that retries blindly repeats both the damage and the cost. An agent only its builder can restart is a liability, and one asked to repair its own configuration may be blocked by its own protections.

What we did. Our research fleet gives each mode a hard timeout, after which it synthesises from whatever has arrived and marks each finding's confidence by how many agents agreed. Salesbot stores each stage's findings as soon as the stage completes, so a call left part-way is kept to its last completed stage; its tool handlers return at once and slow work runs in the background, because a blocked handler is dead air; and a stage still open after 20 exchanges moves on.

Onboarding a resident agent ends with a tested round trip that records which identity answered. An authorised operator changes service settings and credentials with the runtime's own commands, keeping a backup first; changes to the agent's instructions and skills arrive as briefs in its mailbox, and the agent installs them itself. Before a restart the operator checks for a turn in progress. The workspace is a git repository, committed automatically, and claims of unattended operation need restart, offline delivery and resumption verified first. Salesbot's deploy script refuses an uncommitted tree or a call in progress, tests the running service, and restores the previous version if a check fails.

The mistake to avoid. Editing live configuration by hand without a backup, or taking a running service for proof of delivery.

• A timeout, a capped retry and a fallback for every dependency; actions safe to repeat
• An operating guide another person can follow
• Operator changes and agent-owned changes separated
• Versioned workspace; restart and update procedures tested

Step 13: Keep it on course

What to decide. How the agent's work stays aimed at its outcome over weeks, as new requests, findings and corrections arrive.

Why. A resident agent drifts in recognisable ways: research with no finish line, documents in place of the outcome, an accreting state file, briefs with no method, checks of form, and mirroring the latest input.

What we did. Each became a design rule, set out with its evidence in keeping a resident agent on course: a finish line and a date on every assignment; one page of current state; the method named in every brief and read in full; sources read rather than summaries; decisions recorded in exact words, so a change of course is a new decision; and hooks for what instructions can only request.

The mistake to avoid. Treating drift as a prompt problem. It is usually a missing finish line, record or check.

• Every assignment has a finish line and a date
• Every brief names its method
• Course changes are recorded decisions, not conversation

Step 14: Know when to stop or simplify

What to decide. When to narrow the agent's job, step it down to a workflow or a session-bound skill, or retire it.

Why. Machinery added to correct an agent often compensates for a job that is too broad.

What we did. We stopped the always-on strategy agent and kept the skill a person starts. For the marketing manager agent, a fuller structure is designed and accepted in fifteen recorded decisions, and we deferred building it: the agent works with the installed library and its existing records, and structure is added only where use shows a need. The marketing agent's plan lists stages of autonomy, each with a stated reason to advance.

The mistake to avoid. Measuring an agent by how much it does on its own rather than by the outcome in its role.

• A stated reason to advance to each further stage of autonomy
• A review of whether a simpler form would now do the job
• Retirement planned: records archived, channels closed

Working with us

AR Labs builds agents like these. We have built and run our own strategy agent, resident marketing manager agent, coding agents and research agents, and a public browser voice agent; we have designed, not delivered, a system for an outside business. If you want an agent built for a job in your business, how AR Labs builds an agent sets out the process, which follows our client delivery framework, and the about page says how to reach us.