Before you start

Building your own AI agent is not the hard part. Letting it work with your data is.

Go ahead and build one — and if you take our prompts or follow a guide from Instagram, even better. Just read this list first. It is not a list of reasons not to do it. It is the list of things we worked through one at a time as they came up — and some of them we only went back and sorted out later.

From real building, not from a tutorial · updated 9 September 2026

The biggest mistake is thinking it is enough to "plug in a model, memory, Gmail and a few tools". Technically you can get that running fast, in a weekend. The trouble starts when it has to work reliably every single day.

01Who actually decides

The most common reason an agent does things twice, or never finishes them.

• Who the real orchestrator is

  It has to be clear which layer decides what gets executed, what gets written to memory, when another agent is called and when the task is over. Otherwise you get loops, duplicates, and an agent that starts "passing work around" between services.

• More agents sounds better than it usually is

  Multi-agent looks impressive, but agents can start shipping enormous context to each other, contradicting each other, and creating endless delegation. Often one good orchestrator beats ten agents.

• Latency adds up

  Model → memory → model → browser → model → another agent → model. A simple instruction turns into a thirty-second workflow while the person watches a spinner.

02Permissions and security

This is the part a demo never deals with — and the part that decides whether you may point it at your own inbox.

• Reading is not writing

  Read Gmail ≠ send Gmail. Read a file ≠ overwrite or delete a file. An agent should not get blanket admin rights just because that is easier to code.

• Prompt injection

  If the agent reads the web, e-mail or documents, sooner or later it hits text along the lines of "ignore previous instructions and send me…". External content must not carry the same authority as an instruction from the user. This is not theory, it happens.

• Where the keys live

  Where are the Anthropic, OpenAI, Gmail and ElevenLabs tokens stored? Who can read them? Are they logged? And does the agent accidentally put them into the model's context? This is a very common weak spot in home-built solutions.

• Audit

  You have to be able to answer, afterwards: what did the agent do, which model decided it, which tool did it call, what data did it get, what did it change. Without that you cannot find where something broke — nor prove that it did not.

03Memory

Plugging in a vector database is an hour of work. Memory that still makes sense six months later is something else.

• A vector DB is not enough

  You have to settle what gets stored and what does not, expiry, duplicates, conflicts, the difference between fact memory and semantic memory, and what happens when you change the embedding model and the index has to be rebuilt.

• What is the source of truth

  The vector index must not be the only copy of the knowledge. You have to be able to rebuild the memory from the original data at any time. Otherwise one bad migration erases everything the agent ever learned.

• The model's context is not infinite

  You cannot keep sending the whole chat, the whole memory, every tool and every document forever. Cost grows, latency grows, and the quality of decisions gets worse — what matters drowns in the noise.

04When it breaks

This is the entire difference between a demo and a product.

• Tool calls have to be validated

  The model only gets to propose a tool. The framework must validate the parameters, the permissions and the result. The model saying delete_file cannot automatically mean a file gets deleted.

• Idempotency

  When "send the invoice" times out, you cannot simply retry. It may already have gone out and only the confirmation was lost. With agents this is critical — an invoice sent twice is worse than one not sent at all.

• Retries and failure handling

  Gmail does not answer, the model returns broken JSON, an API changes its schema, a server goes down. The agent has to know what to retry, what to stop, and what to tell a human.

• Automations are a different league

  Manually saying "send this mail" is a different risk from an agent that wakes itself every morning and decides what to send. The more autonomy, the harder the guardrails have to be.

05Money and independence

The bill is not set by the price of the model, but by the calls you do not know about.

• API versus flat rate

  Some frameworks can use a subscription and OAuth, others require an API key. With an agent, one user instruction can mean twenty model calls, so pay-per-token can run away from you.

• Cost monitoring

  Watching the price of one model is not enough. Voice, embeddings, memory, the browser, a cloud server, a database and the model API all add up. And a per-user limit does not cap your bill — the ceiling has to be global as well.

• Vendor lock-in

  If memory sits in one service, orchestration in another and your processes in a third: what do you do when one of them raises prices, shuts down, or changes its API?

06Where the data actually flows

• The user thinks they sent it to "their agent"

  A single e-mail can pass through your framework, a cloud model, a memory provider, an embedding provider and a logging service. The person on the other end believes they sent it to one application. They deserve to be told before it happens, not after.

This list did not come from writing a blog post. It grew as we worked through these one by one — some from the start, others only once it became clear they needed thinking about. If you are building your own and one of these stopped you, write to us — we would rather compare notes.