Free AI Agent Deployment Checklist

An AI agent deployment checklist is the operational artifact you fill in before you ship an LLM-backed agent to production. It is not a model card and not a prompt evaluation report; it is the deploy-readiness gate. The reason it exists separately from a normal deployment checklist is that LLM agents fail in shapes that traditional services do not — hallucination, context leakage, prompt injection, ungrounded confidence — and the standard pre-deploy review does not surface those.

The checklist below is the version that has worked on real production LLM systems. It assumes you already have model evals; it covers the operational layer that turns "the model performs well in eval" into "we can run this in production and rollback if it goes wrong." If you are deploying an agent without this layer, the first incident will teach it to you. The cost of learning it that way is higher than reading this page.

When to use it

• Any LLM agent that takes actions on behalf of a user.
• Any retrieval-augmented system serving production traffic.
• Any tool-using agent (function calling, code execution, vendor APIs).
• Any new model rollout for a production agent (e.g. major version bump).

The template structure

This is the structure of the template. Copy it into a Notion page, a Linear doc, or a markdown file in your repo — it works in any of them.

AI AGENT DEPLOYMENT CHECKLIST — [agent name]
Version:    [N]
Owner:      [name]
Deploy date: [date]

CAPABILITY SCOPE
  [ ] One-sentence description of what the agent does.
  [ ] Explicit list of actions the agent can take.
  [ ] Explicit list of actions the agent must NOT take.
  [ ] Authority boundaries: when must the agent escalate to a human?

GROUNDING
  [ ] Sources of truth are named and accessible to the agent.
  [ ] Retrieval citations are stored with every answer.
  [ ] Agent refuses when its sources do not cover the question.
  [ ] Refusal behavior is tested with at least 50 out-of-scope queries.

GUARDRAILS
  [ ] Prompt injection mitigations in place and tested.
  [ ] PII redaction layer tested with synthetic PII corpus.
  [ ] Output filter for [unsafe content classes] is active.
  [ ] Rate limits per user and per org configured.

OBSERVABILITY
  [ ] Every prompt + response logged with trace ID.
  [ ] Token usage by user and by org tracked.
  [ ] Latency histogram per call type.
  [ ] Refusal rate alerted if it deviates more than X% from baseline.
  [ ] Tool call success rate per tool tracked.

EVALS
  [ ] Eval set covers [N] core scenarios.
  [ ] Eval covers refusal cases, not just success cases.
  [ ] Eval includes adversarial inputs (prompt injection, jailbreaks).
  [ ] Eval threshold defined for shipping (regression triggers rollback).

ROLLBACK
  [ ] Previous version pinned and runnable.
  [ ] Rollback command and runbook tested in last [N] days.
  [ ] Feature flag in front of new behavior.
  [ ] Canary plan: % traffic at each stage.

HUMAN ESCALATION
  [ ] When the agent escalates, where does it go?
  [ ] On-call coverage for agent failures defined.
  [ ] Escalation runbook for each failure class.

COST
  [ ] Cost-per-call modeled at 10x expected traffic.
  [ ] Budget alert at [N]% of monthly cap.
  [ ] Per-user / per-org cost cap enforced.

POLICY
  [ ] Data-retention policy for prompts/responses confirmed.
  [ ] Customer notice (if any) shipped.
  [ ] Internal AI policy compliance reviewed.

POST-DEPLOY
  [ ] One-week structured review of: refusal rate, error rate, cost,
      top failure cases.
  [ ] Eval set expanded with any failure cases surfaced in week one.

How to use it well

• Refusal is a first-class capability, not an exception. An agent that always answers is an agent that hallucinates. Eval the refusal behavior with at least as much rigor as the success behavior.
• Citations stored with every answer, not just shown. When a customer complaint comes in three weeks later, you need to see exactly which sources the agent grounded the answer in. Citations in the UI are not enough — they need to be in the log.
• Test rollback in the last seven days, not at deploy time. Rollback runbooks decay. A rollback that worked six months ago does not necessarily work today; the time to discover that is not during an incident.
• Cost modeled at 10x traffic. LLM costs scale linearly with traffic, but viral spikes scale super-linearly. Modeling 10x is the cheapest way to find the cost cliff before it hits the bill.
• One-week structured review post-deploy. Most production agent issues surface in week one but go unnoticed if no one looks. A scheduled review of refusal rate, error rate, and top failure cases catches them.

What to skip

Skip the urge to ship without rollback. "We can just patch the prompt" is not a rollback strategy — prompt changes have their own deploy cycle and their own risks. A real rollback runs the previous version unchanged.

Skip eval sets that only cover happy paths. The expensive failures in production are adversarial inputs and out-of-scope queries. An eval that only measures "does it do the thing" misses the failure modes that actually matter.

Frequently asked questions

Is this template free?

Yes. The checklist above is the template. Drop it into your deploy runbook or your engineering Notion.

Can I edit it?

Yes. Common edits: add a Privacy section if you have specific data-handling rules, add a Multi-Tenant section if your agent serves multiple orgs.

Do I need to give my email?

Not for the template. The download is just a polished Notion version; the email is for our newsletter only.