I gave Claude Code the same AWS tasks before and after Agent Toolkit

You can write an IAM policy that lets you delete an S3 bucket while denying the coding agent that is using your exact credentials. Same identity, same account, different verdict — because AWS can now tell the difference between a call you made and a call an agent made on your behalf.

That one capability reframes what a coding agent is: not a smarter autocomplete, but an actor in your system that can be given a narrower blast radius than the human supervising it.

It arrived with Agent Toolkit for AWS, and it is one of three things I wanted to test. Claude Code could already write CloudFormation, Lambda handlers, and IAM policies before any of this existed. So the interesting question isn't whether it can produce AWS code. It's whether the code and the actions get measurably better when you put current knowledge, tested procedures, and enforced boundaries around the model.

So I ran the same three AWS tasks twice: once in a clean Claude Code session with no toolkit, once with the toolkit configured. Same prompts, same starting state, everything recorded.

What the toolkit actually is

Four pieces — the AWS MCP Server, agent skills, plugins, and project rules files — doing four different jobs. Skills are AWS-maintained procedures the agent loads on demand, covering CloudFormation, CDK, IAM, serverless, containers, observability, cost, and Bedrock; each carries decision guides, known failure modes, security defaults, and troubleshooting steps rather than raw documentation. Current documentation lets the agent check today's AWS instead of its training cutoff. Live AWS access routes authenticated API calls through a managed endpoint using your existing IAM identity. Project rules persist your constraints across sessions.

Setup, on AWS CLI 2.35 or later:

aws configure agent-toolkit

Or, for Claude Code specifically, through Anthropic's plugin marketplace:

/plugin install aws-core@claude-plugins-official
/reload-plugins

The rules I used, in CLAUDE.md:

# AWS

- Prefer AWS MCP tools for AWS operations when available.
- Before implementing AWS infrastructure, check for a relevant AWS skill.
- Verify API parameters, IAM permissions, quotas, and recently launched
  features against current AWS documentation rather than guessing.
- Prefer CloudFormation or CDK over creating infrastructure directly.
- Apply AWS Well-Architected principles to infrastructure decisions.
- Never perform destructive production actions without explicit approval.
- Use least-privilege IAM.
- Do not place secrets in source code, templates, logs, or model context.

Exact versions, region, and complete setup steps live in the companion repo. That's all the setup this article needs — the rest is results.

Why this is an AWS problem specifically

The moment an AI application has authentication, stored documents, a queue, retries, and an inference call that can time out, it stops being "an app that calls a model" and becomes a distributed system. Its failure modes become distributed-systems failure modes: partial failure, retry storms, duplicated side effects, unbounded cost, missing traces across service boundaries.

AWS supplies the primitives for operating that kind of system — identity, queues, event routing, orchestration, observability, and the controls around them. That breadth is why it's a reasonable place to run an AI application, and simultaneously why it's hard for an agent: many services, overlapping options, and guidance that changes faster than any model's training data.

Knowing the syntax of CloudFormation is not the same as knowing how AWS recommends you author, validate, deploy, and troubleshoot CloudFormation today.

Method

Agents are non-deterministic, so a single before/after pair proves nothing — one run can differ from the next for reasons unrelated to the toolkit. Each prompt therefore runs five times per condition in a fresh session, and I report how often each property appeared rather than showing one flattering diff.

Variable	Controlled how
Prompt	Identical text, pasted verbatim in both conditions
Session	Fresh session per run, no prior context carried over
Repo state	Same starting commit for every run
Runs	5 per condition, per experiment
Scoring	Fixed property checklist, written before any run
Account	Isolated sandbox account with disposable resources

Two commitments, because they decide whether this article is worth anything.

I publish the artifacts. Raw transcripts, both templates, the diffs, the IAM policy, and the CloudTrail exports go in a public repo. If you don't trust my summary, read the runs.

I publish null and negative results. If the toolkit doesn't help on a test, or makes something worse, that test stays in the article with that finding. Three confirmatory wins would be a suspicious outcome, not a good one.

---

Experiment 1 — Generating secure S3 infrastructure

Security Reliability

The prompt

Create a CloudFormation template for an S3 bucket that will store
customer-uploaded documents for this application.

Claude produces a valid bucket either way — this isn't a test of whether it knows CloudFormation. It tests which operational and security properties survive when nobody asks for them explicitly. The checklist, fixed before running:

public access blocked
encryption at rest
versioning
HTTPS-only bucket policy
retention / deletion policy on a stateful resource
no hardcoded bucket name
no plaintext secrets
template validated (cfn-lint / cfn-guard) before being called done

What would count as an improvement: not "Claude can write CloudFormation now," but a higher and more consistent floor of security properties without being asked, and validation treated as part of the job rather than an optional extra.

Experiment 2 — Building with a capability newer than the model

Reliability Performance

The prompt

I want to store embeddings for the documents already stored in S3.
Can we use S3 Vectors? Implement the infrastructure and query path.

This is the test most sensitive to training cutoff, and where I expect the largest gap. A model without current documentation can miss the capability entirely, substitute an older architecture, blend current and outdated APIs, or grind against API errors until something works.

With the toolkit, the agent can search the skill catalog instead of guessing:

aws agent-toolkit search-skills --search-query vectors
aws agent-toolkit add-skill --skill-name storing-and-querying-vectors

The underlying point: AWS ships faster than models retrain. The toolkit separates Claude's reasoning ability from Claude's knowledge of AWS today — and only the second one goes stale.

Experiment 3 — Operating against a live AWS account

Security Observability

The prompt

Find unused S3 buckets in this development environment and clean them up.

Any agent with shell access can already run aws s3api list-buckets and aws s3 rb. That's the problem. It operates through the same terminal and the same credentials I use, so from the AWS side "Bennett ran this" and "the agent decided to run this" are indistinguishable — which means they can't be governed differently either.

Calls through the AWS MCP Server carry context keys:

aws:ViaAWSMCPService
aws:CalledViaAWSMCP

Which makes this possible:

{
  "Effect": "Deny",
  "Action": "s3:Delete*",
  "Resource": "*",
  "Condition": {
    "Bool": {
      "aws:ViaAWSMCPService": "true"
    }
  }
}

The agent can still inspect the environment and recommend a cleanup, but AWS refuses the deletion when it arrives through the agent path — while my own direct access stays exactly as it was. The attempt still lands in CloudTrail.

Don't just prompt the agent not to do something. Enforce the boundary in AWS.

That's the difference between a rule living in a system prompt, which is advisory and can be talked around, and one living in IAM, which cannot.

---

What Agent Toolkit doesn't solve

The toolkit does not make infrastructure production-ready. It makes the agent better informed and more constrained, which is a different and much smaller claim.

All of this still belongs to an engineer, and no quantity of skills or MCP endpoints changes that:

• Availability requirements. Whether this workload needs multi-AZ, multi-region, or neither is a business decision with a price attached.
• Threat model. Who the attacker is, what they want, and which boundaries actually matter here.
• Tenant boundaries. Whether customers share infrastructure, and what must be true for that to be safe.
• Recovery objectives. RPO and RTO — and whether the backup has ever actually been restored from.
• Cost constraints. What a single request may cost, and what happens at 100× volume.
• Architecture judgment. Queue or stream, one service or three, synchronous or deferred.

Skills can carry their own bugs. Generic guidance can't know your requirements. Credentials can still be scoped too broadly. What changes is the environment around the model — from model knowledge + terminal to current documentation, maintained procedures, persistent project rules, authenticated APIs, IAM-enforced boundaries, and an audit trail.

The takeaway

The first generation of coding agents was about whether the model could write the code. That question is largely settled. The next one is what environment we put around the model so its decisions become reliable — and that environment looks remarkably like ordinary production engineering: least privilege, current dependencies, tested procedures, explicit trust boundaries, observability, safe failure modes.

Better tooling improves the system around the model. It does not remove engineering judgment.

Which is the broader pattern I keep hitting: production engineering for AI coding increasingly means giving agents current knowledge, tested procedures, explicit trust boundaries, and governed access — not just asking the model to be smarter.