# Before You Hire an AI Agency: A 30-Minute Self-Audit (Part 1 of 2)

Most AI agency pitches collapse under a 30-minute self-audit you can run with no technical skills. Eleven specific questions, the F12 check, and the cost math that lets a 25-person company evaluate any AI vendor without hiring a consultant to evaluate the consultant.

Author: J.A. Watte
Published: May 14, 2026
Source: https://jwatte.com/blog/blog-pre-agency-ai-self-audit-part-1/

---

You're sitting across from an AI agency. They've quoted you between $1,500 and $5,000 a month to "implement AI" for your business. They've shown a slick demo, a roadmap, three case studies (none in your industry), and a contract.

You have no way to evaluate the quote.

This post is the 30-minute self-audit you run before you sign. None of it requires technical skills. You don't need a developer. You don't need to understand prompts. You need a stopwatch, a browser, and a willingness to ask the questions agencies hope you don't think to ask.

Part 2 of the series, [the post-agency migration playbook](/blog/blog-post-agency-ai-migration-part-2/), covers what to do once you've decided to walk away. This part is about whether you should.

## The five-minute version (for when you only have five minutes)

Open the agency's demo product in your browser. Press F12. Click the Network tab. Use their AI feature. Look at where the network traffic goes.

If you see `api.openai.com`, `api.anthropic.com`, `api.cohere.ai`, or any other major model provider's hostname, the agency's "proprietary AI" is a wrapper around someone else's API. That's not necessarily disqualifying (the [vendor markup walkthrough](/blog/blog-spot-ai-vendor-markup/) covers the honest wrapper vs dishonest wrapper line). But it is a tell that the price they're quoting needs to be justified by something other than the AI itself.

If you don't see those hostnames, it just means they're routing the API call through their own backend. You can't tell what's underneath from the browser. Move to the 30-minute version below.

## The 30-minute version

Eleven questions. Most can be asked in front of the agency rep. A few you'll want to research afterward. Run all eleven before deciding.

### Question 1. What model are you actually running?

The right answer is specific. "We use Claude Sonnet 4.6 for drafting and Claude Haiku for classification." Or "we run GPT-4o for the conversational layer and a fine-tuned BERT for retrieval."

The wrong answer is vague. "We use our proprietary AI." "We have access to multiple models." "It's a mix of techniques."

If they won't tell you the model, that's the answer. Don't sign.

### Question 2. What's the dollar cost per query of the underlying model?

A good agency knows this number. They have to; it's their own cost-of-goods-sold. If they say "we don't break it down that way," they're either lying or running their business sloppily. Either way, you don't want to sign.

A typical answer for an average-complexity query in 2026 is somewhere between $0.001 and $0.05 depending on the model. If they say "$2 per query, that's just our cost," they're misleading you. Run the math on the [AI Vendor Cost Reverse-Calculator](/tools/ai-vendor-cost-reverse-calculator/) before you talk to them.

### Question 3. What's your gross margin on this contract?

This sounds rude. Ask anyway. A reasonable agency will say "we target 70-80% gross margin, here's how we get there." That's transparent. An evasive answer ("we don't think about it that way") usually means the gross margin is high enough to be embarrassing.

The math tells you whether the price is fair. If their underlying cost is $40/month and they're charging you $3,500, the gross margin is 98.8%. That number is real. It also tells you how much room there is to negotiate.

### Question 4. Who else has you implemented for, in my industry?

Specific named clients. Not industry verticals. Not anonymized case studies. Actual companies you can call.

If the agency works in seven different verticals with case studies in none of them, they're a general-purpose AI implementation shop. That's fine if your needs are general. It's not fine if they're charging you premium prices for specialized work they haven't actually done.

### Question 5. What happens to the data the AI sees?

The right answer covers three things:

1. Where does the data sit at rest? (Their servers, their cloud, their fine-tuning corpus, the model provider's servers, customer-isolated, or shared.)
2. Who can access it? (Their engineers, their support team, the model provider's staff.)
3. What's the retention policy? (Default: 30 days at OpenAI/Anthropic for API calls. The agency may have additional retention.)

If they can't answer all three precisely, your contract has a data-governance hole. Don't sign without clarifying. If you're in a regulated industry (healthcare, legal, financial), don't sign without legal review either way.

### Question 6. What's the integration we're actually paying for?

Most "AI implementation" projects are really integration projects with an AI shim on top. The agency's value isn't the AI; it's connecting the AI to your QuickBooks, your CRM, your help desk, and your email.

That's legitimate work. Ask for a list of the integrations they'll build. Get specific. "We'll connect Salesforce" is not specific. "We'll build a Salesforce → Claude bidirectional sync with these 12 fields on these 4 objects" is.

If the integration list is short or vague, you're not paying for integrations. You're paying for the AI markup. See questions 2-3.

### Question 7. What's the human-in-the-loop architecture?

Every AI system that sends emails, posts publicly, or moves money should have a human approval step somewhere. Ask: "show me where in your workflow a human reviews before action."

If the answer is "everything is automated end-to-end," the agency hasn't thought about agent safety. That's a $47K-runaway-bill waiting to happen. (Reference: [the agent cost-controls post](/blog/blog-ai-agent-cost-controls-smb/) for the specific incident.)

If the answer is "we leave that up to you," the integration isn't actually finished. The hardest part of agent deployment is the approval-gate UX. If they're handing you a half-built system, you're paying agency prices for DIY.

### Question 8. Can I see the actual prompt/skill text?

The agency's "magic" is usually a prompt file. A skill file. A few hundred words of instructions to Claude that define what it does. Ask to see the text of the prompts/skills they'll be running on your data.

A confident agency will show you. They'll talk about why the instructions are written the way they are. They'll discuss the tradeoffs.

A scared agency will say "that's our proprietary IP." If a 400-word instruction file is the source of competitive advantage, the rest of the offering is probably even thinner.

Run any prompt they show you through the [Claude Skill Linter](/tools/claude-skill-linter/). If it flags more than two critical issues, the agency isn't doing safety work either.

### Question 9. What happens when I want to leave?

The right answer is "we export your data in standard formats, hand you the prompts/skills we built for you, and stay available for 30 days of transition support."

The wrong answer is anything that involves keeping their software, their dashboards, their custom integrations, or their fine-tuned models when you leave. If the lock-in story is "you can't do this without us," they're admitting their value is the lock-in, not the work.

### Question 10. What's the contract length?

If they're pushing for 12 or 24-month minimums, ask why. The only legitimate reason for a long minimum is that the agency is doing significant up-front integration work that won't pay back in fewer months. Ask them to break that out.

For most SMB AI projects, a 3-month evaluation period followed by month-to-month is the right structure. Anything tighter benefits the agency more than you.

### Question 11. What's the total cost of saying no?

If you say no today and decide to build this yourself with a Claude Pro subscription ($20/month), a couple of weekends, and free tools, what does that get you?

For most SMB workflows in 2026, the honest answer is: most of the way to what the agency was offering. The remaining 20% (the hard integrations, the support, the ongoing iteration) is what an agency would be earning their fee on. If that 20% costs you $3,500/month, that's a $150,000/year line item. Compare to the $20/month DIY alternative.

The right question isn't "is this worth it?" It's "is what I'm paying for the right 20%?"

## What good answers look like, in aggregate

A good agency:

- Tells you the model
- Knows the per-query cost
- Quotes a reasonable gross margin (60-80%)
- Has named clients in your industry
- Has a clear data-governance story
- Builds real integrations you couldn't easily build yourself
- Designs human approval into the workflow
- Shows you the prompts
- Has a clean exit path
- Offers a short evaluation period
- Honestly addresses the DIY alternative

If they hit 9 of 11, sign. They're doing real work and they know it.

If they hit 5 of 11, negotiate the price down by 40-60% and re-evaluate.

If they hit fewer than 5, walk. There are better-aligned agencies, and the DIY alternative is honestly close.

## The audit tools that catch this for you

- **[AI Vendor Cost Reverse-Calculator](/tools/ai-vendor-cost-reverse-calculator/).** Plug in their quote + your query volume. Outputs the implied markup multiplier. Use the negotiation prompt it emits.
- **[Claude Skill Linter](/tools/claude-skill-linter/).** Paste any prompt or skill the agency shows you. Flags missing safeguards. If they fail more than two critical checks, the agency isn't doing the safety work you're paying for.
- **[API Secret Leakage Audit](/tools/api-secret-leakage-audit/).** Run on the agency's own product. If they're leaking API keys in their frontend, their attention to detail tells you everything.
- **[FBI Fraud Reflex Card for SMBs](/tools/fbi-fraud-reflex-card/).** Pattern-matching for sales-pitch red flags that AI agencies specifically use.
- **[Third-Party Script Cost](/tools/third-party-script-cost/).** Shows you what their product is actually loading in the browser. If it's running 14 different SaaS pixels alongside the AI, that's a different kind of warning sign about how their team builds.

## What's next

Part 2 of this series, the [post-agency migration playbook](/blog/blog-post-agency-ai-migration-part-2/), walks through what happens after you've decided to walk away. How to extract your data, how to rebuild the integrations in-house or with a better-fit vendor, and how to keep the workflows running during the transition.

## The deeper version

The complete argument for evaluating AI vendors like any other capital expenditure is in [The $20 Dollar Agency](https://www.amazon.com/dp/B0FB17VG3D) (Digital Empire series, $9.99 on Kindle). The 11 questions above are the tactical version. The book is the strategic frame.

## Related reading

- [Before you pay an agency $3,500/month for proprietary AI](/blog/blog-spot-ai-vendor-markup/), the technical companion (F12 check + markup math).
- [Claude for Small Business walkthrough](/blog/blog-claude-for-small-business-walkthrough/), the DIY alternative to most agency pitches.
- [How a small business runs AI agents without a $47K surprise bill](/blog/blog-ai-agent-cost-controls-smb/), the safety baseline.
- [Connector permission cheat sheet](/blog/blog-connector-permissions-claude-smb/), the integration governance the agency should be doing for you.
- [AI fraud reflexes for SMBs in 2026](/blog/blog-ai-fraud-reflexes-smb-2026/), the broader vendor-evaluation pattern.

## Fact-check notes and sources

- OpenAI public pricing per query (the cost-per-query benchmarks used in Question 2) per [openai.com/api/pricing](https://openai.com/api/pricing).
- Anthropic Claude public pricing per [anthropic.com/pricing](https://anthropic.com/pricing).
- OpenAI's [API data retention policy](https://platform.openai.com/docs/models/how-we-use-your-data) (30-day default).
- Anthropic's [data usage policy](https://www.anthropic.com/legal/commercial-terms) (zero retention by default on API).
- $47K agent-loop incident per [Kusireddy](https://medium.com/towards-artificial-intelligence) (Towards AI, October 2025).

*This post is informational, not legal, contract-review, or vendor-due-diligence advice. The 11-question framework is a heuristic, not a substitute for talking to your own counsel or accountant. No specific agency is being accused of fraud or bad practice; the patterns described are aggregate findings from publicly available research and SMB owner accounts. No affiliation with any named platform is implied.*


---

Canonical HTML: https://jwatte.com/blog/blog-pre-agency-ai-self-audit-part-1/
RSS: https://jwatte.com/feed.xml
JSON Feed: https://jwatte.com/feed.json
Hero image: https://jwatte.com/images/blog-pre-agency-ai-self-audit-part-1.webp