← Back to Blog

Ponytail Tells the Agent What to Do. These 9 Repos Check That It Did

· 22 min read A single file of small spheres rolling out through a tall archway of warm amber light and across a deep navy floor.

In October we wrote about Ponytail, a set of written rules that pushes an AI coding agent toward the smallest complete change and gives it a strict audit method. We used that method on a billing portal and it found seven serious problems. We also wrote this sentence: "Whether the agent follows them, and whether its conclusion is true, still needs checking."

This post is about that checking. On November 18 we went back through Trendshift and GitHub's trending pages looking for projects that do the part a prompt cannot. Nine are worth a small business's time. For each one you get what it is for, how to start, and where it stops. We ran three of them ourselves, and one of those runs turned up a conflict with Ponytail that you would otherwise find the hard way.

What rules alone leave open

Ponytail's audit skill is 103 lines of text. It sets a priority order (correct, safe, holds under load, tested, fast, lean), demands a concrete failing case for every finding, and ends each report with a list of what was not checked. That is a good method. It is still text that an agent reads and may or may not follow.

Four questions stay open after any rules-only review:

  1. Did the agent read everything it should have? On a large change it can quietly skip files.
  2. Is each finding true? The agent that found a problem is the worst judge of whether it is real.
  3. Are the tests real? A test that mocks the thing it tests passes forever.
  4. Does the skill change the outcome at all? The only published numbers are the author's.

The method itself has not moved since our article. Ponytail's repository shows no commits under its skills folder after October 21, and the one release since then (5.3.3, October 29) fixed display and startup issues. So the gaps above are not being closed from inside. They are being closed by other projects.

The nine at a glance

Stars are from GitHub at 18:58 UTC on November 18, 2026. Stars measure attention, not quality, and we come back to that near the end.

Scroll to compare columns and rows when the table extends beyond the box.

Repo Gap it closes What you need License Stars
landmines Fake tests A JavaScript or TypeScript project, Node 22.18+ MIT 534
open-code-review Skipped files, drifting line numbers Git 2.41+, a model endpoint or your existing agent Apache-2.0 54,152
modern-web-guidance Outdated web patterns Node Apache-2.0 22,735
security-audit-skill Unverified findings An agent that can run parallel sub-agents, Node MIT 75,275
elephant-goldfish-model Unreviewed plans, stale docs An agent that can start fresh sub-agents Apache-2.0 21,082
codex-security Unverified security fixes Node 22.13+, Python 3.10+, an OpenAI login or API key Apache-2.0 14,077
SkillEvaluator Unmeasured skills Python 3.12 to 3.14, plus Docker and a model key for the live tier Apache-2.0 3,611
rust-anything Rewrites that change behavior A Rust toolchain or the prebuilt binary MIT 3,037
LoopForge No human sign-off Docker, a server, model keys MIT 3,776

Checks that run as code

These three do not ask the agent to behave. They fail a command when it does not.

landmines: tests that cannot lie

What it is. A set of rules for Oxlint, a fast JavaScript and TypeScript linter. The README states the problem better than we can: agents "mock the thing under test, sleep until a race usually goes their way, retry the flake, raise the timeout, and narrate the diff in comments. Every one of those passes CI. None of them is true."

Use it when an agent writes or edits your tests, and especially when you ask for a regression test after a bug fix. A back test (run the new test against the old code and prove it fails) shows that a test can detect the defect. Landmines stops the test from being faked in the first place.

How. The README installs with Bun. We used npm in a clean Node 24 container and it worked:

npm install -D oxlint @interfere/landmines

Then register the testing rules in .oxlintrc.json:

{
  "jsPlugins": [
    { "name": "landmines-testing", "specifier": "@interfere/landmines/testing" }
  ],
  "rules": {
    "landmines-testing/no-test-doubles": "error",
    "landmines-testing/no-fixed-sleep": "error",
    "landmines-testing/no-flake-masking": "error"
  }
}

Run npx oxlint src. Each failure message tells the agent what to do instead, so the fix lands in the same loop that tripped it.

What we measured. We wrote a two-file sample on purpose to trip the rules: a test that mocked the module it was testing, waited a fixed 500 milliseconds, and retried three times. With the full recommended configuration, Oxlint 1.94.0 and landmines 0.1.0 reported four errors in 265 milliseconds. Three were the test tricks. The fourth is the one to know about.

The conflict with Ponytail. Ponytail marks deliberate limits with a shortcut: comment, and its debt command collects them. The landmines recommended configuration includes a no-comments rule that removes prose comments and keeps only functional directives and single-line WARNING: notes. Our // shortcut: comment failed. Our // WARNING: comment passed.

The README on GitHub documents an allow option for adding prefixes. We tried it and got "Options validation failed". The option was merged to the main branch at 18:43 UTC on November 18. The package on npm, version 0.1.0, was published the day before and does not have it. Until the next release, register only the testing plugin as shown above and leave the style plugin out. We ran that exact configuration as well: three test errors, and the shortcut: comment left alone.

Limits. It is three weeks old with 534 stars, and it covers JavaScript and TypeScript with Vitest and MSW. A Python or PHP shop gets nothing from it. The rules are opinionated: banning every mock is a position, not a law of nature, and three of the rules accept a per-file allowance with an exact count so an existing codebase can adopt them and ratchet down.

open-code-review: the pipeline decides what gets read

What it is. A command-line reviewer that Alibaba says began as its internal review assistant. It reads a Git diff, picks the files, groups related ones, matches review rules to each file, and hands each bundle to an agent with its own context. A separate step re-checks where each comment points.

Why it matters next to rules. Its README names three failures of reviewing with a general agent and a skill: incomplete coverage on larger changes, reported issues that do not match the real code location, and quality that swings with small prompt changes. Its diagnosis: "a purely language-driven architecture lacks hard constraints on the review process." That is a direct description of the first gap above.

Use it when:

  • You want every pull request reviewed the same way, in CI or before you push.
  • You have inherited a codebase with no meaningful diff. ocr scan reviews whole files.
  • You have an agent subscription and no separate API key. Delegation mode lets your current agent do the reading while the tool still controls file selection, rules and reporting.

How.

npm install -g @alibaba-group/open-code-review

ocr config provider      # pick a model provider (native mode only)
ocr config model

ocr review                    # staged, unstaged and untracked changes
ocr review --from main        # this branch against main
ocr scan --path src/billing   # whole files, no diff needed
ocr delegate preview          # delegation mode: your agent reviews

For agents that support skills, npx skills add alibaba/open-code-review offers both modes in its installer.

Limits. The README publishes a benchmark built from 200 real pull requests across 50 repositories with 1,505 annotated issues, and reports higher precision at about one ninth of the tokens of a general agent on the same model. It also says this, to its credit: "its Recall is lower than general-purpose agents." Precision means fewer false alarms. Recall means fewer misses. For routine diffs, fewer false alarms is the right trade. For the code that moves money, a miss costs more than a false alarm, so we would keep a slow, deep audit for those paths. The benchmark is the vendor's and we did not run the tool.

modern-web-guidance: the platform already does it

What it is. A set of skills and a command-line tool that put current web platform practice in front of a coding agent. The README says it is stewarded by the Chrome, Edge, Firefox and MDN teams, and gives the reason plainly: "Coding agents often default to older patterns because LLM training data contains vast amounts of legacy code."

Why it matters next to rules. Ponytail tells the agent to prefer what the standard library or platform already provides. That only works if the agent knows what the platform provides this year. This project supplies the list, 262 features at our reading, and a scanner that needs no model at all.

Use it when you maintain a website or web app, an agent writes front-end code for you, or you want a free second look at old JavaScript and CSS.

How. Nothing to install for a first look:

npx modern-web-guidance@latest search "animate a dialog modal backdrop"
npx modern-web-guidance@latest retrieve "animate-to-from-top-layer"
npx modern-web-guidance@latest legacy-scan .
npx modern-web-guidance@latest install     # adds the skills to your agent

What we measured. We ran version 1.0.1 against the source of this site. The search returned ten guide IDs for the dialog question. The legacy scan reported 338 modernization opportunities across 7 guides, and said it had scanned 38 guides with deterministic rules, about 20% of its guidance.

Scroll to compare columns and rows when the table extends beyond the box.

Finding Count
Theme toggle that does not update the color-scheme meta tag 114
innerHTML = "" used to empty an element 79
Enter-key handlers that ignore text composition 72
Older array-copy patterns 37
Hand-rolled grouping loops 16
Underlines faked with borders 10
Everything else (5 smaller rules) 10

Read that table the way the tool tells you to. It labels its own results "provisional" and prints a keep rule and a drop rule for each. The 114 is mostly one inline snippet: 109 of our 360 tool pages carry the same line. That is one template fix, not 114 jobs. The 72 Enter-key findings are the ones we would act on first, because the consequence is concrete: a visitor typing Japanese or Chinese presses Enter to confirm a character and the form submits instead.

Limits. The scanner covers a fifth of the guidance, and static pattern matching produces false positives. The rest of the value depends on your agent actually calling the skill.

A second opinion that is built in

When we audited the billing portal, one agent wrote the report and a second model re-read the code for every finding. That step confirmed all seven and widened one. These three projects make that step part of the tool.

security-audit-skill: three verdicts and a ledger

What it is. Cloudflare's security audit skill, which the README calls the single-repository starting point for the company's larger vulnerability discovery system. It runs six phases: reconnaissance, hunting by coverage, validation, structured output, independent verification of the record, and reporting.

The ideas worth taking, even if you never install it:

  • Three verdicts, not one. A candidate ends as confirmed (a complete source trace and an observed result), needs_validation (an exact unresolved fact, and no severity), or rejected (disproved, and kept on record). A plain findings list has no honest place for "probably, but we could not prove it".
  • The finder never verifies. The README's words: "The agent that checks a finding is never the agent that found it."
  • A coverage ledger. A file records what was examined, and a small validator script checks the ledger and the findings against a schema. "Not checked" stops being a sentence at the end and becomes data.
  • One run is not enough. "In our test runs, a single run found roughly half of the vulnerabilities that repeated runs found in total." Runs against the same repository are additive.

Use it when you are about to launch something that handles logins, payments or customer records, or when you want a second audit to pick up where the first stopped.

How.

npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit

Then ask your agent: security audit this codebase. Output goes to a folder outside the repository by default.

Limits. It needs a model that supports parallel sub-agents, so it spends more than a single pass. To execute the target's own code it requires a sandbox enforced by the operating system with networking disabled. Without one it keeps the lead as needs_validation, which is the right behavior. The repository was last pushed on September 14. We ran a static check on the skill file (see SkillEvaluator below): it is large, about 5,500 estimated tokens, and one of six agent environments tested may cut it short.

elephant-goldfish-model: review the plan with fresh eyes

What it is. A plugin for designing and reviewing software specifications. One agent, the elephant, holds the project context and drafts the document. Reviewer agents, the goldfish, read the draft and the code it refers to without the earlier conversation. You decide. The repository sits under Google's GitHub organization and says it is "not an officially supported Google product".

Why it matters next to rules. An agent that has spent an hour agreeing with you cannot see the assumptions you both made. A reviewer with no memory of the conversation can. It applies the idea from the Cloudflare skill one step earlier, to the plan.

Use it when:

  • A change needs decisions before code: what happens when a delivery fails, or the same event arrives twice.
  • You already have a design document, README or pull request plan and want it checked against the live code.
  • Your documentation has drifted from what the code does.

How (Claude Code).

claude plugin marketplace add google/elephant-goldfish-model
claude plugin install egm@egm

Start a new session, then:

/egm:egm-review docs/designs/webhooks.md
/egm:egm-drift-report docs/designs/

The README lists equivalent installs for Codex and Antigravity.

Limits. It reviews documents. It does not run your code. If your agent cannot create fresh sub-agents, the review stops. The README says so itself: "A review does not replace tests or human judgment."

codex-security: find, validate, patch, verify

What it is. OpenAI's command-line tool and SDK for security scanning. It scans a repository, selected paths or a diff, validates candidate findings, generates patches, and can verify a fix that already exists.

Use it when:

  • You want a scheduled security scan with a spending cap.
  • You want a pull request checked for security problems only.
  • You fixed a hole and want a second look at whether it is closed.
  • You have no SECURITY.md and want a draft.

How.

npm install --global @openai/codex-security
cs login

cs scan .                        # whole repository
cs scan . --diff origin/main     # committed changes only
cs scan . --mode deep            # repeated scans, deduplicated
cs scan . --max-time 15          # advisory 15-minute allowance
cs policy .                      # draft a SECURITY.md outside the repo

The repository includes a weekly GitHub Actions workflow. Its example sets an estimated $5 limit per scan and reports findings without failing the job. Findings export as SARIF, JSON or CSV.

Limits. It is metered. It covers security, not correctness in general. The README notes that some requests and protected findings require a separate access program. Besides OpenAI it lists Amazon Bedrock, OpenRouter, Fireworks AI and Cheaper Inference as providers. We did not run it.

Proof that the change, or the skill, works

SkillEvaluator: is the skill any good

What it is. NVIDIA's tool for checking agent skills in three tiers. Tier 1 is static: format, security scanners, and whether each agent environment will load the skill. Tier 2 looks for guidance that repeats or overlaps. Tier 3 runs an agent on tasks with and without the skill and compares.

Why it matters next to rules. It is aimed squarely at the fourth gap. Our Ponytail article had to say that the benchmark figures were the author's and not ours. Tier 3 is how you would get your own.

How.

pip install skillevaluator

skillevaluator quality-check ./my-skill    # no key, no Docker
skillevaluator harness-check ./my-skill    # will each agent load it?
skillevaluator validate ./my-skill         # all three tiers, needs setup

What we measured. We ran version 1.0.1 on three skills in a clean Python 3.12 container, with no API key.

Scroll to compare columns and rows when the table extends beyond the box.

Skill Static score Loads in
Ponytail audit 88.5 of 100 (B) 6 of 6 agents
Ponytail review 84.8 of 100 (B) 6 of 6 agents
Cloudflare security audit 86.2 of 100 (B) 5 of 6, one degraded

All three pass the default threshold of 70. Now look at what the score is made of. The deductions were for a missing author field, missing tags, a description longer than 200 characters, and no sections titled Purpose, Limitations or Troubleshooting. Those are packaging. None of them says whether the audit finds bugs. The tool is honest about this: every one of our reports ended with "Publication eligibility: Incomplete", because static quality is one part of the evidence.

The loading check was the useful half. The Cloudflare skill file is 22,063 characters, and the tool warned that one agent environment may refuse the read or return a cut prefix at that size. A skill that is silently truncated is a rule your agent never saw. That is worth knowing before you trust a long skill, and it costs nothing to check. The same check noted that the fuller Tier 1 schema test would fail both Ponytail skills for a missing top-level heading, a formatting point the quick score does not cover.

Limits. The README gives its support level as Experimental. Tier 3 needs a model provider, a supported agent, a running Docker sandbox and a dataset you have reviewed, and it can incur charges. We did not run Tier 2 or Tier 3.

rust-anything: record the old behavior first

What it is. A tool for porting code to Rust one piece at a time. It records what the original does on generated edge cases and your own inputs, then replays every recorded case against the new code. Nothing ships until they all match.

Our advice is to take the idea and be careful with the tool. The idea is the oldest safe way to change code you do not fully understand: capture what it does today, change it, and compare. That applies to any refactor an agent proposes, in any language, and it is the same instinct as the back test. A Rust rewrite, on the other hand, is the opposite of the smallest complete change. Most small businesses have no reason to do one.

Use it when you have measured one slow path and the rest of the program is fine.

How.

cargo install rust-anything
rust-anything setup            # registers with the agents it finds

rust-anything profile . -- node bench.mjs     # what is slow
rust-anything record rust/fuzzy-rs            # save the original's behavior
rust-anything diff rust/fuzzy-rs              # every recorded case must match
rust-anything gate rust/fuzzy-rs              # done means this passes

Limits. The README has a section called "What it can't catch" and it is the best part: "Passing means the Rust matches every case that was recorded, nothing more." Inputs nobody recorded are the ceiling. It also warns that recording runs your original code many times, so code that writes files or calls a database does so once per case. Run it on a copy with a test database. The repository was two days old when we read it. The speed figures in the README are the author's.

Process and people

LoopForge: a human gate between steps

What it is. A self-hosted workspace from Tencent where agents, scripts and people take part in one saved workflow, which it calls an SOP. A typical one runs clarify, design, implement, review, test, then waits for a person to approve. The README describes the problem it answers: a single agent run "might skip the plan, forget the tests, review its own code, or stop halfway without anyone noticing."

Why it matters next to rules. Ponytail keeps diagnosis apart from editing by making audit and review read-only. LoopForge makes the separation structural. The reviewing role is a different step with its own prompt and model, and the run cannot continue past a gate until someone approves or sends it back.

Use it when several people ship through agents and you need the same stages and sign-off every time.

How.

git clone https://github.com/Tencent/LoopForge.git
cd LoopForge
./start.sh

New accounts get a demo project whose workflow is one script and one approval, so you can watch a gate work without any API key.

Limits, and they matter. By default it listens on all network interfaces on port 8090, and the first account created becomes the administrator. The README tells you to restrict access to your own address until that account exists. Its sandboxes start through the host's Docker socket, which the README says grants control of the host and suits trusted teams only. Signed-in users can read project data across the instance. Its license file is MIT with third-party notices, which is why GitHub's license detector shows it as unclassified.

For a one-person shop this is too much machinery. A written checklist, a second model re-reading the findings, and a rule that nothing merges without you does the same job. Ponytail's own test applies to process as well as code: name the real load before you build for it.

Seventeen use cases, one line each

Scroll to compare columns and rows when the table extends beyond the box.

You want to Reach for First command or prompt
Stop an agent mocking the thing under test landmines npx oxlint src
Stop retries and raised timeouts hiding a flaky test landmines rule no-flake-masking
Stop fixed waits in tests landmines rule no-fixed-sleep
Review every pull request the same way open-code-review ocr review --from main
Audit a codebase you inherited open-code-review ocr scan --path <folder>
Review without a separate API key open-code-review ocr delegate preview
Find outdated patterns in a website modern-web-guidance npx modern-web-guidance@latest legacy-scan .
Give an agent the current way to build a dialog modern-web-guidance npx modern-web-guidance@latest search "..."
Run a security audit with verified findings security-audit-skill security audit this codebase
Re-audit without repeating covered ground security-audit-skill run it again on the same repository
Check a plan before any code exists elephant-goldfish-model /egm:egm-review <path>
Find where the docs and the code disagree elephant-goldfish-model /egm:egm-drift-report docs/designs/
Schedule a weekly security scan with a cost cap codex-security the included GitHub Actions workflow
Check a pull request for security only codex-security cs scan . --diff origin/main
Confirm your agent will load a skill SkillEvaluator skillevaluator harness-check ./my-skill
Prove a rewrite behaves the same rust-anything record, then diff, then gate
Require a person to approve between agent steps LoopForge the demo workflow, then your own

How we would stack them

Nobody needs all nine. For a small business with one application that takes payments and one person who approves changes, this is the order we would add them, cheapest first.

  1. Keep the rules. Ponytail or a similar rule set for everyday work. It costs nothing extra and makes the diffs smaller.
  2. Add the free scanners. legacy-scan on any website, and the landmines testing rules on any JavaScript or TypeScript project. Neither needs a model or a key.
  3. Check your skills load. One harness-check per skill you depend on, and again after your agent updates.
  4. Review diffs with a pipeline. open-code-review in delegation mode, or a codex-security diff scan, on each pull request.
  5. Audit the money paths deeply, twice. Use the audit method you trust and borrow the Cloudflare format: three verdicts, a different agent to verify, and a ledger of what was read. Then run it again.
  6. Back test every fix. Record what the old code did, prove the new test fails on it, then prove it passes on the fix.
  7. Review plans before code when a change needs real decisions.
  8. Stay the gate. Whatever the tooling, a person approves the merge.

Steps 1 to 3 are free. Steps 4 and 5 spend model usage, which is where an existing subscription or a metered key comes in.

What we left out, and why

A "cage" that is not a sandbox. One trending desktop app presents itself as an isolated virtual machine for working with an AI assistant. Its README describes the purpose as getting past the vendor's account checks: its own network routing, time zone and language matched to the exit address, and management of several subscriptions. That is a route to a closed account, not an audit sandbox. We are not naming it.

Attack tools. Two of this week's fastest risers are autonomous penetration-testing and bug-hunting bundles. They have a proper use on systems you own or are contracted to test. One is under AGPL-3.0, which has source-disclosure terms worth reading before you build on it.

Star counts as proof. One repository on this month's lists shows more than 460,000 stars and 114,000 forks seven months after it was created. Another passed 36,000 stars in 18 days. Maybe. We covered the research on purchased stars in our Trendshift roundup. Rank tells you what people are looking at. It does not tell you whether the thing works.

Very new code. Two of our nine are under a month old, and one of those was two days old when we read it. We included them for the idea each one carries. Pin a version, read what the installer changes, and try it on a copy first.

What we did not test

We ran landmines, the Modern Web Guidance scanner and SkillEvaluator's two keyless checks, each in a throwaway container with no credentials and only a scratch folder writable. Everything else in this post comes from each project's README and repository as read on November 18, 2026. We did not run open-code-review, the Cloudflare skill, elephant-goldfish-model, codex-security, rust-anything or LoopForge, and we did not reproduce any vendor benchmark. Our landmines sample was written to trip the rules, so it shows what the messages look like, not how often the rules catch real problems.

Fact-check notes and sources

  • Our records: three runs on our own server on November 18, 2026, between 19:43 and 19:46 UTC, in Node 24.22.1 and Python 3.12 containers. landmines 0.1.0 with Oxlint 1.94.0: four errors on a two-file sample in 265 ms, and "Options validation failed" when the allow option was passed. modern-web-guidance 1.0.1: 338 findings across 7 guides on this site's source, 38 guides scanned. SkillEvaluator 1.0.1: scores of 88.5, 84.8 and 86.2. A count of our own source found the theme snippet on 109 of 360 tool pages.
  • Trending lists: Trendshift daily, weekly, monthly and yearly pages, and GitHub Trending, read between 18:56 and 18:59 UTC on November 18, 2026. Star counts, licenses and creation dates are from the GitHub API at 18:58 UTC the same day.
  • Ponytail: the audit skill (103 lines), the releases list and the commit history for the skills folder.
  • landmines: README for the rules, requirements and quoted text. The allow option arrived in pull request 18, merged November 18, 2026. Version 0.1.0 was published to npm on November 17.
  • open-code-review: README for the design, commands, the three quoted problems and the recall statement. Benchmark data: AACR-Bench.
  • modern-web-guidance: README for the stewardship statement, the feature count and the commands.
  • security-audit-skill: README for the six phases, the three verdicts, the sandbox requirement and both quotations. Background: Cloudflare, Build your own vulnerability harness.
  • elephant-goldfish-model: README for the roles, install commands, skills and the support statement.
  • codex-security: README for features, commands, providers, the access note and the workflow's $5 example limit.
  • SkillEvaluator: README for the tiers, the threshold of 70, requirements and the Experimental support level.
  • rust-anything: README for the loop, the warning about recording, and the section on what it cannot catch.
  • LoopForge: README for the workflow model, the default port, the Docker socket warning and the access model. License text: LICENSE.

Related reading

This post is informational, not security-consulting or legal advice. Test any tool on a copy of your project before trusting it with the real one. Mentions of Ponytail, Alibaba, Cloudflare, Google, Google Chrome, Microsoft Edge, Mozilla Firefox, MDN, NVIDIA, OpenAI, Codex, Tencent, Interfere, Claude Code, Anthropic, GitHub, Trendshift, Oxlint, Vitest, MSW, Docker, Rust, Node.js and Python are nominative fair use. No affiliation is implied.

← Back to Blog

Accessibility Options

Text Size
High Contrast
Reduce Motion
Reading Guide
Link Highlighting
Accessibility Statement

J.A. Watte is committed to ensuring digital accessibility for people with disabilities. This site conforms to WCAG 2.1 and 2.2 Level AA guidelines.

Measures Taken

  • Semantic HTML with proper heading hierarchy
  • ARIA labels and roles for interactive components
  • Color contrast ratios meeting WCAG AA (4.5:1)
  • Full keyboard navigation support
  • Skip navigation link
  • Visible focus indicators (3:1 contrast)
  • 44px minimum touch/click targets
  • Dark/light theme with system preference detection
  • Responsive design for all devices
  • Reduced motion support (CSS + toggle)
  • Text size customization (14px–20px)
  • Print stylesheet

Feedback

Contact: jwatte.com/contact

Full Accessibility Statement • Privacy Policy

Last updated: April 2026