Validate the manifest, reject on failure, and your plugin client is non-conformant

Agent Plugins 1.0.0 ships a JSON Schema for plugin.json. It sets
additionalProperties: false. So the obvious loader is four lines:

const manifest = JSON.parse(await readFile(join(dir, 'plugin.json')));
if (!validate(manifest)) return reject('invalid manifest');

That loader is wrong, and the specification says so in a sentence most people
never reach. §5.2:

Clients MUST report and ignore each unknown field and MUST continue
loading the plugin if the manifest otherwise satisfies this section.

An unknown top-level field is a schema violation you have to tolerate. §8.1
says the same for an extensions field that isn't an object. Every other schema
violation is fatal. So a validator gives you one boolean where the spec wants
three different outcomes, and the natural implementation is non-conformant in
exactly two cases and correct everywhere else.

That is the kind of bug that doesn't show up in your tests. It shows up as a
plugin that works in one client and not another, six months later, in someone
else's bug tracker.

This has already happened, repeatedly

I went looking before building anything. In the last few months:

• Codex loaded any directory with a root plugin.json through its Agent Plugins loader, which had no hook support. Every hook in .codex-plugin/plugin.json silently stopped running. Two plugins were dead for a week before anyone noticed.
• oh-my-pi routed packages declaring an agent-plugins.org $schema to a strict provider that dropped any SKILL.md with an extra frontmatter key. Downstream, a plugin went from 33 skills to 3. The fix was to delete $schema from the manifest, so conforming to the standard cost them the standard.
• dotnet/skills shipped manifests with no $schema and with skills, agents and mcpServers as top-level fields. Kiro refused them. Adding $schema got past the rejection and then loaded the package with every functional component excluded.
• VS Code, the largest shipping client, has no validation surface at all. Its troubleshooting page tells you to open SKILL.md and check the name field by hand, because "Invalid names cause the skill to be silently skipped."

Different clients, same shape: the loader disagrees with the spec, nothing
errors, and a user finds it.

Testing a loader instead of a package

There are good tools that check whether a plugin you wrote is valid. There was
nothing that checks whether a client that loads plugins behaves the way the
spec requires. Those are different jobs and they need different fixtures.

So the unit I settled on is a pair: a real plugin directory on disk, and the
load report a conformant client has to produce for it.

fixtures/core/AP-5.2-UNKNOWN-FIELD/
├── plugin/
│   └── plugin.json        # canonical $schema, name "demo", plus "skills": []
└── fixture.json

{
  "rejected": null,
  "loaded": { "skills": [], "mcpServers": [] },
  "skipped": [],
  "reported": [{ "field": "skills", "ruleId": "AP-5.2-UNKNOWN-FIELD" }]
}

rejected: null is the whole point of that fixture. Validate-and-reject answers
rejected: "additional-properties" and fails it.

Hooking up a client is one script that takes a directory and prints that JSON:

#!/usr/bin/env node
import { loadPlugin } from 'your-client';

const result = await loadPlugin(process.argv[2]);

console.log(JSON.stringify({
  rejected: result.ok ? null : result.reason,
  loaded: {
    skills: result.skills?.map((s) => s.name) ?? [],
    mcpServers: result.servers?.map((s) => s.name) ?? [],
  },
  skipped: result.dropped?.map((d) => ({ what: d.path })) ?? [],
  reported: result.warnings?.map((w) => ({ field: w.field })) ?? [],
}));

No imports, no plugin API, no language requirement. It shells out, so a Go or
Rust client works the same way.

What it found

I wrote an adapter for a published 1.0.0 loader and ran all 133 fixtures at it.
131 passed. Two failures, both real, both since reported upstream.

The sharpest one is §4.1 containment. The spec lists five failure boundaries for
a path that resolves outside the plugin root. That loader enforces the first,
for plugin.json, and not the two for skills/:

await fs.symlink('../../outside/escaped', '/plugin/skills/escaped');

const result = await loadAgentPlugin(fs, '/plugin');
console.log((await result.plugin.skills.list()).map((s) => s.name));
// [ 'alpha', 'escaped' ]   the spec wants [ 'alpha' ]

A plugin is a directory someone downloaded. A link in it reached outside itself
and the file came back in the model's context, with nothing reported. The
manifest check shows the containment design was already there, which is what
makes it look like two boundaries that were missed rather than a decision.

One client is not enough to trust a corpus

At that point I had a suite that said a real client was wrong twice. That is
exactly the position where you should not believe yourself, because a corpus run
against one implementation cannot tell "the client is wrong" from "the fixture is
wrong".

So I found a second 1.0.0 client, written by someone else, with its own
conformance document, and ran the same 133 fixtures at it.

It passed 132, failed nothing, and passed both of the §4.1 fixtures the first
client fails. That is what turns those two into a defect rather than my opinion.

It also found three bugs in my corpus:

• Two fixtures required a remote MCP entry to be activated, when the rule under test was only that headers and urls are not expanded. The second client validates such an entry and then declines to connect, for a documented reason: its MCP runtime forwards custom headers across redirects and the spec forbids that. Declining is not a violation. One fixture lost the header from its control server, the other became partial with the entry optional.
• A fixture I had filed under core asserted that a non-object value inside extensions is fatal. Both clients read §8.1's report-and-ignore exception as covering member values, against me. Two independent implementations agreeing against my reading is not a defect in them. It moved to disputed, accepts either answer, and I have asked the spec maintainers which one is intended.

If you are building something like this, that second implementation is not a
nice-to-have. It is the only thing standing between a conformance suite and a
list of one person's opinions with a CLI on top.

Three things I got wrong before that

Grading everything equally. §7.1 says a client SHOULD report a skill it
skipped. §5.2 says it MUST report an ignored field. If you fail both, you
have written a linter people turn off. Mismatches on the SHOULDs are warnings.
Whether the component actually loaded is the MUST, and that is checked.

Asserting the rejection string. The spec defines no rejection vocabulary. My
first diff compared rejected as text, which made the suite a naming quiz. It
now compares null against non-null and echoes your string in the failure line.

Pretending everything is decidable. Some outcomes are genuinely open. sse
support is OPTIONAL. §4.1 says symlinks MAY resolve to in-root targets. Agent
Skills lists its frontmatter fields without saying the set is closed, which is
the oh-my-pi argument and it has two reasonable sides. Those fixtures accept
either answer and record which one you gave. A corpus that graded them would be
picking sides in arguments the spec has not settled.

There is a fourth, which is that a load report cannot see everything. Whether
PLUGIN_ROOT reaches a subprocess is not visible in a report about what loaded.
Eight fixtures are marked partial and say in writing what they do and do not
assert, rather than implying coverage they do not have.

Adopting it without a red build on day one

Nobody wires a new conformance suite into CI and gets green. So it gates on
change:

apconform run --adapter ./adapter.mjs --baseline conformance.json --update-baseline

Commit that file. After it, only fixtures that regressed fail. Existing
failures print as KNOWN and don't gate, fixes print as FIXED, and a fixture
that is new after an upgrade has to pass on its own, because new coverage is
exactly the thing you want to hear about.

Try it

npm install --save-dev agent-plugins-conformance-kit
npx apconform run --adapter ./my-adapter.mjs

133 fixtures, 89 rules covering spec sections 4 through 11. Every rule carries
the normative sentence verbatim, and npm run verify:sources proves each quote
is still a byte-identical substring of the published spec, so the corpus can't
quietly drift from the document it claims to implement. CI runs both clients on
every push, so if a fixture starts disagreeing with a known-good implementation,
that shows up as a red build rather than as someone else's wasted afternoon.

https://github.com/Booyaka101/agent-plugins-conformance-kit

If you maintain a client, I would genuinely like to know which fixtures you
disagree with. The four marked disputed are there because I could not
honestly call them either way, and a second implementer's opinion is worth more
to that file than mine is.