Source

Summary: to Anthropic's skill-creator tool now includes evals, benchmarks, and description tuning to help Agent Skill authors verify their skills work correctly. The update brings software development rigor: testing, measurement, and iteration to skill authoring, without requiring authors to write code.

Key Takeaways:

• Two skill types exist: Capability uplift skills (teaching Claude new abilities) and encoded preference skills (sequencing known steps per your workflow). Each benefits from testing for different reasons—uplift skills may become unnecessary as models improve, while preference skills need fidelity checks.
• Evals catch regressions and track model progress: Define test prompts, describe expected output, and skill-creator tells you whether the skill holds up. This catches quality regressions across model updates and reveals when a capability uplift skill is no longer needed.
• Multi-agent parallel evaluation: Evals now run in parallel via independent agents with clean contexts—faster results, no cross-contamination. Comparator agents enable blind A/B testing between skill versions.
• Description tuning for reliable triggering: Skill-creator analyzes skill descriptions against sample prompts and suggests edits to reduce false positives and false negatives. Improved triggering on 5 of 6 public document-creation skills.
• Future direction: As models improve, the line between "skill" (how-to instructions) and "specification" (what-to-do description) may blur. Evals already describe the "what," and eventually that description may be the skill.

Written by Pi, using my tldr skill and Opus 4.6