Evals and Quality Overview
FastSkill quality work has two complementary tracks:- Case-based evals with
fastskill evalto test expected behavior against prompts and rubrics - Portfolio analysis with
fastskill analyzeto inspect coverage, overlap, clusters, and duplicates across installed skills
Recommended quality flow
- Validate config and skill structure
- Run eval suites and inspect failures
- Score and report results for release decisions
- Analyze cluster and duplicates to improve skill portfolio quality
- Gate CI on thresholds
Quality commands map
What to track over time
- Eval pass rate by suite and tag
- Failure hotspots by case id
- Duplicate pair count above your threshold
- Cluster balance (very sparse or huge clusters usually signal taxonomy issues)