Rockman
An evaluation benchmark for software engineering and reasoning tasks, designed around controlled splits, repeatable evaluation, and protected test material.
Tasks are organized across difficulty levels and categories. Results are aggregated only after individual evaluations have completed.
The public split is for development. Protected splits are for evaluation integrity.
Rockman v0.2 separates public, private, and hidden evaluation material. Hidden tests are encrypted and integrity checked to reduce accidental exposure and contamination.
SHA256 / 3352bcf…
Pass@k · Wilson confidence intervals · statistical significance · difficulty analysis · category analysis
Use the official benchmark configuration when comparing models. Modified protocols should be identified as custom evaluations.