OPEN LINK ↗
// 1d agoRESEARCH PAPER
RTLRepairBench Exposes Hidden Scoring Confounds
This paper introduces RTLRepairBench, a 459-case benchmark for testing whether models can repair broken RTL using specifications. Its audit finds that simulation and formal scoring can disagree because of reset and first-cycle semantics, while conventional scores may not reveal whether models used the faulty RTL.
// ANALYSIS
RTL repair benchmarks may be measuring evaluation protocol quirks as much as repair ability. The paper’s strongest contribution is treating benchmark design itself as an object of study.
- –Combines 374 syntactic and 85 behavioral mutations with simulation and formal-equivalence checks
- –Finds specification-only versus specification-conditioned recovery shifts ranging from -8.5 to +18.3 percentage points across six models
- –Shows initialization, reset, first-cycle, and unknown-value semantics can materially change conclusions
- –Recommends reporting oracle coverage, undecided cases, and specification-only baselines
- –Releases benchmark data, construction code, sanitized evidence, and reproduction tooling for hardware-AI researchers
// TAGS
rtlrepairbenchbenchmarkevaluationdebuggingtestingresearch
DISCOVERED
1d ago
2026-10-21
PUBLISHED
1d ago
2026-10-21
RELEVANCE
8/ 10
AUTHOR
arnaudmercier