Sisyphus posts 51x attention speedup

The interesting part here is less the raw loss number and more the systems story: better data plus a cheaper attention path looks like it mattered more than any exotic memory trick. The benchmark claims are strong, but the next real test is whether the model can compile, typecheck, or meaningfully complete Rust tasks beyond looking syntactically plausible.

• –Corpus expansion appears to be the biggest win; the jump from core Rust docs to the broader crate ecosystem likely mattered more than architecture tweaks
• –HybridAttention is the right kind of experiment for small code models: local syntax handling plus a recurrent path for longer-range state without quadratic cost
• –The late-training val-loss rise suggests overfitting or a plateau, so the step-18.5k checkpoint may be the more useful candidate
• –The 51x inference gain is compelling, but it needs an apples-to-apples quality eval to prove the cache strategy is truly free
• –For code models, pass@k, parse/compile rate, and task-level editing success will tell you more than perplexity alone