Shadow API paper hits reproducibility crisis

This is the kind of paper that turns a vague community suspicion into a concrete supply-chain problem for AI research and tooling.

• –The biggest takeaway is not just that shadow APIs are sketchy, but that they are already deeply cited and widely used in peer-reviewed work
• –The paper shows model identity checks and benchmark scores can drift in different ways, so even “looks right” outputs may still be behaviorally wrong
• –The medical and legal benchmark failures make this more than a reproducibility issue; it is also a reliability and safety issue for high-stakes deployments
• –Developers using indirect providers for coding tools, evals, or agents should treat direct official API access and model fingerprinting as basic provenance controls
• –The broader implication is ugly: API provenance is now part of the experimental setup, and papers that omit it are harder to trust