ARC Task Gen Targets ARC-AGI Contamination

ARC Task Gen addresses a real weakness in public benchmarks: models may memorize widely circulated evaluation tasks. Its private, distribution-matched sets are promising, but synthetic benchmark quality still depends heavily on model-generated rules and human validation.

• –Samples grid sizes, colors, and train/test pair counts jointly from the 400-task ARC-AGI-1 evaluation set
• –Uses semantic embeddings to remove near-duplicates and tasks too similar to known evaluation examples
• –Supports OpenAI-compatible endpoints, including local vLLM, Ollama, and LM Studio deployments
• –Standard ARC JSON output makes integration with existing solvers straightforward
• –Solvability is not programmatically verified, and rule-description-based deduplication can miss differently worded duplicates