Code Generation QA
CodeElo: How Qwen Rates LLM Coding With CodeForces Elo
CodeElo, the Qwen benchmark, submits LLM code to live CodeForces for human-comparable Elo. See how it grades 387 problems and what o1-mini scored.
Code Generation QA
Testing, evaluating, and validating AI-generated code and code-generation models.
Code Generation QA
GitHub’s RCT on Copilot Code Quality: What the Data Shows
GitHub ran a randomized controlled trial of Copilot: 202 developers, 1,293 blind reviews. Copilot users were 53.2% more likely to pass all 10 unit tests.
Code Generation QA
Qwen2.5-Coder Benchmarks: How Alibaba Tests AI Code
A practitioner walkthrough of how Alibaba's Qwen2.5-Coder is evaluated: EvalPlus, Aider repair, McEval across 40+ languages, Code Arena, and fill-in-the-middle.
Code Generation QA
How Anthropic Evaluates Claude on SWE-bench Verified
How Anthropic measured Claude 3.5 Sonnet at 49% on SWE-bench Verified using a minimal Bash and Edit agent, graded by real repo unit tests.
Code Generation QA
MLE-bench: How OpenAI Grades AI Agents on Real Kaggle ML
OpenAI's MLE-bench scores AI agents on 75 real Kaggle competitions against human medal thresholds. How it works and what it means for code-gen QA.
Code Generation QA
Qwen2.5-Coder: How Alibaba Benchmarks AI Code
How Alibaba's Qwen2.5-Coder report evaluates AI-generated code across HumanEval, EvalPlus, LiveCodeBench and more, and what it means for code QA.
Code Generation QA
SWE-bench Verified: OpenAI’s Cleaned-Up Coding Benchmark
How OpenAI used 93 developers to build SWE-bench Verified, a 500-task coding benchmark that fixes the broken tests inflating AI agent scores.
Code Generation QA
How Meta’s CyberSecEval 3 Tests LLM Security Code
How Meta's CyberSecEval 3 benchmark evaluates LLM security: 31% of code completions flagged vulnerable by CodeShield, plus offensive uplift testing on Llama 3.
Code Generation QA
BigCodeBench: Testing How LLMs Handle Real Library Calls
How BigCodeBench from Hugging Face / BigCode tests LLMs on 1,140 tasks across 139 libraries, and why the 60% vs 97% gap matters for code QA.
Code Generation QA
BigCodeBench Explained: A Realistic Code-Gen Benchmark
How Hugging Face's BigCodeBench tests AI on 1,140 real Python tasks across 139 libraries, why GPT-4o hit 61.1%, and what it means for code QA.