- CodeXGLUE bundles 14 datasets across 10 tasks spanning four input/output shapes: code-code, text-code, code-text, and text-text.
- Microsoft Research Asia shipped three baseline families with it: CodeBERT (BERT-style, for understanding), CodeGPT (GPT-style, for completion and generation), and a generic encoder-decoder for sequence-to-sequence tasks.
- Text-to-code generation is scored with CodeBLEU, a metric that mixes n-gram overlap with syntax-tree and data-flow matching rather than treating code as plain text.
Overview
CodeXGLUE is a benchmark suite for code intelligence, released by Microsoft Research Asia in February 2021. The name echoes GLUE, the natural-language benchmark that became a standard scoreboard for language models. The goal here is the same idea applied to source code: give researchers one place to train, evaluate, and compare models on a fixed set of tasks, with public baselines and a leaderboard so results stay honest.
For anyone building or testing AI that writes code, it is a useful reference point. It defines what the field decided was worth measuring in 2021, and it exposes the metrics those measurements actually used.
How the benchmark is organized
The 14 datasets map onto 10 tasks, grouped by what goes in and what comes out. That framing matters because a model good at reading code is not automatically good at writing it.
The four task categories
- Code-code — clone detection, defect detection, cloze test, code completion, code refinement, and code-to-code translation. Both ends are source code.
- Text-code — natural-language code search and text-to-code generation. A human prompt goes in, code or a code ranking comes out.
- Code-text — code summarization, where the model writes a natural-language description of a function.
- Text-text — documentation translation across human languages.
The generation-heavy tasks are the ones most relevant to today’s coding assistants: text-to-code generation, code completion, code translation, and code refinement (taking buggy code and producing a fixed version).
The baseline models
Microsoft did not just ship data. They shipped reference models so a new method has something to beat. CodeBERT is the encoder, pretrained on paired code and documentation, and it handles classification-style tasks like defect and clone detection. CodeGPT is the decoder, built for autoregressive generation, and it drives completion and text-to-code. For tasks that need both sides, such as translation and summarization, the suite uses a standard encoder-decoder. This three-way split mirrors how the broader model world divides up the work.
How it evaluates generated code
The evaluation choices are where a QA reader should pay attention, because each task uses a metric tuned to its shape.
- Classification tasks (defect detection, clone detection) use accuracy and F1. Clean to compute, hard to game.
- Completion uses exact-match accuracy and edit similarity at the token and line level.
- Code-to-text and text-to-text use BLEU, the n-gram overlap metric borrowed from machine translation.
- Text-to-code generation uses CodeBLEU.
CodeBLEU is the interesting one. Plain BLEU rewards a model for matching word sequences, which is a weak signal for code: two programs can share almost no tokens and still be identical in behavior, or share most tokens and still be broken. CodeBLEU adds two code-aware terms. It compares the abstract syntax trees of the candidate and reference, and it compares their data-flow graphs. A prediction that gets the structure and the variable dependencies right scores higher even when the surface text drifts. It is a closer proxy for correctness than BLEU, though still a proxy.
Why it matters for code-generation QA
CodeXGLUE set the template most later code benchmarks reacted to, and understanding its limits is half the value of reading it. The headline caveat: none of its generation metrics run the code. CodeBLEU and BLEU measure resemblance to a reference answer, not whether the output compiles, passes tests, or does the right thing. Two years after this paper, the field moved hard toward execution-based evaluation precisely because a high CodeBLEU score does not guarantee working software.
If you are evaluating an AI coding tool today, treat similarity metrics as a coarse first filter, not a verdict. A model can produce plausible-looking, structurally-correct code that fails on the first edge case. Reference-based scoring also penalizes valid solutions that differ from the single reference answer, so it under-credits genuinely good but unconventional output.
The benchmark is still worth knowing for three reasons. The task taxonomy is a clean checklist for thinking about what your own product needs to be tested on, generation is only one slice. The dataset splits remain a reasonable starting corpus for fine-tuning and probing. And the leaderboard makes a point that holds up: published baselines and a fixed test set are what keep eval claims comparable across teams. The right move now is to pair a CodeXGLUE-style task breakdown with execution and unit-test checks, so you measure both whether the code looks right and whether it actually runs.
Read the original: CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation — Microsoft Research, 2021-02-09.