LLM Position Bias Benchmark: Swapped-Order Pairwise Judging
A new open benchmark from developer lechmazur on GitHub reveals that LLM judge models flip their preferred answer nearly half the time when the same two options are simply shown in reverse order. This is a serious problem because AI systems are now widely used to grade other AI s...
GitHub developer lechmazur published the LLM Position Bias Benchmark, a public repository measuring how reliably large language models act as judges when the same two candidate answers are presented in swapped order. The project, hosted at github.com/lechmazur/position_bias, sits alongside related academic work by Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno, and Yoshitaka Ushiku, who published "Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge" on arXiv on February 2, 2026. Together, these efforts expose a reliability crisis sitting at the center of modern AI evaluation infrastructure.
Why This Matters
The median model in this benchmark flips its underlying choice in 44.8% of decisive cases when answer order is reversed, which is closer to a coin flip than a reliable evaluation. Teams building automated model evaluation pipelines, preference-labeling systems, and AI graders are making product decisions on measurements that are, in many cases, more sensitive to prompt formatting than to actual answer quality. With 27 judge models tested across 193 verified story pairs, this is not a theoretical concern built on a handful of examples. If your AI evaluation stack has not been audited for position bias, there is a strong chance your leaderboard rankings are partly fiction.
Daily briefing from 50+ sources. Free, 5-minute read.
The Full Story
The premise of the benchmark is deceptively simple. Show a judge model two candidate answers, ask it to pick the better one, then show it the exact same two answers in reverse order and ask again. If the model is actually judging quality, the winner should stay the same. If the model is influenced by display position, the winner will change. The benchmark runs this test at scale across 27 models and 193 verified story pairs, generating 386 prompts per full model to build statistically meaningful results.
The numbers that emerge are alarming. Across all models tested, the average first-shown pick rate is 63.3%. That means that on average, judge models prefer whichever answer appears first in the prompt roughly 63 times out of 100, regardless of content. The median model flips its underlying choice in 44.8% of decisive swapped-order case pairs. The top performer on the leaderboard is Xiaomi MiMo V2 Pro, ranked first by its order flip rate, meaning it preserves the same winner across order swaps more consistently than any other model tested.
The benchmark's methodology also accounts for an important confound. A model could appear to have a low flip rate simply by calling everything a tie. The "Decisive Pair Coverage" column in the leaderboard tracks how often a model actually picks a winner rather than hedging, so that a low flip rate backed by many decisive choices is weighted as stronger evidence than a low flip rate produced by frequent abstentions.
The academic research from Xu and colleagues adds a specific mechanism to explain why this happens. In rubric-based evaluation, a judge model selects from a list of predefined scoring categories. That selection process, the researchers argue, behaves like a multiple-choice question, and language models carry well-documented position biases into multiple-choice settings. Models do not treat all list positions equally. They favor certain positions, and when the rubric is presented in a fixed order, that preference gets baked silently into every evaluation score the system produces.
The proposed fix from the arXiv paper is a balanced permutation strategy. Instead of running one evaluation with a fixed rubric order, you run multiple evaluations where the score options are systematically rotated through different positions, then aggregate the results. The researchers found this approach improves correlation between LLM judge scores and human expert scores, which is the ultimate validation that the bias correction is doing real work and not just averaging out noise.
Key Details
- The benchmark covers 193 verified story pairs tested across 27 distinct judge models.
- Each full model evaluation runs 386 prompts to produce its statistics.
- The model-average first-shown pick rate across all tested models is 63.3%.
- The median model flips its underlying choice in 44.8% of decisive swapped-order case pairs.
- Xiaomi MiMo V2 Pro ranks first on the leaderboard by order flip rate.
- The related arXiv paper, designated arXiv:2602.02219v1, was published on February 2, 2026, by Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno, and Yoshitaka Ushiku.
- The GitHub repository had 7 stars at the time of publication and is publicly available for community contributions.
What's Next
The benchmark's public snapshot will likely expand as the community contributes additional model evaluations, and the 27-model leaderboard is already large enough to put pressure on AI labs to address position bias in their judge model releases. Practitioners building evaluation pipelines should treat the balanced permutation strategy from the February 2026 arXiv paper as a near-term implementation priority, since the method requires no fundamental infrastructure changes and demonstrably improves agreement with human judgments. Expect major evaluation frameworks to ship permutation-based calibration options within the next few product cycles as awareness of this benchmark spreads.
How This Compares
The LLM-as-a-Judge paradigm has been gaining adoption across the industry for the past two years, with frameworks like DeepEval from Confident AI providing production-ready tooling for teams that want to automate evaluation at scale. Those frameworks have made it easy to plug a judge model into a pipeline, but they have largely treated the judge as a black box whose outputs are trustworthy. This benchmark and the accompanying arXiv paper directly challenge that assumption. For practical coverage of AI tools like these evaluation frameworks, the gap between ease of use and evaluation validity is now impossible to ignore.
The position bias problem is also distinct from the more commonly discussed verbosity bias, where judge models favor longer answers regardless of quality. Verbosity bias has been widely reported and many teams have added length-normalization steps to their pipelines. Position bias has received less attention, which makes the 63.3% first-shown pick rate finding particularly striking. Teams that audited for verbosity bias and called it done may be sitting on a second, equally serious distortion in their evaluation results. You can find deeper analysis of related AI news on how bias affects model rankings across the industry.
For developers who want to understand how to build bias-resistant evaluation systems, the balanced permutation approach described in the arXiv paper is worth studying carefully. It is a practical technique that any team can implement without waiting for model providers to release updated judge models. Check out our guides section for walkthroughs on building robust LLM evaluation pipelines that account for these failure modes.
FAQ
Q: What is position bias in LLM judging? A: Position bias means a judge model tends to prefer whichever answer appears first in the prompt, rather than making its decision purely on quality. In the lechmazur benchmark, the average judge model picks the first-shown answer 63.3% of the time across 193 tested story pairs, which reveals a consistent formatting influence on supposedly objective evaluations.
Q: How can you fix position bias in AI evaluation? A: The fix proposed by Xu and colleagues in their February 2026 arXiv paper is a balanced permutation strategy. You run the same evaluation multiple times with the answer order or rubric option order rotated systematically, then aggregate the results. This neutralizes the positional preference and improves alignment with human expert scores.
Q: Which AI model has the least position bias right now? A: Based on the current public snapshot of the lechmazur benchmark, Xiaomi MiMo V2 Pro ranks first on the leaderboard by order flip rate, meaning it most consistently picks the same winner regardless of which answer appears first. The benchmark covers 27 judge models, so rankings will shift as more models are added.
The lechmazur position bias benchmark and the February 2026 arXiv paper together make a compelling case that automated AI evaluation is less reliable than the industry has been assuming, and that fixing it is tractable with the right methodology. Practitioners who act on these findings now will have more trustworthy evaluation infrastructure than competitors who wait for the problem to become common knowledge. Subscribe to the AI Agents Daily weekly newsletter for daily updates on AI agents, tools, and automation.
Get stories like this daily
Free briefing. Curated from 50+ sources. 5-minute read every morning.