Acta Informatica Pragensia X:X | DOI: 10.18267/j.aip.33664

Self-Preference or Position Bias? A Controlled Multilingual Study of Evaluation Bias in LLM-as-Judge Pairwise Comparison

Tomáš Bruckner ORCID..., Filip Vencovský ORCID..., Oleg Svatoš ORCID...
Department of Information Technology, Faculty of Informatics and Statistics, Prague University of Economics and Business, Prague, Czech Republic

Background: Large language models (LLMs) are increasingly used as automated evaluators, but concerns remain about systematic biases such as self-preference. Prior findings may be confounded by stylistic similarity, shared quality preferences, or presentation order rather than genuine authorship-based bias.

Objective: We test whether LLM judges favor their own generated texts in blind pairwise comparison beyond what shared quality preferences would predict, and we quantify position bias as a competing explanation.

Methods: We conduct a controlled, blind, multilingual experiment. Five near-frontier LLMs from five developers generate short texts in five European languages across three text types, with five generations per cell (375 texts). Each model then blindly compares its own text against each other model’s text, in both presentation orders, yielding 3,000 evaluations. Self-selection is benchmarked against cross-model selection of the same text as a quality-controlled baseline, and we apply equivalence testing to bound the size of small effects.

Results: Across 3,000 evaluations, a correctly specified conditional-logit model (Bradley–Terry candidate strengths plus a self-bias term, controlling for both candidate quality and presentation order) revealed a modest but robust overall own-text preference (odds ratio or OR=1.26; p<.001). This was concentrated in two of the five judges (Gemini OR=2.09, Mistral OR=3.20; both p<.001); one judge showed the reverse (GPT OR=0.43; p<.001), and two showed no significant effect (Claude, Qwen). Model-specific position biases were strong (up to φ=0.62) and constitute the single largest distortion.

Conclusion: Evaluation reveals a genuine own-text preference beyond shared quality, alongside strong position bias; both must be actively mitigated in LLM-as-judge pipelines. We do not attribute the own-text preference specifically to authorship recognition rather than familiarity, which our design does not separate.

Keywords: Large language model; Self-preference bias; Position bias; Evaluation bias; Multilingual evaluation.

Received: June 4, 2026; Revised: October 22, 2026; Accepted: October 23, 2026; Prepublished online: November 14, 2026 

Download citation

References

  1. Agresti, A. (2002). Categorical data analysis (2nd ed.). Wiley. Go to original source...
  2. Ahuja, K., Diddee, H., Hada, R., Ochieng, M., Ramesh, K., Jain, P., Nambi, A., Ganu, T., Segal, S., Axmed, M., Bali, K., & Sitaram, S. (2023). MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, (pp. 4775-4812). ACL. Go to original source...
  3. Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., & Baraniuk, R. G. (2024). Self-consuming generative models go MAD. In Proceedings of the Twelfth International Conference on Learning Representations (pp. 1-28). ICLR. Go to original source...
  4. Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48. https://doi.org/10.18637/jss.v067.i01 Go to original source...
  5. Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324-345. Go to original source...
  6. Bruckner, T., Vencovský, F., & Svatoš, O. (2026a). Self-preference or position bias? A controlled multilingual study of evaluation bias in LLM-as-judge pairwise comparison [Dataset]. Zenodo. https://doi.org/10.5281/zenodo.22395365 Go to original source...
  7. Bruckner, T., Vencovský, F., & Svatoš, O. (2026b). Self-preference or position bias? A controlled multilingual study of evaluation bias in LLM-as-judge pairwise comparison - software [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.22397578 Go to original source...
  8. Chen, Y.-S., Jin, J., Kuo, P.-T., Huang, C.-W., & Chen, Y.-N. (2025). LLMs are biased evaluators but not biased for fact-centric retrieval augmented generation. In Findings of the Association for Computational Linguistics: ACL 2025, (pp. 26669-26684). ACL. https://doi.org/10.18653/v1/2025.findings-acl.1369 Go to original source...
  9. Chen, Z.-Y., Wang, H., Zhang, X., Hu, E., & Lin, Y. (2025). Beyond the surface: Measuring self-preference in LLM judgments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, (pp. 1653-1672). ACL. https://doi.org/10.18653/v1/2025.emnlp-main.86 Go to original source...
  10. Chhun, C., Suchanek, F. M., & Clavel, C. (2024). Do language models enjoy their own stories? Prompting large language models for automatic story evaluation. Transactions of the Association for Computational Linguistics, 12, 1122-1142. https://doi.org/10.1162/tacl_a_00689 Go to original source...
  11. Chiang, C.-H., & Lee, H. (2023). Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), (pp. 15607-15631). ACL. Go to original source...
  12. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
  13. Dubois, Y., Galambosi, B., Liang, P., & Hashimoto, T. B. (2024). Length-controlled AlpacaEval: A simple way to debias automatic evaluators. In Proceedings of the Conference on Language Modeling 2024. COLM.
  14. Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., & Seo, M. (2024). Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, (pp. 4334-4353). ACL. https://doi.org/10.18653/v1/2024.emnlp-main.248 Go to original source...
  15. Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., & Kang, D. (2024). Benchmarking cognitive biases in large language models as evaluators. In Findings of the Association for Computational Linguistics: ACL 2024, (pp. 517-545). ACL. https://doi.org/10.18653/v1/2024.findings-acl.29 Go to original source...
  16. Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355-362. https://doi.org/10.1177/1948550617697177 Go to original source...
  17. Li, Z., Wang, C., Ma, P., Wu, D., Wang, S., Gao, C., & Liu, Y. (2024). Split and merge: Aligning position biases in large language model based evaluators. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, (pp. 11084-11108). ACL. https://doi.org/10.18653/v1/2024.emnlp-main.621 Go to original source...
  18. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, (pp. 2511-2522). ACL. https://doi.org/10.18653/v1/2023.emnlp-main.153 Go to original source...
  19. Needham, J., Edkins, G., Pimpale, G., Bartsch, H., & Hobbhahn, M. (2025). Large language models often know when they are being evaluated. arXiv:2505.23836. https://doi.org/10.48550/arXiv.2505.23836 Go to original source...
  20. Panickssery, A., Bowman, S. R., & Feng, S. (2024). LLM evaluators recognize and favor their own generations. In Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024). NIPS. Go to original source...
  21. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023). NIPS. Go to original source...
  22. Shi, L., Ma, W., Wu, C., Wang, Y., Wang, Y., Xu, J., Cheng, R., Wu, Y., Wu, Y., Xu, H., Wu, Z., & Vosoughi, S. (2025). Judging the judges: A systematic study of position bias in LLM-as-a-judge. Proceedings of the 2025 Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL), (pp. 292-314). ACL. https://doi.org/10.18653/v1/2025.ijcnlp-long.18 Go to original source...
  23. Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2024). AI models collapse when trained on recursively generated data. Nature, 631, 755-759. https://doi.org/10.1038/s41586-024-07566-y Go to original source...
  24. Stureborg, R., Alikaniotis, D., & Suhara, Y. (2024). Large language models are inconsistent and biased evaluators. arXiv:2405.01724. https://doi.org/10.48550/arXiv.2405.01724 Go to original source...
  25. Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., & Sui, Z. (2024). Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), (pp. 9440-9450). ACL. https://doi.org/10.18653/v1/2024.acl-long.511 Go to original source...
  26. Wataoka, K., Takahashi, T., & Ri, R. (2024). Self-preference bias in LLM-as-a-judge. In NeurIPS 2024 Workshop on Safe Generative AI, (pp. 1-12). ICLR.
  27. Wu, M., & Aji, A. F. (2025). Style over substance: Evaluation biases for large language models. In Proceedings of the 31st International Conference on Computational Linguistics, (pp. 297-312). ACL.
  28. Xu, W., Zhu, G., Zhao, X., Pan, L., Li, L., & Wang, W. (2024). Pride and prejudice: LLM amplifies self-bias in self-refinement. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), (pp. 15474-15492). ACL. https://doi.org/10.18653/v1/2024.acl-long.826 Go to original source...
  29. Ye, S., Kim, D., Kim, S., Hwang, H., Kim, S., Jo, Y., Thorne, J., Kim, J., & Seo, M. (2024). FLASK: Fine-grained language model evaluation based on alignment skill sets. In Proceedings of the Twelfth International Conference on Learning Representations, (pp. 1-54). ICLR.
  30. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In NIPS '23: Proceedings of the 37th International Conference on Neural Information Processing Systems, (pp. 46595-46623). NIPS. Go to original source...

This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits use, distribution, and reproduction in any medium, provided the original publication is properly cited. No use, distribution or reproduction is permitted which does not comply with these terms.