Acta Informatica Pragensia X:X | DOI: 10.18267/j.aip.33664
Self-Preference or Position Bias? A Controlled Multilingual Study of Evaluation Bias in LLM-as-Judge Pairwise Comparison
- Department of Information Technology, Faculty of Informatics and Statistics, Prague University of Economics and Business, Prague, Czech Republic
Background: Large language models (LLMs) are increasingly used as automated evaluators, but concerns remain about systematic biases such as self-preference. Prior findings may be confounded by stylistic similarity, shared quality preferences, or presentation order rather than genuine authorship-based bias.
Objective: We test whether LLM judges favor their own generated texts in blind pairwise comparison beyond what shared quality preferences would predict, and we quantify position bias as a competing explanation.
Methods: We conduct a controlled, blind, multilingual experiment. Five near-frontier LLMs from five developers generate short texts in five European languages across three text types, with five generations per cell (375 texts). Each model then blindly compares its own text against each other model’s text, in both presentation orders, yielding 3,000 evaluations. Self-selection is benchmarked against cross-model selection of the same text as a quality-controlled baseline, and we apply equivalence testing to bound the size of small effects.
Results: Across 3,000 evaluations, a correctly specified conditional-logit model (Bradley–Terry candidate strengths plus a self-bias term, controlling for both candidate quality and presentation order) revealed a modest but robust overall own-text preference (odds ratio or OR=1.26; p<.001). This was concentrated in two of the five judges (Gemini OR=2.09, Mistral OR=3.20; both p<.001); one judge showed the reverse (GPT OR=0.43; p<.001), and two showed no significant effect (Claude, Qwen). Model-specific position biases were strong (up to φ=0.62) and constitute the single largest distortion.
Conclusion: Evaluation reveals a genuine own-text preference beyond shared quality, alongside strong position bias; both must be actively mitigated in LLM-as-judge pipelines. We do not attribute the own-text preference specifically to authorship recognition rather than familiarity, which our design does not separate.
Keywords: Large language model; Self-preference bias; Position bias; Evaluation bias; Multilingual evaluation.
Received: June 4, 2026; Revised: October 22, 2026; Accepted: October 23, 2026; Prepublished online: November 14, 2026
References
- Agresti, A. (2002). Categorical data analysis (2nd ed.). Wiley.
Go to original source... - Ahuja, K., Diddee, H., Hada, R., Ochieng, M., Ramesh, K., Jain, P., Nambi, A., Ganu, T., Segal, S., Axmed, M., Bali, K., & Sitaram, S. (2023). MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, (pp. 4775-4812). ACL.
Go to original source... - Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., & Baraniuk, R. G. (2024). Self-consuming generative models go MAD. In Proceedings of the Twelfth International Conference on Learning Representations (pp. 1-28). ICLR.
Go to original source... - Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48. https://doi.org/10.18637/jss.v067.i01
Go to original source... - Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324-345.
Go to original source... - Bruckner, T., Vencovský, F., & Svatoš, O. (2026a). Self-preference or position bias? A controlled multilingual study of evaluation bias in LLM-as-judge pairwise comparison [Dataset]. Zenodo. https://doi.org/10.5281/zenodo.22395365
Go to original source... - Bruckner, T., Vencovský, F., & Svatoš, O. (2026b). Self-preference or position bias? A controlled multilingual study of evaluation bias in LLM-as-judge pairwise comparison - software [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.22397578
Go to original source... - Chen, Y.-S., Jin, J., Kuo, P.-T., Huang, C.-W., & Chen, Y.-N. (2025). LLMs are biased evaluators but not biased for fact-centric retrieval augmented generation. In Findings of the Association for Computational Linguistics: ACL 2025, (pp. 26669-26684). ACL. https://doi.org/10.18653/v1/2025.findings-acl.1369
Go to original source... - Chen, Z.-Y., Wang, H., Zhang, X., Hu, E., & Lin, Y. (2025). Beyond the surface: Measuring self-preference in LLM judgments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, (pp. 1653-1672). ACL. https://doi.org/10.18653/v1/2025.emnlp-main.86
Go to original source... - Chhun, C., Suchanek, F. M., & Clavel, C. (2024). Do language models enjoy their own stories? Prompting large language models for automatic story evaluation. Transactions of the Association for Computational Linguistics, 12, 1122-1142. https://doi.org/10.1162/tacl_a_00689
Go to original source... - Chiang, C.-H., & Lee, H. (2023). Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), (pp. 15607-15631). ACL.
Go to original source... - Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
- Dubois, Y., Galambosi, B., Liang, P., & Hashimoto, T. B. (2024). Length-controlled AlpacaEval: A simple way to debias automatic evaluators. In Proceedings of the Conference on Language Modeling 2024. COLM.
- Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., & Seo, M. (2024). Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, (pp. 4334-4353). ACL. https://doi.org/10.18653/v1/2024.emnlp-main.248
Go to original source... - Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., & Kang, D. (2024). Benchmarking cognitive biases in large language models as evaluators. In Findings of the Association for Computational Linguistics: ACL 2024, (pp. 517-545). ACL. https://doi.org/10.18653/v1/2024.findings-acl.29
Go to original source... - Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355-362. https://doi.org/10.1177/1948550617697177
Go to original source... - Li, Z., Wang, C., Ma, P., Wu, D., Wang, S., Gao, C., & Liu, Y. (2024). Split and merge: Aligning position biases in large language model based evaluators. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, (pp. 11084-11108). ACL. https://doi.org/10.18653/v1/2024.emnlp-main.621
Go to original source... - Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, (pp. 2511-2522). ACL. https://doi.org/10.18653/v1/2023.emnlp-main.153
Go to original source... - Needham, J., Edkins, G., Pimpale, G., Bartsch, H., & Hobbhahn, M. (2025). Large language models often know when they are being evaluated. arXiv:2505.23836. https://doi.org/10.48550/arXiv.2505.23836
Go to original source... - Panickssery, A., Bowman, S. R., & Feng, S. (2024). LLM evaluators recognize and favor their own generations. In Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024). NIPS.
Go to original source... - Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023). NIPS.
Go to original source... - Shi, L., Ma, W., Wu, C., Wang, Y., Wang, Y., Xu, J., Cheng, R., Wu, Y., Wu, Y., Xu, H., Wu, Z., & Vosoughi, S. (2025). Judging the judges: A systematic study of position bias in LLM-as-a-judge. Proceedings of the 2025 Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL), (pp. 292-314). ACL. https://doi.org/10.18653/v1/2025.ijcnlp-long.18
Go to original source... - Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2024). AI models collapse when trained on recursively generated data. Nature, 631, 755-759. https://doi.org/10.1038/s41586-024-07566-y
Go to original source... - Stureborg, R., Alikaniotis, D., & Suhara, Y. (2024). Large language models are inconsistent and biased evaluators. arXiv:2405.01724. https://doi.org/10.48550/arXiv.2405.01724
Go to original source... - Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., & Sui, Z. (2024). Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), (pp. 9440-9450). ACL. https://doi.org/10.18653/v1/2024.acl-long.511
Go to original source... - Wataoka, K., Takahashi, T., & Ri, R. (2024). Self-preference bias in LLM-as-a-judge. In NeurIPS 2024 Workshop on Safe Generative AI, (pp. 1-12). ICLR.
- Wu, M., & Aji, A. F. (2025). Style over substance: Evaluation biases for large language models. In Proceedings of the 31st International Conference on Computational Linguistics, (pp. 297-312). ACL.
- Xu, W., Zhu, G., Zhao, X., Pan, L., Li, L., & Wang, W. (2024). Pride and prejudice: LLM amplifies self-bias in self-refinement. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), (pp. 15474-15492). ACL. https://doi.org/10.18653/v1/2024.acl-long.826
Go to original source... - Ye, S., Kim, D., Kim, S., Hwang, H., Kim, S., Jo, Y., Thorne, J., Kim, J., & Seo, M. (2024). FLASK: Fine-grained language model evaluation based on alignment skill sets. In Proceedings of the Twelfth International Conference on Learning Representations, (pp. 1-54). ICLR.
- Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In NIPS '23: Proceedings of the 37th International Conference on Neural Information Processing Systems, (pp. 46595-46623). NIPS.
Go to original source...
This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits use, distribution, and reproduction in any medium, provided the original publication is properly cited. No use, distribution or reproduction is permitted which does not comply with these terms.

ORCID...