Acta Informatica Pragensia 2018, 7(1), 74-103 | DOI: 10.18267/j.aip.1154611

Přehled přístupů k vyhodnocování inteligence umělých systémů

Ondřej Vadinský
Department of Information and Knowledge Engineering, Faculty of Informatics and Statistics, University of Economics, Prague, W. Churchill Sq. 4, 130 67 Prague 3, Czech Republic

Obecná umělá inteligence usiluje o vytvoření umělých systémů schopných řešit mnoho různých, a to i během vývoje nepředvídaných, úloh, což takové systémy činí svou inteligencí srovnatelné s lidmi. To však vyžaduje existenci vhodných metod vyhodnocujících, zda a nakolik jsou umělé systémy inteligentní. Tento přehledový článek hledá právě takové evaluační metody. Provádí proto rozsáhlou rešerši literatury pokrývající jak filosofické a kognitivní předpoklady inteligence, tak i formální definice a praktické testy vycházející z algoritmické teorie informace. Na základě porovnání představených metod článek odhaluje dvě rozdílné skupiny přístupů založené na principiálně odlišných předpokladech. Zatímco starší přístupy, jako např. Turingův test, jsou založeny na předpokladu, že úspěch v komplexní činnosti je postačující pro přiznání inteligence, nové přístupy, jako např. test algoritmického IQ, kromě toho vyžadují i důkladné ověření úspěšnosti v jednoduchých činnostech. V důsledku tohoto zjištění článek dochází k závěru, že test algoritmického IQ založený na definici univerzální inteligence je v současné době nejlepším kandidátem na vhodný prakticky proveditelný test obecné inteligence umělých systémů. Ačkoliv i tento test má několik známých limitů.

Keywords: Obecná umělá inteligence, definice univerzální inteligence, kdykoliv přerušitelný test inteligence, test algoritmického IQ, vyhodnocování inteligence umělých systémů

An Overview of Approaches Evaluating Intelligence of Artificial Systems

Artificial General Intelligence seeks to create an artificial system capable of solving many different and possibly unforeseen tasks thus being comparable in its intelligence to that of a human. Such an endeavour, however, requires suitable methods that can evaluate whether an artificial system is intelligent, and to what extent. This review paper searches for such evaluation methods. Therefore, an extensive literature overview is conducted that covers both philosophical and cognitive presumptions of intelligence as well as formal definitions and practical tests of intelligence grounded in Algorithmic Information Theory. Based on a comparison of the introduced approaches, the paper identifies two distinct groups based on fundamentally different presumptions. The one group of approaches, such as Turing test, is based on the presumption that success in a complex task is a sufficient condition for intelligence evaluation, while the other group of approaches, such as Algorithmic Intelligence Quotient test, also require explicit verification of success in simple tasks. This paper, therefore, concludes that the Algorithmic Intelligence Quotient test, derived from Universal Intelligence definition, is currently the most suitable candidate for a practical intelligence evaluation method of artificial systems. Although the test has several known limitations.

Keywords: Artificial General Intelligence, Universal Intelligence Definition, Anytime Intelligence Test, Algorithmic Intelligence Quotient Test, Evaluating Intelligence of Artificial Systems

Received: January 29, 2018; Accepted: May 29, 2018; Prepublished online: June 10, 2018; Published: June 30, 2018  Show citation

ACS AIP APA ASA Harvard Chicago Chicago Notes IEEE ISO690 MLA NLM Turabian Vancouver
Vadinský, O. (2018). An Overview of Approaches Evaluating Intelligence of Artificial Systems. Acta Informatica Pragensia, 7(1), 74-103. doi: 10.18267/j.aip.115
Download citation
  • BibTeX (.bib)
  • Bookends (.ris)
  • EasyBib (.ris)
  • EndNote (.enw)
  • EndNote 8 (.xml)
  • ISI WoS (.isi)
  • Medlars (.medlars)
  • Mendeley (.ris)
  • MODS (.xml)
  • Papers (.ris)
  • RefWorks (.txt)
  • RefManager (.ris)
  • RIS (.ris)
  • MS Word (.xml)
  • Zotero (.ris)
    • Open full article

    References

    1. Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., & Qin, Y. (2004). An integrated theory of the mind. Psychological Review, 111(4), 1036-1060. doi: 10.1037/0033-295x.111.4.1036. Go to original source...
    2. Besold, T., Hernández-Orallo, J., & Schmid, U. (2015). Can machine intelligence be measured in the same way as human intelligence? KI - Künstliche Intelligenz, 29(3), 291-297. doi: 10.1007/s13218-015-0361-4. Go to original source...
    3. Bickle, J. (2016). Multiple realizability. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy. Stanford: Metaphysics Research Lab, Stanford University. Retrieved November, 20, 2017, from https://plato.stanford.edu/archives/spr2016/entries/multiple-realizability/.
    4. Bringsjord, S. & Schimanski, B. (2003). What is artificial intelligence? psychometric AI as an answer. In Gottlob, G. (Ed.), Proceedings of the 18th International Joint Conference on Artificial Intelligence (IJCAI'03), (pp. 887-893). Acapulco: IJCAI.
    5. Burge, T. (1979). Individualism and the mental. Midwest Studies in Philosophy, 4(1), 73-121. doi: 10.1111/j.1475-4975.1979.tb00374.x. Go to original source...
    6. Cattell, R. B. (1987). Intelligence: Its structure, growth, and action. New York: Elsevier.
    7. de Mey, M. (1992). The cognitive paradigm. Chicago and London: University of Chicago Press. doi: 10.1007/978-94-009-7956-7. Go to original source...
    8. Dennett, D. C. (1980). The milk of human intentionality. Behavioral and Brain Sciences, (3), 428-430. doi: 10.1017/S0140525X0000580X. Go to original source...
    9. Descartes, R. (1637), [1992]. Rozprava o metodě. Praha: Svoboda.
    10. Dowe, D. L. & Hájek, A. R. (1998). A non-behavioural, computational extension to the Turing test. In Selvaraj, H. & Verma, B. (Eds.), Proceedings of International Conference on Computational Intelligence & Multimedia Applications (ICCIMA'98), Gippsland, Australia, (pp. 101-106). Singapore: World Scientific.
    11. Gardner, H. (1983). Frames of mind: Theory of multiple intelligences. New York: Basic Books.
    12. Goertzel, B. (2010). Toward a formal characterization of real-world general intelligence. In Baum, E., Hutter, M., & Kitzelmann, E. (Eds.), Proceedings of the 3rd International Conference on Go to original source...
    13. Artificial General Intelligence (AGI 2010), Lugano, Switzerland, (pp. 19-24). Amsterdam-Beijing-Paris: Atlantis Press. doi: 10.2991/agi.2010.17. Go to original source...
    14. Goertzel, B. (2014). Artificial general intelligence: Concept, state of the art, and future prospects. Journal of Artificial General Intelligence, 5(1), 1-48. doi: 10.2478/jagi-2014-0001. Go to original source...
    15. Harnad, S. (1991). Other bodies, other minds: A machine incarnation of an old philosophical problem. Minds and Machines, 1(1), 43-54. doi: 10.1007/BF00360578. Go to original source...
    16. Havel, I. M. (2001). Přirozené a umělé myšlení jako filozofický problém. In V. Mařík, O. Štěpánková, & J. Lažanský (Eds.), Umělá inteligence 3 (pp. 17-75). Praha: Academia.
    17. Hernández-Orallo, J. (2000). Beyond the Turing test. Journal of Logic, Language and Information, 9(4), 447-466. doi: 10.1023/A:1008367325700. Go to original source...
    18. Hernández-Orallo, J. (2010). A (hopefully) unbiased universal environment class for measuring intelligence of biological and artificial systems. In Baum, E., Hutter, M., & Kitzelmann, E. (Eds.), Proceedings of the 3rd International Conference on Artificial General Intelligence (AGI 2010), Lugano, Switzerland, (pp. 182-183). Amsterdam-Beijing-Paris: Atlantis Press. doi: 10.2991/agi.2010.18. Go to original source...
    19. Hernández-Orallo, J. (2015). C-tests revisited: Back and forth with complexity. In Bieger, J., Goertzel, B., & Potapov, A. (Eds.), Proceedings of the 8th International Conference on Artificial General Intelligence (AGI 2015), Berlin, Germany, (pp. 272-282). Berlin: Springer. doi: 10.1007/978-3-319-21365-1_28. Go to original source...
    20. Hernández-Orallo, J. (2017). The measure of all minds. Cambridge: Cambridge University Press. doi: 10.1017/9781316594179. Go to original source...
    21. Hernández-Orallo, J. & Dowe, D. L. (2010). Measuring universal intelligence: Towards an anytime intelligence test. Artificial Intelligence, 174(18), 1508-1539. doi: 10.1016/j.artint.2010.09.006. Go to original source...
    22. Hibbard, B. (2009). Bias and no free lunch in formal measures of intelligence. Journal of Artificial General Intelligence, 1(1), 54-61. doi: 10.2478/v10229-011-0004-6. Go to original source...
    23. Hutter, M. (2007). Universal algorithmic intelligence: A mathematical top→down approach. In B. Goertzel & C. Pennachin (Eds.), Artificial General Intelligence (pp. 227-290). Berlin: Springer. doi: 10.1007/978-3-540-68677-4_8. Go to original source...
    24. Hutter, M. (2012). One decade of universal artificial intelligence. In P. Wang & B. Goertzel (Eds.), Theoretical Foundations of Artificial General Intelligence (pp. 67-88). Paris: Atlantis Press. doi: 10.2991/978-94-91216-62-6_5. Go to original source...
    25. Hutter, M. & Legg, S. (2007). Temporal difference updating without a learning rate. In Platt, J. C.,
    26. Koller, D., Singer, Y., & Roweis, S. T. (Eds.), Proceedings of the 21st Annual Conference on Advances in Neural Information Processing Systems (NIPS 2007), Vancouver, Canada, (pp. 705-712). New York: Curran Associates, Inc.
    27. Hyslop, A. (2014). Other minds. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy. Stanford: Metaphysics Research Lab, Stanford University. Retrieved November, 20, 2017, from http://plato.stanford.edu/archives/spr2014/entries/other-minds/.
    28. Insa-Cabrera, J., Dowe, D. L., España-Cubillo, S., Hernández-Lloreda, M. V., & Hernández-Orallo, J. (2011). Comparing humans and AI agents. In Schmidhuber, J., Go to original source...
    29. Thórisson, K. R., & Looks, M. (Eds.), Proceedings of the 4th International Conference on Artificial General Intelligence (AGI 2011), Mountain View, USA, (pp. 122-132). Berlin: Springer. doi: 10.1007/978-3-642-22887-2_13. Go to original source...
    30. Kolmogorov, A. N. (1963). On tables of random numbers. Sankhyā: The Indian Journal of Statistics, Series A, 4(25), 369-376. doi: 10.1016/S0304-3975(98)00075-9. Go to original source...
    31. Kripke, S. A. (1972). Naming and necessity. Cambridge: Harvard University Press. Go to original source...
    32. Legg, S. & Hutter, M. (2007a). A collection of definitions of intelligence. In B. Goertzel & P. Wang (Eds.), Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms (pp. 17-24). Amsterdam: IOS Press.
    33. Legg, S. & Hutter, M. (2007b). Universal intelligence: A definition of machine intelligence. Minds and Machines, 17(4), 391-444. doi: 10.1007/s11023-007-9079-x. Go to original source...
    34. Legg, S. & Veness, J. (2011). AIQ: Algorithmic intelligence quotient [source codes]. Retrieved June, 26, 2017, from https://github.com/mathemajician/AIQ.
    35. Legg, S. & Veness, J. (2013). An approximation of the universal intelligence measure. In D. L. Dowe (Ed.), Algorithmic Probability and Friends. Bayesian Prediction and Artificial Intelligence (pp. 236-249). Berlin, Heidelberg: Springer. doi: 10.1007/978-3-642-44958-1_18. Go to original source...
    36. Levin, J. (2017). Functionalism. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy.
    37. Stanford: Metaphysics Research Lab, Stanford University. Retrieved November, 20, 2017, from https://plato.stanford.edu/archives/win2017/entries/functionalism/.
    38. Levy, D. & Newborn, M. (1991). How computers play chess. New York: Computer Science Press. doi: 10.1007/978-3-642-85538-2_2. Go to original source...
    39. Minsky, M. (1974). A framework for representing knowledge. Technical report. Retrieved November, 20, 2017, from http://web.media.mit.edu/~minsky/papers/Frames/frames.html.
    40. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529-533. doi: 10.1038/nature14236. Go to original source...
    41. Müller, U. (1993). dev/lang/brainfuck-2.lha in Aminet. Retrieved June, 26, 2017, from http://aminet.net/package.php?package=dev/lang/brainfuck-2.lha.
    42. Piaget, J. (1936). Origins of intelligence in the child. London: Routledge & Kegan Paul.
    43. Putnam, H. (1975). Mind, language and reality, chapter The Meaning of 'Meaning', (pp. 215-271). Cambridge: Cambridge University Press. Go to original source...
    44. Rescorla, M. (2017). The computational theory of mind. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy. Stanford: Metaphysics Research Lab, Stanford University. Retrieved November, 20, 2017, from https://plato.stanford.edu/archives/spr2017/entries/computational-mind/.
    45. Schweizer, P. (2012). The externalist foundations of a truly total Turing test. Minds and Machines, 22(3), 191-212. doi: 10.1007/s11023-012-9272-4. Go to original source...
    46. Searle, J. R. (1980). Minds, brains, and programs. Behavioral and Brain Sciences, (3), 417-457. doi: 10.1017/S0140525X00005756. Go to original source...
    47. Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K., & Hassabis, D. (2017). Mastering chess and shogi by self-play with a general reinforcement learning algorithm.
    48. Solomonoff, R. J. (1964a). A formal theory of inductive inference, part 1. Information and Control, 7(1), 1-22. doi: 10.1016/S0019-9958(64)90131-7. Go to original source...
    49. Solomonoff, R. J. (1964b). A formal theory of inductive inference, part 2. Information and Control, 7(2), 224-254. doi: 10.1016/S0019-9958(64)90131-7. Go to original source...
    50. Spearman, C. E. (1927). The abilities of man, their nature and measurement. New York: Macmillan.
    51. Sternberg, R. J. (1984). Beyond IQ: A triarchic theory of human intelligence. Cambridge: Cambridge University Press.
    52. Sun, R. (2007). The importance of cognitive architectures: An analysis based on CLARION. Journal of Experimental & Theoretical Artificial Intelligence, 19(2), 159-193. doi: 10.1080/09528130701191560. Go to original source...
    53. Sutton, R. S. & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge: MIT Press. doi: 10.1016/S0925-2312(00)00324-6. Go to original source...
    54. Thomsen, K. (2013). The cerebellum in the Ouroboros model, the "interpolator hypothesis". In Shimizu, S. & Bossomaier, T. (Eds.), Proceedings of the 5th International Conference on Advanced Cognitive Technologies and Applications (COGNITIVE 2013), Valencia, Spain, (pp. 37-41). Wilmington: IARIA.
    55. Turing, A. M. (1936). On computable numbers, with an application to the Entscheidungsproblem. Proceedings of the London Mathematical Society, 2(42), 230-265. Go to original source...
    56. Turing, A. M. (1950). Computing machinery and intelligence. Mind, 59(236), 433-460. Go to original source...
    57. Tvrdý, F. (2014). Turingův test: Filozofické aspekty umělé inteligence. Praha: Togga.
    58. Veness, J., Ng, K. S., Hutter, M., Uther, W., & Silver, D. (2011). A Monte Carlo AIXI approximation. Journal of Artificial Intelligence Research, 40(1), 95-142. doi: 10.1613/jair.3125. Go to original source...
    59. Watkins, C. (1989). Learning from delayed rewards. PhD thesis, University of Cambridge, Kings College, Cambridge.

    This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits use, distribution, and reproduction in any medium, provided the original publication is properly cited. No use, distribution or reproduction is permitted which does not comply with these terms.


    Return to the content