The Effect of Data Quality in Machine Learning

Through Chess Engine Performance

Austin Lehman & Philip Zeng

Grapevine High School

This project aimed to create a program that will "learn" to play chess from a set of

data gathered from skilled players. Machine learning makes it so that a program will be

able to find the most optimal way to complete the task it is assigned to do. We

hypothesized that it is possible to train a high level chess engine using limited data from

chess matches played by highly rated players. This is important because applying

machine learning to chess can give us insight into a world of possibilities in other areas.

This project required only a computer and a chessboard, no other materials. There was

also no safety risks to any of the participants. In order to start this project we had to have

permission from Erik Bernhardsson to use a program titled “deep-pink”. Once we were

given permission, our project could begin.

Chess is a game with finite legal positions, meaning that a theoretical “solution”

to chess could potentially be derived. However, since there are 10^43 possible positions

in chess, it would take nearly infinite computing power to calculate this solution. We

applied machine learning in order to approximate an evaluation function (a function to

determine the best move for a given position on the board) based on the game data of

highly rated chess players. To do this, we modified the “deep-pink” chess engine (written

by Erik Bernhardsson) which uses a 3 layer convolutional neural network (CNN) trained
on over 100 million chess games to approximate an evaluation function. We trained our

model using game data from the FICS Games Database of chess games played at an

average of 2000 elo. To test the performance of our “deep red” model in comparison to

the older “deep-pink” model, we had both models play against human players and the

Sunfish chess engine.

 Deep-red seems to be worse than Deep-pink when playing the average player, so

it can be concluded that even though it was given higher quality data, the program is

unable to beat people that make less than optimal moves. Deep-pink was able to

capitalize on more mistakes, while Deep-red was able to put players in more difficult

positions (however, it tended to leave pieces open). Machine learning shows promising

potential, but can still be outclassed by people if it is not given sufficient time. Deep-red

made poor moves when its opponent made a move that was not optimal. This makes

sense since it was trained using high quality data, and not data from players who made

mistakes. Given some more time, and a wider spectrum of data, the program would likely

be better against average players.

References:
A Tour of Machine Learning Algorithms. (2017, February 21). Retrieved from

https://machinelearningmastery.com/a-tour-of-machine-learning-algorithms/

David, O. E., Herik, H. J., Koppel, M., & Netanyahu, N. S. (2014). Genetic Algorithms for

Evolving Computer Chess Programs. IEEE Transactions on Evolutionary Computation,

18(5), 779-789. doi:10.1109/tevc.2013.2285111

Erik Bernhardsson. (2014, November 29). Deep learning for... chess. Retrieved from

https://erikbern.com/2014/11/29/deep-learning-for-chess.html

Faggella, D. (n.d.). What is Machine Learning? | Emerj - Artificial Intelligence Research and

Insight. Retrieved from http://www.techemergence.com/what-is-machine-learning/.

Machine Learning: What it is and why it matters. (n.d.). Retrieved from

http://www.sas.com/en_us/insights/analytics/machine-learning.html