2018-12-08

TCEC Season 14. Leela promoted from 3rd division to div2....



Leela's big journey to try to go to premier division of TCEC, has started!
TCEC season 14 is running for the last couple of weeks and Leela has participated in 3rd division of it, finishing in the top position easily and now participates in the 2nd division trying to promote to 1st division.

2018-12-07

AlphaZero paper, and Lc0 v0.19.1

As everyone has already heard, DeepMind has published a detailed paper on AlphaZero!

The announcement can be found here. Scroll down the announcement to get links to the full paper text as well as supplementary materials (including PGNs of games and training pseudocode).

The paper contains additional details that were missing in the original preprint from one year before. There were some aspects that were implemented in Leela differently from AlphaZero, and I'm sure we'll find some more.

Differences found

So, what differences have we found so far? Here is the list!
  • In training games, only first 15 moves (30 ply) are generated with temperature randomness.
    To explore more possibilities during training games, a randomness (including random blunders) was added to the training. The paper preprint told that that happens for all moves. Final paper also says so, but if you look into pseudocode, it turns out that it's only applied during first 15 moves!
    Training new networks with 15-move-temperature setting will possibly help us to improve endgame play. Leela won't longer wait opponent to blunder, having too high eval for drawn positions.
  • When played against stockfish, AlphaZero used a new technique to ensure game diversity.
    What AlphaZero did, is picked a random move with eval within 1% of the best move's eval, for the first 15 moves. Surprisingly, that improved winrate of AlphaZero in those games.
    We can try that too!
  • Action space turned out to be 0..1, not -1..1
    That's more of a technical detail rather than something that changes the algorithm. In AlphaGo paper, loss was encoded as 0 and win as 1. When AlphaZero preprint came out, they wrote that they changed MCTS action values to -1 for loss, 0 for draw and 1 for a win. But in the end it turned out that it wasn't correct understanding. Loss is still 0, and draw is 0.5.
    As I mentioned, it doesn't change algorithms. However, it changes the meaning of some constants from the paper.
  • Cpuct is not a constant
    CPUCT is a constant, which indicates what should be the balance between exploration and exploitation in the search algorithm. Turned out that that constant is not a constant! This value grows as search progresses!
    We had plans to do something along those lines, as there were problems which were seemingly caused by a constant Cpuct. Namely, it usually happend, that at large number of nodes Leela stuck to one move and never switched.
  • First Play Urgency value is known now. It's -1!
    FPU is a fancy name for a node eval for the case the node was never visited. We used a value based on a parent node (assuming that eval of children is roughly the same as parent's eval). Turned out that AlphaZero just considered unvisited nodes as lost (with very little confidence though)
  • When training new network, positions from last 1 000 000 games are used.
    We used 500 000 last games so far, as it was the number mentioned in previous papers.
  • DeepMind generated new network 4 times rarer than we do.
    We were worried that we did that too rare. But it happened that we were fine, in fact it's fine to have 4 times less networks per day.
  • The network architecture has differences.
    See here for the context.

v0.19.1-rc2

What does those findings mean for us?

We want to experiment with new settings in play and training, so we are urgently releasing a new version of Lc0 v0.19.1 (as a release candidate today, the full release will happen during the next days), where we add missing parameters. There are lots of parameters, and many of them are expected to be renamed/rethought for version v0.20. So, please welcome new parameters:


  • --temp-cutoff-move=X
    After move number X, temperature will be fixed to what is set in --temp-endgame flag.
    To reproduce match a0 vs sf8, set this to 16
  • --temp-endgame
    See above for the meaning. This parameter is mostly exposed for training experiments. Default is 0, and it makes sense to keep it like that for play.
  • --temp-value-cutoff=X
    Only moves with eval within x percentage points from the bestmove are considered during temperature pick.
    Set to 1.0 to reproduce match a0 vs sf8
  • --temperature
    This is an old flag, but set to 10.0 to reproduce settings of match a0 vs sf8.
  • --fpu-strategy
    Default is "reduction", old way of handling first play urgency. Set to "absolute" to play like AlphaZero!
  • --fpu-value=X
    Only used in "absolute" FPU mode. -1.0 is the default, and that's what DeepMind used.
  • --cpuct
    That used to be a constant, and it was equal to 3.4 for quite a long time in Lc0.
    Correct value from AlphaZero is 2.5, but it slows down nps (will investigate why), so for now default is 3.0
  • --cpuct-base
    That's that factor which defines how Cpuct grows. The value from DeepMind paper is 19652, and that's now the default.
  • --cpuct-factorThat's the multiplier of the growing part of Cpuct. Default value now is 2, and that's what DeepMind used (well, they didn't have that factor, but as our action space is 2 times larger, we have to scale this parameter).

Those parameters will appear in today's release candidate v0.19.1-rc2, which will be available for download here. (Yesterday there was already v0.19.1-rc1 which had one new parameter, but rc2 will have more!)

Note that most of those parameters probably won't have immediate useful effect. For them to be useful, new networks have to be trained using those parameters.

Also, all those parameters were added into RC2 in a bit of a hurry. It's very probable there will be RC3 with fixes for bugs that I just introduced. If you see a bug, please report!

2018-11-19

Lc0 v0.19.0 has been released.

v0.19.0 is finally out of "release candidate" status, and now is fully released!
It has been quite a long bugfixing run with 5 release candidates, but now all known issues seem to be resolved.

Can be downloaded here.

For the list of differences relative to v0.18, see post for v0.19.0-rc1.

For people contributing training games, there's no need to rush to upgrade, it's fine to use v0.18.

2018-11-13

Where to play Leela online?

The play.lczero.org web site where everyone could quickly play Lc0 online is down pretty often recently.

But even while it doesn't work, there are some options to play Leela online.

The easiest way is to play on lichess.
There is for example a bot called LeelaChess, it is the very first lichess bot.
Also there are other bots of different configurations and strength. Check the all-the-leelas lichess team and pick the one which is online. You are also welcome to host your own Leela and join that team.

If you know other ways to play Leela online (FICS, etc), please reply in comments, I'll add it to this post.

And of course you can always download Leela and set it up to play locally. This blog post describes how to do that.

UPD: Kontrachess has a way to play with LCZero. It seems very to be very nice looking site! I did not try it myself though. (Initially I thought it was a paid site, but one of the site representatives said in comments below that it's actually free).

UPD2: Also NextChessMove has a number of options including different networks of Lc0. From what I can see, it is similar to what play.lczero.org was. It takes some time to get a move from a free version, but probably paid version is faster (again, I don't know anyone who tried that).

2018-11-03

Lc0 v0.19.0-rc1 (UPD: rc2) has been released.

The release candidate of a new Leela version has been released:

(v0.19.0-rc1)
Upd: we are releasing v0.19.0-rc2 immediately as due to mistake in the release procedure rc1 reported its version as v0.19.0-dev rather than v0.19.0-rc1

We expect testing phase to last around 7-10 days, after which proper v0.19.0 will be released.

Download here. Please test it thoroughly and report any bugs that you find.
Note: CudNN builds for Windows are now compiled with CUDA 10. You may need to update you GPU driver to run it.

Please don't use release candidates to generate training games. We only use stable versions for that.

What's new:

Search algorithm changes

When visiting terminal nodes and collisions, instead of counting that as one visit, estimate how many subsequent visits will also go to the same node, and do a batch update.
That should slightly improve nps near terminal nodes and in multithread configurations. Command line parameters that control that:
  • --max-collision-events – number of collision events allowed per batch. Default is 32. This parameter is roughly equivalent to --allowed-node-collisions in v0.18.
  • --max-collision-visits – total number of estimated collisions per NN batch. Default is 9999.

Time management

  • Multiple changes have been done to make Leela track used time more precisely (particularly, the moment when to start timer is now much closer to the moment GUIs start timer).
  • For smart pruning, Leela's timer only starts when the first batch comes from NN eval. That should help against instamoves, especially on non-even GPUs.
  • Also Leela stops the search quicker now when it sees that time is up (it could continue the search for hundreds of milliseconds after that, which caused time trouble if opponent moves very fast).
Those changes should help a lot in ultra-bullet configurations.

Better logging

Much more information is outputted now to the log file. That will allow us to easier diagnose problems if they occur. To have debug file written, add a command line option:
--logfile=/path/to/logfile
(or short option "-l /path/to/logfile", or corresponding UCI option "LogFile")
It's recommended to always have logging on, to make it easier to report bugs when it happens.

Configuration parameters change

Large part of parameter handling has been reworked. As the result:
  • All UCI parameters have been changed to have more "classical" look.
    E.g. was "Network weights file path", became "WeightsFile".
  • Much more detailed help is shown than before when you run
    ./lc0 --help
  • Some flags have been renamed, e.g.
    --futile-move-aversion
    is renamed back to
    --smart-pruning-factor.
  • After setting a parameter (using command line parameter or uci setoption command), uci command "uci" shows updated result. That way you can check the current option values.
  • Some command-line and UCI options are hidden now. Use --show-hidden command line parameter to unhide them. E.g.
    ./lc0 --show-hidden --help

Also, in selfplay mode the per player configuration format has been changed (although probably noone knew that anyway):
Was: ./lc0 selfplay player1: --movetime=14
Became: ./lc0 selfplay --player1.movetime=14

Other

  • "go depth X" uci command now causes search to stop when depth information in uci info line reaches X. Not that it makes much sense for it to work this way, but at least it's better than noting.
  • Network file size can now be larger than 64MB.
  • There is now an experimental flag --ramlimit-mb. The engine tries to estimate how much memory it uses and stops search when tree size (plus cache size) reaches RAM limit. The estimation is very rough. We'll see how it performs and improve estimation later.
    In situations when search cannot be stopped (`go infinite` or ponder), `bestmove` is not automatically outputted. Instead, search stops progress and outputs warning.
  • Benchmark mode has been implemented. Run run, use the following command line:
    ./lc0 benchmark
    This feature is pretty basic in the current version, but will be expanded later.
  • As Leela plays much weaker in positions without history, it now is able to synthesize it and do not blunder in custom FEN positions. There is a --history-fill flag for it. Setting it to "no" disables the feature, setting to "fen_only" (default) enables it for all positions except chess start position, and setting it to "always" enables it even for startpos.
  • Instead of output current win estimation as centipawn score approximation, Leela can how show it's raw score. A flag that controls that is --score-type. Possible values:
    • centipawn (default) – approximate the win rate in centipawns, like Leela always did.
    • win_percentage – value from 0 to 100.0 which represents expected score in percents.
    • Q – the same, but scales from -100.0 to 100.0 rather than from 0 to 100.0

2018-10-28

Lc0 training.




 If you are new to Leela (Lc0) Chess and have begun contributing games either using Google Cloud or some other online service or your own home computer, you may be wondering where all those games go and how training of Leela happens.

2018-10-19

Leela beats Fire promoting to Semi-Final of TCEC Cup!




 Leela in a classic drama style, promoted in TCEC Cup Semi-Finals and it will face Stockfish today!
While in CCCC blitz tournament she is still at 3rd place ahead of Komodo, Ethereal and Fire and behind Stockfish and Houdini.

2018-10-12

CCCC Blitz is running.... Leela on top 3!




CCCC blitz tournament is running and till now Leela is having a good performance being steadily on the top 3.


Conditions for the tournament are:
33 engines play a 4x Round Robin tournament with each engine that will play each other 4 times(2 with black and 2 with white) in a total of 128 games per engine, with no opening books or predefined positions used.
This implies a problem though as an engine will play each other twice with white and twice with black so the question becomes: how variety of play will be assured to not have duplicate games? Obviously they will rely on the non determinism of multithreaded search(traditional engines that use more than 1 threads/cores are not deterministic, even Leela that uses more than 1 CPU threads it's not(Leela mainly uses GPU for its search, but uses also CPUs)). This is of course not that wise decision and they should use predefined positions for the second part of the Round Robin.

•Time control of 5 minutes per game plus 2 seconds added time per move.

2018-10-11

Draw in Chess. Some odd cases.




Chess is a game where there are 3 distinct results. White wins or black wins or it is a draw and nobody wins.  Draw can achieved in many ways in Chess. These are:

•Stalemate position. A position where the player to move does not have a legal move to play and his King is not in check. Game immediately ends as a draw.

2018-10-10

Understanding Training against Q as Knowledge Distillation.


Article by Cyanogenoid, member of Leela Chess Zero development team .



Recently, Oracle investigated training the value head against not the game outcome z, but against the accumulated value q for a position that is obtained after exploring some number of nodes with UCT [Lessons From AlphaZero: Improving the Training Target.]. In this post, we describe some of the experiments with Knowledge Distillation (KD) and relate them to training against q.


Background

Knowledge Distillation (KD) [1 , 2] is a technique where there are two neural networks at play: a teacher network and a student network. The teacher network is usually a fixed, fully-trained network, perhaps of bigger size than the student network. Through KD, the goal is usually to produce a smaller student network than the teacher -- which allows for faster inference -- while still encoding the same "knowledge" within the network; the teacher teaches its knowledge to the student. When training the student network, instead of training with the dataset labels as targets (in our case this is the policy distribution and the value output), the student is trained to match the outputs of the teacher.

2018-10-09

Leela promotes to round of 16 in TCEC Cup with 2 nice wins!



 Leela after 6 consecutive draws in the series of games against Laser(division 1 engine) she won last 2 games so she promoted to next round where she will face Ethereal(premier division engine) that beat Rodent(division 4 engine) easily with 5-0 and promoted too. The games against Ethereal will probably take place on this Sunday.

 Leela's performances generally seem very odd, since when she is playing top engines like Stockfish and getting countless draws with performances like around 40 Elo short of Stockfish(like it is on TCEC bonus games) you expect to crush weaker engines. But this doesn't really happen.
 It's a general observation that Leela underperforms against weaker engines and a good analysis of this can be found HERE where it was found that the usual Elo curve does not fit Leela's results well.

2018-10-08

Achilles heel of Chess engines.... Neural net engines and Leela the only hope!





 When first Chess engines were born were very weak. First real Chess program(even though in 1912 there was a King Rook versus King solver and in 1951 Turing wrote only in paper a Chess program) was written in 1951 by Alan Turing's colleague, Dietrich Prinz, but it was not able to play a whole game but could only solve small "mate in 2" problems.
 The interest in computer Chess was growing and more and more researchers in artificial intelligence(AI) used Chess as a platform to showcase their progress on AI. As hardware has started to evolve more and more, many researchers started to be able to use the already known algorithm, that has proved to be the backbone for future Chess engines: the Minimax algorithm, that was invented many years before by the pioneer in so many fields, John Von Neumann.

TCEC CUP running. Leela in danger of elimination from the 1st round against Laser! UPDATE: Leela won!



TCEC Cup continues and we already know 6 engines that have promoted to next round.
Leela right now has a big fight with Laser and after 4 games it is on 2-2 with 4 draws.

It would be a big surprise if Leela will not manage to promote but with just 8 games everything can happen. The peculiar strength adaption to the opponent of Leela is well known anyway, this means it can hold Stockfish to very close result but playing against much weaker engines she doesn't perform accordingly.



UPDATE:     After 6 draws in the first 6 games, Leela won the last 2 games to promote in the next round. In the game with black pieces she slowly outplayed Laser obtaining eventually a won position, while with white pieces, in a reversed opening from previous game, she managed to easily win heavily outplaying Laser in a French defense where Leela knew even sooner than Cfish(a clone of Stockfish that can run with 7 piece tablebases somewhat better than Stockfish) running on 88 cores, what is going on in the position!
Some analysis of the games will follow.....

2018-10-06

TCEC Cup started! CCCC variants tournament continues....




The TCEC Cup has started and 2 match pairs will be played every day.
We remind that 32 engines will compete in a Knockout elimination tournament, and in each pair match, the best out of 8 games will promote to next round.
That means if an engine completes first, in the 8 games, 4.5 points will promote. As per their decision if an engine completes first 4.5 or 5 points, the match is stopped and the engine promotes to next round. So not all 8 games will be played if not necessary.
Also the tournament would not necessarily be continuous like normal TCEC, since only 2 pairs will be played every day even if there is time for more. The games will begin 12:30 UTC every day.

Till now Stockfish crushed Ivanhoe with a 5-0 score and promoted and Gull won 4.5-2.5 against Texel and will play Stockfish in next round.

2018-10-03

Leela wins the match series against Komodo and wins a Pawn odds game against Stockfish!



 Leela won 16-14 against Komodo the 30 games match to determine 3rd position in CCCC1, while on the variants tournament she won a game against Stockfish while being handicapped having a Pawn less!!

 So the 3rd place Komodo-Leela match has ended with Leela winning 16-14, having 5 wins, 3 losses and 22 draws, so Leela had finished 3rd in the first CCCC tournament according to Chess.com changed rules.

Lc0 v0.18.1 has been released.

It turned out that the version v0.18.0, that was released a few days ago, had a bug that caused it to miss tablebase wins sometimes.

Because of that, v0.18.1 is released. You can download it here.

If you don't use tablebases, there is no need to update, v0.18.0 is fine.
Also version v0.18.1 is not necessary for training game contribution, version 0.18.0 is totally fine for that too.

CCCC season 2 will use v0.18.1.
TCEC cup uses v0.18.0 (probably).

2018-10-02

Contributing to Leela Chess Zero. Creating the Caissa of Chess engines....




 Deepmind's latest paper release (Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm) on 5 December 2017, presented a method of reinforcement learning for convolutional neural networks that its result was fantastic as they created 3 engines, one for GO, one for Chess and one for Shogi that were beating easily the best available engines of the time.

 For Chess the known AlphaZero for Chess, beat Stockfish 8 easily, but in order AlphaZero to be created it required generating games and training for "only" 4 hours, having generated 44 million games! Then these games were used to make the engine learn observing these games and become stronger.
Generating games is the hard part of the whole process anyway. As it is the most time consuming thing. And one has to generate millions of games to start becoming really strong.

CCCC Leela-Komodo 3rd place match started and TCEC Cup is about to start.



TCEC Cup is about to start in a few days and Chessdom.com just announced the complete pairings up to the final.

Leela is on the Stockfish side and if she will promote from the first match against Laser she will probably face Ethereal in round of 16, so it will be a tough match already.
But first has to overcome Laser of course.
Leela in TCEC Cup will use the v18 of Lc0 and 11248 net of test10.

Here is the format of TCEC Cup:

2018-09-30

Lc0 v0.18.0 has been released.

v0.18.0 is out of "release candidate" status, and now is fully released!

Can be downloaded here.

It has no changes relative to RC2. For the list of differences relative to v0.17, see posts for v0.18.0-rc1 and v0.18.0-rc2.

For people contributing training games, there's no need to rush to upgrade, it's fine to use v0.17.


There is a small known issue with time management. The version v0.18.0 times out sometimes when all of the following is true:
  • Ponder is ON
  • Time is added every X moves (e.g. every 40)
  • There is per move increment
Most probably there will be v0.18.1 soon where this issue is fixed.


2018-09-28

TCEC Cup in the next days!



 The TCEC(Top Chess Engine Championship) tournament is currently running with engines playing in the premier division and this division will finish in a few days. But this time, after premier division end, there would not be an immediate starting of the superfinal between the top 2 placed engines in premier, but a new tournament is going to happen that Chessdom has announced, the TCEC CUP, with all engines of TCEC but in a different format, that of knockout games, one that brings in mind the traditional Cup competitions of football.
(TCEC superfinal will he held after the TCEC Cup of course)

2018-09-27

CCCC Leela-Komodo event for 3rd place and Chess variants tournament!




Chess.com has announced that after the CCCC superfinal between Stockfish and Houdini(Stockfish is ahead 27.5-20.5 till now) will finish, then 30 games of Komodo versus Leela will be played to determine the 3rd place.
This is a surprise since there were no such plans initially announced, but it is welcomed for Leela and Komodo fans. Probably Chess.com did it since Leela fan base is high and they want to take advantage of this.

The most interesting thing on the announcement is that the top 6 engines, Stockfish, Houdini, Komodo, Leela, Ethereal and Fire, will play a 10x Round-Robin tournament from each of the following 5 predefined positions/Chess variants.
Each engine will play one with white and one with black from each position in a total of 50 games for each engine(50 rounds).

And here comes the dangerous part about Leela.

2018-09-26

Lc0 v0.18.0-rc2 has been released.

The "Release Candidate 2" for the Lc0 version v0.18 has been published!
Available to download here.

Change:
  • Fixed a severe race condition bug when only happened when --out-of-order-eval is on (which is the default value now).
    Could produce incorrect search results.

CCCC stage 2 ended. Leela 4th with a good performance! Stockfish undefeated!



Chess.com Computer Chess Championship stage 2 has finally ended. After a 280 games battle only the top 2 engines will continue to the superfinal.
Stockfish and Houdini for a 200 games "fight". Well no one really expects it would be a fight and Stockfish will probably win this by a huge margin.


Leela finished 4th, with 37.5 points, in a strong position where it was very close just 1 point short to the 3rd Komodo and a comfortable 6.5 points ahead of the 5th Ethereal. Fire was 6th with 29 points and Booot and Andscacs took the last 2 places with 25.5 and 24.5 point respectively.
Houdini with 43 points won easily the 2nd place 4.5 point ahead of Komodo while Stockfish remained undefeated with an easy lead 8 full points ahead of the 2nd Houdini!
So till now Stockfish has not lose a single game out of the 116 of stage 1 and 2. Let's see if it will lose in the 200 games of the superfinal. Logic says it will but Stockfish on CCCC hardware appears to be invincible! Even though it should definitely had lost one game against Leela since they reached a dead-won endgame position.


Leela managed against the big 3 to have a fantastic score in this tournament of 2 wins, 3 losses and 31 draws!! While against Stockfish she was the only engine in this tournament that lost just 1 time as it had 1 loss and 11 draws against the mighty Stockfish.

In stage 2 there were also some really dubious predefined opening lines, like one that Leela had to deal with in 2 games against Ethereal(the lines always had been played on reverse too so for a specific pair of engines would not be unfair) the 1.e4 Nf6 2.e5 Ne4? line where Leela won with white while she lost with black. Other 2 such dubious lines had been played but Leela did not have any of the other.

From the 280 played games, white won the 80(28.6%), black only the 23(8%) and 177(63%) ended in a draw.


2018-09-24

Lc0 v0.18.0-rc1 has been released.

The release candidate of a new Leela version has been released:

v0.18.0-rc1

We expect testing phase to last around 7-10 days, after which proper v0.18.0 will be released.

Download and full changelog here. Please test it thoroughly and report any bugs that you find.

Please don't use release candidates to generate training games. We only use stable versions for that.

Change highlights:

  • Full Syzygy tablebase support (both WDL and DTZ) (read here for more context what it is).
  • MultiPV support is added.
  • Time management is optimized. Leela should take more time in first half of the game now, and insta-move rarer.

2018-09-21

GUIDE: Setting up Leela on a Chess GUI.





 Leela Chess Zero is a project started before some months inspired by Deepmind's papers about AlphaGO Zero and AlphaZero, which is based on a new paradigm of Chess engines by not using traditional AlphaBeta search with handcrafted evaluation function but uses a variant of MCTS search called puct and for evaluation function it uses a self-taught neural network that learns by deep learning methods by playing against itself million times.

Now after 6.5 months of training it seems that it has reached a place among the top 5 strongest engines available.

 Its use is not that straightforward like traditional engines since it requires some extra things one that has to make in order to run it. It's easy thought. This guide will be about running it on Microsoft Windows on various GUIs.


Leela Chess Zero works with the help of 2 things:

CCCC stage 2 continues.... Leela 3rd place. Missed 2 chances to beat Stockfish!





Stage 2 continues with Leela doing excellently being 3rd, half a point ahead of Komodo!
Stockfish
continues its unbeaten run, being the only engine with no loss, stage 1 and 2 combined!

This definitely comes as a surprise since Komodo one of the big 3 engines in the Chess world, after 31 games should already be ahead in a well defined top 3 but Leela persists to not lose! She now had in the last 17 games, 2 wins and 15 draws. 
Komodo even had 15 consecutive draws!

 She only lost once from Andscacs and that was only because of a heavy blunder of her in an equal position, while she has 13 draws with the big 3 engines Komodo, Houdini and Stockfish and if we count stage 1 results too she has 19 draws out of 19 games with the big 3.  Also, she beat Fire twice in 2 excellent games outplaying Fire, but the big games were the 2 Leela versus Stockfish games where Leela was close enough to beat Stockfish in the first and one small step for that in the 2nd. Leela outplayed Stockfish in these 2 games especially in the 2nd, to reach winning positions.
 In the first as we mentioned in the previous post, she missed a nice move that other nets find(e.g the strong 11250 finds it after 1.5 million nodes) that would lead to a huge advantage and perhaps a win if Leela could convert, but in the second Leela was really close to beating Stockfish! That would be the first loss of Stockfish in this tournament. But Leela blundered and lost the win. Afterwards analysis show that this 11089(CCCC) net with endgame tablebases would not be able to beat SF from that position while net 11250 with endgame tablebases would do it rather easily.

Fire is also underperforming being currently last in 8th position while Booot continues to surprise being 6th for now.
Stockfish is a clear leader 3 points ahead of 2nd, wining against everyone except Leela.



Standings so far:

2018-09-19

CCCC Stage 2 has started and running.....





Stage 2 of CCCC has already started before 2 days. Not without problems initially with repeating restarts but at last now it's running flawlessly.
Stockfish continues to be invincible! As it hasn't lost a single game in the first 46 games of stage 1 plus the 15th till now in stage 2.
While the so far unbeatable too Komodo, in this stage lost to Houdini.

Leela on the other hand had after 6 games with the big 3, 6 draws as in stage 1. So now she is the only engine that has not lost a game from the big 3. She is going fine also generally till now since she won 4 games also and shares 3rd place with Komodo. She lost 1 game though due to a huge blunder with a blind spot of her in an equal position.

Every engine will play 10 times all the other so 70 games for each engine will be played in this stage.
Top 2 will promote to the super final.


Below are the results, the games of Leela with some comments as also the standings and ratings.

2018-09-16

CCCC rounds 43,44 and 45. First stage is over!




1st stage is almost over, 1 round remains but the top 8 engines that will participate in stage 2 are known.
Stockfish, Komodo, Houdini, Leela, Fire, Ethereal, Andscacs and Boot promote to next round.
Next round, unlike this round, would be without pondering and every engines will play every other 10 times so we would have a total of 70 games per engine.

Andscacs won Arasan in the final round so Shredder has no hope to promote.



ROUND 43
 In this round Leela destroyed Shredder's hopes for promotion by winning him after a superb game where it completely outplayed Shredder with a startling exchange sacrifice that Shredder(as also other engines like the mighty Stockfish) had no idea about it. The defense that would lead to draw was very thin and Shredder didn't manage to find it. To him as also to other engines most moves drew easily. But that was not the case as the genius Leela knew in this game!

TB Rescoring

While test20 runs, we are running test30 in parallel to test two ideas. First test30 uses a different method to initialize the first random net, and a slightly different LR (Learning Rate) strategy for the first few nets. The idea is to see if this eliminates the large spikes seen in some key metrics -- policy loss, mse loss, reg term (see our expanded glossary).

The second idea is to test TB (Endgame Tablebase) rescoring. This may improve the accuracy of the value head. The procedure is:

  • Clients continue generating self-play games as before, no TB involved.
  • The data from these games are sent to the server.
  • The server parses these games, and when a position reaches a new WDL entry, the game result for all positions up to that point is changed to the TB result.
  • An WDL entry is when there are 5 or fewer pieces and a capture or pawn push just happened, and neither side can castle (9000 internet points for anyone who can find an Lc0 self-play game where the castling exception applies!)
  • The game is parsed for the next WDL entry, and again all positions in this new section are marked with the TB result.

This is an alternative to doing temperature reduction or other similar methods, but does not require changing the way clients generate data. So it’s easier to test and compare with and without this method. Also by doing this on the server side we don’t require all users to download TBs.

2018-09-15

CCCC quiz. 12 impressive moves!




In CCCC there have been played some very nice games and moves till now.
Can you find them? Play the move on the board and if it's correct it will show the game continuation with a comment. If not, go back with arrows and try again or press show solution if you can't find it.

CCCC rounds 37 to 42. Leela promoted to 2nd stage!





CCCC continues and Leela, along with Stockfish, Houdini, Komodo and Fire, already has secured a place in the 2nd stage.
Ethereal after a strong finish, mainly winning in the last games, has almost secured the promotion. Booot has also many chances.
While Shredder and Andscacs are the ones that are competing for the final 8th place right now with Shredder to be ahead, but having difficult opponents to face in the next 4 rounds.



ROUND 37
 Leela won a fine game against Fritz with an opposite colors Bishops ending with Rooks also. Leela outplayed Fritz that made some mistakes being short on time since it played in the beginning very slowly.

2018-09-13

CCCC rounds 32,33,34,35 and 36. Leela with 5 draws and an "almost win" against mighty Komodo!





First round CCCC approaches to its end and situation is clear by now.
Houdini, Komodo and Stockfish will be in top 3 advancing to stage 2, with Stockfish and Komodo to fight for the 1st place in this 1st stage, thatdoesn't mean much anyway, Fire and Leela would most probably advance to the next round also with some minor chances of not achieving it, and Shredder, Booot, Ethereal and Andscacs will fight for the other 3 places for stage 2.

Leela had 5 draws in these 5 rounds with 2 easy and 3 very hard opponents and she has 7 consecutive draws.

Simulations using Bayeselo give right now(after and the Andscacs-Inavhoe 1-0 game of round 37):
Probability to promote:
Stockfish 100 %
Komodo    100 %
Houdini   100 %
Fire      100 %
Lc0       99.67 %
Shredder  90.33 %
Booot     81.67 %
Andscacs  77.67 %
Ethereal  44.67 %


ROUND 32
Leela had an extra-disastrous opening with Senpai and never managed to recover so it ended in a draw. Against the very weak in this CCCC Senpai, Leela had to win but the selection of the opening moves was a nightmare and even with Senpai playing sub optimal moves, Leela never really managed to get an advantage so it was easy draw.

2018-09-12

A Standard Dataset

When doing machine learning it helps to use a standardized dataset such that methods can be compared in an objective manner. For machine vision, one of the earliest standard datasets is MNIST, a set of handwritten characters that was also used in the (arguably) first deep learning paper.

We should define such a dataset in the world of chess programming as we try to improve training algorithms for our new chess engines based on neural networks. This blogpost introduces such a dataset called the CCRL Dataset (also giving a huge hint as to where it comes from).

Introducing the CCRL Dataset

This dataset was constructed from CCRL 40/40 and 40/4 data combined. It consists of 2'500'000 games, 20% of which is the testset and 80% the trainingset. You can download the dataset in pgn-format (539M) and v3-format (11G).



This figure shows a distribution of all the gameover types within the testset of this data. Games with over 500 plies have been excluded from this figure to keep it readable, and as such the game count is slightly smaller than 500'000 with 0.02% ignored. The double bands for black/white wins show wins on checkmates and resignations. 38% was won with white, 30% with black and 32% draw. Finally, the testset has ~86% unique positions (including history planes).

Baseline results

For training a simple baseline network the following yaml scheme was used as input to train.py:

%YAML 1.2 --- name: '128x10-base' gpu: 0
dataset: num_chunks: 2500000
train_ratio: 0.80
input_train: '/home/fhuizing/Downloads/chess/computer/cclr/v3/train/'
input_test: '/home/fhuizing/Downloads/chess/computer/cclr/v3/test/'
training: batch_size: 1024
num_batch_splits: 1
test_steps: 1000
train_avg_report_steps: 500
total_steps: 200000
checkpoint_steps: 10000
shuffle_size: 250000
lr_values: - 0.1
- 0.01
- 0.001
- 0.0001
lr_boundaries: - 80000
- 140000
- 180000
policy_loss_weight: 1.0 value_loss_weight: 0.01 path: '/mnt/storage/home/fhuizing/chess/networks'
model: filters: 128
residual_blocks: 10
...

This resulted in an accuracy of 47.0583%, policy loss of 1.591 and mse loss of 0.10882. The network can be downloaded as ccrl-baseline.pb.gz. The tensorboard graphs can be downloaded as leelalogs-base.tgz.

Potential ideas

For inspiration, here's a list of ideas where using such a dataset may be useful:
  • Testing a multigpu training algorithm
  • Different neural network architectures
  • Different input encoding (e.g. removing history planes)
  • Different move encoding on the policy head
  • Finishing resign or adjudicated games to get more endgame data with n-man tablebases
And much more...

Final notes

This dataset is very different from our selfplay data runs which improve over time through reinforcement learning. Sliding a training window across the vast number of games produced by clients as new networks are trained. As such one can only test a subset of ideas/parameters using this data. Still, it's probably safe to say one should never regress in terms of performance on this dataset.

Have fun experimenting and please share your results (good or bad, as both can be very useful)!

2018-09-11

CCCC rounds 28, 29, 30 and 31. Leela draws Stockfish and wins in drawn position again!




 It seems Leela is hiding something that is beyond us. How else can be explained that after Fizbo a 3325 CCRL Elo engine with endgame tablebases in a dead 4-men position lost to Leela, now Leela in another dead draw position made Xiphos, a 3200 CCRL Elo engines, to lose!
At round 29 there was the first loss in this tournament of one of the big 3, as Houdini with black lost to Ethereal!


ROUND 28
Andscacs lost to Stockfish and lost ground for the battle of 8th place but Ethereal and Booot lost to Komodo and Leela respectively so they did not took advantage of it so things remained the same.
Fire won against Ivanhoe so now it has a comfortable place inside top 8.

 Leela playing with Booot a French defense(what else?) opening, she took a very easy win by playing simple Chess. She initially pushed all her Pawns, h4,f4,g4, in from of the castled King gaining all the space in Kingside and first thoughts were that she is planning an attack. But suddenly she forced the
exchange of Queens getting a huge Knight on d6 and from then on some almost forced trades have been made going to an endgame that as it seems it was almost lost for black since its King was too far away and white had a 3-2 Pawn majority on Queenside.

2018-09-10

CCCC rounds 26 and 27. Leela with a crazy game as also her first loss!




Leela had its first loss in the tournament after 27 games that went undefeated. It lost against Ethereal with black after she blundered in endgame.
But before that, a real circus game happened in the Leela game of course again. It was the Leela-Fisbo game with Fizbo the known drawmaster with 19 draws out of 25 games but Leela managed to hypnotize Fizbo to lose, while Fizbo was playing with tablebases and the position was a drawn with just 4 pieces remaining!


ROUND 26
In this round Arasan upset Shredder by drawing him, while the big 3 won their games as usual and Fire won against Nemorino. While for the battle of the top 8 place Booot, Ethereal and Andscacs won their games, even though all had the black pieces.

The real craziness happened in the Leela-Fisbo game. Fizbo till now is drawing most of its games with 19 draws out of 25 games and 9 draws out of the last 10!

2018-09-09

CCCC rounds 23, 24 and 25. Leela with 2 nice wins.




ROUND 23
Andscacs won Arasan in an important win for top8 place, while Ethereal and Booot drew their games.
Top 3 continued with wins but Shredder drew.
Leela took advantage of it and won against Laser so she is now alone in 4th place. Leela's game was great, since while in the opening the game seemed dry with no much play but Leela dominated with clever Pawn moves and showing great long term understanding of the position it created a positionaly winning position.
Fizbo after 9 consecutive draws (17th in the tournament so far) it couldn't hold and lost against the mighty Stockfish.


265 Andscacs vs Arasan 1-0
266 Xiphos vs Booot 1/2-1/2
267 Crafty vs Wasp 1/2-1/2
268 Vajolet vs Ethereal 1/2-1/2
269 Fire vs Texel 1-0
270 Stockfish vs Fizbo 1-0
271 Fritz vs Shredder 1/2-1/2
272 Senpai vs Gull 0-1
273 Houdini vs Pedone 1-0
274 Nirvana vs Ivanhoe 1-0
275 Komodo vs Nemorino 1-0
276 Lco vs Laser 1-0

First half of CCCC ended. 23/46 rounds. Leela is 4th! Results and conclusions....


First half of the CCCC tournament has ended.
It's a 46 rounds tournament and so 23 rounds have been played. Time control is 15'+5" and engines will play all against all twice, one with black and one with white, while pondering is on and no opening books are used and 6 pieces tablebases are being used.
Top 8 engines promote to next round.


Till now there are 4 distinctive groups that have been formed. The top 3 that are consisted of the known "big 3" of computer Chess this moment and for the last 4-5 years, Stockfish, Komodo and Houdini, the other group the engines that fight for the top 8, Leela, Shredder are the leaders of this group that show they might have the advantage and the other Fire, Booot, Ethereal and Andscacs.
There is a group 8-9 engines that seems to just be in the middle without having any prospects of going to top 8 and there is the low end of the tournament that does not do that good.
There are 4 engines that till now haven't lost any game so far. Stockfish, Houdini, Komodo and Leela.


The standings after round 23, finishing the first half of the tournament.

2018-09-08

CCCC rounds 20, 21 and 22. Leela with 3 draws.




As it seems 6 engines will fight hard for 5 places that lead to next stage provided that Stockfish, Houdini and Komodo will have no problems to get the 3 top places.
Leela, Shredder, Fire, Andscacs, Ethereal and Boot will fight for the other 5. Booot actually came out of nowhere to fight for getting in the top 8 and right now seems the outsider even though it has a good performance thus far. Leela and Shredder seem a bit safe right now but with 23 more rounds remaining everything is possible.

After Komodo beat Shredder in round 21, now the only undefeated engines are Stockfish, Komodo, Houdini and Leela.


ROUND 20
This round had Ethereal beating Andscacs and now it took its place on the 8th place where it gives the promotion.
Wasp after drawing Leela, managed to get a draw with white from Houdini also. Laser also with white drew its mighty opponent Stockfish also.

While Leela having black played with Shredder a sharp Sicilian Najdorf variation with opposite castles, Leela gave the exchange giving her Rook for a Bishop trying to attack but the defense of Shredder was spectacular and there was no way for Leela to continue the attack. But the exposed white King gave no way to Shredder to try for an advantage due to being an exchange up so game ended with 50 move rule shuffling.

2018-09-07

CCCC rounds 17, 18 and 19. Leela with 2 wins and 1 draw.



 CCCC continues with Houdini on the lead a full point ahead of the 2nd Stockfish. Houdini till now seems unstoppable and is killing every other engine except for the top ones. Leela is 5th for now even though she won 2 games and drew the other.

ROUND 17
Houdini won against Fire in a nice game where after just a single mistake of Fire leaving its Rook in b2 Houdini took the chance and punished it winning the exchange of a Rook for a Knight and after some deep maneuvering it won a Pawn also and the game was essentially over.

CCCC standings, ratings and statistics after round 17.


Chess.com Computer World Championship continues with engines battling for the first 8 positions that would bring them to next stage.
Round 17 has been completed out of 46 that will be played.

Here are some statistics from the games so far:

The crosstable of the engines. Only the first 8 promote to the next round.

2018-09-06

CCCC rounds 14,15 and 16. Leela draws against Crafty....



After another 3 rounds have completed and we reached round 16 of 46 that would be played, Houdini continues its comfortable lead being 1 full point ahead of the second. Leela had a nice win against Fritz but didn't manage to win against the weakest engine of the tournament Crafty!


ROUND 14


The round of draws. Out of 12 games, the 10 were drawn and only 2 were decided.
Leela was one of them by winning against Fritz. A nice game where in late middlegame she outplayed Fritz creating a strong Kingside attack step by step, with Fritz making not optimal moves and losing tempos for Leela to obtain a Kingside attack.

Another clash of titans with Houdini-Komodo game where in a Ruy Lopez, after all Bishops were traded already in the opening, it got an interesting position with black to have a pressure with double Rooks on the Kingside but white was very solid and black couldn't make any progress so it ended quickly with a draw. Not that quickly over the board of course, since even though after move 68 it was a dead draw for these engines, they were forced to play till insufficient material till move 135.
Shredder while in first 6 games with white had 6 wins, it now dropped another half point with white by drawing with Booot.

157 Ivanhoe vs Andscacs  1/2-1/2
158 Houdini vs Komodo 1/2-1/2
159 Laser vs Gull 1/2-1/2
160 Fritz vs Lco 0-1
161 Nemorino vs Fizbo 1/2-1/2
162 Fire vs Nirvana 1-0
163 Pedone vs Ethereal 1/2-1/2
164 Crafty vs Senpai 1/2-1/2
165 Shredder vs Booot 1/2-1/2
166 Arasan vs Stockfish 1/2-1/2
167 Texel vs Xiphos 1/2-1/2
168 Wasp vs Vajolet 1/2-1/2

Analysis of the Leela-Fritz very nice game: