2019-03-25

Upgrading to Lc0 v0.21.1 in CuteChess

This a quick note for people who use CuteChess and just upgraded to v0.21.1 from earlier versions.

Meaning of FPU-related parameters are changed in v0.21.1, along with default values of those parameters.

If you upgraded your Lc0 to v0.21.1 in CuteChess by just replacing the files, most probably CuteChess still remembers old defaults and sets them.

To make CuteChess see correct default values, either install Lc0 v0.21.1 as a fresh engine, or try pressing "Detect" / "Restore to defaults" in engine settings in cutechess.

After update, there should be:

  • no FpuReduction parameter, 
  • FpuValue should be equal to 1.20000 (was -1.00000 before the change)

UPD: Similar issue also happens in Arena. The fix is similar: "Restore to defaults" in engine settings, or reinstall the engine fresh.

2019-03-21

lczero.org server will be unavailable on April 5th


Summary

Important!

 The lczero.org server will be not available for ~24 hours starting at April 4th 20:00 UTC.

Training game generation will also not be possible during the downtime.

For people who use payed services for generating games: don't forget to stop your instances for this period in order not to waste money.

Details

As some of you know, the lczero.org server is located in my apartment. I'm moving to another apartment on April 5th, so the server has to move too.

Because of that, lczero.org will not be available for at least 24 hours.
  • Starting time of downtime: Thursday 4th April, 20:00 UTC
  • Planned end of downtime: Friday 5th April, 20:00 UTC

The following sites will not be available:
  • lczero.org -- LCZero website, also used by clients to send games
  • lc0.org -- our link shortening service
  • board.lczero.org (aka testtraining.lczero.org) -- Tensorboard
data.lczero.org will be available, you will be able to download network files from there (although it's not easy to guess network id from filename).

The new apartment should already have Internet connection at that day, but it can happen that it won't immediately work. For that case I have a backup plan, but it will mean that the downtime is extended as I have to travel further with a server in my pocket. I have no way to check in advance whether the connection in new apartment works, as I only have one set of fiber equipment.

Also there is a small chance that Internet in old location will be cut before April 4th. While this is not very probable, I'd suggest people who use payed services to run client to watch the news and stop their instances if that happens.

Stay tuned.

UPD:
Downtime started.
Latest networks list is here: http://data.lczero.org/networks.html All networks have been copied to http://data.lczero.org/files/networks/

2019-03-18

Test40 update

We recently did the 2nd LR drop for T40, and usually this is when the net approaches it's strongest point (gains after the final LR drop are very small). External Elo tests show T40 is close to T30, and may have already passed. We will know more in the following few weeks. But coincidentally we found two major issues at the same time as the LR drop:

Negative gammas not handled correctly. This was a bug in the Lc0 NN code, but since most Nets do not have negative gammas we didn't notice it. As a coincidence at the 2nd LR drop many nets started to have negative gammas. When this happens, the Net becomes very erratic (-100 Elo or worse), so our gating caught these and prevented further permanent damage to T40 training. This was fixed and released in v0.21.0.

Pawn promotion issues. T40 developed a blind spot when a pawn can either capture+promote, or ignore the capture and promote normally straight ahead. In this case, it placed nearly 100% of Policy on the capture+promote move. This lead to many blunders where Lc0 never considered the opponent would play the normal promotion move. The problem is related to details of how Batch Normalization is done. Starting from net 41546, we are using Batch Renormalization.

You may have heard this issue described as a problem with too large gammas. There were tests done with regularization on gamma, but they were slow to fix the capture promotion. Batch Normalization has 4 main parameters (gamma plus mean, variance, and beta), and they all interact. So we tried Batch Renormalization to improve all 4 of these parameters, and it quickly fixed the pawn promotion problem. It's still early to know if we have a final solution or if there are further changes are needed. A major drawback of using Batch Renormalization (and one reason we didn't use it from the beginning) is it comes with even more hyperparameters that require tuning, especially during the early LR stages.

2019-03-16

Leela falls just short in first TCEC Superfinal appearance

Lc0 did well in it's first TCEC Superfinal appearance, but fell just short of winning the match, losing 49.5 to 50.5. Here are some overviews of the Superfinal:

  • The official TCEC summary
  • Assaf Wool's blog about the Superfinal.
  • And to see the games themselves, see the TCEC archive

    We know this update is long overdue, if you're willing to help us write blog entries please let us know on our Discord