2019-05-17

Training server is down

UPDATE: The person who has access to the server is currently travelling, until Sunday. That means that updated information about the server is expected in the beginning of the next week.

--- original message ---

As many of you heard, the training of Lc0 has been suspended yesterday.

Here is a short summary what's happened.

(context)
In April 2018 when we were looking for a training server, one of our contributors kindly volunteered to help and provided 3 servers (with middle-class GPUs), together than a space to cost them and covering the electricity bills.

In May 2018 we croudfunded and bought another server (with two GTX 1080ti, which were high end consumer GPUs at that time), and located it at the same place. Since then it have been our training server.
(/end of context)

Yesterday, May 16 2019, our training server went offline (three other servers are online, but they are too weak to be useful), and so far we have not been able to reach the person who hosts it.

We don't know what happened to that machine (it's likely that just reboot would help, or maybe there's some part to replace, or maybe it's something more serious).
We also don't know how long will it take to reach to the person who can take a look.


FAQ
Q: Why not to move training to cloud service?

A: Several reasons:

  1. We had our own server, and it was just fine.
  2. It takes time to set up (so if we choose this option, or decide to buy a new machine and put to some other place, it will take up to a few days to bring it online).
  3. It is expensive (at least $600/months a the configuration that we need), and we are trying to stay away from accounting.

2019-05-11

Competition Season!


Today, May 10, 2019, Leela is playing in the first game of the high profile TCEC Season 15 Superfinal. Leela qualified after she cleared Division P undefeated, winning at least once head-to-head against every opponent except Houdini. Her opponent in the superfinal is Stockfish, who lost once to Leela head-to-head in Division P, but finished with a higher score by winning more games than Leela against the other Division P competitors.

In the just-concluded TCEC Cup III, seeding was determined by the engines' finishing position in their divisions, with the exception of Leela, who was automatically seeded #1 as the Defending Cup Champion. Leela easily handled Round 32 and Round 16 of TCEC Cup 3, knocking out Marvin (5-0) and Booot (4.5-0.5). The quarterfinals featured the same engines as Division P with Leela knocking out Fire (5-3). In the semifinal, Leela scored two wins over Houdini ending with a 4.5-2.5 score to advance to the cup final versus Stockfish. In the final, Leela and Stockfish played their first eight games to a tie, each with one win and one loss. In the first tie-breaker pair, Leela won as white and held Stockfish to a draw on the reverse, winning the match 5.5-4.5 for her second consecutive TCEC Cup.

In the hours before the superfinal, the TCEC team ran various unofficial tests at disparate time controls. One such test match featured Stockfish at 30' + 5" time control and Leela 1/10th of that at 3' + 0.5". Leela still won a game and drew the match despite the large time disadvantage. Prior to the Cup, Leela faced the supercharged 176-thread, 7-man tablebase "Bluefish" version of the Stockfish engine that is used as a kibitzer for the TCEC eval graphs. Again, Leela won a game and drew the match.

The Leela net T41800 (used in Division P) is approximately 10-20 Elo stronger than the T32930 net that narrowly lost to Stockfish in the previous superfinal and the T40.T8.610 net selected for this superfinal is approximately 35 Elo stronger than T32930. Given Leela's progress, many chess fans think that Leela is likely to win this superfinal.

The net T40.T8.610 that is playing in the superfinal is a fork of T40 that has been refined by @jhorthos. While more detail is available in the documentation he put together, it can be summarized as a T40 net forked from 42000 with an additional 3x LR (learning rate) drop applied at the fork and partial 7-man tablebase rescoring. It was trained only on games produced in the main training pipeline and doesn't include any non-zero features that distinguish it from any other T40 net (LR drops and TB rescoring are a standard part of Leela's training process). This fork is a short-term Elo boost that allows for extra polishing of a competition net without affecting the LR schedule of the main training run. The main T40 run is expected to surpass this net and will have another LR drop in the coming weeks/months before it eventually plateaus and is retired to open up resources for the next run.

Leela is also competing in the Computer Chess Championship CCC 8: Deep Dive as the last engine up in the escalation format, playing four games against each of the other 23 engines and looking to overtake the current top scores from Antifish, Leelenstein, and undefeated Stockfish.