2018-08-23

Tablebase support and Leela weirdness in endgame

As it has been announced earlier, Leela has a partial endgame tablebase support now.

The support in v0.17.0 is partial only, only WDL tables are probed, but not DTZ.
That means, that Leela is only able to query tablebase for positions immediately after captures and pawn moves, and for other positions it has to think by itself.

While it improves strength of the play in average, often lack of DTZ queries causes weird endgame playing effects and losing play.

For example, Leela may have a 7-men position with considerable advantage (probability of win 99%) and then "simplify" to 6-men "won" position by just giving up the material. That 6-men position is "won" from the point of view of tablebases, so it has probability of win 100%, and Leela happily goes there.
However, after that move Leela has to play by itself, and that position may be really hard for Leela to win without tablebases. It's not rare that that leads to drawn or lost games, which Leela would win or draw if it played without tablebases at all.



The code for DTZ support is ready, but not tested, and we are not releasing it in v0.17.0 which will be used in CCCC.

We are looking at CCCC test which is currently running and depending on how it looks like, possibly will ask CCCC team to disable tablebase usage for LCZero this season (but probably won't).

12 comments:

  1. Looks like you might want to disable it after that sudden dramatic loss to Nirvana of all bots. For now let's just see how far she can get with the suped-up hardware.

    ReplyDelete
    Replies
    1. That particular game was really lost long before approaching 6-men position, so I doubt that is related to tablebases.

      Delete
    2. I seemed like Leela trapped its bishop. Very unlike her!

      Delete
  2. Well, the thing is that Leela was against Nirvana, which was by far the weakest of Div2 on TCEC. It's very possible that she could have worked it into a draw rather than fumbling her own bishop. By that point in the game she was definitely hitting the EGTB quite a bit. It could easily have been leading her down a complicated path towards a draw that she wasn't accustomed to playing. Meanwhile, Nirvana had no TB. It appears that in that game the TB was working *against* Leela, rather than for her. Should probably analyze that game thoroughly to see where things went wrong, rather than assuming it wasn't this TB bug.

    ReplyDelete
  3. Crem, i watched the game against nirvana yesterday and leela seemed to calculate little, being quite behind in tactics. Is the engine including TB slower than the old one? Do you know which leela ratio was achieving? Which NN was playing yesterday. After seeing the games yesterday, i am a bit pessimistic about performance in cccc. Leela could only play at about 3250 elo cccc.
    Is the NN to be sent decided?

    ReplyDelete
  4. Since leela doesn't evaluate as many nodes as conventional engines could it be possible to probe at every position without losing performance?

    ReplyDelete
  5. I think that it's only meaningful to probe the TB if the position was reached by a capture. If there was no capture the position either is not in the TB (because there are more than 6 pieces left), or the answer from the TB would be the same as the answer already received (for this branch).

    ReplyDelete
    Replies
    1. The problem is AFTER you enter TB territory if you do not have DTZ the engine should find the path to victory on its own. If that path is very narrow then there is a problem (especially if the engine converted from an easier but longer path to a 100% won but hard)

      Delete
    2. Surely one could probe the TB all the time and limit the moves to those that don't spoil the position (and hopefully Leela's understanding will make her favor the better moves that "make sense").

      Delete
  6. If DTZ support code is ready as you say, is one week of testing by several users not enough to pinpoint possible bugs? Or is it that the the version submitted to CCCC is already frozen?

    ReplyDelete
  7. After more than a thousand training steps i trust leela can handle endgames by herself.

    ReplyDelete
  8. what kind of settings are best for long time control? (correspondence chess, hours per move)

    ReplyDelete