|
|
NN
May 29, 2022 11:56:12 GMT
Post by matejst on May 29, 2022 11:56:12 GMT
I analyze my opening lines with several engines, and what I have noticed is that from a certain size (10mb), the evaluation of the NNs tends to be very, very similar. I use lately FF2, Rebel15x2, Berserk 9-dev -- absolutely no differences at similar depths in relatively positionally complicated positions. The differences start to occur in positions that were less trained -- late middlegame, simple positions between the middlegame and the endgame, where some engines are clearly better.
So... when I have a look at the rating lists, I started to expect that the rating differences would be the product of differences in search speed and reliability. But still, some engines do not fit this pattern. Let's have a look.
01. Stockfish 311221 NN dev : 3492.00 (40 mb NN)
02. Dragon 3 NN (Komodo) : 3480.86 (30 mb NN)
03. Lc0 0.28.2 611062 GPU : 3480.31 (60 mb NN, special case)
04. Fire 8 MC.3 NN : 3389.13 (20 mb NN, SF NN)
05. Koivisto 8.6 NN : 3382.86 (10 mb NN)
06. SlowChess Blitz 2.83 NN : 3376.95 (2-4 mb NNs, 16 specialized NNs)
07. Berserk 8.5.1 NN : 3373.63 (2-3 mb NN -- I was too lazy to check the NN format to know the precise size)
08. Revenge 2.0 NN : 3365.73 (I don't know)
09. rofChade 2.321 NN dev : 3360.57 (20 mb NN)
10. RubiChess 20220223 NN : 3346.63 (20 mb NN)
------------------------------------------------------------------------------------------------------------------------
11. Ethereal 13.25 NN : 3346.30 (I don't know)
12. Seer 2.5.0 NN : 3344.73 (72 mb NN)
13. Arasan 23.3 NN : 3298.46 (I don't recall; did not work on my system)
14. Igel 3.0.10 NN : 3275.84 (I don't know)
15. Halogen 10.23 NN dev : 3258.98 (small net, ultra fast)
16. Nemorino 6.09 NN dev : 3255.56 (20mb net)
17. Clover 3.1 NN : 3251.35 (I don't know; no compiles for my PC)
18. Booot 7.0 NN dev : 3238.77 (I don't know)
19. Tucano 10.00 NN : 3238.43 2370 53.7 (I don't know)
20. Wasp 5.53 NN dev : 3234.12 1500 51.6 320 908 272 774.0 60.5 11.07 3223.77 10.26 50.0
------------------------------------------------------------------------------------------------------------------------
21. Minic 3.18 NN : 3232.65 (I think it is about 30mb; had crashes with previous versions)
22. Rebel 15 NN : 3214.60 (20/40 mb NN)
23. Fritz 18 NN (Ginkgo) : 3193.57 (I don't know)
24. Zahak 10.0 NN : 3183.14 (10 mb NN)
Some comments. All the engines are relatively close, with relatively close evaluations, with the exceptions of Berserk, which has a very, very fast search. SCB has several nets, some of which are specialized for endings, where it kills all the other engines but the top three. It is weaker in the openings. Fire uses a SF net. These nets are usually better. Minic is low on the list, I don't know why. Rebel has a slow and buggy search. Zahak's evaluation was mainly PST (the words of the author), and it is still reflected in the net. Wasp has a small net, based on blitz games, but it seems to work well with a good search.
The new Berserk has switched to a bigger net, and I expect some improvements -- it seems that most improvements are when the engine jumps from a really small net to a medium one; after that, the progress is linked, IMHO, to the distribution of the data: more data from simple positions and endgames seem to improve the quality of the net. I don't know enough about nets' architecture and how it influences the quality of networks.
|
|
|
|
NN
May 30, 2022 21:05:34 GMT
matejst likes this
Post by Ozymandias on May 30, 2022 21:05:34 GMT
There's also the search. There's a reason why SF is still king. No matter how well KD's net is trained (I'd venture just as well as SF's) it never manages to reach SF's level. And that's in the face of SF "influence" over the years. Either you do a clone or you do a weaker engine (usually both ;-)). 06. SlowChess Blitz 2.83 NN : 3376.95 (2-4 mb NNs, 16 specialized NNs) [...] SCB has several nets, some of which are specialized for endings, where it kills all the other engines but the top three. That's some strange engine. Does it use more than one net during the course of a game? |
|
|
|
NN
May 31, 2022 8:33:06 GMT
Post by matejst on May 31, 2022 8:33:06 GMT
Today, the search makes all the difference, I fully agree, although I haven't emphasized it enough in my post. Based on Dietrich Knappe posts at CCC, I believe that the difference in NN quality between SF, Dragon and, e.g. Seer and Rebel is 10 to 20 ranking points, not more. The rest is search (hundreds of Elos). The best nets are fully comparable, and it is a question mostly of computing power and data set choice to create them.
And it is not only about speed -- but quality also. NNs allow a lot of pruning, but check/tactical extensions have to be baked in the search. Yesterday, I tried the new Berserk 9-dev against Koivisto, and she (I saw that some English native speakers use "she" for engines...) blundered at depth 20 at the end of the opening line. Jay Honnold, who probably had the best HCE of the "new hopes" (the authors around Open Bench) hasn't fully adapted the search yet, although I think his net could be better than Koivisto's, having started from a better HCE.
The style of play is similar too. Let's take Rebel. The first net was über-aggressive -- like Benjamin. After Ed's experiments, where he -- just like all the authors -- focused mainly on "strength", his nets, today, are drawish. I generally think he should have avoided reinforcement learning, especially since he has not focused on ending/simple positions, where Rebel is not good enough.
Yes, SCB uses all his nets in relation with the type of positions. She is weaker in the opening (small net), compensate a smaller net with a good, thorough search in the middlegame, and is simply very, very good in simple positions and endings, where it usually easily beats other NN engines (but the top three). After reading a remark from John Stanback, I tested it against Wasp, and indeed, the described pattern above is what happened: SCB was tactically aware, did not lose in complicated positions, and then simply obliterated Wasp in simpler positions.
|
|
|
|
NN
May 31, 2022 9:45:47 GMT
matejst likes this
Post by Ozymandias on May 31, 2022 9:45:47 GMT
That approach has been advocated by many, specially applied to Lc0. Given it's tactically flawed nature (MCTS) a combination of nets could surely improve it overall. For some reason, the devs never fully explored this path.
About HCE. With all the SF clones and derivatives, why hasn't anyone focused on the search changes introduced to the engine in the past couple of years? Are all of them bad for anything other than the NNUe implementation. What about piece values? Can't they be optimized playing against the official version? A stronger, pure HCE AB engine would be interesting to many.
|
|
|
|
NN
Jan 4, 2023 19:16:20 GMT
Post by matejst on Jan 4, 2023 19:16:20 GMT
I remember some of the discussions on OpenBench's Discord about the HCE. In general, it seems that Jay Honnold was the one who best understood HCE, disseminated his knowledge/helped other authors, and tried to improve his engine.
HCE is not exclusively an engineering problem, but a creative/chess problem. All the fundamental ideas have been harvested from the work of older authors and refined. Now, a new paradigm would have to be found. Programming knowledge is very difficult and does not give enough Elo (the obsession of all the new authors and of the SF team). We see, nonetheless, that Mark Uniacke has continued to improve his HCE engine and hasn't switched to NN yet. It is possible, but everybody, today, tends to transform programming in a technical problem, which can be solve by technical means and more processor power.
Let's take some examples: I recently tested two engines with TBs and several time I saw the winning engine giving back material to get in a TBs won position (instead of mating rapidly). The games became nonsensical, but it much easier to use TBs instead of programming endgame knowledge. Many authors ranted that such knowledge gave so little in Elo terms, but none of them, generally, tried to implement knowledge as a whole.
I am also skeptical about the results of NNUE in the opening, and perhaps, here, Jonathan Rosenthal's approach to use exclusively FRC games for net training could be wise.
Anyway, few are the authors interested by pure chess when creating a chess engine. Chris W., perhaps; Jonathan R. -- he is a FM, Larry K., although I doubt he still works on Komodo/Dragon (I think he stopped his work after Don's death); perhaps somebody else, but I don't see who. Marco, Andrea -- but they don't seem good enough as authors, while Omar Khalid refused to open his code.
BTW, contrary to many, I have a good opinion of Norman Schmidt, and lately he has produced an relatively original engine -- Fire Zero (or something like this).
|
|
|
|
NN
Jan 4, 2023 20:47:10 GMT
Post by Ozymandias on Jan 4, 2023 20:47:10 GMT
If you want chess played better, although at the same strength, you need the human component. You need a centaur. Engines alone can only go so far, beauty is not in their nature to seek.
|
|
|
|
NN
Jan 4, 2023 21:07:11 GMT
Post by matejst on Jan 4, 2023 21:07:11 GMT
I played blindfold two games today. Lost the first one against Maia 1600. I was so mad at myself: I hurried to win a won position and lost. I resigned halfway the second in a completely won position -- I was too tired. I played it against Maia 1500, won a K first, then the Q in a position of the Semi-Slav where I knew the motives, so it was easy to play.
But the first game again: it is so difficult to be patient at the board. My position was easily won, I had only to finish developing. Even then, had I not made a blunder (I thought black was in mate after Qh7+, Qh8), I still had a winning advantage, but I couldn't just stop a bit, wait, assess the position.
My vision is not good. I have to focus deeply, for a very long time, to really see. Otherwise, I often rely on pieces' relative positions, see only a tiny part of the board. In the second game I tried to focus even more and it worked for part of the game, but I was extenuated at the end.
I'll try another game tomorrow, or simply to read a game in a book without a board.
The only thing that is really positive is that I find good, attacking, aggressive moves easier, using the techniques I learned from a book.
BTW, I could send you some books about training if you are OK with it.
|
|
|
|
NN
Jan 4, 2023 21:14:16 GMT
Post by matejst on Jan 4, 2023 21:14:16 GMT
If you want chess played better, although at the same strength, you need the human component. You need a centaur. Engines alone can only go so far, beauty is not in their nature to seek. Centaur chess could be revived with the use of old engines -- like Fritz 3, some bean counters to help avoid blunders, but the man would still be the one searching for positional ideas. I often analyze under Fritz 5 with Hiarcs 8 Bareev. It is slow, positionally safe, and does not replace your ideas, but helps you clarify them. After a session with H8Bareev, I understand the position much better than if I use a newer NN engine which does all the job for me. In general, the most powerful engines are useful only to top players imho. |
|
|
|
NN
Jan 4, 2023 23:17:23 GMT
Post by Ozymandias on Jan 4, 2023 23:17:23 GMT
I could never finish a game playing blind, not one of my talents. In general, I feel like going backwards, I miss the most obvious of things, even spending on simple positions more time than average (for other players). It's really disappointing.
Centaur chess can't be revitalized, the kind of competition you mention would need to be played with a referee present.
|
|
|
|
Post by matejst on Jan 5, 2023 0:23:03 GMT
I could never finish a game playing blind, not one of my talents. In general, I feel like going backwards, I miss the most obvious of things, even spending on simple positions more time than average (for other players). It's really disappointing. Centaur chess can't be revitalized, the kind of competition you mention would need to be played with a referee present. I did not know that I could play blindfold until I tried, for a bet, against a club member. And I won my first two games. But it was 30 years ago.
I think that you are able to play blindfold, and that everyone is. It is difficult, but most of the time, you don't need to see the board, just a part, and you need to remember some motives, relative position of pieces. Krogius, in his book about chess psychology, asserted that most GM did not really "see" the board playing blindfold. Jonathan Rowson, otoh, was worried because, from a certain age, "he had difficulties seeing the board". I don't believe Rowson.
Playing part of a game blindfold is a great training, almost replacing (and nicely complementing) real otb play. When I restarted playing, last year, I could not remember the position of all the pieces after 10-15 moves in a game. When I replayed games, they were completely different from what I believed/thought. I made a pause, then retried, then repaused, then retried again. It becomes easier with training. The mental picture is more clear, more truthful, and one needs less time to reconstruct it. You still see just an abstraction, but you have a mental "picture" you can use. It is just a question of practice.
I met/knew several GMs from the Goša club (Martinovic, Ristic, Rajkovic, Barlov, Cebalo) and played football better, did maths better, and, in general, found them very average. The most intelligent player I knew was an MK, Djuretanovic, but he never wanted to become a master, to be able to play blitz for money. The difference is only training, practice, and nothing else. I am absolutely sure that if I manage to work 4 hours on chess daily (something I did not manage so far), in one year I'll be 100 Elo over my best results, and in two 200 Elos. Anybody can be a master. Having played very little -- 60 LTC games, perhaps, I got somewhere near 2200, without a coach, with just a few old books. I am older, but I am sure I can achieve more today with all the means I have at my disposition.
|
|
|
|
NN
Jan 5, 2023 0:27:51 GMT
Post by matejst on Jan 5, 2023 0:27:51 GMT
Centaur chess as a form of competition indeed cannot be saved, and as you wrote well, correspondance chess is also dead. But, at least, it was fun while it lasted.
|
|
|
|
NN
Jan 5, 2023 6:59:08 GMT
matejst likes this
Post by Ozymandias on Jan 5, 2023 6:59:08 GMT
It was, but I arrived so very late to the game. I played the last two PAL/CSS tours, but I was starting, so I didn't have much fun in those. After that, it was chaos.
|
|
|
|
NN
Jan 5, 2023 20:48:55 GMT
Post by matejst on Jan 5, 2023 20:48:55 GMT
Today's blindfold game: [White "Boban Stanojevic"]
[Black "Maia 1600 5s"]
[Event "Blindfold game"] [Site "Lucas Chess R 2.01c2"] [Date "2023.01.05"] [Round "?"] [Result "1-0"] [ECO "C62"] [Opening "Ruy Lopez: old Steinitz defence"] [Termination "Mate"] 1.e4 e5 2.Nf3 Nc6 3.Bb5 d6 4.d4 exd4 5.Nxd4 Bd7 6.O-O Nf6 7.Nc3 Be7 8.Nxc6 bxc6 9.Bd3 O-O 10.f4 Bg4 11.Qe1 Re8 12.Qg3 Bd7 13.Be3 Ng4 14.Bd2 Bh4 15.Qf3 Rb8 16.h3 Nf6 17.Kh2 Rxb2 18.g3 Bxg3+ 19.Qxg3 Nh5 20.Qg5 Qxg5 21.fxg5 g6 22.Rae1 Re5 23.Rb1 Rxb1 24.Rxb1 d5 25.exd5 cxd5 26.Rb8+ Kg7 27.Ne2 c5 28.Bc3 d4 29.Nxd4 cxd4 30.Bxd4 f6 31.Bxe5 fxe5 32.Rb7 Nf4 33.Rc7 Nxd3 34.cxd3 Kf7 35.Rxd7+ Ke6 36.Rxh7 Kf5 37.h4 Kg4 38.Kg2 a5 39.Kf2 a4 40.h5 gxh5 41.g6 h4 42.g7 h3 43.g8=Q+ Kf5 44.Rh5+ Kf6 45.Qg5+ Ke6 46.Qxe5+ Kf7 47.Rh7+ Kf8 48.Qe7+ Kg8 49.Qg7# 1-0 I made a blunder somewhere near the end, when I thought that the black pawn was still on c7. Otherwise, decent blindfold play against a weak opponent, especially until move 30, when I lost focus. |
|
|
|
NN
Jan 5, 2023 21:20:24 GMT
Post by Ozymandias on Jan 5, 2023 21:20:24 GMT
I lose track of the board at around move 12.
|
|
|
|
NN
Jan 5, 2023 21:45:53 GMT
Post by matejst on Jan 5, 2023 21:45:53 GMT
Just try to play a game every day. It's disheartening at the beginning, but with every game you'll play longer and better. At the end of a game, it is perhaps the most difficult: one has forgotten the place of a piece (or that a piece is still in play), pieces are distant from each other, and one has to focus deeply and to "see" the board to be fully aware of their places.
With time, you will remain aware of a bigger part of the board easily -- pawn structure, place of pieces. You will forget less and less moves. Sometimes, you will have to replay the game in your head to remember the place of a piece, but it will become less and less often. Just trust yourself.
And don't force yourself. At first, in this game, I was with my eyes closed next to the PC, but then I went to smoke, then have a bite to eat, I thought about my position while drinking a coffee without full "vision". So, rest between moves when you can. Playing blindfold is a combination of vision and logic.
All serious players incorporate this type of training into their routine. Korchnoi read games in bulletins, Spassky played a handicap blindfold match against eight opponents before meeting Petrosian, etc. This considerably improves calculation and overall awareness.
|
|
