AI and voice30 Jun 2026

Turning a player's worst moves into drills the engine grades

11 / 11 regression positions tagged

My DataTalks.Club AI Engineering Buildcamp capstone. It syncs a player's whole history, finds the moves that cost the most with Stockfish, brings them back as graded drills on a spaced schedule, and answers questions through a PydanticAI agent with twelve tools, which can also rank each practice session and write its prompts. It has since grown into a full trainer: labelled game review, bots that blunder by rating, a sparring partner built from past opponents, and an opening repertoire on its own schedule.

Live app ↗Code on GitHub ↗

Club players collect hundreds of online games and rarely turn them into a training plan. An engine can say a move lost 2.3 pawns, but not which habit caused it: a loose piece, a missed check, an uncastled king, a queen that wandered in the opening. And nothing brings the same position back a week later, so the lesson fades.

I built Chess Coach Agent as my capstone for the DataTalks.Club AI Engineering Buildcamp. It imports a player's Chess.com, Lichess or PGN games, finds the moves that cost the most, and turns them into board drills that Stockfish grades and a schedule brings back. A PydanticAI agent answers questions about the game on screen, and when an API key is set the same agent also picks each practice session and writes its questions. It is told to take every chess fact from Stockfish and the stored games. In the latest demo recording the sample account holds 2,777 Chess.com games and 7,880 positions ready to drill.

After the course I kept going, against what Chess.com Game Review, Lichess and Aimchess already offer. The review now labels every move, eight bots and a sparring partner play full games, an opening repertoire runs on its own schedule, a screenshot of a board can become a position, and a Today page says what to do next. The rule from the first version still holds: every screen starts from the player's own games, and Stockfish decides what is true.

2,777

games synced for the sample account

7,880

positions from those games ready to drill

11 / 11

regression positions tagged with the expected theme

3.0 → 5.5

judge score for coaching text, out of 6

Starting from how players get better

The research on chess improvement points at practice more than play. Campitelli and Gobet's review concludes that a large amount of deliberate practice is necessary for master level, though not sufficient on its own. Deliberate practice means working on your own weak spots with feedback, which a casual player rarely has a coach to give. Memory research adds the other half: Cepeda and colleagues pooled 317 experiments and found that spaced study beats cramming, and that the best gap grows with how long you want to remember.

I compared twelve products against that before writing code, from Chess.com Game Review and Lichess to LLM coaches like Chessy and Chessiro. Each one covers part of the loop and none covers all of it.

So the brief stayed narrow: take the player's own games, find the moments that cost them, explain them in a coach's words, and bring each one back on a schedule. Stockfish decides what is true. The language model explains and organises.

Deciding what counts as a mistake

A full history is too much to review by hand. The sample account has 2,777 games, and a coach that flagged a fixed number per game would invent mistakes in clean games and hide them in messy ones.

I let the engine decide. A background job pulls up to 5,000 games, skips any it has already analysed, and runs Stockfish on every move. A game can come back with no critical moments at all, or with a long list.

Centipawns alone mislead. Losing two pawns of advantage matters in a level game and hardly at all when you're a rook up. So scores become winning chances first, with the formula Lichess publishes, and a mistake is a drop in those chances.

winning_chances = 2 / (1 + exp(-0.00368208 × cp)) − 1     (cp clamped to ±1000)
loss            = winning_chances_before − winning_chances_after

A move Stockfish itself picked as best is never labelled, which keeps out the noise where two searches disagree slightly. Set the evaluation before and after a move to see where the lines fall.

Only after the engine has confirmed a loss do the theme detectors run. They explain a real loss in human terms and never create one.

Labelling every move

Mistakes alone make a review read like a list of faults. Players coming from Chess.com expect every move labelled and an accuracy for each side, and without them the review felt thinner than the free tools they already use.

I kept the same winning-chances scale, so a label and a drill never disagree about one move. Each move goes through the checks in a fixed order and takes the first label that fits.

Accuracy uses the Lichess formula: each move's win-percentage loss becomes a 0 to 100 score, and the game figure averages a mean weighted by how volatile the position was with a harmonic mean, so one blunder in a calm game costs more than it would in chaos.

The review page keeps its panels under four tabs, Moments, Moves, Coach and Summary, and adds an eval bar and graph, a Retry this move button on any labelled mistake, a heatmap of which side controls each square, and an annotated PGN download. The coach can now point as well as talk. Its answer carries up to four arrows and six highlighted squares, and the server drops any arrow that isn't a legal move in that position before the board draws it. The browser's own speech engine reads answers aloud, and each answer has helpful and not-helpful buttons.

Turning a mistake into a drill

Reviewing a mistake once doesn't make it stick. Chess.com's Retry and Lichess's “Learn from your mistakes” replay the position inside that one game, and then it's gone.

Stockfish grades every attempt, so the model never decides whether you were right.

I store each mistake as a training position, and without a model a session takes the most severe ones first. Every drill comes from the player's own games. The Today page opens on what is due, the latest game and the top leak, ranked by points lost per 10 games, and one button starts a session on that leak.

The drill adapts to the player: with three or more attempts on a theme, accuracy over the last ten sets the difficulty; before that, rating does. Beginners pick from three moves with a hint on offer, intermediate players from four, and advanced players type the move. The wrong choices are forcing moves, checks first, because those are the ones a player is tempted by.

The grade then sets when the position returns.

Add attempts to one position below and watch the gap stretch after each clean solve and drop back after a miss.

A player with few stored mistakes on a theme would get a short session, so rated Lichess puzzles fill the gap. The library holds 1,250 of them, 250 per theme, mapped from Lichess tags such as hangingPiece to the app's themes. They are picked near the player's rating, 150 points lower for a beginner and 150 higher for an advanced player, and the search widens in steps until it finds ones the player hasn't seen. The player's own positions always come first; puzzles only top up the session.

Grading had one bug worth naming. Stockfish reports a forced mate as a score of 100,000, and the grader subtracted those raw, so missing a mate counted as a loss of about a thousand pawns. Both scores are now clamped to ±1000 centipawns before the subtraction, the same clamp the move labels use.

Building bots that lose like people

Playing full games against Stockfish teaches little below master level, because it never makes the mistakes a club opponent makes. Turning its depth down doesn't help either: a shallow engine plays strange moves, not human ones. Maia, the network trained to predict human moves, would be the right tool, but it needs lc0, which the container doesn't ship.

So I put a move model on top of Stockfish. It asks for the eight best moves, converts each to a win percentage, and samples from them. Lower ratings sample more loosely and sometimes blunder on purpose, more often in sharp positions with checks and captures on the board.

temperature    = max(0.6, 14 × exp(−(rating − 600) / 600))
blunder_chance = min(0.30, 0.25 × exp(−(rating − 600) / 450)) × complexity     (complexity 0.5 to 1.6)
move           ~ softmax(−loss / temperature + style bonus)

Eight bots run from 600 to 2200 in three styles. Aggressive bots score checks, captures and moves toward the king higher. Tricky ones like quiet moves that leave a piece en prise as bait. Solid ones prefer castling, development and recaptures. Each plays book moves from the Lichess openings list for its first six to fourteen plies, more for stronger bots.

The ninth opponent is a sparring partner. It reads the first sixteen plies of every stored game, keeps the moves the player's real opponents chose, and replays them by frequency, with a note like “Your opponents played c6 here in 5 games.” Once it runs out of history it hands over to the same engine model at the player's own rating.

Scheduling an opening repertoire

Opening mistakes repeat because the same player meets the same openings. The game library already shows which lines a player reaches, so the repertoire builder starts from it rather than from a blank tree.

For the first twelve plies, it takes the move the player chose most often in games that scored at least 40%, and adds the opponent's three most common replies. If half the games with that move labelled it an inaccuracy or worse, the engine's move goes in instead, with a comment saying how often the player chose the old move and what the engine prefers, and the line stops. Positions are stored by FEN without move counters, so transpositions meet in one node.

The drill shows the board after the shortest route to each due position and accepts any move in the repertoire from there. These intervals are longer than the mistake drills' 30-day cap, because an opening line is meant to be kept for years.

Grounding the coach in the engine

A language model will describe a chess position with confidence and get it wrong. It can't calculate, and a plausible variation reads the same as a real one. Chess is a bad place to trust it on facts.

I gave the agent twelve tools and told it to treat engine evaluations, legal moves and stored game facts as its only evidence. It answers in a panel beside the board on the review, practice, analysis and play pages. The Moments and Summary tabs never call the model.

Chess is a bad place to trust a language model on facts.

Four tools work on the selected game, the lessons and the board in front of the player. The other eight reach the database of stored games or Stockfish, and three of those exist for building practice sessions.

The server adds verified context before the model runs, the answer must validate as a Pydantic model, and the browser keeps the conversation so the server holds no state. Step through one real answer from the evaluation set.

question: "Why did I lose my queen on move 13 after Qe2? What should I have played instead?"

When the model is missing, fails or passes 60 seconds, the server sends an offline answer built from engine data, and the panel labels it as offline. It uses the moment on the player's board if there is one, and otherwise the game's top moment.

Letting the agent build practice sessions

Practice has gone back and forth. In June a twelve-tool agent sat behind the Start practice button, and I took it out along with the public chat routes, so that sessions came straight from Stockfish data. The plain version worked, but every drill asked the same question, “What would you play in this position?”, and the order ignored which habits a player keeps repeating.

So the agent is back as an option. With a key set, starting a session runs the same twelve-tool agent: it reads the stored moments, ranks them against the player's recurring themes, builds the session and writes a prompt and hint for each position. The merge kept the puzzle top-up, and the model can only reorder moments that already exist. Ids it invents are dropped, and hints are dropped for everyone above beginner.

The trade is latency for prompts that name what happened in the position, and I haven't measured how long that wait is. It sits in the one place I had removed it from. Sessions have no overall deadline the way coach answers do, only a timeout on each model request. Stockfish still grades every answer, so a weak prompt can make a drill read badly but can't mark a wrong move as right. The prompt text itself is checked by nothing except its instruction not to invent moves.

Reading a board from a screenshot

Players often meet a position outside their own games: a puzzle in a book, a stream, a friend's phone. Typing it in square by square is slow enough that most never do.

The Set up position page takes an image. A vision model returns the board as eight rows of eight piece letters with the orientation, side to move and its confidence, and python-chess turns that into a position. Then the rules check it: one king each, no more than eight pawns or sixteen pieces a side, no pawns on the back ranks, and the side not to move not in check. If a check fails, the problems go back to the model for one more try, and the player can fix any square by hand before analysing.

Six boards is a small set, and it isn't in the repository yet, so this result can't be rerun the way the others can. The position checks matter more than the model choice: they catch the errors that would otherwise reach the engine as an impossible position.

Choosing retrieval by measurement

The coach needs the right lesson to cite, and the knowledge base is small: four hand-written lessons on tactical vision, loose pieces, opening principles and king safety, cut into chunks of about 500 tokens. On a set that small, the obvious choice of vector search may not win.

I wrote 20 queries with known answers and benchmarked four strategies. At start-up, the server reads the results file and picks the winner.

Judging the coach before trusting it

Good-sounding coaching text can still be wrong about the position, and nobody reads hundreds of answers by hand. The checks had to run on their own and be cheap enough to repeat.

I built four layers. Each answers one question, and each uses data I wrote rather than data a model generated.

For the analysis text, an LLM judge scored theme, grounding and coaching quality against my reviewer notes, 0 to 2 each. A short format averaged 3.0 out of 6 and passed half the cases. A grounded format, with the played move, the engine move, the principle and a drill, scored 5.0. The judge then pointed out that the opening-drift answer never linked repeated queen moves to lost development and delayed castling. I rewrote it around that, the case went from 4 to 6, and the average reached 5.5. I also reviewed the two original cases by hand: both acceptable, with one improvement noted.

Good-sounding coaching text can still be wrong about the position.

The live coach got its own set: 15 questions phrased the way a player asks them, each with acceptable moves, forbidden moves and key points, every one checked with Stockfish at depth 20. I ran two system prompts on three models. Pick a measure, then a row.

Haiku and Gemini Flash tie on the judge, but Haiku answers in about a third of the time, so Haiku is the default in the repository. The judge is also Haiku, so it could favour itself; Gemini matching it on the same scale suggests the gap to MiniMax is real. Neither prompt won clearly. With one run of 15 questions, half a point is noise, so the baseline prompt stays. This run used the nine tools the coach had at the time. I haven't rerun it on the twelve-tool agent.

The whole comparison, 90 answers and 90 judge calls, took 8.5 minutes and at most $0.38. The tests swap in PydanticAI's test model and a fake judge, so CI runs the same pipeline with no key.

The practice agent is the thinnest part of this. Ten scenarios list the tools it should call, but the published 10 out of 10 is an offline run that feeds the expected tools back in and uses a judge that passes everything, so it proves only that the pipeline runs. A --live flag scores real runs, and no live result is in the repository. The check I can stand behind is a manual one: three scenarios reviewed by hand, all acceptable, with no invented positions.

Watching every answer in production

Once real players use it, the questions change: how often does the model fail over to the offline answer, which tools does it call, and do players find the explanations useful?

I log in two layers. A JSONL event log works with no account. With a Logfire token set, PydanticAI is instrumented, so every coach question becomes a coach_ask span with each model request and each tool call, arguments and results included.

1. Collect

   Each critical moment and coach answer has helpful and not-helpful buttons. A moment rating stores the moment, theme, FEN and comment.

2. Export

   make feedback-export turns moment ratings and not-helpful coach answers into rows marked as candidates.

3. Review

   A person checks each row and adds the PGN. Noisy ratings never become ground truth on their own.

4. Re-run

   Accepted rows join the dataset, and the theme and judge evaluations run on them.

Shipping one container

A capstone is graded on whether someone else can run it. Stockfish, a database, a Python API and a React app are four things to install, and each is a chance for a reviewer to give up.

I put them in one image: the built React app, Stockfish, Alembic migrations, lesson ingestion and FastAPI. Render runs that image with managed Postgres and pgvector; locally, SQLite needs no credentials.

1. Sync once

   A durable job per player fetches the whole history and reports progress the page polls.

2. Store

   Postgres in production, SQLite on a laptop. Games, moments and attempts stay relational; only the lesson chunks get vectors.

3. Test

   The 154 pytest tests need no network or key. CI runs it, both offline evaluations, Postgres migrations and the frontend build.

4. Deploy

   One Render blueprint, with /api/health as the health check and the API key asked for at setup.

The game library was the slowest screen. It sent every game's stored analysis in one response, 78 MB for the sample account. Now the server keeps a light index of each player's games in memory and loads the heavy analysis only for the page on screen, 50 games at a time, and the first page is 26 KB.

Against the course rubric, the project scores 35 of 35 on its own scorer script. A strict reviewer might take two points off, because the docker-compose frontend proxies the API to the wrong host.

Naming what is still weak

The numbers above come from small sets I wrote myself. They show the loop works, not that the coach is right about chess in general.

Notes

• The engine numbers in the traced answer are from the evaluation run on 29 October 2026. Move choices in the drill figure are one shuffle of the distractors the training code picks, and its prompt and hint are the templates used without a model.
• Screens are frames from the demo recording in the repository, made on 29 October 2026. The live demo's database starts empty, so the sample account's data isn't there until someone syncs it.
• The live demo reports minimax/minimax-m3 as its model, while the repository's blueprint sets Claude Haiku 5.5. The evaluation numbers above are for the models named in them.
• On Render's free plan the event log lives in /tmp, so the quality page resets on redeploy.
• The repository holds the sample games, the four lessons and every evaluation dataset, so each number here can be rerun.

Sources

1. Campitelli and Gobet, Deliberate practice: necessary but not sufficient ↗, Current Directions in Psychological Science, 2011.
2. Cepeda, Pashler, Vul, Wixted and Rohrer, Distributed practice in verbal recall tasks ↗, Psychological Bulletin, 2006.
3. Lichess accuracy and Win% ↗, and the 0.10 / 0.20 / 0.30 thresholds in the Lichess source ↗.
4. PydanticAI ↗, the agent framework, and Logfire ↗ for tracing.