It Started With an Email
One morning we received a detailed email from a player who had been testing our bots. Their findings were eye-opening: bots labeled as club players were actually playing at Candidate Master level, far above what their difficulty tier suggested. A bot in the Apprentice tier should not be crushing experienced club players. On platforms like Chess.com, players use bots as personal milestones. If our difficulty levels are wrong, players misjudge their own level, and beginners get crushed by opponents they believe should be within reach. We decided to tackle this on two fronts: first, recalibrate every bot so the difficulty progression is smooth and correctly ordered, and second, remove Elo ratings from the app interface entirely. Showing a number like 1500 invites direct comparison with other platforms, but Xiangqi rating scales vary wildly between sites. Instead, we organize bots into clear difficulty tiers so players can focus on finding the right challenge level without worrying about a number that might not mean what they think it means.
Why Pikafish Makes Calibration Hard
Xiangqi One uses Pikafish, the strongest open-source Xiangqi engine, as the brain behind every bot. Pikafish is incredibly powerful, but that power is a double-edged sword. Even at very low search settings, Pikafish plays far above beginner level. Its strength grows exponentially with search depth, not linearly. Going from depth 1 to depth 3 does not triple the strength, it can multiply it by ten. This means you cannot simply map engine depth to a linear Elo scale and call it a day. Each bot on Xiangqi One is not just a raw engine. It is a personality built from several parameters: search depth, how many candidate moves to consider (MultiPV), how many intentional mistakes the bot makes, variance in error rate, and whether the bot picks the best candidate or chooses more randomly. Getting all these knobs right for 22 different bots, spanning from absolute beginner to near-grandmaster level, is the core of the calibration challenge.
Building the Tournament Infrastructure
To measure actual bot strength, we needed bots to play each other at scale, not dozens of games, but tens of thousands. We built a tournament system that schedules matchups between every pair of nearby bots, alternates colors so both sides play Red and Black equally, and stores every game result in a database. Each tournament round generates around 400 games per bot pair, which gives us statistically meaningful results. After all games are played, we export the results as PGN files and feed them into Ordo, a Bradley-Terry rating system that computes Elo ratings from game outcomes. Ordo is the same tool used by the computer chess community to benchmark engines. It outputs a rating for each bot with error margins, giving us a precise picture of true relative strength.
The First Results: How Far Off We Were
The first tournament results were a wake-up call. Our initial bot profiles, configured with ratings ranging from 280 to 2500, were wildly miscalibrated. The chart below shows the initial state: each dot is a bot, the X axis is the configured rating in our code, and the Y axis is the measured Elo from the tournament.
Important note: the measured Elo is relative to the bot pool (Ordo anchors the average at 1000), so the absolute numbers do not need to match the configured ratings. What matters is that the bots are correctly ordered and evenly spaced.
The results revealed serious problems. The top bots (Elena, Jun-so, Katya) were clustered together at nearly identical strength despite being configured hundreds of points apart. Several bots were out of order entirely, with supposedly weaker bots beating supposedly stronger ones. The progression chart makes these issues even more visible. Instead of a smooth staircase of increasing difficulty, there were massive clumps where several bots played at nearly identical strength, and large gaps where difficulty jumped abruptly.
A player moving from Erik (680) to Ông Minh (760) would barely notice a difference, but jumping from Dewi to Xinping was like hitting a wall. This is not the experience we wanted.
The Tuning Process: Four Tournament Iterations
Fixing the calibration required multiple rounds of adjustment. After each tournament, we analyzed the results, identified the worst offenders, tweaked the engine profiles, and ran another tournament. In the first round, we adjusted the configured ratings to better match measured strength and identified the bots that needed the biggest profile changes. One recurring pattern we discovered was that defensive-style bots consistently overperformed their profiles. A bot with a defensive playstyle would measure hundreds of Elo points above a similarly-configured aggressive bot. We had to significantly nerf several defensive bots, particularly Nour and Vo Nguyen Giap. By tournament four, we had every bot respecting the intended hierarchy, with reasonably even spacing between difficulty levels. The biggest individual changes included boosting Sun Tzu by over 200 Elo points (giving him a deeper search and more accurate play), nerfing Jun-so by 270 points (he was far too strong for his tier), and completely reworking the bottom four bots to create a gentler introduction for new players. Those retuned tiers are the ones you meet today when you play against an AI bot.
Before and After: Measuring the Improvement
To validate that our changes actually worked, we ran a special comparison tournament: all 22 bots in their original configuration (v1) played against all 22 bots in their tuned configuration (v2) on the same Elo scale. This gave us a direct, apples-to-apples measurement of how each bot changed.
The arrows show the direction and magnitude of change for each bot. Blue arrows pointing right mean the bot got stronger; red arrows pointing left mean we successfully weakened it. The biggest nerfs went to bots that were dramatically overperforming: Vo Nguyen Giap dropped 374 Elo points, Jun-so dropped 270, and Jules Cesar dropped 234. On the other side, Sun Tzu gained 202 points to take his rightful place among the strongest historical bots.
The combined view below links each bot across both versions, showing exactly how the calibration shifted.
The final calibration chart confirms that after tuning, the bots are correctly ordered with no inversions, and the spacing between difficulty levels is much more even.
What Comes Next: Turning Games Into Puzzles
The tens of thousands of games generated by our tournament system are not just useful for calibration. Every game is a goldmine of tactical and positional patterns. Our next step is to mine these games for puzzle material. The tournament runner already stores full position-by-position engine evaluations for every game. By looking for positions where the evaluation swings sharply, one player had a winning move but the opponent missed it, we can automatically extract puzzle candidates. This approach gives us an almost unlimited supply of fresh, rated puzzles drawn from games with realistic positions, not composed endgame studies. Stay tuned for our next article. If you would rather know how to pick a level and use bots to improve, our guide to playing Chinese chess against the computer covers the practical side.
