Hi everyone! I am excited to share a new Pokémon bot named Jaxcalibur. It only plays Gen 9 Random Battles, but it plays them extremely well. Over the past three weeks, it has spent time at #1 on the ladder and reached a peak Elo of 2557 and peak GXE of 95.3. As far as I know, it is the first AI to reach the level of top human players at Pokémon battling, albeit in a narrow format. For comparison, the strongest previous bot I'm aware of, FoulPlay, peaked around 2341 Elo and 88 GXE, which is roughly top-50 rather than top-1.
That said, I can't say Jaxcalibur is the world's best random battler because one human outperformed it over this time period! MichaelderBeste2 held #1 on the ladder about 50% of the time compared to Jaxcalibur's 20%, and reached a slightly higher GXE of 95.5. I am really impressed that anyone can play better than a bot trained over millions of games and simulating thousands of possibilities each turn; that is an incredible feat!
How it works
Jaxcalibur uses a neural net trained with self-play reinforcement learning combined with pUCT search, similar to AlphaGo Zero. It is built on poke-env and Jax (which is where the name comes from). There is much more on how it works in the technical write-up here.
Some observations on how it plays: it is fairly conservative, preferring safe lines over hard reads. It is especially good at endgames because it can calculate several turns ahead. I think its clearest weaknesses are an imperfect understanding of PP stalling and a tendency to become predictable in some situations.
Current status
As of August 28, Jaxcalibur has stopped searching for ladder games and will let its rating decay. However, it will still accept randbats challenges if you'd like to play against it. It can only play one battle at a time, so it might not accept your challenge right away. Jaxcalibur currently plays random battles only, but I think it could be extended to other singles formats and, with more work, to doubles battles. It can't build teams itself, but you could use self-play with different teams to evaluate which ones are strongest
The open-sourcing question
When I started this project, I was intending to fully open-source Jaxcalibur, but it's strong enough I'm now a bit worried about how this could affect the community.
The case for releasing it: for me, the biggest benefit is that others could learn from it and build on it. Strong bots could also make single-player Pokémon more challenging, serve as practice partners, and be powerful tools for team building or balancing formats.
The case against: cheating is the big one. Someone could use Jaxcalibur to climb the ladder, obtain suspect reqs, or potentially even win tournaments in a way that's hard to detect. This feels especially relevant given the recent discussion about bots and suspect reqs. I also wonder whether easily accessible information about which teams, leads, and openings perform best would make team building less interesting.
One possible middle ground would be to release the code but not the model weights. However, for now I am holding off releasing either. I'd love to hear what people here think: should Jaxcalibur be open-sourced? If so, what would be a responsible way to release it? Strong, widely available Pokémon bots are probably coming sooner or later. This may be a useful chance to establish some norms around how they should be released and used.
Links
I also posted this on Smogon here, where there’s already a good discussion about the open-sourcing question.