Still working on trying to reproduce in a minimalistic example (no luck so far). However, one thing I did notice and fix, which seemed to make things a bit better, was the placement of the arena walls, which I discovered after some visual inspection and placing with arena wall placement in the input...