Expanding non-player character (NPC) navigation behavior in video games has the potential to induce novel player experiences. Current industry standards represent traversable world geometry by utilizing a Navigation Mesh (NavMesh); however NavMesh complexity scales poorly with additional navigation abilities (e.g. jumping, wall-running, jet-packs, etc.) and increasing world scale. Deep Reinforcement Learning (DRL) allows for an NPC agent to learn how to navigate environmental obstacles with any navigation ability without NavMesh dependence. Despite the promise of DRL navigation, adoption in industry remains low due to the required expert knowledge in agent design and the poor training efficiency of DRL algorithms. In this work, we utilize the off-policy Soft-Actor Critic (SAC) DRL algorithm to investigate the importance of different local observation types and agent scalar information to agent performance across three topologically distinct environments. We implement a truncated n-step returns method for minibatch sampling which improves early training efficiency by up to 75% by reducing inaccurate off-policy bias. We empirically evaluate environment partial observability with observation stacking where we find that 4–8 observation stacks renders the environments sufficiently Markovian.

https://link.springer.com/chapter/10.1007/978-981-99-8391-9_23

Part of the Lecture Notes in Computer Science book series (LNAI,volume 14472)