Multiagent reinforcement learning for discrete optimization of hole based photonic integrated circuits

This work explores Reinforcement Learning (RL) as a robust alternative to Direct Binary Search (DBS) and standard gradient-based topology optimization for the inverse design of hole-based silicon photonic devices. While algorithms like Bandit Proximal Policy Optimization (BPPO) and Bandit Actor-Critic (BAC) have been recently introduced to nanophotonics, their application to hole-based topologies remains unexplored. Addressing the combinatorial challenges of discrete optimization, we utilize BPPO and BAC to decompose the design space into hundreds of cooperative agents. We evaluate this framework through the design of a mode de-multiplexer on the Silicon-on-Insulator (SOI) platform. Additionally, we conduct an ablation study comparing implicit neural-network parameterization against explicit array parameterization to isolate the core contributions of the reinforcement learning update mechanisms. We find that BPPO consistently outperforms the strong DBS baseline without any additional hyperparameter tuning, whereas BAC cannot match its performance. Our experiments yield a device that achieves highly efficient modal routing with insertion losses of just 0.34 dB for the TE0 mode and 0.46 dB for the converted TE1 mode, demonstrating the efficacy of BPPO for complex, discrete nanophotonic inverse design.