Constrained Reinforcement Learning for the Dynamic Inventory Routing Problem under Stochastic Supply and Demand
Green hydrogen has multiple use cases and is produced from renewable energy, such as solar or wind energy. It can be stored in large quantities, decoupling renewable energy generation from its use, and is therefore considered essential for achieving a climate-neutral economy. The intermittency of renewable energy generation and the stochastic nature of demand are, however, challenging factors for the dynamic planning of hydrogen storage and transportation. This holds particularly in the early-adoption phase when hydrogen distribution occurs through vehicle-based networks. We therefore address the Dynamic Inventory Routing Problem (DIRP) under stochastic supply and demand with direct deliveries for the vehicle-based distribution of hydrogen. To solve this problem, we propose a Constrained Reinforcement Learning (CRL) framework that integrates constraints into the learning process and incorporates parameterized post-decision state value predictions. Additionally, we introduce Lookahead-based CRL (LCRL), which improves decision-making over a multi-period horizon to enhance short-term planning while maintaining the value predictions. Our computational experiments demonstrate the efficacy of CRL and LCRL across diverse instances. Our learning methods provide near-optimal solutions on small scale instances that are solved via value iteration. Furthermore, both methods outperform typical deep learning approaches such as Proximal Policy Optimization, as well as classical inventory heuristics, such as (s,S)-policy-based and Power-of-Two-based heuristics. Furthermore, LCRL achieves a 10% improvement over CRL on average, albeit with higher computational requirements. Analyses of optimal replenishment policies reveal that accounting for stochastic supply and demand influences these policies, showing the importance of our addition to the DIRP.