Central Path Proximal Policy Optimization

• Authors: Nikola Milosevic, Johannes Müller, Nico Scherf
• First public date: 2025-05-31
• arXiv: 2506.00700
• Preprint year: 2025
• Status: Published
• Publication type: Workshop paper
• Publication year: 2025
• Proceedings: Exploration in AI Today Workshop at ICML 2025
• Publication details: Author’s publication list

Abstract

In constrained Markov decision processes, enforcing constraints during training is often thought of as decreasing the final return. Recently, it was shown that constraints can be incorporated directly into the policy geometry, yielding an optimization trajectory close to the central path of a barrier method, which does not compromise final return. Building on this idea, we introduce Central Path Proximal Policy Optimization (C3PO), a simple modification of the PPO loss that produces policy iterates, that stay close to the central path of the constrained optimization problem. Compared to existing on-policy methods, C3PO delivers improved performance with tighter constraint enforcement, suggesting that central path-guided updates offer a promising direction for constrained policy optimization.

Associated SAiS members

• Johannes Müller

Research areas

• Numerics
• Machine Learning

BibTeX

@article{arxiv250600700,
  title = {Central Path Proximal Policy Optimization},
  author = {Nikola Milosevic and Johannes Müller and Nico Scherf},
  year = {2025},
  eprint = {2506.00700},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  journal = {The Exploration in AI Today Workshop at ICML 2025}
}