MHPO: Modulated Hazard-Aware Policy Optimization
for Stable Reinforcement Learning

Smooth Ratio Control with Bounded Gradient Multipliers and Asymmetric Hazard Penalties

Hongjun Wang1 Wei Liu2 Weibo Gu2 Xing Sun2 Kai Han1†
1 HKU The University of Hong Kong 2 Youtu Tencent Youtu Lab

† Corresponding author

MHPO Overview

Overview of MHPO. (1) Token-level importance ratios are high-variance and heavy-tailed. (2) The Log-Fidelity Modulator (LFM) produces smooth, bounded log-space modulation. (3) The Decoupled Hazard Penalty (DHP) adds asymmetric hazard-aware regulation. (4) The resulting gradient multiplier is smooth, bounded, and stable. (5) MHPO achieves the largest gains across all benchmarks. (6) Per-sample gradient shifts: MHPO concentrates updates in a stable band.

Abstract

Regulating the importance ratio is central to the training stability of Group Relative Policy Optimization (GRPO) frameworks. Prevailing ratio-control methods, such as hard clipping or direction-agnostic masking, produce non-differentiable boundaries and vanishing gradient regions, and fail to preserve gradient fidelity. These methods also lack a hazard-aware mechanism that adaptively suppresses extreme deviations, which leaves the optimization process vulnerable to abrupt policy shifts. To address these limitations, we propose Modulated Hazard-aware Policy Optimization (MHPO), a framework for stable reinforcement learning. MHPO introduces a Log-Fidelity Modulator (LFM) that maps unbounded importance ratios into a bounded differentiable domain, thereby preventing high-variance outlier tokens from destabilizing the loss landscape while ensuring global gradient stability. MHPO further introduces a Decoupled Hazard Penalty (DHP) that integrates cumulative hazard functions from survival analysis to regulate positive and negative policy shifts separately. The resulting framework achieves fine-grained regulation of asymmetric policy shifts, mitigating mode collapse from over-expansion and preventing policy erosion from catastrophic contraction within a stabilized trust region. Evaluations on mathematical reasoning benchmarks demonstrate that MHPO achieves the strongest overall performance among existing methods.

Key Contributions

Method

The MHPO Objective

The MHPO objective replaces the clipped surrogate in standard GRPO with a smooth, bounded transformation:

$$\mathcal{L}_{\text{MHPO}}(\theta) = -\mathbb{E}\left[\frac{1}{K}\sum_{i=1}^{K}\frac{1}{T_i}\sum_{t=1}^{T_i} \exp\!\Big(\psi(r_t^i(\theta)) - \zeta(r_t^i(\theta))\Big)\,\hat{A}_t^i\right]$$

where $\psi(\cdot) = c\tanh(\log(\cdot)/c)$ is the Log-Fidelity Modulator and $\zeta(\cdot) = (s(\psi(\cdot))/\lambda_+)^{k_+} + (s(-\psi(\cdot))/\lambda_-)^{k_-}$ is the Decoupled Hazard Penalty.

Log-Fidelity Modulator (LFM)

The LFM projects the importance ratio into log-space and bounds its contribution via a smooth saturation function. It exhibits three key properties: (P1) High-fidelity local mapping near the on-policy anchor; (P2) Smooth gradient attenuation in the tails via $\text{sech}^2$; (P3) $C^\infty$ differentiability preventing Adam optimizer perturbations.

LFM Analysis

Figure 2: Characteristics of the Log-Fidelity Modulator. (a) The LFM operator with different values of $c$, showing reciprocal antisymmetry and smooth bounding. (b) The derivative demonstrating smooth gradient attenuation while preserving local fidelity.

Decoupled Hazard Penalty (DHP)

The DHP uses the Weibull cumulative hazard to shape the optimization landscape with direction-aware penalties. Positive shifts are regulated mildly to preserve exploration, while negative shifts receive stronger suppression to prevent irreversible policy erosion caused by branch-wise occupancy decay.

DHP Analysis

Figure 3: Analysis of the Decoupled Hazard Penalty. (a) The penalty $\zeta(\psi)$ with different $(\lambda, k)$ configurations, showing asymmetric suppression. (b) The survival weight $w = \exp(-\zeta) \in (0, 1]$ — the DHP can only attenuate, never amplify.

Training Algorithm

Algorithm 1: MHPO Training
Require: Policy $\pi_\theta$, dataset $\mathcal{D}$, LFM bound $c$, DHP params $(k_+, \lambda_+, k_-, \lambda_-)$
for each iteration do
Sample prompts $p$; generate $K$ responses per prompt using $\pi_\theta$
Compute group-normalized advantages $\hat{A}_t^i$
for each response $q^i$, each token $t$ do
$r_t^i(\theta) \gets \pi_\theta(q_t^i \mid p, q_{<t}^i) / \pi_{\theta_{\text{old}}}(q_t^i \mid p, q_{<t}^i)$ // importance ratio
$\psi_t \gets c \cdot \tanh(\log r_t^i(\theta) / c)$ // LFM
$\zeta_t \gets (s(\text{sg}(\psi_t)) / \lambda_+)^{k_+} + (s(-\text{sg}(\psi_t)) / \lambda_-)^{k_-}$ // DHP
end for
$\mathcal{L} \gets -\frac{1}{K}\sum_i \frac{1}{T_i}\sum_t \exp(\psi_t - \zeta_t) \cdot \hat{A}_t^i$; update $\theta$ via gradient descent
end for

Stability Certificate

Theorem (MHPO Multiplier-to-Moment Stability Certificate)

The gradient multiplier $\mathcal{M}(r)$ satisfies $0 \le \mathcal{M}(r) \le M_\psi(c)$ for all $r > 0$, where $M_\psi(c) = \frac{2}{1+\sqrt{1+c^2}} \exp\!\left(\frac{c^2}{1+\sqrt{1+c^2}}\right)$. Under any behavior distribution $\mu$, the normalized mini-batch gradient satisfies $\mathbb{E}_\mu[\|g(\theta)\|^2] \le \sigma_A^2 G^2 M_\psi(c)^2$ and $\Pr_\mu(\|g(\theta)\| \ge \tau) \le \sigma_A^2 G^2 M_\psi(c)^2 / \tau^2$.

Experimental Results

Qwen3-4B-Base (Dense Foundation Model)

Avg@32 (%) for the best checkpoint. All methods trained on DAPO-Math-17k-Processed with verifiable outcome-based rewards.

Method MATH500 HMMT25 AMC23 AIME25 AIME24 Avg.
Baseline 57.23.841.35.78.823.4
GRPO 64.811.181.330.840.445.7
GSPO 66.212.483.534.039.747.2
DAPO 64.014.975.829.234.243.6
SAPO 66.411.581.432.939.646.4
CISPO 69.011.085.333.543.748.5
MASPO 67.512.882.634.540.847.6
MHPO (Ours) 70.9 19.4 88.2 41.4 44.7 52.9

Qwen3-30B-A3B-Base (MoE Foundation Model)

Avg@32 (%) for the best checkpoint. Demonstrates scalability to large Mixture-of-Experts architectures.

Method MATH500 HMMT25 AMC23 AIME25 AIME24 Avg.
Baseline 76.51.643.69.014.229.0
GRPO 82.835.378.428.931.651.4
GSPO 84.136.980.629.832.452.8
DAPO 84.537.281.030.232.853.1
SAPO 83.234.676.830.729.450.9
CISPO 85.437.882.331.633.154.0
MHPO (Ours) 86.6 37.3 83.6 32.4 33.8 54.7

Training Stability Analysis

Per-sample effective gradient shifts: hard-clipped baselines produce many dead or saturated tokens, whereas MHPO concentrates updates in a stable band while preserving decayed tail signals.

Training Stability Comparison

Figure 4: Training stability comparison across methods. MHPO keeps updates in a stable band around the on-policy anchor while preserving decayed tail signals, consistent with the theoretical stability certificate.

Ablation Studies

Hyperparameter sensitivity sweeps on Qwen3-4B-Base (macro-average Avg@32, %).

LFM Bound $c$

$c=0.5$: 50.9%
$c=1.0$: 51.3%
$c=1.5$: 52.0%
$c=2.0$: 51.7%

Weibull Shape $k$

$k=1.0$: 52.0%
$k=1.5$: 52.3%
$k=2.5$: 52.6%
$k=3.0$: 52.9%

Weibull Scale $\lambda$

$\lambda=0.5$: 52.5%
$\lambda=0.8$: 52.9%
$\lambda=1.0$: 52.7%
$\lambda=2.0$: 51.9%

Key Findings

The LFM alone (hazard-free) is stable but remains below the full MHPO configuration, confirming the added value of the hazard penalty. Larger $k$ improves over near-linear damping (superlinear Weibull regime), and intermediate $\lambda$ best balances exploration and tail control. The default setting stays in a high-performing region across all sweeps.

BibTeX

@inproceedings{wang2026mhpo,
  title     = {MHPO: Modulated Hazard-Aware Policy Optimization 
               for Stable Reinforcement Learning},
  author    = {Wang, Hongjun and Liu, Wei and Gu, Weibo 
               and Sun, Xing and Han, Kai},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}