World Feedback Outperforms Model Uncertainty for RL Safety.

This paper challenges the common practice in safe model-based reinforcement learning (MBRL) of using internal model uncertainty as a proxy for risk. Instead, it advocates for deriving learning signals directly from "world feedback," aligning with the broader RLxF (Reinforcement Learning from eXternal Feedback) program. The authors distill this principle into three concrete design guidelines for developing safer MBRL systems. Empirical studies across various world-model architectures demonstrate a critical flaw in relying on model uncertainty. While model performance varied, MPC planning remained statistically equivalent. Crucially, dynamics-based uncertainty penalties were found to increase collision rates, suggesting an anti-correlation with actual safety in the tested regime. In contrast, replacing these internal proxies with direct world-feedback signals—such as sensor-derived margins, time-to-collision, or outcome-supervised feedback models—dramatically reduced collision rates without requiring retraining of the world model or planner. The underlying mechanism is structural: model uncertainty operates in state-prediction space, whereas task risk is defined by constraint boundaries, showing a weak empirical correlation. This research concludes with three RLxF principles applicable to both model-based control and LLM alignment: ground risk in world outcomes, validate proxies rigorously, and use outcome-trained feedback models when direct world signals are unavailable.