Hierarchical Reinforcement Learning with Symbolic Planning and Incremental Knowledge Boosts Sample Efficiency

Summary

A neurosymbolic hierarchical reinforcement learning framework where a symbolic planner (e.g., D*) guides low-level, goal-conditioned neural policies by operating on a progressively updated world model. This structure improves sample efficiency in sparse-reward navigation tasks.

Why it matters

Sample efficiency is one of the most persistent bottlenecks in reinforcement learning, particularly for long-horizon and sparse-reward problems common in robotics. This work bridges the practical gap between neurosymbolic planning and online learning by allowing the symbolic knowledge base to be updated during discovery rather than being fixed at the design stage. It is directly relevant to embodied AI agents that must reason in partially known and continuously changing environments.

Highlights

• The symbolic high-level planner (D*) reasons over a progressively updated world representation, unlike hierarchical reinforcement learning (HRL) baselines that rely on static knowledge.
• Low-level goal-conditioned neural modules learn action primitives through reward shaping rather than end-to-end rewards.
• When prior world knowledge is available, Belief World Tree Search is introduced for optimal symbolic planning.

Critical take

The evaluation is limited to relatively suitable and structured navigation tasks for symbolic planning. Generalization to high-dimensional, continuous, or adversarial environments has not been tested. Additionally, the hand-designed symbolic InK representation leaves open questions regarding scalability and robustness to noisy and ambiguous sensor inputs in real-world deployment.