question

active

question:how-does-active-inference-compare-to-reinforcement-learning-in-environments-with-no-rewards-or-uninformative-prior-preferences

How does active inference compare to reinforcement learning in environments with no rewards or uninformative prior preferences?

Core question addressed by the simulations when rewards are removed.

Source paper

extracted_from

Active inference: demystified and compared

(2021) · Noor Sajid · Philip J. Ball · Thomas Parr · Karl J. Friston

Neighborhood — ranked by edge-count

Findings (2)

finding

In the absence of any reward signal, Q-learning (epsilon=0.1) learns a deterministic circular policy with score 0.00 and does not explore purposefully.
answered_by
Table 2 first row; reward shaping section.
In the absence of prior preferences, Active Inference null model and Bayesian RL maintain exploration with average scores of 44.00 and 39.94 respectively, whereas Q-learning does not explore.
answered_by
Table 2 first row; reward shaping section.

Claims (1)

claim

Active inference agents can carry out epistemic exploration and account for uncertainty about their environment in a Bayes-optimal fashion.
gates
Abstract and §1, summarizing a key property.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

In active inference, reward can simply be treated as another observation we have a preference over, rather than a special signal.claim0.853
Abstract; central distinction.
Active inference agents can learn their own reward function (prior preferences) by interacting with the environment, bypassing the need for an explicit reward signal.claim0.841
Abstract and §3, preference learning section.
Can active inference agents learn their own prior preferences without explicit reward signals?question0.833
Question answered by the preference learning experiments.
There is an implicit behavioral equivalence between Bayesian model-based reinforcement learning and active inference when prior preferences are treated as a reward function.claim0.825
§3, reward shaping conclusion.
Reinforcement learning can be regarded as a limiting or special case of model-based approaches in general — or active inference in particular — when epistemic value is removed.claim0.823
§3 Discussion.
Active inference agents engage in information-seeking behavior in reward-free FrozenLake environments, contrasting with Q-learning but similar to Bayesian RL.finding0.822
Empirical demonstration on FrozenLake; shows epistemic value drives exploration absent reward signal.
The natural curiosity emerging in active inference contrasts with handcrafted exploration in reinforcement learning such as epsilon-greedy or ad hoc novelty bonuses.claim0.819
§2, comparing exploration mechanisms.
Active inference agents can learn epistemic habits in the absence of extrinsic rewards through ambiguity minimization.claim0.806
§2, summarizing information-seeking behavior.