Sutton, Richard S. and Szepesvári, Csaba and Geramifard, Alborz and Bowling, Michael H. (2008) Dyna-style planning with linear function approximation and prioritized sweeping. In: UAI 2008. Proceedings of the 24th conference in uncertainty in artificial intelligence. Helsinki, 2008..

Text linearDyna.pdf - Published Version Restricted to Registered users only Download (279kB)

Abstract

We consider the problem of efficiently learning optimal control policies and value functions over large state spaces in an online setting in which estimates must be available after each interaction with the world. This paper develops an explicitly model-based approach extending the Dyna architecture to linear function approximation. Dyna-style planning proceeds by generating imaginary experience from the world model and then applying model-free reinforcement learning algorithms to the imagined state transitions. Our main results are to prove that linear Dyna-style planning converges to a unique solution independent of the generating distribution, under natural conditions. In the policy evaluation setting, we prove that the limit point is the least-squares (LSTD) solution. An implication of our results is that prioritized-sweeping can be soundly extended to the linear approximation case, backing up to preceding features rather than to preceding states. We introduce two versions of prioritized sweeping with linear Dyna and briefly illustrate their performance empirically on the Mountain Car and Boyan Chain problems.

Item Type:	Conference or Workshop Item (Paper)
Subjects:	Q Science > QA Mathematics and Computer Science > QA75 Electronic computers. Computer science / számítástechnika, számítógéptudomány
Depositing User:	Eszter Nagy
Date Deposited:	11 Dec 2012 15:32
Last Modified:	11 Dec 2012 15:32
URI:	https://eprints.sztaki.hu/id/eprint/5601

Update Item