Smaller world models for reinforcement learning

Loading...
Thumbnail Image

Date

2023-08-10

Journal Title

Journal ISSN

Volume Title

Publisher

Alternative Title(s)

Abstract

Model-based reinforcement learning algorithms try to learn an agent by training a model that simulates the environment. However, the size of such models tends to be quite large which could be a burden as well. In this paper, we address the question, how we could design a model with fewer parameters than previous model-based approaches while achieving the same performance in the 100 K-interactions regime. For this purpose, we create a world model that combines a vector quantized-variational autoencoder to encode observations and a convolutional long short-term memory to model the dynamics. This is connected to a model-free proximal policy optimization agent to train purely on simulated experience from this world model. Detailed experiments on the Atari environments show that it is possible to reach comparable performance to the SimPLe method with a significantly smaller world model. A series of ablation studies justify our design choices and give additional insights.

Description

Table of contents

Keywords

Model-based reinforcement learning, World models, Discrete latent space, VQ-VAE, Atari

Subjects based on RSWK

Bestärkendes Lernen (Künstliche Intelligenz), Weltmodell, Diskretes System, Atari

Citation