Abstract
Large Language Models (LLMs) have shown impressive performance across a wide range of tasks, yet the mechanisms behind their response behavior remain poorly understood. Drawing on insights from cognitive science and the computational framework of model-free and model-based reinforcement learning (RL), we examined conditions under which GPT-3.5 and four open-weight LLMs can exhibit behavioral signatures of model-based RL in the two-step task, a well-established paradigm for studying human learning and decision-making. Across nine simulation studies, we varied the task complexity and prompt design to examine conditions under which previously reported model-based RL signatures could be replicated. We found that GPT-3.5 showed model-based RL signatures in simplified task settings, but that these effects disappeared under de-simplified task versions. Crucially, model-based reasoning signatures could be re-established when incorporating an inner speech-analog prompt, requiring the LLM agent to reflect on previous trials and anticipate consequences of future actions. For the open-weight LLMs tested, we observed diminished signatures of model-based reasoning under both de-simplified and facilitative task conditions. Our fine-grained analyses of behavioral learning dynamics highlight the role of structured reflective and anticipatory linguistic processes in enabling complex response patterns in LLMs and reveal both similarities to and differences from human decision-making patterns.