Abstract
Large Language Models (LLMs) have shown impressive performance across a wide range of tasks, yetthe mechanisms behind theirresponse behaviorremain poorlyunderstood. Drawing on insights fromcognitive science and the computational framework of model-free and model-based reinforcementlearning (RL), we examined conditions under which GPT-3.5 and four open-weight LLMs can exhibitbehavioral signatures of model-based RL in the two-step task, a well-established paradigm forstudying human learning and decision-making. Across nine simulation studies, we varied the taskcomplexity and prompt design to examine conditions under which previously reported model-basedRL signatures could be replicated. We found that GPT-3.5 showed model-based RL signatures insimplifiedtasksettings,butthattheseeffectsdisappearedunderde-simplifiedtaskversions.Crucially,model-based reasoning signatures could be re-established when incorporating an inner speech-analog prompt, requiring the LLM agent to reflect on previous trials and anticipate consequences offuture actions. For the open-weight LLMs tested, diminished signatures of model-based reasoningevolved under both de-simplified and facilitative task conditions. Our fine-grained analyses ofbehavioral learning dynamics highlight the role of structured reflective and anticipatory linguisticprocesses in enabling complex response patterns in LLMs and reveal both similarities and differencesto human decision-making patterns.