Abstract
Continual low-rank adaptation enables large language models (LLMs) to absorb sequentially arriving tasks with limited trainable parameters, but repeated updates can cause catastrophic forgetting and drift in autoregressive generation behavior. We formulate this problem as sequence-consistent continual LoRA and align the current model with its previous-stage snapshot at the token-distribution, hidden-state, and short-horizon rollout levels. A fuzzy–Bayesian reliability model assigns uncertainty-aware weights to historical anchor prompts, while a stable–plastic decomposition separates retentionoriented and adaptation-oriented low-rank components. We evaluate the method on a four-stage stream comprising SQuAD v2, SAMSum, MBPP, and GSM8K. Under the explicitly defined raw task-score averaging protocol, the final model obtains an average performance of 48.4 and average old-task retention of 48.8, while achieving 7.1 forgetting and 75.4 generation consistency. Component-wise ablations show complementary contributions from the three consistency losses, reliability weighting, and stable–plastic decomposition. These results support behavior-level regularization as a useful complement to parameter-space control in continual parameter-efficient adaptation.