Abstract
Abstract
Large language models (LLMs) have demonstrated strong capabilities in document key–value pair(KVP) extraction, yet controlled evaluations of their robustness to optical character recognition (OCR)output remain limited. This leaves an important gap in understanding their reliability within real-world OCR-to-LLM pipelines. Unlike end-to-end Vision-Language Models (VLMs) that conflate visualperception with semantic extraction, modular pipelines allow errors to be explicitly isolated andaudited. To bridge this evaluation gap, we introduce a controlled, large-scale benchmarking frameworkdesigned to distinguish downstream LLM extraction behavior from upstream OCR degradation. Thisbenchmark evaluates 136 experimental configurations and 17,688 deterministic document-level inferenceinstances across five instruction-tuned open-weight LLMs (2B–8B parameters), three complementarydatasets, and four text-quality conditions. This extensive matrix pairs a full zero-shot comparisonand targeted one- to three-shot experiments with a confirmatory sensitivity analysis comprising 40configurations across 20 frozen demonstration sets. By separating Key Recall (annotated-field recovery)from Exact Match and Value F1 (exact and partial value recovery, respectively), we reveal whetherOCR degradation affects field identification and value reproduction differently across models. Crucially,our findings challenge three practical evaluation assumptions: (1) that clean-text performance is areliable proxy for real-world robustness, (2) that model rankings remain consistent across annotation-derived Gold and OCR-derived text conditions, and (3) that providing few-shot demonstrationsmonotonically improves extraction accuracy. By exposing vulnerabilities such as model-rankingreversals and unstable few-shot gains, this benchmark provides a rigorous and reproducible frameworkfor evaluating LLM reliability under noisy document conditions. To facilitate reproducible research,our entire benchmarking framework, dataset splits, and evaluation scripts are open-sourced.