DocOCR-Eval Selects Best OCR Tools Without Ground Truth

Document parsing is a foundational step for understanding unstructured scanned documents, converting them into structured data. With a multitude of Optical Character Recognition (OCR) engines and multimodal large language models (MLLMs) available, selecting the most suitable solution for a specific document collection can be challenging, especially when manual annotations (ground truth) are scarce or non-existent. This research introduces DocOCR-Eval, an innovative framework designed to evaluate and select OCR tools without requiring ground-truth labels. The framework employs a three-staged approach involving correction and ranking strategies to approximate the performance ordering typically achieved with annotations. Experiments show that by aggregating results from multiple MLLMs, DocOCR-Eval progressively improves its alignment with ground-truth-based rankings, offering a reliable and practical method for deploying document parsing systems across diverse, label-limited real-world document collections.