A new study demonstrates that visual pretraining on raw visual documents bypasses OCR/parsing limitations and consistently outperforms traditional text-only pretraining for building language intelligence.

Text-only pretraining is reaching its semantic limits; visual pretraining on raw documents is a cleaner, more powerful way to build intelligent models that understand the layout and layout-dependent context of documents.

• –Bypassing the Parser Bottleneck: Training on raw document images captures layout, tables, and figures that OCR and text extraction strip away.
• –Consistent Improvements: The method consistently outperforms text-only pretraining across multiple backbones on standard benchmarks.
• –Simplified Data Pipelines: Bypassing OCR reduces pipeline complexity and eliminates errors introduced during HTML/text parsing.