TrOCR-mT5 hybrid fails Hindi OCR tasks

Swapping decoders isn't just about matching hidden sizes; it's about latent space alignment that rarely works out of the box without a dedicated curriculum.

• –Cross-attention weights are initialized randomly during the swap, requiring a "warm-up" phase where the encoder is frozen to prevent gradient corruption.
• –mT5's massive 250k token vocabulary introduces significant sparsity that can drown out visual signals in small-sample training environments.
• –Character repetition is a classic symptom of a decoder that has lost its visual grounding and is falling back on its language model priors.
• –Utilizing standardized wrappers like Hugging Face's VisionEncoderDecoderModel is critical for managing the complex interplay between disparate encoder-decoder architectures.