Grafy News
Try Grafy free

Meta and NYU Show Encoder Pretraining Unnecessary for Multimodal Models

A joint study from Meta FAIR and NYU trained over 200 multimodal LLMs from scratch.

Illustration for the news story: Meta and NYU Show Encoder Pretraining Unnecessary for Multimodal ModelsSource: Reddit ML

Meta FAIR and NYU released a systematic investigation into the role of vision encoders in multimodal large language models (MLLMs). By training more than 200 models from the ground up on an identical dataset, the researchers isolated three architectural choices that are usually bundled together: (1) whether the vision encoder receives a separate pretraining phase, (2) whether the model contains parameters that are accessed only by vision tokens, and (3) where the encoder sits in the processing pipeline relative to the language model. The findings overturn a common assumption that heavy encoder pretraining is a prerequisite for strong multimodal performance, while confirming that dedicated vision‑only parameters are still required.

What was the experimental setup?

The team built a large experimental grid in which every model, including the language backbone, was trained from scratch on the same multimodal corpus. This contrasts with most published MLLMs, which differ in data volume, compute budget, and often retain a pretrained vision encoder from a separate vision‑only model. By keeping the training data and compute constant across the 200+ runs, the study could attribute performance differences directly to the three encoder‑related choices. The total lifetime compute for the entire suite was reported as a substantial figure, emphasizing the scale of the effort.

Which encoder designs were compared?

Three binary decisions define the encoder configuration:

  1. Separate pretraining – the encoder is first trained on a vision‑only task before being inserted into the multimodal model.
  2. Vision‑only parameters – the model allocates weights that are never touched by text tokens.
  3. Placement before the LLM – the encoder processes visual inputs before they are handed to the language model, the conventional design. By toggling each decision independently, the researchers produced four families of models: fully pretrained encoders, encoders trained jointly with the LLM, models without any vision‑only weights, and variants that place the encoder after the language component.

What did the results reveal about encoder pretraining?

Across the board, models that omitted a separate vision‑encoder pretraining stage matched or exceeded the performance of their pretrained counterparts on standard multimodal benchmarks. This suggests that, given sufficient joint training data and compute, the encoder can learn useful visual representations from scratch. The conclusion aligns with recent anecdotal evidence from models such as Gemma 4 12B, which dropped the encoder entirely, and Kimi K3, which built its encoder from the ground up.

Why are vision‑only parameters still needed?

When the researchers removed the subset of weights that are accessed exclusively by vision tokens, performance dropped sharply. The loss was consistent across tasks involving image captioning, visual question answering, and cross‑modal retrieval. The result indicates that while the encoder itself can be learned jointly, the model still benefits from a dedicated “vision‑specific” capacity that does not interfere with the language pathway. Inkling 2 Small, which re‑introduced an encoder after earlier attempts to omit it, exemplifies this trade‑off.

How does this study compare to prior claims?

Previous MLLM releases have offered conflicting evidence about the necessity of vision encoders. Some, like Gemma 4 12B, argued that the encoder could be eliminated without harming results, while others, such as Inkling 2 Small, re‑added it to recover lost accuracy. The Meta‑NYU work clarifies the picture by controlling for data and compute, showing that the encoder’s pretraining is not the decisive factor; rather, the presence of vision‑only parameters and the encoder’s position in the architecture drive the observed differences.

Practical takeaways for developers and researchers

  • Skip costly encoder pretraining – When building a new multimodal system, allocating compute to joint training may be more efficient than maintaining a separate vision‑only pretraining pipeline.
  • Reserve vision‑only capacity – Ensure that the model architecture includes parameters that are exclusive to visual tokens, even if the encoder itself is learned jointly.
  • Experiment with placement – While the study kept the encoder before the language model for most runs, the systematic grid suggests that alternative placements could be explored without sacrificing performance, provided vision‑only weights are retained.
  • Use Grafy’s tools to replicate or extend – Researchers can upload the paper to Grafy Reader to ask detailed questions, or build a branching experiment log with Grafy Graph to track variations in encoder design across their own model runs.

How to explore the findings with Grafy

The study’s methodology maps naturally onto Grafy’s workflow. Load the PDF of the paper into Grafy Reader and pose queries like “What impact did removing vision‑only parameters have on VQA accuracy?” to extract specific numbers. Then, use Grafy Graph to construct a visual map of the four encoder configurations, attaching performance metrics as nodes. If you want to prototype a multimodal model without encoder pretraining, you can experiment with Grafy’s Chat interface, selecting a vision‑capable model from the catalog and prompting it to generate synthetic training data for a joint training run. While Grafy does not host full‑scale model training, its suite of tools helps you design, document, and analyze the experiments described in the paper.


Source: Reddit MachineLearning post linking to the paper (https://www.alphaxiv.org/abs/2611.17334) and accompanying blog/FAQ (https://unbundling.github.io).


Source: Reddit ML — [Meta FAIR + NYU] Unbundling Multimodal Vision Encoding: training 200+ MLLMs to find encoder pretraining is optional but vision-only parameters are not [R]