Grafy News
Try Grafy free

New VL-DE Framework Enables Variable-Length Multimodal Generation

Researchers have introduced VL-DE, a framework that allows diffusion models to generate data of arbitrary lengths and multiple modalities simultaneously.

Illustration for the news story: New VL-DE Framework Enables Variable-Length Multimodal GenerationSource: Grafy Research Lab

Researchers have developed a new generative modeling framework called Variable Length Diffuse Everything (VL-DE) that can handle data of any length and multiple different types of information at once. This approach addresses a major limitation in current diffusion and flow-based models, which typically require data to have a fixed dimension or length.

Why is variable-length generation difficult?

Most existing generative models assume that the input data follows a rigid structure, such as an image with a specific number of pixels or a fixed-length vector. However, many real-world applications involve data where the length is not known until the content is actually created. For example, in layout generation, a model must decide how many bounding boxes to place and what their specific coordinates and types are.

According to the paper by Kevin Rojas, Felix X.-F. Ye, and Molei Tao, current methods often try to solve this by "padding" data—adding empty filler to make everything a fixed size—or by using complex specialized objectives. These methods can be computationally inefficient and lack the flexibility needed for truly diverse datasets. Furthermore, while some models can handle different types of data (multimodal), they often force all those types to share the same length, which is not how real-world data behaves.

How does the VL-DE framework work?

Instead of using padding, VL-DE utilizes an insertion-based mechanism. The researchers built this framework by combining two existing concepts: Generator Matching and Diffuse Everything. To make the math work for variable lengths, they augmented the state space with an auxiliary variable that tracks the operations used to create a state. By marginalizing this variable, they created a way to train the model on a tractable objective.

Figure 1 shows an overview of how this works in a layout example. Each element, such as a coordinate, is assigned an "insertion time" and is denoised on its own individual clock. This allows all elements to reach a "clean" or finished state at the same time, even if they were added to the sequence at different stages of the generative process. This decoupled approach allows the model to handle continuous values, like coordinates, alongside discrete values, like object types, without needing them to be the same length.

What are the performance results?

To test the effectiveness of VL-DE, the authors conducted experiments across several domains, including layout generation on datasets like RICO and PubLayNet, molecule generation on QM9, and even generating arithmetic equations.

In the context of layout generation, the model was able to match the performance of "oracle-length" models—models that are artificially told exactly how many elements they need to generate—without actually needing that information beforehand. Figure 8 illustrates the results of unconditional layout generation, comparing VL-DE against other methods. The results suggest that the framework can produce high-quality, diverse layouts that respect the underlying data distribution.

Figure 8: Unconditional layout generation. Rows are methods, columns eight layouts of a seeded batch; colors are categories. LayoutFlow is given the number of elements, t…
Figure 8. Unconditional layout generation. Rows are methods, columns eight layouts of a seeded batch; colors are categories. LayoutFlow is given the number of elements, the other methods generate it. Figure 8 of “Variable Length Diffuse Everything: Arbitrary Length Generative Modeling on Arbitrary State Spaces” by Kevin Rojas, Felix X. -F. Ye, Molei Tao (arXiv.org), CC BY 4.0.

Can this be applied to other data types?

The versatility of VL-DE comes from its ability to operate on arbitrary state spaces. This means it is not limited to just layouts or text. The researchers demonstrated its utility in generating molecules and mathematical equations, where the number of symbols or atoms varies significantly. Because the model uses a non-autonomous operator splitting scheme for higher-order sampling, it can achieve high accuracy even when using larger step sizes during the generation process, making it more efficient for practical use.

How to explore this research with Grafy

While VL-DE is a specialized research framework for training new types of generative models, the implications for multimodal AI are significant. For researchers looking to dive deeper into the mathematical proofs or the specific implementation details of this paper, the Reader app is the best tool. You can upload the full PDF of the arXiv paper to Reader to ask specific questions about the "non-autonomous operator splitting" or the "insertion-based mechanism" used by the authors.

Frequently asked questions

Does VL-DE require padding to handle different data lengths?

No, VL-DE does not use padding. Instead, it handles variable lengths through an insertion-based mechanism. This allows the model to generate an arbitrary number of elements by assigning each one an insertion time and a unique denoising clock, which avoids the complexity and inefficiency of forcing data into a fixed-size format.

What makes VL-DE different from existing multimodal models?

Unlike many existing multimodal models that require all modalities to share a single fixed length, VL-DE allows for arbitrary lengths and different state spaces for each modality. It can jointly generate continuous coordinates and discrete types even if they do not follow the same structural constraints or sequence lengths.

How does the sampling process in VL-DE improve efficiency?

The model uses a novel non-autonomous operator splitting scheme for general Markov processes. This allows for higher-order sampling, which improves the quality of the generated samples per step. This is particularly useful in the large step size regime, allowing the model to produce accurate results more quickly during the generation phase.


Source: Grafy Research Lab — Variable Length Diffuse Everything: Arbitrary Length Generative Modeling on Arbitrary State Spaces