A multimodal model is an LLM that can take in more than text: images, audio, sometimes video, all sharing the same token stream as words. The elegant part is that almost nothing new is invented. You reuse a language model, reuse a vision model, and train a small bridge between them. Here is how the pieces wire together, and why the frontier is now tearing the bridge out.

Core Architecture & Mental Model

A large multimodal model (LMM, also called a vision-language model or VLM when it handles images) extends an LLM so that images, audio, or video can share the same token stream as text. The mental model is a universal translator into the LLM’s token space: whatever the modality, convert it into vectors the language model can attend to alongside words, then let the same next-token machinery do the reasoning.

The workhorse design has three parts wired in series: a vision encoder that turns an image into embeddings, a projector that reshapes those embeddings into the LLM’s representation space, and the LLM decoder that consumes the combined stream and generates text. This is the LLaVA-style recipe, and it became mainstream precisely because it is cheap to build: reuse a pretrained vision encoder, reuse a pretrained LLM, and train mostly the small bridge between them.

The Mechanics (Under the Hood)

1. Encode. A Vision Transformer (ViT), or a contrastively-trained encoder such as SigLIP, splits the image into fixed patches (a common grid is many 14×14 or 16×16 pixel patches) and encodes each patch into a vector. An image becomes a sequence of visual tokens, exactly like text becomes a sequence of text tokens.
2. Project. A small projector, usually a two-layer MLP, maps each visual token into the LLM’s embedding space so the language model treats it like any other token. This adapter is the component most teams actually train.
3. Fuse. The visual tokens are concatenated with the text tokens and fed to the decoder. This is early fusion: vision and language interact from the first layers. An older approach, Flamingo-style gated cross-attention, injected vision through separate attention layers; it has largely been superseded by simpler concatenation.
4. Generate. The LLM autoregressively produces text conditioned on both the image tokens and the prompt.

Training runs in two phases. A feature-alignment phase trains the projector (and often keeps the vision encoder and most of the LLM frozen) so visual tokens land in the right region of the language space. A visual instruction-tuning phase then trains on image-question-answer data to teach the model to follow multimodal instructions. Freezing the encoder and training only the projector plus the top few LLM layers can cut training compute dramatically while retaining most of the quality.

The 2026 frontier is moving past the bolt-on bridge toward native multimodality, where image, audio, and video tokens enter a single early-fused stream and the vision representation is a first-class citizen of the transformer rather than a translated guest. Native designs reduce visual hallucination and capture spatial relationships better, at the cost of far more expensive end-to-end training. The newest branch, vision-language-action (VLA) models, adds motor control as an output modality for robotics.

Key Technical Trade-offs

Real-World Implementation

In practice you send an image inline (base64 or URL) alongside text in the same message. The critical variable you control is how many visual tokens the image consumes, which you tune with resolution or a detail setting.

import base64
from openai import OpenAI

client = OpenAI(base_url="https://api.your-provider.com/v1", api_key=KEY)

with open("firewall_topology.png", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()

resp = client.chat.completions.create(
    model="vlm-2026",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "List every subnet visible in this diagram."},
            {"type": "image_url", "image_url": {
                "url": f"data:image/png;base64,{img_b64}",
                "detail": "high",   # high = more visual tokens, more cost, more precision
            }},
        ],
    }],
)
print(resp.choices[0].message.content)

Rule of thumb: a detail: high image can cost more tokens than several pages of text. Downscale before sending unless you truly need to read fine print, and never assume the model perceives the image at full native resolution.

Common Pitfalls & Antipatterns

• Token-cost blindness. Images are expensive in tokens. A batch job over thousands of high-res screenshots can cost more than the text workload it supports. Right-size resolution deliberately.
• Resolution loss on dense content. Many encoders downsample to a fixed grid. Small fonts, dense tables, and fine schematics degrade. Tile or crop the region of interest instead of shrinking the whole image.
• OCR overconfidence. LMMs read text in images impressively but not perfectly. For invoices, config screenshots, or legal documents, verify extracted values; do not treat them as ground truth.
• Visual hallucination. The model may describe objects that are not present, especially with late-fusion designs. Ask it to say when something is not visible, and cross-check safety-critical claims.
• Image-borne prompt injection. Text embedded in an image (a caption, a sign, a watermark) can act as an instruction. An attacker can hide directives in a picture the same way they hide them in a web page. Treat image content as untrusted input, exactly like retrieved text.

This is one of four. The others: large language models (the autoregressive engine underneath), small language models and edge AI (that engine shrunk to run on-device), and diffusion video models (the same transformer, denoising instead of predicting).