Multimodal AI: Combining Text, Vision, and Audio in Modern Architectures | FromZero2AI Blog

Historically, artificial intelligence architectures were strictly unimodal. Natural Language Processing (NLP) models processed text, Computer Vision (CV) networks analyzed pixel grids, and digital signal processing systems operated on audio waveforms. However, human perception is inherently multimodal. We perceive the world through the simultaneous integration of sight, sound, and language.

Modern AI architectures have achieved a breakthrough by bridging these modalities, creating systems capable of tasks like Visual Question Answering (VQA), Image-to-Text generation (image captioning), and cross-modal search. This article explores the architecture of multimodal AI systems, detailing how individual encoders are combined and aligned to construct a unified semantic workspace.

The Multimodal Paradigm: Encoders, Alignment, and Fusion

A modern multimodal system consists of three foundational building blocks:

Modality-Specific Encoders: Specialized networks that extract high-level feature representations from raw input formats (text, images, or audio).
Alignment Layers (Projection): Linear or non-linear projections that map representations from different modality spaces into a shared, unified embedding space.
Fusion and Generation Networks: Downstream architectures (often Autoregressive Transformers) that process the aligned embeddings to perform classification, retrieval, or generation.

Multimodal AI: Text, Vision, and Audio Aligning text, image, and audio embeddings into a joint space Multimodal AI: Text, Vision, and Audio Aligning text, image, and audio embeddings into a joint space

1. Modality-Specific Encoders

Each input type requires a specialized architecture designed to exploit the spatial or temporal dependencies of that specific modality.

Vision Encoders (Image and Video)

While Convolutional Neural Networks (CNNs) like ResNet were once standard, the Vision Transformer (ViT) is the dominant architecture today. A ViT processes an image by:

Splitting the image into non-overlapping patches (e.g., 16 \times 16 pixels).
Flattening the patches and projecting them linearly into a 1D vector sequence.
Prepending a learnable [CLS] token (analogous to BERT) to capture global image semantics.
Adding positional embeddings to maintain spatial awareness.
Passing the sequence through a standard Transformer Encoder.

Text Encoders

Text encoders convert discrete words or tokens into dense continuous vectors. Typically, models use:

Autoregressive decoders (e.g., GPT-style causal transformers) when the downstream task is generation.
Bi-directional encoders (e.g., BERT-style) when the goal is representation learning or contrastive alignment.

Audio Encoders

Audio signals are inherently temporal and continuous. Processing typically proceeds as follows:

The raw waveform is converted into a 2D time-frequency representation, such as a Log-Mel Spectrogram.
This spectrogram is treated as an image, processed either by 1D/2D convolutional layers to downsample the temporal resolution (as in Whisper) or passed directly into an Audio Spectrogram Transformer (AST).

2. Modality Alignment: Bridging the Representational Gap

The main challenge in multimodal AI is that vision and text encoders output vectors of different dimensions and scaling factors. For example, a ViT might output a 1024-dimensional visual vector, whereas a BERT model outputs a 768-dimensional text token embedding. These spaces do not share a common coordinate system.

To align them, architectures use Projection Adaptors:

Linear Projections: A simple learned matrix W_{proj} \in \mathbb{R}^{d_{vision} \times d_{joint}} that translates the visual vector into the text token space.
MLP Adaptors: Multi-layer perceptrons with non-linear activations (e.g., GELU) that learn more complex, non-linear mappings.
Query-Transformers (Q-Former): Pioneered in BLIP-2, a Q-Former uses a set of learnable query vectors that interact with the visual encoder features via cross-attention. This compresses visual representations into a fixed number of tokens that are compatible with the input embedding layer of a Large Language Model.

Contrastive Alignment (CLIP)

Contrastive learning aligns modalities by training twin encoders to maximize the cosine similarity between corresponding image-text pairs while minimizing the similarity for mismatching pairs. For a batch of N image-text pairs, the model calculates the symmetric InfoNCE loss:

\mathcal{L}_{InfoNCE} = -\frac{1}{2} \left[ \sum_{i=1}^{N} \log \frac{e^{\text{sim}(I_i, T_i) / \tau}}{\sum_{j=1}^{N} e^{\text{sim}(I_i, T_j) / \tau}} + \sum_{i=1}^{N} \log \frac{e^{\text{sim}(I_i, T_i) / \tau}}{\sum_{j=1}^{N} e^{\text{sim}(I_j, T_i) / \tau}} \right]

where \text{sim}(I, T) = \frac{I \cdot T}{\|I\| \|T\|} and \tau is a learnable temperature parameter.

3. Multimodal Fusion: Conceptual PyTorch Implementation

Below is a complete, runnable PyTorch model demonstrating how to align an image encoder and text encoder using projection layers, followed by a multimodal fusion network that combines the features using cross-attention.

import torch
import torch.nn as nn
import torch.nn.functional as F

class JointMultimodalAligner(nn.Module):
    def __init__(
        self, 
        image_dim: int = 1024, 
        text_dim: int = 768, 
        shared_dim: int = 512,
        num_classes: int = 10
    ):
        super(JointMultimodalAligner, self).__init__()
        
        # Projection Layer for Vision Modality
        self.vision_projector = nn.Sequential(
            nn.Linear(image_dim, shared_dim),
            nn.LayerNorm(shared_dim),
            nn.GELU(),
            nn.Linear(shared_dim, shared_dim)
        )
        
        # Projection Layer for Text Modality
        self.text_projector = nn.Sequential(
            nn.Linear(text_dim, shared_dim),
            nn.LayerNorm(shared_dim),
            nn.GELU(),
            nn.Linear(shared_dim, shared_dim)
        )
        
        # Cross-Attention Fusion Block
        self.cross_attention = nn.MultiheadAttention(
            embed_dim=shared_dim, 
            num_heads=8, 
            batch_first=True
        )
        
        # Norm and MLP Classifier
        self.layer_norm = nn.LayerNorm(shared_dim)
        self.classifier = nn.Sequential(
            nn.Linear(shared_dim, 256),
            nn.ReLU(),
            nn.Dropout(0.1),
            nn.Linear(256, num_classes)
        )

    def forward(self, raw_image_features: torch.Tensor, raw_text_features: torch.Tensor) -> torch.Tensor:
        """
        Args:
            raw_image_features: Tensor of shape (batch_size, num_patches, image_dim)
            raw_text_features: Tensor of shape (batch_size, seq_len, text_dim)
        Returns:
            logits: Predicted class probability distributions
        """
        # Step 1: Project modalities to the shared dimension
        # Shape: (batch_size, num_patches, shared_dim)
        projected_vision = self.vision_projector(raw_image_features)
        
        # Shape: (batch_size, seq_len, shared_dim)
        projected_text = self.text_projector(raw_text_features)
        
        # Step 2: Perform Cross-Attention
        # We query the image features using the text features to find spatial-linguistic correlations
        # query = projected_text, key = projected_vision, value = projected_vision
        attention_out, _ = self.cross_attention(
            query=projected_text, 
            key=projected_vision, 
            value=projected_vision
        )
        
        # Residual connection and normalization
        fused_features = self.layer_norm(attention_out + projected_text)
        
        # Step 3: Pool features (Global average pooling across sequence dimension)
        # Shape: (batch_size, shared_dim)
        pooled_representation = torch.mean(fused_features, dim=1)
        
        # Step 4: Classify
        logits = self.classifier(pooled_representation)
        return logits

# Example Usage
if __name__ == "__main__":
    batch_size = 4
    num_patches = 196  # e.g., 14x14 patches from ViT
    seq_len = 32       # max token length of text
    
    # Mock output from standard pre-trained encoders
    mock_image_embeds = torch.randn(batch_size, num_patches, 1024)
    mock_text_embeds = torch.randn(batch_size, seq_len, 768)
    
    model = JointMultimodalAligner(image_dim=1024, text_dim=768, shared_dim=512, num_classes=5)
    output_logits = model(mock_image_embeds, mock_text_embeds)
    
    print(f"Input Image Features Shape: {mock_image_embeds.shape}")
    print(f"Input Text Features Shape: {mock_text_embeds.shape}")
    print(f"Output Logits Shape: {output_logits.shape}")
    assert output_logits.shape == (batch_size, 5)

4. Visual Question Answering (VQA) and Generation

In generative multimodal models (like LLaVA or GPT-4V), the aligned visual features are treated as virtual text tokens.

For instance, an image is encoded and projected into N visual embeddings v_1, v_2, \dots, v_N. The text question is tokenized into word embeddings t_1, t_2, \dots, t_M. The input to the LLM decoder is simply the concatenated sequence of tokens:

X_{input} = [v_1, v_2, \dots, v_N, t_1, t_2, \dots, t_M]

The autoregressive language model treats visual tokens exactly like text words, predicting the next textual token using causal self-attention. This elegant integration allows modern models to understand complex diagrams, perform spatial reasoning, and execute optical character recognition (OCR) natively in a unified generation step.