15 KiB
JTP-3 Hydra Model Architecture
This document provides comprehensive documentation of the JTP-3 Hydra model architecture. It is designed for ML/inference agents that need to understand the model's components, data flow, and extension system.
Model Overview
| Attribute | Value |
|---|---|
| Architecture | naflexvit_so400m_patch16_siglip+rr_hydra2 (NaFlexViT + HydraPool + LinearHead) |
| Base Model | SigLIP-400m (So400m) |
| Tags Supported | 8,888 (Hydra 3.5) |
| Patch Size | 16x16 pixels |
| Sequence Length | Up to 1024 patches |
| Model Source | HuggingFace RedRocket/Hydra |
Architecture Components
High-Level Structure
┌─────────────────────────────────────────────────────────────────────┐
│ JTP-3 Hydra Model │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ NaFlexVit Backbone │ │
│ │ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ │
│ │ │ embeds │─▶│ 27 │─▶│ norm │ │ │
│ │ │ (patch +│ │ Blocks │ │ │ │ │
│ │ │ pos) │ │ (ViT) │ │ │ │ │
│ │ └───────────┘ └───────────┘ └───────────┘ │ │
│ └───────────────────────┬───────────────────────────────┘ │
│ │ │
│ ▼ (image_features) │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ HydraPool Classifier Head │ │
│ │ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ │
│ │ │ q (learned│ │ attention│─▶│ out_proj │─▶ logits│ │
│ │ │ queries)│ │ pool │ │ │ │ │
│ │ └───────────┘ └───────────┘ └───────────┘ │ │
│ └─────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
NaFlexVit Backbone
The NaFlexVit is a flexible Vision Transformer based on SigLIP-400m, modified for variable sequence length support.
Implementation
File: siglip2.py
Components
NaFlexVit
├── embeds (NaFlexEmbeds)
│ ├── pos_embed: Learned positional embeddings (1x16x16x1152)
│ └── proj: Linear(768 → 1152)
├── blocks: ModuleList[27 × NaFlexBlock]
│ ├── attn (NaFlexAttn)
│ │ ├── qkv: Linear(1152, 3456) # Q,K,V split into 3×1152
│ │ └── proj: Linear(1152, 1152)
│ ├── mlp (NaFlexMlp)
│ │ ├── fc1: Linear(1152, 4304)
│ │ └── fc2: Linear(4304, 1152)
│ ├── norm1: LayerNorm(1152)
│ └── norm2: LayerNorm(1152)
└── norm: LayerNorm(1152)
Forward Methods
| Method | Description |
|---|---|
forward(x, sizes, valid) |
Full forward pass with stacked patches |
forward_features(x, sizes, valid) |
Return features without classification head |
forward_head(features, valid) |
Apply attention pool + classification head |
forward_varlen(x, sizes, cu_seq, max_seq) |
Variable-length sequence forward |
Variable Sequence Length Support
The model supports variable sequence lengths via two modes:
# Standard forward - stacked batch (fixed max sequence length)
def forward(self, x, sizes=None, valid=None):
output = self.forward_features(x, sizes, valid)
return self.forward_head(output.pop("features"), output.pop("valid"))
# Variable length forward - jagged tensor (efficient for varied sizes)
def forward_varlen(self, x, sizes, cu_seq, max_seq=1024):
output = self.forward_features_varlen(x, sizes, cu_seq, max_seq)
return self.forward_head_varlen(output.pop("features"), output.pop("cu_seq"), output.pop("max_seq"))
Key Dimensions
| Component | Dimension |
|---|---|
| Input Patches | (batch, seq_len, 768) |
| Hidden Dimension | 1152 |
| Attention Heads | 16 |
| Head Dimension | 1152 / 16 = 72 |
| MLP Hidden | 4304 (×3.75) |
| Number of Blocks | 27 |
| Output Features | (batch, seq_len, 1152) |
HydraPool Classifier Head
HydraPool is a novel attention-based classification head that learns per-tag queries for multi-label classification.
Implementation
File: pool.py
Architecture
HydraPool
├── q: Parameter(n_heads, n_classes, head_dim)
│ # Learned query vectors per tag
├── kv: Linear(input_dim, attn_dim × 2)
│ # Key-Value projection
├── qk_norm: RMSNorm(head_dim)
│ # Query/key normalization
├── ff: _FeedForward
│ ├── norm: LayerNorm(attn_dim)
│ ├── proj_in: SwiGLU(attn_dim, ff_dim)
│ └── proj_out: Linear(ff_dim, attn_dim)
├── mid_blocks: ModuleList[n × _MidBlock]
# Optional mid-level processing blocks
Forward Process
def forward(self, x, attn_mask=None):
# x: (batch, seq_len, attn_dim)
# 1. Attention pooling
x, k, v = self._forward_attn(x, attn_mask)
# 2. Feedforward with residual
x = x + self.ff(x)
# 3. Mid blocks (if any)
for block in self.mid_blocks:
x = block(x, k, v, attn_mask)
return x # (batch, n_classes, attn_dim)
Attention Mechanism
def _forward_attn(self, x, attn_mask):
# x: (batch, seq_len, attn_dim)
# 1. Expand learned queries to batch size
q = self._forward_q().expand(*x.shape[:-2], -1, -1, -1)
# q: (batch, n_heads, n_classes, head_dim)
# 2. Compute KV from features
x = self.kv(x)
k, v = rearrange(x, "... s (n h e) -> n ... h s e", n=2).unbind(0)
# k, v: (batch, n_heads, seq_len, head_dim)
# 3. Normalize keys (queries are pre-normalized at init)
k = self.qk_norm(k)
# 4. Scaled dot-product attention
x = scaled_dot_product_attention(q, k, v, attn_mask=attn_mask)
# x: (batch, n_heads, n_classes, head_dim)
return rearrange(x, "... h s e -> ... s (h e)"), k, v
Query Normalization
Queries are pre-normalized at initialization time via qk_norm to improve training stability:
with torch.no_grad():
self.q.copy_(self.qk_norm(self.q))
Image Processing Pipeline
Overview
Input Image → Orientation detection → Crop → sRGB conversion → Alpha flatten → Resize → Autorot → Patchify → Model Input
Implementation
File: image.py, via _open_profile() with open_srgb() entry point
Steps
1. Orientation Detection & Crop
EXIF orientation is detected from image metadata. If a crop region is specified, it is applied before color transformation.
2. ICC Color Transform
img = img.icc_transform(
profile, embedded=True,
intent=Intent.RELATIVE,
black_point_compensation=True,
depth=depth,
)
Converts the image to the sRGB profile, respecting embedded ICC profiles.
3. Alpha Flattening
if img.hasalpha():
img = img.flatten(background=background)
If the image has an alpha channel, it is flattened against the configured background color (0 = black, 127 = grey, or 255 = white per classifier.background metadata).
4. Resize (via Kernel)
Resizing uses the configured kernel (Kernel.LANCZOS3 or Kernel.MKS2013 for linear) and may optionally apply pre/post LUTs for linear-light resizing.
5. Autorotate
img = img.autorot()
Applies EXIF rotation after resize.
6. Patch Extraction
Patches are extracted by the patchify() function in image.py which spreads the image into patches and flattens into a sequence of patch vectors.
Resize Sizing Algorithm
File: model.py
def get_image_size_for_seq(image_size, patch_size=16, max_seq_len=1024, ...):
"""Determine max image size within sequence constraint."""
h, w = image_size
max_py = int(max((h * max_ratio) // patch_size, 1))
max_px = int(max((w * max_ratio) // patch_size, 1))
if (max_py * max_px) <= max_seq_len:
return max_py * patch_size, max_px * patch_size
# Binary search for aspect-ratio-preserving size
...
return py * patch_size, px * patch_size
Patch Extraction Implementation
File: image.py, function patchify()
def patchify(img, patch_size=16):
img = spread(img, patch_size)
return img.flatten(1, 2).flatten(-3)
Output
The image pipeline produces a single tensor (not patches/coords/valid tuple):
| Variable | Type | Description |
|---|---|---|
img_tensor |
Tensor (H, W, 3) | Processed image in uint8 (0-255) sRGB |
Patch extraction and stacking happen at inference time via hydra_image.patchify() and the model's from_srgb() normalization.
Example
from hydra.model import load_model
model = load_model("models/hydra-3.5.safetensors")
# Load and process image (returns a tensor)
img_tensor = model.load_image("artwork.png")
# Patchify and normalize
patches = hydra_image.patchify(img_tensor, 16)
patches = model.from_srgb(patches) # uint8 → bfloat16, normalize to [-1, 1]
sizes = torch.tensor([[h, w]], dtype=torch.int32)
Model Inference
Complete Inference Flow
import torch
from hydra import image as hydra_image
from hydra.model import Hydra, load_model
# 1. Load model
model: Hydra = load_model("models/hydra-3.5.safetensors")
model.eval()
model.requires_grad_(False)
tag_list = [label.label for label in model.labels]
# 2. Load and process image
img_tensor = model.load_image("artwork.png") # returns (H, W, 3) uint8
# 3. Patchify and normalize
h = img_tensor.shape[0] // 16
w = img_tensor.shape[1] // 16
patches = hydra_image.patchify(img_tensor, 16) # (B, seq, 768)
sizes = torch.tensor([[h, w]], dtype=torch.int32)
# Normalize: [0, 255] → [-1, 1] (via from_srgb)
patches = model.from_srgb(patches)
patches = patches.to(device="cuda")
sizes = sizes.to(device="cuda")
# 4. Run inference
with torch.no_grad():
logits = model.forward(patches, sizes) # (1, num_tags)
# 5. Process outputs
probs = torch.sigmoid(logits[0].float()).cpu()
values, indices = probs.topk(200)
# 6. Get tag predictions
tags_with_probs = []
for idx, val in zip(indices, values):
tag = tag_list[idx.item()]
prob = val.item()
tags_with_probs.append({"tag": tag, "prob": prob})
Output
| Variable | Type | Description |
|---|---|---|
logits |
Tensor (1, num_tags) | Raw model outputs |
probs |
Tensor (num_tags,) | Sigmoid probabilities (0-1) |
tags_with_probs |
List[dict] | Top-K predictions sorted by probability |
Extension System
Extensions allow adding new classification tags to the model without retraining.
Extension Format
- File Format: SAFETENSORS
- Required Metadata:
modelspec.implementation:redrocket.extension.label.v1orv2modelspec.architecture: Must match base modelclassifier.label: Tag nameclassifier.label.category: Category ID (0-8, 100-111)classifier.label.implies: Space-separated implied tags
Extension Weights
| Key | Shape | Description |
|---|---|---|
q |
(1, 1, head_dim) | Query vector |
out_proj.weight (saved) → loaded as head.proj.weight |
(1, attn_dim, 2) | SwiGLU output projection |
mid_blocks.{n}.q_cls |
(1, n_heads, head_dim) | Mid block queries |
Loading Extensions
from hydra.model import Hydra, load_model, Extension
# Load model with extensions
model: Hydra = load_model("models/hydra-3.5.safetensors")
model.load_extensions(Extension.discover("extensions/"))
print(f"Total tags: {len(model.labels)}")
print(f" {path}: {info['label']}")
Extension Directory Structure
extensions/
├── character_tag1.safetensors
├── character_tag2.safetensors
└── species_tag.safetensors
Model Loading Details
From safetensors
from safetensors import safe_open
with safe_open(path, framework="pt", device="cpu") as f:
metadata = f.metadata()
state_dict = {key: f.get_tensor(key) for key in f.keys()}
# Extract metadata
arch = metadata["modelspec.architecture"]
labels = metadata["classifier.labels"].split("\n")
Model Metadata
| Key | Description |
|---|---|
modelspec.architecture |
Model architecture string |
modelspec.implementation |
Implementation identifier |
classifier.labels |
Newline-separated tag names |
Performance Considerations
Memory Usage
| Component | GPU Memory |
|---|---|
| JTP-3 Model (bfloat16) | ~2GB |
| Input (batch=1) | ~50MB |
| Gradients | ~0 (inference) |
| Total (single worker, CUDA) | ~2.1GB |
Multi-Worker Memory Scaling
Each Gunicorn worker runs its own Flask process with a separate model load:
| Workers | GPU Memory |
|---|---|
| 1 | ~2.1GB |
| 2 | ~4.2GB |
| 4 | ~8.4GB |
For production deployments, ensure your GPU has enough VRAM for the number of workers (recommended: 4GB per worker minimum).
Inference Speed
| Device | Approximate Time |
|---|---|
| RTX 4090 | ~100ms |
| RTX 3090 | ~150ms |
| CPU (i9-13900K) | ~2000ms |
Optimization Flags
# Enable TF32 on Ampere+
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True
# bfloat16 inference
model = model.to(dtype=torch.bfloat16)
Integration with Flask
The Flask app in app.py wraps the model:
from hydra.model import load_model
from hydra import image as hydra_image
# Load on startup
MODEL_PATH = os.getenv('MODEL_PATH', 'models/hydra-3.5.safetensors')
DEVICE = os.getenv('DEVICE', 'cuda' if torch.cuda.is_available() else 'cpu')
PATCH_SIZE = 16
model = load_model(MODEL_PATH)
if DEVICE == 'cpu':
model = model.float()
else:
model = model.to(dtype=torch.bfloat16, device=DEVICE)
model.requires_grad_(False)
model.eval()
# In predict route:
img_tensor = model.load_image(image_path)
h = img_tensor.shape[0] // PATCH_SIZE
w = img_tensor.shape[1] // PATCH_SIZE
patches = hydra_image.patchify(img_tensor, PATCH_SIZE)
sizes = torch.tensor([[h, w]], dtype=torch.int32)
patches = model.from_srgb(patches)
patches = patches.to(device=DEVICE)
sizes = sizes.to(device=DEVICE)
with torch.no_grad():
logits = model.forward(patches, sizes)