15 KiB
e621tagger Architecture
This document provides a comprehensive system overview for agent understanding. It describes component relationships, data flows, and the overall system architecture.
System Overview
e621tagger is a Flask-based web application that provides automatic image tagging for furry artwork using the JTP-3 Hydra deep learning model. The system consists of three main layers:
- Web Layer - Flask HTTP handling, security, rate limiting
- Inference Layer - Image processing, model execution, tag prediction
- Model Layer - JTP-3 Hydra architecture, NaFlexVit backbone
┌─────────────────────────────────────────────────────────┐
│ Web Layer (Flask) │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌────────────┐ │
│ │ HTTP │ │ Rate │ │ Security│ │ Template │ │
│ │ Routes │ │ Limiter │ │ Headers │ │ Rendering │ │
│ └─────────┘ └─────────┘ └─────────┘ └────────────┘ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Inference Layer (app.py) │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Image │ │ Patch │ │ Model │ │ Tag │ │
│ │ Upload │ │ Extract │ │ Inference│ │ Output │ │
│ │ Handler│ │ │ │ │ │ │ │
│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Model Layer │
│ ┌─────────────────────────────────────────────────┐ │
│ │ JTP-3 Hydra Model │ │
│ │ ┌─────────────┐ ┌─────────────────────────┐ │ │
│ │ │ NaFlexVit │ │ HydraPool Head │ │ │
│ │ │ Backbone │ │ (8,888 tags) │ │ │
│ │ └─────────────┘ └─────────────────────────┘ │ │
│ └─────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────┘
Component Relationships
Core Components
| Component | File | Responsibility |
|---|---|---|
| Flask App | app.py |
HTTP routing, request handling, security, subcategory display names |
| Model Library | hydra/ |
Model loading, inference, labels, pooling, head |
| Image Processor | hydra/image.py |
sRGB conversion, patch creation |
| Subcategory Map | hydra/_subcat.py |
6,263 general tags mapped to 12 subcategories |
| Label Classifier | hydra/label.py |
Label.subcategory property |
| Loader | utils/loader.py |
Multi-process image loading for batching |
Component Interaction Diagram
User Upload
│
▼
┌────────────────┐
│ app.py │
│ (Flask) │
│ │
│ 1. Validate │
│ 2. Save/Temp │
│ 3. Extract │
└────────┬───────┘
│
▼
┌────────────────┐
│ model.py │
│ │
│ 1. load_ │
│ image() │
│ 2. patchify_ │
│ image() │
└────────┬───────┘
│
patches, patch_coords,
patch_valid
│
▼
┌────────────────┐
│ NaFlexVit │
│ (siglip2.py) │
│ │
│ 1. forward() │
│ 2. blocks[] │
│ 3. norm() │
│ 4. forward_ │
│ head() │
└────────┬───────┘
│
image_features
│
▼
┌────────────────┐
│ HydraPool │
│ (pool.py) │
│ │
│ 1. attn() │
│ 2. ff() │
│ 3. out_proj()│
└────────┬───────┘
│
logits (8888)
│
▼
┌────────────────┐
│ Top-K Filter │
│ + Metadata │
│ + Auto Tags │
│ = Response │
└──────────────┘
Data Flow
Image Classification Flow
1. Upload Phase
User → POST /predict → Flask validates file (type, size)
│
2. Preprocessing Phase
PIL Image.open() → open_srgb() → resize to sequence limit
→ patchify() → patches, patch_coords, patch_valid tensors
│
3. Model Inference Phase
NaFlexVit.forward() → image_features
→ HydraPool.forward() → logits (per-tag confidence)
│
4. Postprocessing Phase
sigmoid(logits) → probabilities
→ top-k selection → category lookup → auto meta-tags
→ JSON response
Key Data Structures
| Structure | Type | Description |
|---|---|---|
patches |
Tensor (seq_len, 768) | Image patches as pixel values |
patch_coords |
Tensor (seq_len, 2) | Spatial coordinates (y, x) |
patch_valid |
Tensor (seq_len,) | Valid patch mask |
logits |
Tensor (num_tags,) | Raw model outputs |
probs |
Tensor (num_tags,) | Sigmoid activations |
tags_with_probs |
List[dict] | Sorted tag predictions |
Image Processing Pipeline
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Input │────▶│ Resize │────▶│ Patchify │
│ Image │ │ (max seq) │ │ (16x16) │
└──────────────┘ └──────────────┘ └──────────────┘
│
┌───────────────────┼───────────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ patches │ │patch_coords │ │patch_valid │
│ uint8 (768) │ │ int16 (2) │ │ bool │
└──────────────┘ └──────────────┘ └──────────────┘
Configuration Flow
Environment Variables (app.py)
│
├── MODEL_PATH → hydra_load_model() → Hydra
├── DEVICE → CUDA/CPU selection
├── MAX_SEQ_LEN → image resize constraint
├── SAVE_UPLOADS → file persistence
└── USE_PROXY → ProxyFix middleware
Security Architecture
Request Pipeline
1. Client Request
│
2. ProxyFix (if USE_PROXY=true)
│ - x_for, x_proto, x_host, x_prefix
│
3. Flask Router
│ - /, /predict, /health, /static/*
│
4. Rate Limiter
│ - 500/hour, 20/minute for /predict
│
5. Security Headers
│ - HSTS, X-Frame-Options, CSP, etc.
│
6. Response
Security Headers Set
| Header | Value |
|---|---|
| Strict-Transport-Security | max-age=31536000; includeSubDomains |
| X-Frame-Options | DENY |
| X-Content-Type-Options | nosniff |
| Referrer-Policy | strict-origin-when-cross-origin |
| Permissions-Policy | geolocation=(), microphone=(), camera=() |
| Content-Security-Policy | (see app.py line 176 for full CSP string) |
Extension System
The model supports extension files for adding new classification tags. Extensions are loaded via the Extension.discover() function in model.py.
Extension Format
- Format: SAFETENSORS with metadata
- Architecture: Must match base model (
naflexvit_so400m_patch16_siglip+rr_hydraornaflexvit_so400m_patch16_siglip+rr_hydra2) - Metadata required:
modelspec.implementation:redrocket.extension.label.v1orredrocket.extension.label.v2classifier.label: Tag nameclassifier.label.category: Category ID (0-8, 100-111)classifier.label.implies: Space-separated implied tags
Extension Loading
extensions directory
│
├── extension1.safetensors → Extension.load()
└── extension2.safetensors → HydraPool.load_extensions()
Logging Architecture
The application uses structured logging with emoji prefixes for easy parsing:
| Prefix | Context |
|---|---|
| 👤 | User request (method, path, IP, device, duration) |
| 📤 | Prediction request |
| 📥 | File upload |
| ✅ | Successful operation |
| ⚠️ | Warning |
| ❌ | Error |
| 🔄 | Health check |
| 📄 | Debug request |
Log Format
2024-01-01 12:00:00 [INFO] 👤 🟪 192.168.1.1 GET / 📱 iOS/Safari 200 🟢 0.2ms
2024-01-01 12:00:01 [INFO] 📥 🟪 192.168.1.1: uploading file 'image.png'
2024-01-01 12:00:02 [INFO] ✅ 🟪 192.168.1.1: file 'image.png' processed, top=200 tags (auto=5)
2024-01-01 12:00:02 [INFO] 📤 🟪 192.168.1.1 POST /predict 200 🟢 2500.0ms
Log Format Pattern:
| Position | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | |
|---|---|---|---|---|---|---|---|---|---|---|
| Part | emoji | color+IP | method | path | flag | device | ua | status | duration | |
| Example | 👤 | 🟪 192.168.1.1 | GET | / | 📱 | iOS/Safari | 200 | 🟢 | 0.2ms |
IP Color Squares
The application generates a consistent color square emoji for each IP address using a simple hash of the IP's octets. This allows you to visually track user sessions in logs:
| Color | Emoji | Usage |
|---|---|---|
| Black | ⬛️ | Index 0 |
| Brown | 🟫 | Index 1 |
| Purple | 🟪 | Index 2 |
| Blue | 🟦 | Index 3 |
| Green | 🟩 | Index 4 |
| White | ⬜ | Index 5 |
| Yellow | 🟨 | Index 6 |
| Orange | 🟧 | Index 7 |
| Red | 🟥 | Index 8 |
The same IP always maps to the same color, making it easy to trace user activity chains through logs.
Deployment Architecture
┌─────────────────────────────────────────────────────────────────┐
│ Production │
│ ┌─────────────────────────────────────────────────┐ │
│ │ Traefik (Reverse Proxy) │ │
│ │ - TLS termination │ │
│ │ - Route to container │ │
│ └─────────────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────────────────────────┐ │
│ │ Docker (e621tagger:latest) │ │
│ │ - Gunicorn (2 workers) │ │
│ │ - PyTorch + CUDA │ │
│ │ - Model in memory │ │
│ └─────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
Memory Requirements
| Component | GPU | RAM | Notes |
|---|---|---|---|
| Model (JTP-3 Hydra 3.5) | ~2GB | ~2GB | Single model load (bfloat16) |
| Image patches (batch) | ~50MB | ~50MB | Per-request processing |
| Flask/Gunicorn | - | ~100MB | Application overhead |
| Total (single worker, CUDA) | ~2.1GB | ~2.2GB |
Multi-Worker Memory
When using multiple Gunicorn workers, each worker loads its own model copy:
| Workers | GPU Memory | RAM (app) | Recommended GPU |
|---|---|---|---|
| 1 | ~2GB | ~2.2GB | 4GB VRAM |
| 2 | ~4GB | ~4.4GB | 6GB VRAM |
| 4 | ~8GB | ~8.8GB | 12GB VRAM |
Note: Each additional Gunicorn worker adds ~2GB of model memory. For production with redundancy, use 2 workers with a GPU that has at least 6GB VRAM.
File Structure
e621tagger/
├── app.py # Flask application, subcategory display names
├── hydra/ # Model library (model, pool, head, image, labels, subcat)
│ └── _subcat.py # _SUBCATEGORY_MAP: 6,263 general tags → 12 subcategories
├── utils/ # Utilities (loader, workqueue)
├── templates/
│ ├── index.html # Main UI template
│ └── service-worker.js # PWA service worker (Jinja2 template rendered at runtime)
├── static/
│ ├── css/style.css # Frontend styles
│ ├── js/script.js # Frontend logic
│ ├── js/sw-init.js # SW registration
│ ├── manifest.json # PWA manifest
│ └── icons/ # App icons
├── .github/
│ ├── workflows/
│ │ └── docker-publish.yml
│ ├── compose.yml
│ └── compose-test.yml
└── Dockerfile
Note: The service worker (
templates/service-worker.js) is a Jinja2 template that gets rendered at runtime with theAPP_VERSIONvariable. It is served via the/service-worker.jsroute, not as a static file.