# e621tagger Architecture This document provides a comprehensive system overview for agent understanding. It describes component relationships, data flows, and the overall system architecture. ## System Overview e621tagger is a Flask-based web application that provides automatic image tagging for furry artwork using the JTP-3 Hydra deep learning model. The system consists of three main layers: 1. **Web Layer** - Flask HTTP handling, security, rate limiting 2. **Inference Layer** - Image processing, model execution, tag prediction 3. **Model Layer** - JTP-3 Hydra architecture, NaFlexVit backbone ``` ┌─────────────────────────────────────────────────────────┐ │ Web Layer (Flask) │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌────────────┐ │ │ │ HTTP │ │ Rate │ │ Security│ │ Template │ │ │ │ Routes │ │ Limiter │ │ Headers │ │ Rendering │ │ │ └─────────┘ └─────────┘ └─────────┘ └────────────┘ │ └─────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ Inference Layer (app.py) │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ Image │ │ Patch │ │ Model │ │ Tag │ │ │ │ Upload │ │ Extract │ │ Inference│ │ Output │ │ │ │ Handler│ │ │ │ │ │ │ │ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ └─────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ Model Layer │ │ ┌─────────────────────────────────────────────────┐ │ │ │ JTP-3 Hydra Model │ │ │ │ ┌─────────────┐ ┌─────────────────────────┐ │ │ │ │ │ NaFlexVit │ │ HydraPool Head │ │ │ │ │ │ Backbone │ │ (8,888 tags) │ │ │ │ │ └─────────────┘ └─────────────────────────┘ │ │ │ └─────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────┘ ``` ## Component Relationships ### Core Components | Component | File | Responsibility | |-----------|------|----------------| | Flask App | `app.py` | HTTP routing, request handling, security, subcategory display names | | Model Library | `hydra/` | Model loading, inference, labels, pooling, head | | Image Processor | `hydra/image.py` | sRGB conversion, patch creation | | Subcategory Map | `hydra/_subcat.py` | 6,263 general tags mapped to 12 subcategories | | Label Classifier | `hydra/label.py` | `Label.subcategory` property | | Loader | `utils/loader.py` | Multi-process image loading for batching | ### Component Interaction Diagram ``` User Upload │ ▼ ┌────────────────┐ │ app.py │ │ (Flask) │ │ │ │ 1. Validate │ │ 2. Save/Temp │ │ 3. Extract │ └────────┬───────┘ │ ▼ ┌────────────────┐ │ model.py │ │ │ │ 1. load_ │ │ image() │ │ 2. patchify_ │ │ image() │ └────────┬───────┘ │ patches, patch_coords, patch_valid │ ▼ ┌────────────────┐ │ NaFlexVit │ │ (siglip2.py) │ │ │ │ 1. forward() │ │ 2. blocks[] │ │ 3. norm() │ │ 4. forward_ │ │ head() │ └────────┬───────┘ │ image_features │ ▼ ┌────────────────┐ │ HydraPool │ │ (pool.py) │ │ │ │ 1. attn() │ │ 2. ff() │ │ 3. out_proj()│ └────────┬───────┘ │ logits (8888) │ ▼ ┌────────────────┐ │ Top-K Filter │ │ + Metadata │ │ + Auto Tags │ │ = Response │ └──────────────┘ ``` ## Data Flow ### Image Classification Flow ``` 1. Upload Phase User → POST /predict → Flask validates file (type, size) │ 2. Preprocessing Phase PIL Image.open() → open_srgb() → resize to sequence limit → patchify() → patches, patch_coords, patch_valid tensors │ 3. Model Inference Phase NaFlexVit.forward() → image_features → HydraPool.forward() → logits (per-tag confidence) │ 4. Postprocessing Phase sigmoid(logits) → probabilities → top-k selection → category lookup → auto meta-tags → JSON response ``` ### Key Data Structures | Structure | Type | Description | |----------|------|-------------| | `patches` | Tensor (seq_len, 768) | Image patches as pixel values | | `patch_coords` | Tensor (seq_len, 2) | Spatial coordinates (y, x) | | `patch_valid` | Tensor (seq_len,) | Valid patch mask | | `logits` | Tensor (num_tags,) | Raw model outputs | | `probs` | Tensor (num_tags,) | Sigmoid activations | | `tags_with_probs` | List[dict] | Sorted tag predictions | ### Image Processing Pipeline ``` ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Input │────▶│ Resize │────▶│ Patchify │ │ Image │ │ (max seq) │ │ (16x16) │ └──────────────┘ └──────────────┘ └──────────────┘ │ ┌───────────────────┼───────────────────┐ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ patches │ │patch_coords │ │patch_valid │ │ uint8 (768) │ │ int16 (2) │ │ bool │ └──────────────┘ └──────────────┘ └──────────────┘ ``` ## Configuration Flow ``` Environment Variables (app.py) │ ├── MODEL_PATH → hydra_load_model() → Hydra ├── DEVICE → CUDA/CPU selection ├── MAX_SEQ_LEN → image resize constraint ├── SAVE_UPLOADS → file persistence └── USE_PROXY → ProxyFix middleware ``` ## Security Architecture ### Request Pipeline ``` 1. Client Request │ 2. ProxyFix (if USE_PROXY=true) │ - x_for, x_proto, x_host, x_prefix │ 3. Flask Router │ - /, /predict, /health, /static/* │ 4. Rate Limiter │ - 500/hour, 20/minute for /predict │ 5. Security Headers │ - HSTS, X-Frame-Options, CSP, etc. │ 6. Response ``` ### Security Headers Set | Header | Value | |--------|-------| | Strict-Transport-Security | max-age=31536000; includeSubDomains | | X-Frame-Options | DENY | | X-Content-Type-Options | nosniff | | Referrer-Policy | strict-origin-when-cross-origin | | Permissions-Policy | geolocation=(), microphone=(), camera=() | | Content-Security-Policy | (see app.py line 176 for full CSP string) | ## Extension System The model supports extension files for adding new classification tags. Extensions are loaded via the `Extension.discover()` function in `model.py`. ### Extension Format - Format: SAFETENSORS with metadata - Architecture: Must match base model (`naflexvit_so400m_patch16_siglip+rr_hydra` or `naflexvit_so400m_patch16_siglip+rr_hydra2`) - Metadata required: - `modelspec.implementation`: `redrocket.extension.label.v1` or `redrocket.extension.label.v2` - `classifier.label`: Tag name - `classifier.label.category`: Category ID (0-8, 100-111) - `classifier.label.implies`: Space-separated implied tags ### Extension Loading ``` extensions directory │ ├── extension1.safetensors → Extension.load() └── extension2.safetensors → HydraPool.load_extensions() ``` ## Logging Architecture The application uses structured logging with emoji prefixes for easy parsing: | Prefix | Context | |--------|---------| | 👤 | User request (method, path, IP, device, duration) | | 📤 | Prediction request | | 📥 | File upload | | ✅ | Successful operation | | ⚠️ | Warning | | ❌ | Error | | 🔄 | Health check | | 📄 | Debug request | ### Log Format ``` 2024-01-01 12:00:00 [INFO] 👤 🟪 192.168.1.1 GET / 📱 iOS/Safari 200 🟢 0.2ms 2024-01-01 12:00:01 [INFO] 📥 🟪 192.168.1.1: uploading file 'image.png' 2024-01-01 12:00:02 [INFO] ✅ 🟪 192.168.1.1: file 'image.png' processed, top=200 tags (auto=5) 2024-01-01 12:00:02 [INFO] 📤 🟪 192.168.1.1 POST /predict 200 🟢 2500.0ms ``` **Log Format Pattern:** | Position | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | |----------|---|---|---|---|---|---|---|---|---|---| | Part | emoji | color+IP | method | path | flag | device | ua | status | duration | | Example | 👤 | 🟪 192.168.1.1 | GET | / | 📱 | iOS/Safari | 200 | 🟢 | 0.2ms | ### IP Color Squares The application generates a consistent color square emoji for each IP address using a simple hash of the IP's octets. This allows you to visually track user sessions in logs: | Color | Emoji | Usage | |-------|------|-------| | Black | ⬛️ | Index 0 | | Brown | 🟫 | Index 1 | | Purple | 🟪 | Index 2 | | Blue | 🟦 | Index 3 | | Green | 🟩 | Index 4 | | White | ⬜ | Index 5 | | Yellow | 🟨 | Index 6 | | Orange | 🟧 | Index 7 | | Red | 🟥 | Index 8 | The same IP always maps to the same color, making it easy to trace user activity chains through logs. ## Deployment Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ Production │ │ ┌─────────────────────────────────────────────────┐ │ │ │ Traefik (Reverse Proxy) │ │ │ │ - TLS termination │ │ │ │ - Route to container │ │ │ └─────────────────────────────────────┘ │ │ │ │ │ ┌─────────────────────────────────────────────┐ │ │ │ Docker (e621tagger:latest) │ │ │ │ - Gunicorn (2 workers) │ │ │ │ - PyTorch + CUDA │ │ │ │ - Model in memory │ │ │ └─────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ ``` ## Memory Requirements | Component | GPU | RAM | Notes | |-----------|-----|-----|-------| | Model (JTP-3 Hydra 3.5) | ~2GB | ~2GB | Single model load (bfloat16) | | Image patches (batch) | ~50MB | ~50MB | Per-request processing | | Flask/Gunicorn | - | ~100MB | Application overhead | | **Total (single worker, CUDA)** | ~2.1GB | ~2.2GB | | ### Multi-Worker Memory When using multiple Gunicorn workers, each worker loads its own model copy: | Workers | GPU Memory | RAM (app) | Recommended GPU | |---------|------------|-----------|-----------------| | 1 | ~2GB | ~2.2GB | 4GB VRAM | | 2 | ~4GB | ~4.4GB | 6GB VRAM | | 4 | ~8GB | ~8.8GB | 12GB VRAM | > **Note:** Each additional Gunicorn worker adds ~2GB of model memory. For production with redundancy, use 2 workers with a GPU that has at least 6GB VRAM. ## File Structure ``` e621tagger/ ├── app.py # Flask application, subcategory display names ├── hydra/ # Model library (model, pool, head, image, labels, subcat) │ └── _subcat.py # _SUBCATEGORY_MAP: 6,263 general tags → 12 subcategories ├── utils/ # Utilities (loader, workqueue) ├── templates/ │ ├── index.html # Main UI template │ └── service-worker.js # PWA service worker (Jinja2 template rendered at runtime) ├── static/ │ ├── css/style.css # Frontend styles │ ├── js/script.js # Frontend logic │ ├── js/sw-init.js # SW registration │ ├── manifest.json # PWA manifest │ └── icons/ # App icons ├── .github/ │ ├── workflows/ │ │ └── docker-publish.yml │ ├── compose.yml │ └── compose-test.yml └── Dockerfile ``` > **Note:** The service worker (`templates/service-worker.js`) is a Jinja2 template that gets rendered at runtime with the `APP_VERSION` variable. It is served via the `/service-worker.js` route, not as a static file.