21 KiB
Tag Classification and Implications
This document provides comprehensive documentation of the tag classification system used by e621tagger. It covers tag categories, subcategory classification, implication logic, threshold handling, and output presentation.
Tag System Overview
e621tagger classifies images using 8,888 tags (Hydra 3.5 model) organized into e621 categories. Tags are assigned confidence scores and can have implications (hierarchical relationships) that affect final output. Each general-category tag is further classified into one of 12 fine-grained subcategories for better UI organization.
graph TD
A[Raw Model Output (8,888 logits)] --> B[Sigmoid Activation (0.0 - 1.0)]
B --> C[Implication Application]
C --> D[Threshold Filtering]
D --> E[Top-K Selection]
E --> F[Subcategory Classification]
F --> G[Category + Meta Tag Assignment]
Tag Categories
Tags are organized into e621 categories with numeric IDs:
Category Mapping
| Category ID | Name | Description | Example Tags |
|---|---|---|---|
| 0 | General | General subject matter (includes 100-111) | female, male, anthro, solo |
| 1 | Artist | Artist names | artist:fenrir784, oc:request |
| 2 | Contributor | Contributor tags | (rarely used) |
| 3 | Copyright | Series/franchise | sonic_the_hedgehog, nintendo |
| 4 | Character | Fictional characters | miles_tails_prower |
| 5 | Species | Species/body types | wolf, dragon, fox |
| 6 | Invalid | Invalid tags | (rarely used) |
| 7 | Meta | Rating and technical | safe, questionable, hi_res |
| 8 | Lore | Story elements | backstory, backview |
| 100 | Accessories, Items, Clothing | Clothing and items | scarf, hat, belt |
| 101 | Actions, Positions, State | Actions and poses | sitting, walking, sleeping |
| 102 | Body Color | Color markings | red_fur, blue_eyes, black_markings |
| 103 | Body Features | Body parts/features | tail, wings, horns |
| 104 | Effects, Fluids | Visual effects | cum, sparkles, glowing |
| 105 | Fetishes, Specifics, Interactions | Adult content | anal, footjob, bondage |
| 106 | Genders, Demographics | Gender-related | male, female, herm, femboy |
| 107 | Locations, Backgrounds, Setting | Environments | forest, bedroom, beach |
| 108 | Poses, Scenarios, Situations | Scene setup | profile_view, closeup, action_pose |
| 109 | Style, Perspective | Art style | chibi, realistic, comic |
| 110 | Text, Symbols, UI, Vocalization | Text elements | speech_bubble, sound_effect, onomatopoeia |
| 111 | Other | Miscellaneous | Tags from unrecognized categories |
Note: When copying tags to clipboard, categories 100-111 are merged into "General" to maintain e621 compatibility. The web interface displays all 21 categories separately for better visual organization.
From Code
# hydra/label.py
TAG_CATEGORIES = [
"general", # 0
"artist", # 1
"contributor", # 2
"copyright", # 3
"character", # 4
"species", # 5
"invalid", # 6
"meta", # 7
"lore", # 8
]
# Subcategories 100-111 are handled via _SUBCATEGORY_MAP and Label.subcategory
Subcategory Classification
General tags (category 0) are further split into 12 fine-grained subcategories for better UI organization. The subcategory is determined by Label.subcategory in hydra/label.py.
Subcategory Mapping
| Subcategory | Code | Description | Examples |
|---|---|---|---|
| Accessories, Items, Clothing | accessory |
Clothing, accessories, items | scarf, hat, belt, sword |
| Actions, Positions, State | action |
Actions and dynamic states | kissing, running, sitting_on_lap |
| Body Color | color |
Color markings | red_fur, blue_eyes, black_markings |
| Body Features | body_feature |
Body parts/features | tail, wings, horns, claws |
| Effects, Fluids | effect |
Visual effects, fluids | sparkles, glowing, cum |
| Fetishes, Specifics, Interactions | fetish |
Adult content categories | anal, footjob, bondage |
| Genders, Demographics | demographic |
Gender-related | male, female, herm, femboy |
| Locations, Backgrounds, Setting | setting |
Environments | forest, bedroom, beach |
| Poses, Scenarios, Situations | pose |
Scene setup | profile_view, closeup, splits |
| Style, Perspective | style |
Art style | chibi, realistic, comic, perspective |
| Text, Symbols, UI, Vocalization | text |
Text elements | speech_bubble, onomatopoeia |
| Other | other |
Miscellaneous | Unrecognized / uncategorized tags |
Implementation
File: hydra/_subcat.py
The subcategory is resolved via Label.subcategory in hydra/label.py:102-121:
@property
@cache
def subcategory(self) -> str | None:
if self.category == "general":
if self.label in _SUBCATEGORY_MAP:
return _SUBCATEGORY_MAP[self.label]
if (
self.label.startswith(COLOR_PREFIXES)
and not self.label.endswith(COLOR_EXCEPTIONS)
):
return "color"
return None
if (
self.category == "meta"
and self.label in ("safe", "questionable", "explicit")
):
return "rating"
return None
Mapping Strategy
The _SUBCATEGORY_MAP in hydra/_subcat.py is a pre-built dictionary of 6,263 tag→subcategory mappings. At inference time, the subcategory is resolved by Label.subcategory via:
- Direct lookup in
_SUBCATEGORY_MAP(covers the majority of general tags) - Color prefix heuristic in
label.py— tags starting with color prefixes (e.g.red_,blue_,black_) that aren't in the map but aren't color exceptions are classified as"color" - Fallback —
Nonefor tags that don't match either
| Lookup | Description | Tags Covered |
|---|---|---|
1. _SUBCATEGORY_MAP |
Pre-built dictionary mapping each known general tag to its subcategory | 6,263 |
| 2. Color prefix | COLOR_PREFIXES in label.py (18+ prefixes like red_, blue_, dark_) |
Remaining color tags |
| 3. Fallback | Returns None |
Unclassified tags |
Coverage Validation
The subcategory mapping was validated by 12 independent sub-agents, each reviewing every general tag in their domain (6,263 total). The validation process identified 1,951 corrections across all 12 subcategories:
| Subcategory | Corrections Applied |
|---|---|
| action | 527 |
| accessory | 283 |
| pose | 227 |
| body_feature | 219 |
| other | 202 |
| effect | 123 |
| demographic | 92 |
| style | 88 |
| color | 73 |
| text | 58 |
| setting | 33 |
| fetish | 26 |
| Total | 1,951 |
Subcategory in the API
The /predict endpoint returns subcategory via the category field:
{
"tag": "female",
"prob": 0.95,
"category": "Genders, Demographics"
}
The SUBCATEGORY_DISPLAY_NAMES map in app.py:32-45 converts internal subcategory codes to display names.
Web UI Display
The frontend in script.js uses displayCategoryOrder (line 36) to render all 12 subcategories as separate visual categories, each with its own color-coded header. When copying to clipboard, subcategories are merged back into "General" for e621 compatibility.
Implication System
Tags can imply other tags in hierarchical relationships. For example:
sunglasses_on_headimplieseyewear_on_headandsunglassesvibratorimpliessex_toy(in some contexts)water_bottleimpliesbottle
Implication Modes
The pipeline supports several modes for handling implications, configured via the implications parameter in classification.py:
| Mode | Description | Behavior |
|---|---|---|
off |
No implications | Simple threshold filtering only |
preserve |
Qualify implications | Require implied tags to meet threshold |
inherit |
Tags inherit highest probability | If A implies B, B's prob = max(B's prob, A's prob) |
constrain |
Tags constrained to lowest | If A implies B, A's prob = min(A's prob, B's prob) |
remove |
Exclude implied tags | Remove all tags implied by other tags |
constrain-remove |
Combination | Constrain then remove |
enforce |
Enforce implications | Remove tags whose implied tags don't meet threshold |
enforce-inherit |
Enforce + inherit | Enforce then inherit |
enforce-constrain |
Enforce + constrain | Enforce then constrain |
enforce-remove |
Enforce + remove | Enforce then remove |
Implementation
def _inherit_implications(
outputs: dict[str, float],
antecedent: str, prob: float,
labels: dict[str, tuple[Label, float]]
) -> None:
if (label := labels.get(antecedent)) is None:
return
for consequent in label[0].implies:
if outputs.get(consequent, float("+inf")) < prob:
outputs[consequent] = prob
_inherit_implications(outputs, consequent, prob, labels)
def _constrain_implications(
outputs: dict[str, float],
consequent: str, prob: float,
labels: dict[str, tuple[Label, float]]
) -> None:
if (label := labels.get(consequent)) is None:
return
for antecedent in label[0].implied_by:
if outputs.get(antecedent, float("-inf")) > prob:
outputs[antecedent] = prob
_constrain_implications(outputs, antecedent, prob, labels)
def _remove_consequents(
outputs: dict[str, float], antecedent: str,
labels: dict[str, tuple[Label, float]]
) -> None:
if (label := labels.get(antecedent)) is None:
return
for consequent in label[0].implies:
outputs.pop(consequent, None)
_remove_consequents(outputs, consequent, labels)
Example
Given:
female: 0.9futanari: 0.8 (impliesfemale)intersex: 0.7 (impliesfemaleandmale)
| Mode | female | futanari | intersex | male |
|---|---|---|---|---|
| Off | 0.9 | 0.8 | 0.7 | - |
| Inherit | 0.9 | 0.8→0.9 | 0.7→0.9 | - |
| Constrain | 0.9→0.8 | 0.8 | 0.7 | - |
| Remove | removed | removed | removed | - |
| Constrain-Remove | 0.9→0.8 then remove | 0.8 then remove | removed | - |
Meta Tag Categories
Tags from category 7 (Meta) are automatically detected based on image properties.
Auto-Detected Tags
| Tag | Condition |
|---|---|
animated |
GIF with more than 1 frame |
thumbnail |
Both dimensions ≤ 250px |
low_res |
Both dimensions ≤ 500px |
hi_res |
Width ≥ 1600px OR height ≥ 1200px |
absurd_res |
Width ≥ 3200px OR height ≥ 2400px |
superabsurd_res |
Both dimensions ≥ 10000px |
4k |
3840×2160, 2160×3840, 4096×2160, or 2160×4096 |
long_image |
Aspect ratio ≥ 4:1 or ≤ 1:4 |
tall_image |
Aspect ratio ≤ 1:4 (subset of long_image) |
2025, 2026, ... |
Current year (added server-side in /predict via str(datetime.now().year) with 1.0 confidence, Meta category; frontend filters via Apple-style switch addCurrentYearTag in localStorage e621tagger-settings, default true) |
Aspect Ratio Tags
Standard aspect ratios are detected:
1:1, 2:1, 1:2, 3:1, 1:3, 3:2, 2:3,
4:3, 3:4, 5:3, 3:5, 5:4, 4:5,
6:5, 5:6, 7:4, 4:7, 7:3, 3:7,
16:10, 10:16, 11:8, 8:11, 14:9, 9:14,
16:9, 9:16, 21:9, 9:21
Additionally, widescreen is detected for 16:9 or 16:10 aspect ratios.
Implementation
# app.py - detect_meta_tags_for_image_path
def detect_meta_tags_for_image_path(image_path):
tags = set()
with Image.open(image_path) as im:
w, h = im.size
# Animation detection
if fmt == 'GIF':
if im.n_frames > 1:
tags.add('animated')
# Resolution tags
if w <= 250 and h <= 250:
tags.add('thumbnail')
if w <= 500 and h <= 500:
tags.add('low_res')
if w >= 1600 or h >= 1200:
tags.add('hi_res')
if w >= 3200 or h >= 2400:
tags.add('absurd_res')
if w >= 10000 and h >= 10000:
tags.add('superabsurd_res')
# 4K detection
if (w == 3840 and h == 2160) or (w == 2160 and h == 3840) ...:
tags.add('4k')
# Aspect ratio
ratio = w / h
if ratio >= 4 or ratio <= 0.25:
tags.add('long_image')
# Specific ratios
for tagname, a, b in ratios:
if w * b == h * a:
tags.add(tagname)
# Current year tag (added in /predict endpoint, always; frontend may hide via switch)
from datetime import datetime
tags.add(str(datetime.now().year))
# Frontend: uploadImage() filters with `if (t === currentYearTag && !addCurrentYearTag) return`
# and copy uses perTagAutoDisable set toggled by switch change listener
Threshold System
Threshold Presets
The frontend exposes three threshold presets:
| Preset | "All" Threshold | "Confident" Threshold |
|---|---|---|
| Conservative | 0.70 | 0.80 |
| Standard | 0.60 | 0.70 |
| Liberal | 0.50 | 0.60 |
| Custom | User-defined | User-defined |
Threshold Semantics
- All Threshold: Tags with probability ≥ this value are considered "valid"
- Confident Threshold: Tags with probability ≥ this value are considered "confident" and highlighted differently in UI
Implementation
# Frontend (script.js) threshold logic
const presets = {
conservative: { all: 0.70, confident: 0.80 },
standard: { all: 0.60, confident: 0.70 },
liberal: { all: 0.50, confident: 0.60 }
};
// Applied in frontend for display:
// - Purple: prob >= confidentThreshold
// - Blue: prob >= allThreshold
// - Gray: prob < allThreshold
Per-Tag Thresholds
For fine-tuning, use calibration files:
tag,threshold
female,0.55
anthro,0.50
furry,0.45
intersex,0.60
Tag Output Format
JSON Response
{
"success": true,
"tags": [
{
"tag": "female",
"prob": 0.95,
"category": "General"
},
{
"tag": "anthro",
"prob": 0.89,
"category": "Species"
}
],
"auto_meta": ["hi_res", "16:9"]
}
e621 Format (Space-Separated)
female anthro solo furry blue_eyes smile
PostyBirb Format (Comma-Separated)
female, anthro, solo, furry, blue_eyes, smile
Tag Rewriting
Certain tags are rewritten for e621 compatibility (use --original-tags to disable):
| Original | Rewritten |
|---|---|
vulva |
pussy |
Note: The Flask API (
/predictendpoint) returns raw model tags without rewriting.
| -x lore | Exclude Lore category |
| --original-tags | Keep original tag names (disable vulva→pussy rewrite for diffusion) |
Display Order
There are two category orderings:
Copy Action (e621 format)
Tags are presented in e621 category order when copying to clipboard. Categories 100-111 (fine-grained General sub-categories) are merged into General:
- Copyright
- Character
- Species
- Meta
- General (includes tags from IDs 100-111)
- Lore
Site Display (Web UI)
The web interface displays all 21 categories separately for better visual organization. Body Color and Lore are placed at the bottom:
- General
- Artist
- Contributor
- Copyright
- Character
- Species
- Invalid
- Meta
- Accessories, Items, Clothing
- Actions, Positions, State
- Body Features
- Effects, Fluids
- Fetishes, Specifics, Interactions
- Genders, Demographics
- Locations, Backgrounds, Setting
- Poses, Scenarios, Situations
- Style, Perspective
- Text, Symbols, UI, Vocalization
- Other
- Body Color
- Lore
Within each category, tags are sorted by probability (highest first).
CSV Output Format
When using batch processing with CSV output:
filename,female,anthro,furry,solo,...
image1.png,0.95,0.89,0.87,...
image2.png,0.91,0.78,0.65,...
Frontend Tag Display
Visual Categories
| Tag State | Background | Text | Condition |
|---|---|---|---|
| Confident | #8b4aff (purple) | White | prob >= confidentThreshold |
| Valid | #3c6b8f (blue) | White | prob >= allThreshold |
| Low | #e0e5ec | #5a6b7c | prob < allThreshold |
| Added | #2e9a5c (green) | White | User manually added |
| Removed | #c53030 (red) | White | User manually removed |
Tag Interaction
- Click: Toggle include/exclude (green/red)
- Long press/Right-click: Show e621 wiki description
Category Header Colors
| Category | Color |
|---|---|
| Safe | #1f9d55 |
| Questionable | #d69e2e |
| Explicit | #c53030 |
These are rendered in the rating display based on tag presence.
Metadata File Format
The tag metadata CSV files contain:
tag,category,implications
female,0,
male,0,
futanari,0,female male
intersex,0,female male
anthro,5,
Fields
| Field | Description |
|---|---|
tag |
Tag name |
category |
Category ID (0-8, 100-111) |
implications |
Space-separated list of implied tags |
Tag Count Limits
Top-K Selection
The API returns a configurable number of tags:
| top_k | Description |
|---|---|
| 50 | Minimal tags |
| 75 | Few tags |
| 100 | Standard |
| 150 | Extended |
| 200 | Default |
| 300 | Comprehensive |
Implementation
# app.py
DEFAULT_TOP_K = 200
ALLOWED_TOP_K = {50, 75, 100, 150, 200, 300}
# In predict route:
probs = torch.sigmoid(logits[0].float()).cpu()
values, indices = probs.topk(top_k)
Rating Tags
e621 uses specific tags for content rating:
| Tag | Meaning |
|---|---|
safe |
Acceptable for all audiences |
questionable |
May be unsuitable for some |
explicit |
pornographic content |
These are detected from the model predictions, not auto-generated.
Integration Points
API Response Building
# app.py:462-474
for idx, val in zip(indices, values):
label = model.labels[int(idx.item())]
prob = val.item()
if label.subcategory and label.subcategory in SUBCATEGORY_DISPLAY_NAMES:
category_name = SUBCATEGORY_DISPLAY_NAMES[label.subcategory]
else:
category_name = TAG_CATEGORIES.get(label.category, label.category.title())
tags_with_probs.append({
"tag": label.label,
"prob": prob,
"category": category_name
})
Frontend Rendering
See JS.md for frontend display logic.
Category Hiding/Exclusion
Users can hide entire tag categories (e.g., Fetishes, Body Color) to reduce visual clutter and exclude them from copy output.
UI Interaction
Each .category-block renders a faint ✕ toggle button in the top-right of its header area:
| Action | Control | Effect |
|---|---|---|
| Hide | Click ✕ | Gray out category (opacity 0.35), disable tag interaction, exclude from copy |
| Show | Click ✓ (formerly ✕) | Restore full opacity, re-enable interaction, include in copy |
| Persist | Click "Always hide" text (appears when hidden) | Save preference to localStorage, survives page reload |
| Unpersist | Click "Always hide" again | Remove localStorage preference |
Behavior
- Hidden categories: The
.category-blockgets.category-hiddenclass → opacity 0.35. All.tagchildren getpointer-events: none, making them non-interactive. - Copy exclusion: Hidden categories are filtered out in
filterTags()before clipboard copy. - Per-upload reset:
hiddenCategories(session-only) is cleared on each new upload.alwaysHiddenCategories(persistent) is re-applied from localStorage. - Manual tag corrections (
addedTags/removedTags) are preserved across hide/unhide operations. - Always-hidden reordering: Categories in
alwaysHiddenCategoriesare sorted to the end of the display, keeping them out of the way. Removing the "always hide" preference restores their original position.
State Management
| Variable | Type | Scope | Persistence |
|---|---|---|---|
hiddenCategories |
Set |
Session (per upload) | Not persisted |
alwaysHiddenCategories |
Set |
Global | localStorage via e621tagger-settings |
localStorage
Stored as alwaysHiddenCategories array inside the e621tagger-settings JSON object:
{
"alwaysHiddenCategories": ["Fetishes, Specifics, Interactions", "Body Color"],
"allThreshold": 0.60,
...
}
Reset
The Reset button in settings clears both alwaysHiddenCategories and removes it from localStorage.
Server-Side Tag Blocking
Tags can be excluded at the API level via the BLOCKED_TAGS set in app.py. This prevents tags from ever being returned to clients, unlike frontend hiding which only affects display.
Configuration
# app.py
BLOCKED_TAGS = {'mature'} # Add more tags as needed
Behavior
- Complete exclusion: Blocked tags are filtered during
/predictresponse building — they never appear in the JSON response - No frontend workaround: Clients cannot access blocked tags since they're not in the response
- Easy maintenance: Add/remove tags from the set without code changes elsewhere
- Precedence: Runs before threshold filtering and top-k selection
Use Cases
- Deprecated tags (e.g.,
matureon e621) - Policy-violating tags
- Internal/utility tags not meant for public API
LocalStorage Key
None — this is server-side only. No client-side configuration.