- Remove .agents/context/ from git index (already in .gitignore, files kept locally) - Correct local-plugins/ from "sandbox" to canonical community plugin repo clone (silvertakana/wwv-plugins) - Correct local-seeders/community/ and local-seeders/private/ as independent git clones with own remotes - Remove deprecated worldwideview-plugins repo references; rename wwv-seeders-community → wwv-seeders throughout - Add nested-git-clone invariant to AGENTS.md, ecosystem-repositories.md, directory-structure.md, monorepo-workflow.md, plugin-architecture.md, data-engine-architecture.md, migrate-legacy-plugin skill Co-Authored-By: Claude Opus 4.7 <[email protected]>
6.1 KiB
description, paths
| description | paths | |||
|---|---|---|---|---|
| Architecture rules and dependency management guidelines for wwv-data-engine seeders, pnpm workspace dependency loading, and WebSocket payload contracts. |
|
Data Engine V2 Architecture & Dependency Loading
1. Engine Versions: V2 vs Legacy
The current and only active engine is wwv-data-engine-v2.
- Legacy Engine: Previously, data sources ran as isolated, standalone microservices running their own Node environments.
- V2 Engine: Acts as a unified "Host Environment" runner. It is a single Node.js process that dynamically downloads and executes multiple independent seeder bundles (both public community seeders and private seeders).
2. Deployment Architecture
- Single Docker Image: V2 is deployed via Coolify as a single Docker image, natively mapping environment variables continuously into the container shell.
- Automated Restarts: Code pushes to the
wwv-data-engine-publicrepository automatically trigger a service restart via a secureCOOLIFY_API_TOKENconfigured in the CI/CD pipeline (replacing unreliable generic webhooks). - Dual-Output Engine: Seeders in V2 expose both a WebSocket stream (
/stream) for real-time instantaneous updates, and a REST API endpoint (/api/:id) for fetching live data snapshots directly from Redis.
3. Pnpm Workspaces & Dynamic Dependency Loading
To prevent seeders from ballooning in size by bundling a massive amount of dependencies, the system heavily relies on pnpm workspaces.
- Host Environment: The V2 engine natively provides common geospatial and utility packages (e.g.,
zod,ws,node-cron,undici,satellite.js,geoip-lite). Seeders MUST NOT bundle these standard dependencies, and should leave them external intsup. - Dynamic Workspaces: When V2 loads a compiled
dist/index.mjsvianode, it dynamically resolves imports by traversing up to the workspacenode_modules.
4. Production Deployment & Downloader
In production, the wwv-data-engine-v2 does NOT mount the local file system. Instead:
download-seeders.tsruns on container startup.- It fetches the latest compiled
seeders.zipfrom GitHub Releases for both community and private repositories. - CRITICAL STEP: After unzipping the workspaces into a staging directory (
/app/seeders), the script dynamically generates a rootpackage.jsonandpnpm-workspace.yaml. - It then runs a single workspace-aware
pnpm install --prod. This ensures all custom unbundled dependencies declared by the extracted plugins are downloaded directly into the container and properly linked together.
5. Namespace Separation Rule
To prevent collisions and 404 or ERR_MODULE_NOT_FOUND errors, seeders MUST NOT exist simultaneously in both the community (wwv-seeders, cloned to local-seeders/community/) and private (wwv-seeders-private, cloned to local-seeders/private/) repositories. Namespace overlaps will cause module resolution failures when the V2 engine attempts to load them. Private seeders have priority.
6. Plugin ID Contract — Single Source of Truth
See ADR-0002 (
docs/architecture/decisions/adr-0002-seeder-exported-name-as-canonical-plugin-id.md) for full rationale and alternatives considered.
Rule: Every seeder MUST export a name field that exactly matches the corresponding frontend plugin's id field (kebab-case).
// ✅ Correct — seeder src/index.ts
export default {
name: "gps-jamming", // MUST equal the frontend plugin id
cron: "0 0 * * *",
fn: seedGpsJam,
};
// ❌ Wrong — do not use folder names, camelCase, or abbreviations
export default {
name: "gpsjam", // mismatch — frontend plugin is "gps-jamming"
...
};
How the engine uses it: seeder-loader.ts reads each seeder's exported name as the canonical plugin ID. This ID is used for:
- The
/manifestendpoint'spluginsarray (what the frontend checks for local routing) - WebSocket data messages (
pluginIdfield) - Redis keys (
data:<name>:live) - Scheduler logs
Folder names are organizational only. The engine warns at startup if toKebabCase(folderName) !== seeder.name, but this is cosmetic — the seeder's exported name always wins. Folder renames are a future cleanup, not a functional requirement.
Frontend side: resolveEngineUrl(pluginId) checks localEngineHasPlugin(pluginId) against the manifest. If the names match, the plugin routes to ws://localhost:5000/stream. No alias maps or translation layers are needed.
7. WebSocket Payload Format Contract
When a seeder calls setLiveSnapshot(pluginId, payload, ttl), the engine broadcasts:
{ "type": "data", "pluginId": "my-plugin", "payload": <whatever you passed> }
WsClient payload handling rules (in order):
- Plugin has
mapWebsocketPayload(payload, existingEntities)→ called with raw payload → must returnGeoEntity[] - No handler + payload is a flat
GeoEntity[]array → used directly (timestamp normalized) - No handler + payload is any other object → silently dropped with a console warning
This means:
| Seeder sends | Frontend needs | Notes |
|---|---|---|
[{id, latitude, longitude, ...}] |
Nothing (WsClient handles) | ✅ Simplest for basic point data |
{ items: [{...}] } |
mapWebsocketPayload |
Standard engine snapshot format |
{ satellites: [{...}] } |
mapWebsocketPayload |
Named-collection format |
Recommendation: If the seeder and frontend plugin are developed together, use a flat GeoEntity[] array — no mapWebsocketPayload needed. Use named objects (e.g. { satellites: [...] }) only when the seeder data has a domain-specific shape worth preserving; in that case you must implement mapWebsocketPayload on the frontend plugin.
Caution
The most common silent bug: seeder sends
{ items: [...] }(an object), frontend plugin omitsmapWebsocketPayload. Data arrives but WsClient drops it with a console warning. The globe stays empty. Always implementmapWebsocketPayloadfor any plugin whose seeder sends an object payload.