Compare commits

..
706 changed files with 924295 additions and 17256 deletions

No files matched your search

-169
View File
@@ -1,169 +0,0 @@
---
trigger: model_decision
description: Architecture rules and conventions for the WorldwideView Next.js 16 frontend application, state management, and CesiumJS integration.
---
# WorldwideView Application Architecture
## 1. Core Framework
- Built on **Next.js 16 (App Router)** with `output: "standalone"`.
- Uses **React 19** and **TypeScript 5** (strict mode).
- **Styling**: Strictly Vanilla CSS. No Tailwind or CSS-in-JS.
- **Component File Size**: Max 150 lines per file. Split components, extract helpers, and use hooks for anything larger.
## 2. State & Event Bus
- **Zustand**: Slices handle global state (`globe`, `layers`, `timeline`, `ui`, `filter`, `data`, `config`, `favorites`, `geojson`).
- **DataBus**: Custom typed pub/sub singleton (`DataBus.getInstance().emit()`) decoupling React components from background WebSocket streams.
## 3. Edition System
Features are controlled by `NEXT_PUBLIC_WWV_EDITION`:
- `local`: Self-hosted, full features, auth.
- `cloud`: Managed instance.
- `demo`: Public demo, no auth.
## Source Layout
```
worldwideview/
├── .agents/ # AI agent workflows, rules, memories, skills
├── docs/ # Public-facing documentation
├── artifacts/
│ └── context/ # ← THIS FOLDER — private source of truth (not committed)
├── packages/ # Plugin packages (workspaces)
├── public/ # Static assets (Cesium assets, GeoJSON data files)
├── scripts/ # Build/utility scripts (copy-cesium.mjs, scaffold-osm-plugin.mjs, etc.)
└── src/
├── app/ # Next.js App Router (layouts, pages, API routes)
├── components/ # React UI components (HUD, LayersPanel, Panels, Sidebar)
├── core/
│ ├── data/ # DataBus, PollingManager, CacheLayer
│ ├── globe/ # Cesium hooks, viewer lifecycle, primitive collections
│ └── plugins/ # Plugin registry, lifecycle, loaders
├── lib/ # Shared utilities, server-side logic
│ └── marketplace/ # Marketplace bridge (auth, token, CORS, repository)
└── plugins/ # Built-in feature plugins (aviation, maritime, borders, etc.)
```
---
## Module Breakdown
### `src/core/data/` — The Heartbeat
- **DataBus**: Decentralized event pipeline for all system actions. Plugins emit, components subscribe.
- **PollingManager**: Intelligent scheduler for external API calls with exponential backoff.
- **CacheLayer**: 2-stage persistent caching — In-Memory + IndexedDB (`worldwideview-cache`).
### `src/core/globe/` — Rendering Engine
- **Cesium Integration**: Low-level control over the Cesium Viewer and Primitive Collections.
- **EntityRenderer**: High-performance Primitive-based renderer. See `Primitives vs. Entities` below.
- **Spiderifier & Clustering Engine** (`StackManager.ts` & `stackAnimation.ts`):
- Automatically identifies overlapping entities and groups them into geometrically balanced clusters (concentric rings for <18 density, Fermat's golden spiral for massive stacks).
- **Mobile Layout Constraints**: Integrates responsive coordinate geometry. Modifies logical icon dimensions via `iconUpscaler.ts` (32px mobile / 48px desktop) and scales the radial expansion offsets proportionally by 70% on mobile displays to ensure tight, aesthetically pleasing clusters without massive gaps.
- **Z-Fighting Mitigation**: Clustering involves highly frequent stack rebuild cycles on any zoom/pan delta. To prevent chaotic render flickering, the global `AnimationLoop` explicitly yields visibility management to `stackAnimation.ts` via `isEntityClustered()`. Items transitioning into the `collapsed` state intentionally bypass animation lerping (to prevent 220ms ghost visibility bleed) and mathematically cloak themselves instantly.
- Globe is initialized via `GlobeView.tsx`. Viewer instance passed via `CesiumContext`.
### `src/core/plugins/` — Plugin System
- **PluginManager**: Registers, boots, and destroys plugins. Enforces capability checks.
- **Loaders**: Plugins are dynamically imported at runtime as ES module bundles (`loadPluginFromManifest`). The legacy `StaticDataPlugin` and `DeclarativePlugin` runtimes are fully deprecated.
- **Manifest Requirement**: All plugins MUST include a `"worldwideview"` manifest block in their `package.json` (containing `id`, `icon`, `category`, and `format`). This is strictly required by the Marketplace in order to parse display attributes correctly.
---
## Performance: Primitives vs. Entities
**Core design decision:** Use Cesium `Primitive` API instead of the high-level `Entity` API for high-count datasets.
| Feature | Entity API | Primitive API (WWV) |
|---|---|---|
| **Abstraction** | High (easy to use) | Low (direct GPU access) |
| **JS Overhead** | Significant per-entity objects | Minimal (batched draw calls) |
| **Max Capacity** | ~1,000 entities | **100,000+ points/billboards** |
| **WWV Use Case** | Info window / detail panel content | Live data point visualization |
Specifically: `PointPrimitiveCollection` and `BillboardCollection` ensure 60FPS even with dense global data. The full migration from Entity-based rendering to Primitives was completed to eliminate severe FPS drops.
---
## Globe Occlusion & Depth Testing
- **Correct approach:** `depthTestAgainstTerrain = false` — uses GPU to depth-test against the smooth WGS84 ellipsoid. Icons visible on the near hemisphere, correctly occluded behind Earth.
- **Avoid:** `eyeOffset`, `disableDepthTestDistance` hacks that push entities in front of everything. These cause icons to bleed through the globe.
- The Plugin SDK's `CesiumEntityOptions` exposes this via `disableDepthTestDistance` when absolutely needed for specific usecases.
---
## Data Pipeline (Example: Aviation)
```
Remote API (e.g. OpenSky)
↓ [raw JSON, every ~30s via PollingManager]
Plugin (e.g. AviationPlugin)
↓ [parse, filter, sanitize → GeoEntity[]]
DataBus.emit("dataUpdated", { entities })
↓
EntityRenderer [batched Primitive update]
↓
GPU (CesiumJS) → 60FPS visualization
```
---
## Database (Unified PostgreSQL)
**Location:** We have migrated from a fragmented SQLite + PostgreSQL setup to a unified **PostgreSQL** architecture using Prisma 7 with the `@prisma/adapter-pg` driver. This enables true multi-tenant cloud capability, real-time sync, and scalable connection pooling via `DATABASE_URL`.
**Next.js Core Tables:**
| Model | Table | Purpose |
|---|---|---|
| `User` | `users` | Auth accounts (email, bcrypt password, role) |
| `InstalledPlugin` | `installed_plugins` | Marketplace plugin install records |
| `Setting` | `settings` | Key-value app config (Cesium token, map style, etc.) |
| `Tenant` | `tenants` | Multi-tenant organization boundaries |
**Data Engine Tables:**
Managed explicitly via optimized queries in `packages/wwv-data-engine/src/db.ts` targeting PostgreSQL to maximize ingest performance.
Includes tables like `iranwar_events`, `earthquakes`, `wildfires`, and `maritime_history`.
> [!WARNING]
> **Catastrophic Time-Series Polling (The Lookback Trap):**
> High-frequency polling (e.g. 6,000 aircraft every 10s) accumulates millions of rows rapidly.
> - **DO NOT** query unindexed columns (like `ts`) for lookback/freshness without a dedicated index, otherwise you will trigger full table scans and 500 Route timeout errors.
> - **REQUIRED:** Implement aggressive data retention / hourly pruning via `scheduler.ts` to prevent infinite table growth.
> - **PREFERRED:** Keep real-time / spatial lookbacks in Redis (like `getLiveSnapshot()`), and leave PostgreSQL strictly for historical chronological replays that hit standard indices.
**Hot-Caching (Redis) & Connection Rules:**
The `wwv-data-engine` polls external APIs, dumps raw payloads to PostgreSQL for historical persistence, and broadcasts active snapshots via WebSockets while persisting to Upstash Redis. The Next.js frontend fetches initial state from Redis to ensure horizontal scalability.
*TLS Connection Requirement:* The `REDIS_URL` mapped in Coolify and `.env.local` must use the `rediss://` protocol for TLS encryption. Attempting to use plain `redis://` with Upstash will cause the connection to drop (`ECONNRESET`) and throw a `MaxRetriesPerRequestError (which is 3)` inside the data engine.
*Upstash Request Limit Throttling:* Due to Upstash Serverless billing (capped at 10k requests/day or 500k/month on free tier), the data engine decouples WebSocket live broadcasting from Redis persistence. High-frequency scrapers (like Aviation) run every 15 seconds to push WebSockets, but the backend forces a **5-minute throttle** before saving the next snapshot to Upstash using `zlib` compression. This executes just 2 commands (`SET` and `EXPIRE`) per 5 minutes per scraper, completely eliminating `max requests limit exceeded` errors while retaining real-time frontend updates.
**Resilience (Data Self-Healing):**
Even with persistent PostgreSQL, critical history (like `iranwarlive` events) should be bundled as `seed.json` or `history.csv` artifacts. If the Data Engine detects an empty table, it MUST parse its built-in fallback seed to hydrate the database and Redis, ensuring 100% data recovery without manual intervention.
**Dynamic Engine URL Resolution:**
The decoupled Data Engine lives at `http://localhost:5001` during local development, and deploys to `https://dataengine.worldwideview.dev` in production. The Next.js frontend injects this context directly into the plugin via the `ctx.env` object (specifically reading from `NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL` environment variables). Plugins resolve their required backend natively by parsing `this.context.env.DATA_ENGINE_URL`. The legacy `globalThis.__WWV_ENGINE_URL__` proxy approach has been fully deprecated.
---
## Settings Architecture
User-configurable values live in the `settings` table (not `.env`). They can be updated without rebuilds.
| Setting Type | Where | Example |
|---|---|---|
| **Infrastructure** | `.env` | `DATABASE_URL`, `NEXT_PUBLIC_WWV_EDITION` |
| **User settings** | `settings` table (DB) | Cesium token, map style, API keys |
**Setup Wizard:** If `cesium_token` is missing from the DB on first load, the app shows a setup wizard. Plugin-level wizards work the same way — plugin declares required settings in its manifest; if missing, it renders a setup form instead of data.
---
## Design Principles
- **Single Responsibility:** Plugins only handle data mapping — they don't know about UI or cache.
- **Dependency Inversion:** `PluginManager` communicates via the `WorldPlugin` interface — new plugins don't require core changes.
- **Event-Driven:** Components subscribe to what they need via DataBus — no prop-drilling.
- **Defensive Programming:** All external calls wrapped in error boundaries + `PollingManager` backoff.
-34
View File
@@ -1,34 +0,0 @@
---
trigger: glob
description: Strict guidelines for rendering geospatial entities, handling 3D terrain clipping, and applying dynamic animation properties against Cesium's runtime engine.
globs: "src/core/globe/**/*"
---
# Cesium Rendering Rules
## Purpose
Strict guidelines for rendering geospatial entities, handling 3D terrain clipping, and applying dynamic animation properties against Cesium's runtime engine.
## When to Apply
Whenever writing code for plugins, visual layers, or modifying components like `EntityRenderer` or `GlobeView`.
## Critical Limitations
### `HeightReference.CLAMP_TO_GROUND` Warning
> [!WARNING]
> Do **not** apply `HeightReference.CLAMP_TO_GROUND` to label text rendering (`LabelGraphics`). Due to a known issue in Cesium 1.121+, clamping tens of thousands of dynamic labels blocks the UI thread continuously because Cesium recalculates the terrain intersection each frame.
### Primitive Restrictions
When creating primitive objects:
- **Points** (`PointPrimitive`): Can use `size`, `outlineWidth`, and `outlineColor`.
- **Billboards** (`BillboardGraphics` for SVG/Icons): Must **NOT** use `size`, `outlineWidth`, or `outlineColor`. You must use `iconScale` to resize them. Mixing point properties on billboards causes severe shader clipping.
## Performance Culling
> [!TIP]
> Do not rely solely on depth buffering (`disableDepthTestDistance`) when viewing the backside of the globe. Use CPU-side calculating (dot-product against the Earth's radius) to implement manual horizon culling. Render nothing if the scalar projection is behind the tangential threshold.
## LOD & Stacking
1. Convert distant entities to `billboard` representations.
2. If closely grouped, the `StackManager` should spiderify components circularly. Non-hub objects fade to 0.4 opacity when expanded to reduce visual clutter.
3. Use the `useModelRendering` hook exclusively for transitioning high-importance entities (like aircraft or ships) into full 3D GLTF models on approach.
-217
View File
@@ -1,217 +0,0 @@
---
trigger: model_decision
description: Cloud edition implementation details, PostgreSQL Row-Level Security (RLS) isolation, cross-subdomain auth, and cryptographic license keys.
---
# WorldWideView — Cloud & Auth Architecture
> [!NOTE]
> **This file contains the master plan and advanced implementation details for the future Cloud edition.**
> It covers single-container multi-tenancy, PostgreSQL Row-Level Security (RLS) isolation, cross-subdomain auth, and cryptographic license keys. It is intended for AI reference when building multi-tenant or auth-related features.
---
## Auth Strategy
**Auth.js (NextAuth)** is the single auth API across all editions. `app.worldwideview.dev` owns **all auth UI** — login, registration, password reset. No other subdomain ever shows a login or registration form.
```typescript
// src/lib/auth.ts
const providers = isCloud
? [SupabaseProvider({ /* delegates to Supabase Auth (GoTrue) */ })]
: [Credentials({ /* local bcrypt + PostgreSQL */ })];
export const { auth, signIn, signOut } = NextAuth({ providers });
```
Application code is **identical** across editions — always calls `auth()`, `signIn()`, `signOut()`.
**One account for the entire ecosystem:**
- Cloud instance (`[user].app.worldwideview.dev`)
- Marketplace (browse, install, publish) — via SSO redirect
- Local instance Bridge API connection
**Marketplace never owns auth UI** — it redirects to `app.worldwideview.dev/login?redirect_to=marketplace...` for all sign-in.
---
## Registration & Auto-Provisioning
Registration: `app.worldwideview.dev/register`
```
1. User fills: email, password, username (= subdomain)
2. Account created (Supabase Auth)
3. Tenant row inserted instantly → [username].app.worldwideview.dev is live
4. User lands on their instance → Setup Wizard (Cesium token)
```
**Subdomain validation:** alphanumeric + hyphens, min 3 chars, not already taken.
**Reserved names:** `www`, `app`, `api`, `demo`, `marketplace`, `docs`, `status`, `admin`, `cdn`, `blog`, `changelog`
"Provisioning" is just inserting a database row — no container to spin up (multi-tenant shared app).
---
## Multi-Tenant Architecture
**One Docker container, RLS isolation.** All cloud tenants share one Next.js deployment. Data isolation enforced at the **PostgreSQL database level** via Row-Level Security:
```
nginx (wildcard *.app.worldwideview.dev)
→ reads subdomain → passes X-Tenant-ID header
→ single WWV Docker container (Next.js)
→ middleware reads X-Tenant-ID
→ sets app.tenant_id in DB connection
→ RLS scopes ALL queries to that tenant
```
```sql
ALTER TABLE installed_plugins ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON installed_plugins
USING (tenant_id = current_setting('app.tenant_id')::uuid);
```
PostgreSQL **physically refuses** to return other tenants' rows — even application bugs cannot leak data.
> Schema-per-tenant (stronger isolation) can be offered later for enterprise. The `tenant_id` design makes migration straightforward.
### Security Analysis
| Concern | Risk | Mitigation |
|---|---|---|
| Data bleed between tenants | 🔴 Critical | RLS enforced at PostgreSQL level |
| One tenant crashes the app | 🟡 Medium | Next.js request isolation. Rate limit per tenant at nginx |
| One tenant hammers CPU | 🟡 Medium | Rate limiting at nginx (req/min per subdomain) |
| Plugin code between tenants | 🟢 Low | Plugins run in the user's **browser**, not the server |
| Server crash from users | 🟢 Low | Users can't run arbitrary server code. Rate limits handle abuse |
---
## Storage & Database
| Feature | Local | Cloud |
|---|---|---|
| **Database queries** | Prisma → PostgreSQL | Prisma → PostgreSQL (Supabase) |
| **Auth** | None (single owner) | Supabase Auth (GoTrue) |
| **File storage** | Local filesystem (`data/`) | Supabase Storage |
| **Storage limit** | ∞ (your disk) | Quota enforced per tier |
**Prisma is write-once, deploy-anywhere** — the same queries and schema work on both databases. `DATABASE_URL` in `.env` is the only difference.
---
## Tier Matrix
| Deployment | `NEXT_PUBLIC_WWV_EDITION` | License Key | Users | Storage |
|---|---|---|---|---|
| **Demo** | `demo` | none | ∞ (anonymous, read-only) | N/A |
| Local Free | `local` | none | 1 (owner) | ∞ |
| Local Pro | `local` | Pro key | 1 (owner) | ∞ |
| Local Team | `local` | Team key | Unlimited | ∞ |
| Cloud Free | `cloud` | none | 3 | 500 MB |
| Cloud Pro | `cloud` | Pro key | 20 | 5 GB |
| Cloud Enterprise | `cloud` | Enterprise key | Unlimited | 50 GB+ |
### Cryptographic License Keys
Paid features gated by **RSA-signed JWTs** — users cannot forge them by flipping flags:
```typescript
// Marketplace signs (PRIVATE key — only you have this):
const licenseKey = jwt.sign(
{ tier: "pro", exp: "2027-01-01", org: "acme-corp" },
PRIVATE_KEY, { algorithm: "RS256" }
);
// WWV verifies (PUBLIC key — hardcoded in codebase):
const license = jwt.verify(process.env.WWV_LICENSE_KEY, PUBLIC_KEY);
// license.tier = "free" | "pro" | "team"
```
### Storage Quota (Cloud Only)
- **Counts:** Imported GeoJSON, plugin data history, uploaded assets
- **NOT counted:** Built-in plugin real-time data (transient)
- **Quota exceeded:** Upgrade prompt, reject new imports, allow read access
---
## Environment Variables
### Local `.env`
```env
NEXT_PUBLIC_WWV_EDITION=local
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/wwv?schema=public
```
### Cloud `.env`
```env
NEXT_PUBLIC_WWV_EDITION=cloud
DATABASE_URL=postgresql://...
NEXT_PUBLIC_SUPABASE_URL=https://[project].supabase.co
NEXT_PUBLIC_SUPABASE_ANON_KEY=eyJ...
SUPABASE_SERVICE_ROLE_KEY=eyJ...
WWV_TENANT_MODE=rls
WWV_STORAGE_PROVIDER=supabase
```
### Demo `.env`
```env
NEXT_PUBLIC_WWV_EDITION=demo
DATABASE_URL=postgresql://...
```
`.env` is **infrastructure only** — things the operator sets once. Everything user-configurable lives in the `settings` DB table.
---
## Edition Adapters
Three thin files are all that differs between editions — everything else is edition-unaware:
| Adapter | Local | Cloud | Demo |
|---|---|---|---|
| `src/core/auth.ts` | Auth.js `Credentials` (bcrypt + PostgreSQL) | Auth.js `@auth/supabase-adapter` | Disabled |
| `src/core/storage.ts` | `fs.writeFile()` to `data/` | Supabase Storage SDK | Read-only |
| `src/core/tenant.ts` | No-op (single tenant) | Sets `app.tenant_id` for RLS | No-op |
---
## Bot Prevention (Layered)
| Layer | What | Cost |
|---|---|---|
| **Email verification** | Supabase Auth built-in | Free |
| **Rate limiting** | 3 signups per IP per hour (nginx) | Free |
| **Cloudflare Turnstile** | Invisible CAPTCHA on signup | Free |
| **Cloudflare Bot Fight Mode** | Auto-enabled on free tier | Free |
---
## CI/CD Pipeline
```
Push to silvertakana/worldwideview
→ GitHub Actions: build Docker image
→ Push to ghcr.io/silvertakana/worldwideview:latest
→ Coolify webhook triggers
→ Rolling restart (zero downtime)
```
### `worldwideview-cloud/docker-compose.yml` (concept)
```yaml
services:
wwv-cloud:
image: ghcr.io/silvertakana/worldwideview:latest
environment:
NEXT_PUBLIC_WWV_EDITION: cloud
DATABASE_URL: ${SUPABASE_PG_URL}
SUPABASE_URL: ${SUPABASE_URL}
SUPABASE_ANON_KEY: ${SUPABASE_ANON_KEY}
labels:
- "traefik.http.routers.wwv.rule=HostRegexp(`{user:[a-z0-9-]+}.app.worldwideview.dev`)"
```
The cloud repo (`worldwideview-cloud`) holds **no application logic** — it is purely a thin deployment configuration layer that pulls versioned Docker images.
-21
View File
@@ -1,21 +0,0 @@
---
trigger: always_on
---
# Context and Memory Synchronization
## 1. Purpose
This specification ensures that any agent maintains persistent, accurate project context by pointing them directly to the Single Source of Truth for this repository.
## 2. Synchronization Target
The agent MUST orient itself by immediately reading:
1. `c:\dev\worldwideview\AGENTS.md` (The root index of all technical stack details)
2. `c:\dev\worldwideview\.agents\rules\platform-architecture.md` (The high-level platform goals)
If deep technical rules are required for a specific domain (e.g., UI state, Map Rendering, Monorepo management), the agent MUST consult the relevant file in `c:\dev\worldwideview\.agents\rules\`.
## 3. Reflecting Changes
If the architecture fundamentally changes (e.g., migrating off Zustand, or removing Cesium), the agent MUST proactively update the `AGENTS.md` and relevant files in `.agents/rules/` to reflect this new reality.
## 4. Goal
A new chat instance MUST be able to reconstruct an accurate, comprehensive understanding of the project solely from reading the root `AGENTS.md` file and branching out to the specific rules as necessary. This prevents context bloat and ensures accuracy.
-28
View File
@@ -1,28 +0,0 @@
---
trigger: always_on
description: Guidelines for ongoing project maintenance, cleanliness, and self-correction during everyday development tasks.
---
# Continuous Improvement & Maintenance
As an AI agent working within the WorldWideView project, you are expected to practice the **Boy Scout Rule**: always leave the codebase (and its context) cleaner than you found it.
While executing your primary programming tasks, you MUST actively abide by these maintenance protocols:
## 1. Documentation & Rule Parity
- **Self-Correction**: If you notice that an architectural choice you are implementing contradicts an existing rule in `.agents/rules/`, you must halt and update the rule file to reflect the new paradigm.
- **Dead Context**: If you encounter context, plans, or architectural files referencing deprecated concepts (e.g., legacy microservices, `StaticDataPlugin`), proactively prune or update them.
## 2. Codebase Cleanliness
- **Dead Code**: If you refactor a file and leave behind unused functions, imports, or CSS classes, clean them up immediately. Do not leave "commented out" code unless explicitly requested for future use.
- **Console Logs**: Remove debugging `console.log` statements before finalizing a feature, except for structured error logging or critical startup info.
- **Sandbox Hygiene**: Any throwaway scripts or debugging tools placed in `/local-scripts/` should be deleted once the underlying issue is resolved, unless they provide permanent utility.
## 3. Dependency & Typings Checks
- **Type Safety**: Never use `any` or `@ts-ignore` as a shortcut when refactoring. Always build the proper TypeScript interfaces.
- **Manifest Validation**: Whenever you touch a plugin, ensure its `package.json` `"worldwideview"` manifest block is completely accurate and synced with its capabilities.
## 4. The "See Something, Say Something" Rule
If you spot an unrelated but obvious bug, security vulnerability, or architectural anti-pattern in the file you are editing:
1. Fix it if it's a minor, isolated issue (and document the fix in your summary).
2. For larger issues, explicitly call it out to the user so it can be added to the technical debt backlog.
-31
View File
@@ -1,31 +0,0 @@
---
trigger: model_decision
description: Architecture rules and dependency management guidelines for wwv-data-engine seeders.
---
# Data Engine V2 Architecture & Dependency Loading
## 1. Engine Versions: V2 vs Legacy
The current and only active engine is **`wwv-data-engine-v2`**.
- **Legacy Engine**: Previously, data sources ran as isolated, standalone microservices running their own Node environments.
- **V2 Engine**: Acts as a unified **"Host Environment" runner**. It is a single Node.js process that dynamically downloads and executes multiple independent seeder bundles (both public community seeders and private seeders).
## 2. Deployment Architecture
- **Single Docker Image**: V2 is deployed via Coolify as a single Docker image, natively mapping environment variables continuously into the container shell.
- **Automated Restarts**: Code pushes to the `wwv-data-engine-public` repository automatically trigger a service restart via a secure `COOLIFY_API_TOKEN` configured in the CI/CD pipeline (replacing unreliable generic webhooks).
- **Dual-Output Engine**: Seeders in V2 expose both a WebSocket stream (`/stream`) for real-time instantaneous updates, and a REST API endpoint (`/api/:id`) for fetching live data snapshots directly from Redis.
## 3. Pnpm Workspaces & Dynamic Dependency Loading
To prevent seeders from ballooning in size by bundling a massive amount of dependencies, the system heavily relies on `pnpm` workspaces.
- **Host Environment**: The V2 engine natively provides common geospatial and utility packages (e.g., `zod`, `ws`, `node-cron`, `undici`, `satellite.js`, `geoip-lite`). Seeders **MUST NOT** bundle these standard dependencies, and should leave them external in `tsup`.
- **Dynamic Workspaces**: When V2 loads a compiled `dist/index.mjs` via `node`, it dynamically resolves imports by traversing up to the workspace `node_modules`.
## 4. Production Deployment & Downloader
In production, the `wwv-data-engine-v2` does NOT mount the local file system. Instead:
1. `download-seeders.ts` runs on container startup.
2. It fetches the latest compiled `seeders.zip` from GitHub Releases for both community and private repositories.
3. **CRITICAL STEP:** After unzipping the workspaces into a staging directory (`/app/seeders`), the script dynamically generates a root `package.json` and `pnpm-workspace.yaml`.
4. It then runs a single **workspace-aware `pnpm install --prod`**. This ensures all custom unbundled dependencies declared by the extracted plugins are downloaded directly into the container and properly linked together.
## 5. Namespace Separation Rule
To prevent collisions and `404` or `ERR_MODULE_NOT_FOUND` errors, seeders **MUST NOT** exist simultaneously in both the community (`wwv-seeders-community`) and private (`wwv-seeders-private`) repositories. Namespace overlaps will cause module resolution failures when the V2 engine attempts to load them. Private seeders have priority.
+3 -3
View File
@@ -4,8 +4,8 @@
When making database schema changes, you must ensure that generated migrations are resilient against environmental discrepancies.
- **Review Generated SQL**: Always review the `.sql` files generated by `npx prisma migrate dev`.
- **Add "IF EXISTS" clauses**: Prisma can sometimes produce brittle drop statements (`DROP INDEX "my_index"`). To prevent migrations from crashing if a state desynchronization occurs, manually edit the migration file to use `DROP INDEX IF EXISTS "my_index";` or `DROP TABLE IF EXISTS "table_name";`.
- **Data Preservation**: When dropping columns or renaming, carefully review the migration to ensure data is correctly mapped. In PostgreSQL, ensure that cascaded drops don't accidentally remove required data.
- **Add "IF EXISTS" clauses**: SQLite and Prisma can sometimes produce brittle drop statements (`DROP INDEX "my_index"`). To prevent migrations from crashing if a state desynchronization occurs (e.g., the index was already dropped or never created), manually edit the migration file to use `DROP INDEX IF EXISTS "my_index";` or `DROP TABLE IF EXISTS "table_name";`.
- **Data Preservation**: When dropping columns or renaming, SQLite does not cleanly support it and requires table recreation. Carefully review the `INSERT INTO "new_table" SELECT ...` clauses to ensure data is correctly mapped.
## 2. Managing Failed Migrations
If a migration fails in production:
@@ -13,4 +13,4 @@ If a migration fails in production:
- If you use `resolve --applied`, Prisma will permanently skip that migration in the future, leaving your database structurally out of sync with your Prisma Client.
## 3. Backups Before DDL
Never perform direct DDL operations (ALTER, DROP, CREATE) on a production database without first confirming an automated backup exists, or manually taking a `pg_dump` backup of the database.
Never perform direct DDL operations (ALTER, DROP, CREATE) on a production database (via Prisma or direct SQLite access) without first confirming an automated backup exists, or manually taking a backup of the `wwv.db` file.
-304
View File
@@ -1,304 +0,0 @@
# Deployment, Testing, and Operations
## Development Commands
```bash
pnpm install # Install all workspace dependencies
pnpm run setup # Generate .env.local with AUTH_SECRET (first-time setup)
pnpm dev # Next.js frontend only (auto-runs prisma db push + copy-cesium)
pnpm dev:all # Frontend + wwv-data-engine (via Docker Compose) concurrently
pnpm dev:backends # Starts the local data engine + Redis via Docker Compose
pnpm build # Production build
pnpm test # Run all Vitest tests (scoped to src/lib, src/core, src/plugins)
pnpm db:reset # Reset and re-migrate the frontend database (destructive)
pnpm start:backends # Legacy command for standalone Fastify backends
pnpm clean:backends # Wipe all plugin database records
pnpm run scaffold-osm-plugin <name> # Generate a new plugin from scaffold
pnpm dev:plugins # File watcher for local-plugins/ directory (runs automatically in dev)
node packages/wwv-cli/dist/index.js create <name> --local # Scaffold a new local plugin
node packages/wwv-cli/dist/index.js link <name> # Promote a local plugin to packages/
```
Frontend runs at `http://localhost:3000`.
## Deployment
- **Docker**: Multi-stage Dockerfile using the Extractor Pattern (`deps` → `builder` → `runner`). The `node_modules` folders must be explicitly untracked in from `git` across the workspace to prevent corrupted BuildKit contextual cache overlaps during the `COPY . .` stage.
- **Standalone output**: `next.config.ts` uses `output: "standalone"`.
- **Cesium assets**: Copied to `public/cesium/` via `scripts/copy-cesium.mjs` at build time, excluded from output tracing.
- **Prisma Configuration**: `prisma.config.ts` must export a native javascript object instead of dynamically importing CLI wrapper binaries (`prisma/config` or `dotenv`). The standalone Next.js tracer strips CLI devDependencies during the build, which will cause fatal runtime container crashes if imported.
- **Data Engine**: The single runner container defined in `docker-compose.yml`, proxied via `next.config.ts` rewrites.
- **Coolify**: Deployed via Dockerfile builder natively mapping environment variables continuously into the container shell.
- **Docker volumes**: Ensure PostgreSQL data and Redis volumes are mounted for persistence.
## Testing Strategy
- **Framework**: Vitest with jsdom environment
- **Coverage**: `src/lib/**`, `src/core/**`, `src/plugins/**`
- **Run**: `pnpm test` (or `vitest run`)
- **Key test files**: `rateLimit.test.ts`, `edition.test.ts`, `demoAdmin.test.ts`, `DeclarativePlugin.test.ts`, `cors.test.ts`, `repository.test.ts`, `marketplaceToken.test.ts`
## Security Headers
Configured in `next.config.ts` `headers()`:
- **CSP**: Restrictive with exceptions for CesiumJS (`unsafe-eval`, `unsafe-inline`), camera streams (`http: https:`), and analytics
- **X-Frame-Options**: DENY
- **X-Content-Type-Options**: nosniff
- **Referrer-Policy**: strict-origin-when-cross-origin
- **Permissions-Policy**: camera/microphone disabled, geolocation self-only
---
## Package Manager
The project uses **pnpm workspaces**. All `npm` commands in older docs/scripts should be treated as `pnpm`.
```bash
pnpm install # install all workspace deps
pnpm run dev # start dev server ONLY (worldwideview/)
pnpm run dev:all # start ALL services concurrently (worldwideview, data-engine, marketplace)
pnpm run build # production build
```
---
## Local Dev Setup
```
1. Clone repo
2. pnpm install
3. pnpm run setup # generates .env.local with AUTH_SECRET, copies .env.example
4. pnpm run dev:all # predev: prisma migrate deploy + copy-cesium.mjs → then starts all services
5. Open http://localhost:3000
```
**Marketplace:** `cd worldwideview-marketplace && pnpm run dev` → http://localhost:3001
**Database auto-creates** on first run via `predev` script — no manual DB setup required.
---
## Database Operations
**All SQL from `c:\dev\worldwideview`:**
```powershell
npx prisma db execute --sql "<SQL>"
```
> Stop the dev server first to avoid "database is locked" errors.
### Common Commands
```powershell
# View row counts (all tables)
npx prisma db execute --sql "SELECT 'users' AS t, COUNT(*) AS n FROM users UNION ALL SELECT 'aviation_history', COUNT(*) FROM aviation_history UNION ALL SELECT 'installed_plugins', COUNT(*) FROM installed_plugins UNION ALL SELECT 'settings', COUNT(*) FROM settings;"
# List installed plugins
npx prisma db execute --sql "SELECT pluginId, version, installedAt FROM installed_plugins ORDER BY installedAt DESC;"
# Clear all installed plugins
npx prisma db execute --sql "DELETE FROM installed_plugins;"
# List settings
npx prisma db execute --sql "SELECT key, value FROM settings;"
# Set/update a setting
npx prisma db execute --sql "INSERT INTO settings (key, value) VALUES ('<key>', '<value>') ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value;"
# Prune aviation history older than 7 days
npx prisma db execute --sql "DELETE FROM aviation_history WHERE timestamp < NOW() - INTERVAL '7 days';"
# Vacuum (reclaim disk space after pruning)
npx prisma db execute --sql "VACUUM FULL;"
```
---
## Data Engine & Data Recovery
The secondary backend (Data Engine) handles high-volume API ingestion and pushes active snapshots to Upstash Redis. Because its PostgreSQL database is completely wiped on fresh Coolify deployments if volumes aren't preserved, it uses **Self-Healing Architecture**.
**How Recovery Works:**
1. Upon startup, or during the 60-second cron cycle, the seeder checks if the target table (e.g., `iranwar_events`) is empty.
2. If empty, the engine instantly hydrates the PostgreSQL database and the Upstash Redis cache from the committed fallback payload (e.g., `data/fallback/iranwar_seed.json`).
3. To **force a rapid recovery** locally (such as when testing new scraper data), simply wipe the table and let the minute-tick refill it via `npx prisma db execute --sql "DELETE FROM iranwar_events;"`.
**OSINT Scraping & Hydration Best Practices:**
Utility scripts (like `hydrate.ts`) exist to repair or enrich the fallback seed block offline. Important takeaways from building these scrapers:
- **Location Mapping:** Always append structured `_osint_meta.coordinates: {lat, lng}`. Without exact coordinates, the SDK defaults to Null Island or scatters points uniformly across the globe (declutter grid effect).
- **Tracking URL Resolution:** News aggregators (like Bing News RSS) wrap URLs in tracking redirect pages (`apiclick.aspx`). Never serve these directly to the frontend. Use `cheerio` to fetch the tracking page, parse the `<noscript><meta refresh="URL=..."></noscript>` node, extract the *actual* news URL, and then scrape the destination article for `og:image` and `og:video` tags.
- **Frontend Sync:** Ensure Next.js schema mapping in the plugin (`wwv-plugin-xyz/src/index.ts`) perfectly mirrors the enriched fields so they render correctly in the UI.
### Schema Changes
```powershell
npx prisma db push # sync schema changes to DB (non-destructive)
npx prisma migrate dev --name "<description>" # create + apply a migration
npx prisma generate # regenerate Prisma client types after schema change
npx prisma studio # visual DB browser at http://localhost:5555
```
### Full Reset
```powershell
# Stop dev server first
Remove-Item c:\dev\worldwideview\data\wwv.db
npx prisma db push
pnpm run dev
```
### Browser Cache Clearing
```javascript
// Clear IndexedDB entity cache
indexedDB.deleteDatabase("worldwideview-cache");
// Clear localStorage user API keys
Object.keys(localStorage)
.filter(k => k.startsWith("wwv_"))
.forEach(k => localStorage.removeItem(k));
```
Known `wwv_*` localStorage keys (defined in `src/lib/userApiKeys.ts`):
- `wwv_key_google_maps`
- `wwv_key_nasa_firms`
Hard refresh: `Ctrl+Shift+R`
---
## DB Troubleshooting
| Error | Fix |
|---|---|
| "Database is locked" | Stop the dev server first |
| "Table does not exist" | Run `npx prisma db push` |
| "Plugin still shows after clearing" | Hard-refresh (`Ctrl+Shift+R`) or clear IndexedDB |
| DB file is very large | Prune aviation history + run `VACUUM` |
| Prisma client types are stale | Run `npx prisma generate` |
---
## Creating Static OSM Plugins
See full guide: `.agents/skills/osm-static-plugin-creation.md`
**Quick start:**
```bash
node scripts/scaffold-osm-plugin.mjs '{
"name": "volcanoes",
"displayName": "Volcanoes",
"osmTag": "natural=volcano",
"icon": "Mountain",
"color": "#ef4444",
"category": "Natural Disaster"
}'
```
**Manual Overpass API query (PowerShell):**
```powershell
$tag = '"natural"="volcano"'
$body = "data=[out:json][timeout:300];(node[$tag];way[$tag];relation[$tag];);out center;"
Invoke-RestMethod -Uri 'https://overpass-api.de/api/interpreter' -Method Post -Body $body -OutFile 'tmp/raw_data.json'
```
**Note:** OSM data is licensed under **ODbL** — include attribution in plugin READMEs.
---
## npm Plugin Publishing
See full guide: `.agents/skills/version-control/npm-plugin-publishing.md`
```bash
# First-time publish
pnpm publish --workspace=packages/wwv-plugin-<name> --access public
# Future versions — via Trusted Publishing (GitHub Actions CI/CD)
```
---
## Git / Version Control
See full guide: `.agents/skills/version-control/git-version-control.md`
**Commit convention:** `feat:`, `fix:`, `chore:`, `docs:` prefixes.
**Commit when:** A new feature is completed (per project rules).
---
## Security Checks
See full guide: `.agents/skills/security/security-check.md`
---
## Deployment
**Host Server:**
The production Coolify instance runs on a local server at `192.168.68.64`. Direct SSH access: `ssh [email protected]`.
**CI/CD Flow:**
1. Push to `silvertakana/worldwideview`
2. Coolify webhook triggers → automated BuildKit deployment on the host server.
**Coolify Deployment Rules & Gotchas (The "Redis 500k" Rule):**
- **Coolify Pulls from GitHub:** Coolify does *not* deploy your local workspace files. It strictly clones the repository from GitHub.
- **Push Before You Deploy:** If you make a critical infrastructure or optimization fix locally (e.g., swapping a massive `HSET` pipeline for a compressed 5-min throttled `setLiveSnapshot`), you **must** `git commit` and `git push` those changes to the `main` branch before triggering a Coolify deployment.
- **The "Rollback DDOS" Risk:** If you update environmental variables (like a new Upstash Redis URL) inside Coolify and deploy *without* pushing your local optimized code to GitHub, Coolify will pull the *older, unoptimized* repository code and aim it at your new endpoints. This will immediately exhaust the 500,000 requests/month Upstash free limit.
- **Redis Connection Errors:** If the data engine crashes on startup with `MaxRetriesPerRequestError: Reached the max retries per request limit (which is 3)` and `ECONNRESET`, the `REDIS_URL` in your Coolify environment variables is incorrectly using `redis://` instead of the TLS-required `rediss://`.
- **Immediate Rollback on Failure:** Whenever a Coolify deployment fails or introduces a critical bug (such as rate-limit exhaustion), **immediately select the last known working deployment in the Coolify dashboard and redeploy it.** Do not leave a broken deployment hanging while trying to debug directly in production. Roll back to stability first, debug locally, commit the verified fix, and then deploy forward.
**Local verification:**
Before pushing infrastructure changes, run `docker build -t wwv-test .` to verify that the Next.js `standalone` isolated multi-stage build works correctly.
**Critical Docker/BuildKit Patterns:**
- **No Tracked `node_modules`:** Never track `node_modules` in git. It will severely break the Docker `COPY . .` BuildKit cache stage, as it attempts to overwrite the internally symlinked modules from the `deps` stage with the non-symlinked folders from git, causing cryptic `cannot copy to non-directory` errors.
- **Prisma 7 in Standalone:** Prisma v7 strictly forbids `url = env("DATABASE_URL")` in `schema.prisma` if `prisma.config.ts` exists. You *must* provide the `url` property in `prisma.config.ts`.
- **The `dotenv` Paradox:** Next.js `standalone` output does NOT automatically trace CLI tool devDependencies (like `dotenv`). If you import `dotenv` in `prisma.config.ts`, the production runner container will crash in a reboot loop. *Solution:* `prisma.config.ts` must use native Node `fs` and `path` to gracefully read from `.env` as a fallback, exporting a plain Javascript object without CLI wrappers.
- **Standalone Scripts & Next.js Typechecking:** Next.js `pnpm build` triggers `tsc`. By default, this will aggressively typecheck the `scripts/` directory. One-off migration scripts that import backend-only dependencies (like `pg`) will fail the global build. Always ensure `"scripts"` is added to the `"exclude"` array in `tsconfig.json`.
- **WSL2 / Docker Memory Bloat:** Next.js `standalone` + `pnpm workspaces` caching takes roughly 7GB to 10GB of virtual disk space and maxes out WSL2 memory during `docker-compose build`. Avoid building complex monorepos locally unless necessary. To reclaim local BuildKit disk space: `docker builder prune -a -f`. To reclaim WSL RAM: `wsl --shutdown`.
- **TypeScript NodeNext ESM Extensions:** By default in strict NodeNext ESM, backend scripts compiled via TypeScript (like `server.ts`) require dynamic imports to explicitly list the extension. You MUST write `await import('./routes/index.js')`, even if the underlying code is a `.ts` file, or the Docker compilation will fail with `TS2834`.
- **Git vs Docker Context Bloat:** Ensure `.dockerignore` ruthlessly excludes generated artifacts (e.g., `artifacts/`, `performance/`). Ensure `.gitignore` strips large binary caches like heavy database seed files inside `packages/*/data/` to prevent GitHub from bouncing commits exceeding the 100MB chunk limit.
- **Host Resource Exhaustion & Implicit Failures:** Complex multi-stage pnpm builds generate massive amounts of intermediate layer cache. If the Coolify host server reaches 100% disk capacity during an active deployment, the build logs will silently truncate and die abruptly with no explicit `pnpm` error. Worse, services like PostgreSQL (`coolify-db`) will instantly crash because they cannot write lock files, causing total system outages. Always maintain at least 30-40GB of free space on deployment hosts before auto-deploying via Coolify.
**Local update:** `git pull && pnpm install && pnpm run dev`
---
## Disk Space Management & Monitoring
The Docker build uses `standalone` output to minimize image size. The `copy-cesium.mjs` script copies only the required Cesium assets into `public/` — avoids shipping the entire Cesium package (300MB+) in the build.
**Host Server Maintenance:**
On Coolify deployment targets (e.g., `192.168.68.64`), dangling BuildKit caches and orphaned images will eventually consume 100% of the disk.
- To recover a crashed server: SSH into the host and run `docker system prune -a -f`.
- **Proactive Monitoring:** Use the `netdata-remote` MCP server to poll or setup alerts for the host disk space. It is highly recommended to set a warning alert when disk usage exceeds 80%.
**Application Database Growth:**
Aviation history can grow large. Prune regularly:
```powershell
npx prisma db execute --sql "DELETE FROM aviation_history WHERE timestamp < datetime('now', '-7 days');"
npx prisma db execute --sql "VACUUM;"
```
---
## Terminology Quick Reference
| Term | Definition |
|---|---|
| **Edition** | `local`, `cloud`, or `demo` — set via `NEXT_PUBLIC_WWV_EDITION`. Drives auth/storage/tenant adapters |
| **Bridge API** | Routes in WWV (`/api/marketplace/*`) that the marketplace calls to install/manage plugins |
| **Setup Wizard** | First-run config UI — app-level (Cesium token) and plugin-level (plugin API keys) |
| **Runtime Settings Store** | `settings` DB table — replaces `.env` for everything user-facing |
| **RLS** | Row-Level Security — PostgreSQL feature enforcing tenant data isolation at the DB level |
| **Snapshot Capture** | WWV periodically stores plugin data per tenant for history/timeline. Pro/Local only |
| **Trust Stamping** | WWV always overwrites `manifest.trust` server-side — no client claims are accepted |
| **NpmCache** | DB table caching NPM registry metadata to decouple from live NPM polling |
-69
View File
@@ -1,69 +0,0 @@
# Directory Structure & Repositories
## Directory Structure
```
worldwideview/
├── src/
│ ├── app/ # Next.js App Router (pages, API routes, layouts)
│ │ ├── api/ # Server-side API routes (auth, aviation, camera, etc.)
│ │ ├── login/ # Login page
│ │ ├── setup/ # First-time setup page
│ │ └── globals.css # Root stylesheet
│ ├── components/
│ │ ├── common/ # Shared UI: BootOverlay, FloatingWindow, PluginIcon
│ │ ├── layout/ # AppShell, Header, SearchBar, DataBusSubscriber
│ │ ├── panels/ # LayerPanel, EntityInfoCard, FilterPanel, GraphicsSettings
│ │ ├── timeline/ # Timeline component
│ │ ├── marketplace/ # Plugin install/unverified dialogs
│ │ ├── ui/ # Tooltip, ReloadToast
│ │ └── video/ # Floating video manager
│ ├── core/
│ │ ├── data/ # DataBus, PollingManager, CacheLayer, SmartFetcher
│ │ ├── filters/ # filterEngine (applies plugin filters to entities)
│ │ ├── globe/ # GlobeView, EntityRenderer, AnimationLoop, StackManager,
│ │ │ │ CameraController, InteractionHandler, SelectionHandler,
│ │ │ │ ModelManager, ImageryProviderFactory
│ │ │ └── hooks/ # useCameraActions, useEntityRendering, useModelRendering, etc.
│ │ ├── hooks/ # useBootSequence, useIsMobile, useMarketplaceSync
│ │ ├── plugins/ # PluginManager, PluginRegistry, PluginManifest,
│ │ │ │ loadPluginFromManifest, validateManifest, InstalledPluginsLoader
│ │ │ └── loaders/ # mapJsonToEntities (Legacy loaders like DeclarativePlugin are deprecated)
│ │ └── state/ # Zustand store + slices (config, data, globe, layers, timeline, ui, etc.)
│ ├── lib/ # auth, db, rateLimit, analytics, AIS stream, marketplace APIs
│ ├── plugins/ # GeoJSON plugin registrations
│ ├── styles/ # HUD animations CSS
│ ├── types/ # GeoJSON types, Umami types
│ └── generated/ # Prisma generated client (gitignored)
├── local-plugins/ # Local sandbox for developing plugins (gitignored)
├── packages/ # pnpm monorepo workspace packages
│ ├── wwv-plugin-sdk/ # Plugin SDK: type definitions, manifest schema
│ ├── wwv-plugin-aviation/
│ ├── wwv-plugin-maritime/
│ ├── wwv-plugin-wildfire/
│ ├── wwv-plugin-borders/
│ ├── wwv-plugin-camera/
│ ├── wwv-plugin-military-aviation/
│ ├── wwv-plugin-satellite/
│ ├── wwv-plugin-iranwarlive/ # Standalone plugin with custom endpoints
│ └── wwv-plugin-{airports,embassies,lighthouses,nuclear,seaports,spaceports,volcanoes}/
├── prisma/ # schema.prisma, migrations/
├── public/ # Static assets, Cesium workers, plugin GeoJSON data
├── scripts/ # Build scripts (copy-cesium, scaffold-osm-plugin, setup)
├── data/ # PostgreSQL data volume (gitignored)
├── Dockerfile # Multi-stage production build
├── docker-compose.yml # Main app + data engine runner + local seeders
└── .agents/ # Agent documentation, rules, skills, workflows
```
## Related Repositories
| Repo | Purpose |
|---|---|
| `worldwideview` | Main application (this repo) |
| `wwv-data-engine` | Generic data engine runner (PUBLIC, runs via Docker) |
| `wwv-seeders-community` | Open-source community seeders (PUBLIC) |
| `wwv-seeders-private` | Proprietary seeder scripts (PRIVATE, downloaded in prod) |
| `worldwideview-marketplace` | Plugin marketplace web app |
| `worldwideview-plugins` | Published npm plugin packages (frontend source) |
| `worldwideview-web` | Marketing / landing page |
-34
View File
@@ -1,34 +0,0 @@
---
trigger: model_decision
description: A comprehensive guide to the WorldWideView ecosystem repositories in the local C:\dev environment. Use this to determine which repository to edit based on the feature being developed.
---
# Ecosystem Repositories & Development Structure
The WorldWideView platform is not a single monolith; it is distributed across several repositories. In the local development environment, these repositories are organized as sibling directories under `C:\dev\`.
If you are asked to implement a feature or fix a bug, you **MUST** determine the correct repository to edit before writing code.
## Ecosystem Map (C:\dev\)
| Repository Folder | Purpose & When to Edit |
|---|---|
| **`worldwideview`** | The main application, Next.js frontend, Cesium globe, UI components, and the core plugin SDK. **Edit here** for anything related to the UI, state, map rendering, or built-in plugin frontend configurations. |
| **`wwv-data-engine-public`** | The V2 Data Engine (Host Environment) that runs the seeders. **Edit here** if you need to modify how the engine parses WebSocket streams, implements REST APIs, downloads seeder bundles, or handles Redis caching. |
| **`wwv-seeders`** | The community (public) data seeders. **Edit here** to create or modify background polling scripts (e.g. `aviation.ts`, `wildfire.ts`) that fetch data from public APIs and normalize it into `GeoEntity` arrays. |
| **`wwv-seeders-private`** | The proprietary (private) data seeders. **Edit here** for closed-source, premium, or high-security data ingestion scripts. |
| **`worldwideview-marketplace`** | The Plugin Marketplace web application. **Edit here** if you are modifying the platform where users browse, purchase, or manage plugin installations. |
| **`worldwideview-plugins`** | Independent repository for published npm plugin packages (typically frontend source bundles). |
| **`worldwideview-web`** | The marketing site, landing pages, and documentation. |
| **`wwv-test`** | A testing ground repository (e.g., for docker-compose setups, scratch testing). |
## Cross-Repository Workflows
1. **Adding a New Data Source:**
- Write the backend fetcher inside `C:\dev\wwv-seeders` (or `private`).
- Write the frontend plugin interface inside `C:\dev\worldwideview` (or dynamically loaded via `worldwideview-plugins`).
2. **Fixing Real-Time Streaming Issues:**
- Investigate `C:\dev\worldwideview` (for frontend `WsClient` or `DataBus`).
- Investigate `C:\dev\wwv-data-engine-public` (for backend Fastify websocket broadcasting).
3. **Pnpm Workspaces:**
- The Data Engine V2 automatically combines `community` and `private` seeders at runtime using pnpm workspaces. When testing seeders locally, refer to `C:\dev\worldwideview\local-seeders`.
-17
View File
@@ -1,17 +0,0 @@
# Environment Variables
| Variable | Required | Description |
|---|---|---|
| `DATABASE_URL` | Yes | PostgreSQL connection string |
| `AUTH_SECRET` | Yes | JWT signing secret (generate with `openssl rand -hex 32`) |
| `NEXT_PUBLIC_CESIUM_ION_TOKEN` | No | Cesium Ion access token |
| `NEXT_PUBLIC_BING_MAPS_KEY` | No | Bing Maps imagery |
| `NEXT_PUBLIC_GOOGLE_MAPS_API_KEY` | No | Google 3D Tiles |
| `NEXT_PUBLIC_WWV_EDITION` | No | `local` / `cloud` / `demo` (default: `local`) |
| `NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL` | No | Override engine WebSocket URL (default: cloud) |
| `OPENSKY_CREDENTIALS` | No | Comma-separated `id:secret` pairs for credential rotation |
| `WWV_BRIDGE_TOKEN` | No | Shared secret for marketplace → WWV install bridge |
| `WWV_DEMO_ADMIN_SECRET` | No | Demo edition admin password |
| `IRANWARLIVE_BACKEND_URL` | No | Override for IranWarLive custom backend URL |
Secrets go in `.env.local` (gitignored). Non-secrets go in `.env` (committed).
-222
View File
@@ -1,222 +0,0 @@
---
trigger: model_decision
description: Guidelines for the WorldwideView plugin marketplace, database interactions, and dynamic plugin installation.
---
# Marketplace Architecture
## 1. InstalledPluginsLoader
Marketplace plugins are dynamically loaded via `InstalledPluginsLoader`. At startup, this service queries the PostgreSQL database (`installed_plugins` table) for dynamically installed manifests.
## 2. Dynamic Bundle Loading
Plugins from the marketplace use the `bundle` format and are imported at runtime from CDN endpoints (e.g., unpkg) via `loadPluginFromManifest()`.
## 3. Database Sync
- Marketplace installations mutate the Prisma database.
- `pluginManager.loadFromManifest()` validates and registers these plugins.
- Ensure any schema changes to the marketplace tables follow the `database-migrations.md` rule.
## Current Implementation State
- **Stage 4 (Bridge API + One-Click Install)** ✅ COMPLETE
- Plugin data currently served from a volume-backed **PostgreSQL database** — plugins are submitted by entering an NPM package name in the Admin UI
- The marketplace auto-fetches metadata from the NPM registry and parses the `"worldwideview"` block in `package.json`
- A background `NpmCache` table caches NPM registry metadata to decouple from live NPM dependency
- Future: Supabase DB for full publisher workflow (Stage 5)
---
## Plugin Submission Flow (Current)
1. Developer publishes their plugin to npm with a `"worldwideview"` block in `package.json`
2. Admin opens the Marketplace Admin UI
3. Admin enters the npm package name (e.g. `@worldwideview/wwv-plugin-my-data`)
4. Marketplace fetches metadata from NPM registry, parses the manifest block, stores in PostgreSQL DB
5. Plugin appears in the browse catalog
> Static plugins still need their GeoJSON files placed in the host application's `public/data/` directory.
---
## One-Click Install Flow (Redirect-Based)
**No static API keys.** Auth is entirely session-based. The marketplace redirects the browser to WWV, which checks the user's existing login session.
```
User clicks "Install" on marketplace
↓
Marketplace: check localStorage for instance URL
↓ (if not configured)
Show "Connect Your Instance" modal → user enters http://localhost:3000
↓
MP → GET /api/marketplace/grant-token?redirectTo=...
↓
WWV: check Auth.js session → issue 30-day JWT → redirect ?token=<jwt>
↓
MP stores JWT in localStorage
↓
MP: Navigate browser to /api/marketplace/install-redirect
?pluginId=borders&manifest=<b64>&redirectTo=...
↓
WWV: check session → upsertPlugin() → issueMarketplaceToken()
↓
Redirect back ?installed=borders&token=<jwt>
↓
MP: Show ✓ Installed — store JWT for future Manage-page calls
```
### Token Flow
After successful install, WWV issues a **30-day HS256 JWT** (`scope: "marketplace"`, signed with `AUTH_SECRET`). The marketplace stores it in `localStorage` and sends it as `Authorization: Bearer <token>` on all subsequent API calls.
`validateMarketplaceAuth` accepts auth in priority order:
1. Active Auth.js session (same-origin browser)
2. Marketplace JWT Bearer token (cross-origin Manage page)
3. Legacy static `WWV_BRIDGE_TOKEN` (backward compat)
---
## WWV Bridge API Routes (Implemented)
| Route | Method | Purpose |
|---|---|---|
| `/api/marketplace/install-redirect` | GET | Browser redirect install — checks session, installs plugin, issues JWT |
| `/api/marketplace/grant-token` | GET | Issues marketplace JWT without install (for Manage page) |
| `/api/marketplace/status` | GET | Returns all installed plugins (session or JWT auth) |
| `/api/marketplace/uninstall` | POST | Deletes an installed plugin record |
| `/api/marketplace/load` | GET | Returns valid non-built-in manifests for client startup (no auth) |
| `/api/auth/setup-status` | GET | CORS-enabled connection test endpoint |
### Auth Module: `src/lib/marketplace/`
| File | Purpose |
|---|---|
| `auth.ts` | `validateMarketplaceAuth()` — session → JWT → legacy token |
| `marketplaceToken.ts` | `issueMarketplaceToken()` / `verifyMarketplaceToken()` — HS256 JWT via jose |
| `cors.ts` | CORS headers for marketplace bridge routes |
| `repository.ts` | `upsertPlugin()`, `uninstallPlugin()`, `getInstalledPlugins()` |
| `registryClient.ts` | Fetches and Ed25519-verifies the signed plugin registry. 5-min in-memory cache. Stamps trust at install time |
| `trustedPlugins.ts` | `localStorage` helpers for tracking user-approved unverified plugins |
---
## Marketplace API Endpoints (Target State)
### Public
| Method | Path | Description |
|---|---|---|
| GET | `/api/plugins` | List/search plugins |
| GET | `/api/plugins/:id` | Plugin detail |
| GET | `/api/plugins/:id/bundle` | Download plugin bundle |
| GET | `/api/categories` | List categories |
| GET | `/api/registry` | Ed25519-signed verified-plugins registry — fetched by WWV at install time |
### Authenticated (Consumer)
| Method | Path | Description |
|---|---|---|
| POST | `/api/instances` | Register a WWV instance |
| PUT | `/api/instances/:id/default` | Set default instance |
| POST | `/api/plugins/:id/install` | Trigger install to default instance |
| GET | `/api/me/plugins` | List user's installed plugins |
### Authenticated (Publisher)
| Method | Path | Description |
|---|---|---|
| POST | `/api/publisher/plugins` | Submit a new plugin |
| PUT | `/api/publisher/plugins/:id` | Update plugin |
| POST | `/api/publisher/plugins/:id/versions` | Publish new version |
| GET | `/api/publisher/analytics` | Publisher analytics |
---
## Data Model
### WWV Local DB (Prisma — current)
```prisma
model InstalledPlugin {
id String @id @default(uuid())
pluginId String
version String
config String // Full manifest JSON
installedAt DateTime @default(now())
}
model NpmCache {
id String @id @default(uuid())
packageName String @unique
metadata String // JSON — cached NPM registry response
lastFetched DateTime @default(now())
}
```
### Marketplace DB (Supabase — target, Stage 5+)
```
users ─── published_plugins ─── plugin_versions
│
reviews
```
Key fields on `published_plugins`: `slug`, `publisher_namespace` (`publisher.plugin-name`), `status` (`draft | review | published | rejected`), `manifest` (jsonb).
### Cloud DB (Supabase PostgreSQL — RLS-scoped)
```
tenants ─── installed_plugins
└── settings
```
---
## Security
| Concern | Mitigation |
|---|---|
| No static API keys | Install flow uses session redirects — no token to leak |
| Marketplace JWT | 30-day HS256, scope-locked, verified via `jose` |
| CSRF | Install/grant-token are GET redirects — user explicitly navigates |
| Open redirect | `redirectTo` validated against allowlist |
| Manifest tampering | Base64 only — no code execution. Validated via `validateManifest()` |
| Malicious plugins | Sandboxed in Web Workers (unverified). CSP headers. Code review (verified) |
| Supply chain | Plugin bundles checksummed (SHA-256), integrity verified |
| Plugin API keys | Stored encrypted in `settings` table, never exposed to browser |
| Trust forgery | Registry signed with Ed25519 — `verified` cannot be claimed for unlisted plugins |
| Trust stamping | WWV always overwrites `manifest.trust` server-side — client/marketplace input ignored |
| Unverified consent | Warning dialog before any unverified plugin loads; approval stored per-plugin in `localStorage` |
---
## API Key Proxy & Rate Limiting (For `apikey` plugins)
Response caching makes server-as-proxy viable at scale:
```
1000 users viewing earthquake plugin:
→ 1000 requests hit your proxy
→ Proxy: "I fetched this URL 3s ago, cache still valid"
→ Returns cached response to all 1000
→ 1 actual upstream request to USGS
```
| Tier | Proxy req/min | Cache TTL |
|---|---|---|
| Free | 60 | 30s |
| Pro | 300 | 10s |
| Enterprise | Unlimited | 5s |
---
## History & Timeline
**Third-party history (`historyUrl`):** Plugin manifest declares a history URL with `%t` placeholder. WWV replaces it with the scrubber timestamp — third-party provides the endpoint. No storage on your side. Available on all tiers.
**WWV Snapshot Capture:** Your server periodically stores plugin data per tenant. Counts toward storage quota. Pro/Local only.
| Feature | Free (cloud) | Pro / Local |
|---|---|---|
| Live data (declarative `url`) | ✅ | ✅ |
| Third-party history (`historyUrl`) | ✅ If supported by API | ✅ |
| WWV snapshot capture | ❌ | ✅ Counts toward quota |
| Built-in aviation/maritime history | ✅ Limited (24h) | ✅ Full |
-30
View File
@@ -1,30 +0,0 @@
---
trigger: model_decision
description: Strict rules for operating within the pnpm monorepo workspace environment structure across frontend and backend services.
globs: ""
---
# Monorepo Workflow Guidelines
## Purpose
Strict rules for operating within the `pnpm` monorepo workspace environment structure across frontend and backend services.
## Directory Isolation
- The Next.js frontend is located at root `.`.
- Individual standalone plugins are located at `packages/wwv-plugin-[name]`.
- For heavy plugin processing, data engine seeders are located at `local-seeders/community/[name]` or `local-seeders/private/[name]`.
## Critical Workspace Rule
> [!IMPORTANT]
> Because plugins use internal workspace references, you MUST strictly use `"workspace:*"` instead of `"*"` in `package.json` dependencies (e.g. `"@worldwideview/wwv-plugin-sdk": "workspace:*"`). Using `"*"` can cause pnpm to eagerly resolve cached registry versions instead of strictly linking your local code, resulting in opaque TypeScript and compilation errors.
> **Whenever you add a new plugin package or adjust the SDK, you MUST run `pnpm install` then `pnpm build` from the project ROOT directory to propagate TS types properly.**
## Start/Exec Flow
Running `pnpm dev:all` from the project root employs `docker-compose` to spin up the data engine runner and your local seeders alongside the Next.js frontend.
## Next.js Monorepo Exceptions
When modifying `next.config.ts`, if new UI components or plugins are added that require SSR/transpilation, ensure they are appended to the `transpilePackages: []` array, or they will throw Next-Router unresolved import errors on production build.
-120
View File
@@ -1,120 +0,0 @@
---
trigger: model_decision
description: High-level platform goals, product vision, business model, multi-repository layout, and the Edition System.
---
# WorldWideView — Platform Architecture
> **This folder is the canonical source of truth for the project. It is NOT committed to git.**
> It contains live architectural decisions, future plans, and operational knowledge.
---
> [!TIP]
> **For the fundamental system architecture, component layout, and data pipelines, please read [`docs/ARCHITECTURE.md`](../../../docs/ARCHITECTURE.md) first.**
> This file focuses strictly on the advanced business vision, multi-repository layout, and the Edition System.
## Product Vision & Business Model
WorldWideView operates on a **VS Code distribution model**. It is a visualization platform, not a compute provider. Plugin authors bring their own backends to display real-time geospatial intelligence.
---
## Repository Structure
Five repositories, modelled after the n8n pattern:
| Repo | License | Purpose |
|---|---|---|
| `silvertakana/worldwideview` | Elastic License 2.0 | Core 3D globe app (Next.js + CesiumJS + Prisma) |
| `silvertakana/wwv-data-engine` | Open Source | Generic runner (Fastify + Redis + WS) |
| `silvertakana/wwv-seeders-community` | Open Source | Open-source community seeders |
| `silvertakana/wwv-seeders-private` | **Private** | Proprietary seeder scripts (aviation, maritime, etc.) |
| `silvertakana/worldwideview-marketplace` | Open Source | Plugin catalog, signed registry, admin UI |
| `silvertakana/worldwideview-plugins` | Open Source | Published plugin frontend source |
| `silvertakana/worldwideview-web` | Open Source | Landing page, docs, static content |
| `silvertakana/worldwideview-cloud` | **Private** | Cloud deployment config — Docker Compose, nginx, provisioning scripts |
| `silvertakana/plugin-sdk` | MIT | `@worldwideview/plugin-sdk` — shared types for repos + 3rd-party authors |
**n8n Parallel:**
| n8n | WorldWideView |
|---|---|
| `n8n-io/n8n` | `silvertakana/worldwideview` |
| `n8n-io/n8n-cloud` (private) | `silvertakana/worldwideview-cloud` (private) |
| `n8n.io` landing | `worldwideview.dev` (static on Vercel) |
| `app.n8n.cloud` dashboard | `app.worldwideview.dev` |
| `[user].app.n8n.cloud` | `[user].app.worldwideview.dev` |
| `@n8n/workflow` npm | `@worldwideview/plugin-sdk` npm |
---
## Domain Architecture
| URL | What | Stack |
|---|---|---|
| `worldwideview.dev` | Static marketing/landing | Next.js (static export) on Vercel |
| `app.worldwideview.dev` | Auth, accounts, dashboard | Next.js + Supabase Auth |
| `[user].app.worldwideview.dev` | User's cloud WWV instance | Next.js (multi-tenant, RLS) |
| `marketplace.worldwideview.dev` | Plugin catalog, publishing, install | Next.js (SSO via WorldWideView auth) |
| `demo.worldwideview.dev` | Public demo — pre-configured, read-only | Next.js (WWV app) |
| `docs.worldwideview.dev` | Documentation | TBD (Mintlify/Nextra) |
| `status.worldwideview.dev` | Uptime / incident | Uptime Kuma |
| `api.worldwideview.dev` | Built-in data services (aviation, maritime) | Next.js API routes or standalone |
| `cdn.worldwideview.dev` | Plugin bundle storage | Cloudflare R2 |
**Reserved subdomains:** `www`, `app`, `api`, `demo`, `marketplace`, `docs`, `status`, `admin`, `cdn`, `blog`, `changelog`
---
## Infrastructure
All on **Coolify** running on a repurposed **ThinkPad T480s** (i5-8350U, 16GB RAM, 256GB NVMe, Ubuntu). Zero monthly hosting cost.
```
ThinkPad T480s (home server, $0/mo) Supabase (hosted)
├── Coolify ──→ ├── PostgreSQL (cloud DB)
│ ├── worldwideview.dev (landing, Vercel) ├── Auth (GoTrue)
│ ├── app.worldwideview.dev (auth/dashboard) └── Storage (file uploads)
│ ├── WWV Cloud App (Docker)
│ ├── Marketplace App (Docker) Cloudflare (free)
│ ├── Demo Instance (Docker) ├── DNS + wildcard routing
│ ├── api.worldwideview.dev ├── CDN caching
│ ├── Uptime Kuma ├── DDoS protection
│ └── Plausible Analytics └── R2 (plugin bundle CDN)
└── Ubuntu Server
```
**CI/CD:** Push to `silvertakana/worldwideview` → GitHub Actions builds Docker image → pushes to `ghcr.io/silvertakana/worldwideview:latest` → Coolify webhook triggers rolling restart.
**Monitoring:** Uptime Kuma (all subdomains), Plausible/Umami analytics.
---
## Business Model
**Distribution platform, not compute provider.** Plugin authors handle their own backends.
| Revenue Stream | How |
|---|---|
| **Cloud hosting** | `[user].app.worldwideview.dev` — free/pro/enterprise tiers |
| **Marketplace cut** | % of paid plugin subscriptions via Stripe Connect |
| **Verified tier** | Plugin authors pay for review + verified badge |
| **Pro/Enterprise license** | More quota, faster cache TTL, advanced features |
| **Extension packs** | Curated bundles of related plugins |
Plugin authors can monetize independently (VS Code + GitHub Copilot model). Selling through the marketplace is **incentivized** (managed billing, featured placement, analytics) but not mandatory.
---
## Editions
Three editions; same codebase, gated by `NEXT_PUBLIC_WWV_EDITION`:
| Edition | Value | Database | Auth | Plugin Install |
|---|---|---|---|---|
| **Local** | `local` | PostgreSQL | Auth.js Credentials | Yes |
| **Cloud** | `cloud` | PostgreSQL (Supabase) | Auth.js + Supabase | Yes |
| **Demo** | `demo` | PostgreSQL (read-only) | Disabled | No |
Three thin adapter files handle edition differences: `src/core/auth.ts`, `src/core/storage.ts`, `src/core/tenant.ts`. Everything else is edition-unaware.
-393
View File
@@ -1,393 +0,0 @@
---
trigger: model_decision
description: The standard operating procedure for instantiating, modifying, and registering a new data source plugin within the engine.
---
# Plugin Architecture & Data Flow
## Purpose
The standard operating procedure for instantiating, modifying, and registering a new data source plugin within the engine.
## The WorldPlugin Contract
All data ingest flows through `WorldPlugin` (defined entirely within `@worldwideview/wwv-plugin-sdk`).
A valid plugin class MUST provide:
1. `id` and `version` (The `version` MUST be dynamically imported via `import pkg from "../package.json"` to prevent duplicate tracking during CI/releases for built-in plugins).
2. `fetch(timeRange)` method logic.
3. `renderEntity(GeoEntity)` to determine visual style.
4. `getPollingInterval()` (defaults fallback securely to standard store config).
## Plugin Architectures (Manifest Formats)
Plugins are defined via `PluginManifest` and operate on an **All-Bundle** architecture.
All frontend extensions must eventually be compiled into a JS bundle that exports a `WorldPlugin`.
1. **`bundle` (Dynamic CDN Loaded / Internal)**: Plugins dynamically imported at runtime via ES module references (e.g., `unpkg.com` or local `/plugins/myplugin/frontend.mjs`). Handled by `loadPluginFromManifest` referencing `manifest.entry`.
2. **`static` (Legacy)**: Now compiled via `wwvStaticCompiler` into a dynamic bundle. The raw `StaticDataPlugin` runtime class has been removed.
3. **`declarative` (Legacy)**: Now compiled or wrapped into bundles.
No matter the source, the engine evaluates it by directly loading the `entry` file dynamically:
`const module = await import(/* webpackIgnore: true */ entry);`
## Package Metadata (`package.json`)
All plugins MUST define their identity and compatibility via a `"worldwideview"` block in their `package.json`. This acts as the source-of-truth for generating the `PluginManifest` for the registry. If a plugin does not define `"type"`, validation functions will reject the manifest entirely or rely on legacy fallbacks.
```json
{
"name": "@worldwideview/wwv-plugin-myplugin",
"version": "1.0.0",
"worldwideview": {
"id": "myplugin", // REQUIRED: Must match the directory name & API routes.
"type": "data-layer", // REQUIRED: "data-layer" or "extension"
"format": "bundle", // REQUIRED: "bundle", "declarative", or "static"
"category": "Aviation", // REQUIRED: Matches PluginCategory union in SDK
"icon": "Plane", // RECOMMENDED: Lucide icon name
"capabilities": [ // REQUIRED: What the plugin can do
"data:own", // (e.g. injects its own data)
"globe:overlay", // (e.g. renders 3D elements)
"network:fetch"
]
}
}
```
## The Registration Pipeline
> [!NOTE]
> All plugin interactions run through three singleton services:
> `PluginRegistry` / `InstalledPluginsLoader` -> `PluginManager` -> `PollingManager` -> `DataBus`
### `InstalledPluginsLoader`
Scans the PostgreSQL database at startup for dynamically installed marketplace manifests. Parses, validates (`validateManifest`), and registers valid plugins via `pluginManager.loadFromManifest`.
### `PluginManager`
The orchestrator. Exposes `registerPlugin`, `enablePlugin`, `disablePlugin`. Never bypass the manager to push data manually to the cache. Supports dynamic loading via `loadFromManifest`.
### `PollingManager`
Coordinates interval execution. Implements **exponential backoff**. When a plugin errors, it retries at increasing scales (e.g., 2s → 4s → 8s) automatically.
### `DataBus`
The typed generic event bus separating React UI components from background execution.
```typescript
// How data gets pushed onto the rendering surface securely:
DataBus.getInstance().emit('dataUpdated', {
pluginId: "mytracker",
entities: newEntities
});
```
## Local Plugin Development (Sandbox)
To rapidly iterate on plugins without polluting the core monorepo `packages/` directory or git history, use the local devkit:
1. **Create**: Run `node packages/wwv-cli/dist/index.js create <name> --local` to scaffold a full plugin structure inside the `local-plugins/` directory.
2. **Develop**: Running `pnpm dev` or `pnpm dev:all` automatically spawns the `dev:plugins` watcher. Any changes made to plugins in `local-plugins/` will be instantly rebuilt using Vite (externalizing large globals) and synced to `public/plugins-local/` for hot-reloading.
3. **Promote**: Once the plugin is stable and ready to be committed or published, run `node packages/wwv-cli/dist/index.js link <name>` to automatically move it to `packages/` and clean up the sandbox.
## When to Apply
When writing `fetch` implementations for plugins, or tracing why an entity dropped off the map. Ensure missing data correctly triggers the cache fallback via the DataBus. Ensure all new external plugins use the `bundle` format and exist in the marketplace registry to avoid CDN 404 hydrating crashes.
## Capability Declarations
Every plugin declares what it can access. Undeclared capabilities are **blocked at runtime**.
```typescript
type PluginCapability =
| "data:own" // Fetch and render its own data layer
| "data:read:<id>" // Read entities from another plugin
| "ui:detail-panel" // Inject UI into entity detail panel
| "ui:sidebar" // Add a sidebar section
| "ui:toolbar" // Add toolbar buttons
| "ui:settings" // Add a settings page
| "globe:overlay" // Draw on the globe (polylines, polygons, heatmaps)
| "globe:camera" // Control camera movement
| "storage:read" // Read from local/cloud storage
| "storage:write" // Write to local/cloud storage
| "network:fetch" // Make external HTTP requests
```
---
## Trust Tiers
| Trust Tier | Who | Capabilities Allowed | How Determined |
|---|---|---|---|
| **Built-in** | Ships with WWV | All | Hardcoded in `AppShell` at build time |
| **Verified** | WorldWideView-reviewed | Any declared capability | Plugin ID in the **signed registry** |
| **Unverified** | Community / 3rd-party | `data:own`, `ui:settings` only | Plugin ID NOT in the registry |
### Signed Plugin Registry
Trust is determined by a cryptographically signed JSON file:
```
GET marketplace.worldwideview.dev/api/registry
→ { plugins: ["aviation", "maritime", ...], issuedAt, signature }
```
- Signed with **Ed25519** private key (held only by registry maintainer)
- WWV instances have the **public key hardcoded** — nobody can forge a fake verified list
- WWV fetches and **verifies signature server-side** at install time
- Trust is **always stamped server-side** — no client-side trust claims accepted
- Registry response cached for **5 minutes** (in-memory on each WWV instance)
- If registry unreachable, last cached result used; new unknowns default to `unverified`
- **Managing:** Edit `src/data/verifiedPlugins.ts` in marketplace repo and redeploy
- **Revoking:** Remove plugin ID and redeploy — WWV instances drop trust on next cache refresh
### Unverified Plugin Warning
When an unverified manifest is encountered:
1. Plugin held in pending queue (not loaded)
2. `UnverifiedPluginDialog` shown: "This plugin is unverified. Install at your own risk."
3. User can Allow (stores approval in `localStorage`) or Deny
4. Once approved, dialog won't reappear for that plugin ID
---
## VS Code Design Patterns Applied
**Activation events** — plugins declare when they activate, not at startup:
```json
"activationEvents": ["onView:aviationLayer", "onCommand:wwv.showCockpit"]
```
**Contribution points** — declarative UI, no code needed for sidebar/menu entries:
```json
"contributes": {
"commands": [{ "command": "wwv.enterCockpit", "title": "Enter Cockpit View" }],
"settings": [{ "key": "refreshInterval", "type": "number", "default": 10 }]
}
```
**Extension dependencies** — plugins can require other plugins (auto-installed if missing):
```json
"extensionDependencies": ["worldwideview.aviation"]
```
**Publisher namespaces** — `publisher.plugin-name` (e.g., `silvertakana.aviation`). Prevents naming conflicts.
**Extension packs** — bundles of related plugins:
```json
{ "id": "worldwideview.military-suite", "extensionPack": ["aviation", "military-bases", "radar-overlay"] }
```
---
## Plugin Auth Types
Plugins declare auth type in the manifest:
| Auth Type | Who Calls the API | Server Role |
|---|---|---|
| `none` | Browser, directly | Nothing |
| `apikey` | Server proxy | Injects key from DB, caches response |
| `token-exchange` | Browser, directly (after exchange) | One-time key→token swap |
| `oauth` | Browser, directly (after OAuth) | OAuth callback handler, stores token |
**Recommended for scale:** `token-exchange` — your server is never in the hot path for data requests.
---
## Plugin Development Workflows
Starting in 2026, WorldWideView adopted the **Architect and Toolbox** paradigm using NPM global tools, ensuring a fully decoupled developer experience. Plugin repositories contain *pure code* with absolutely no platform hosting boilerplate (e.g. no embedded docker-compose files).
### The Developer Experience
1. **Scaffold**: `node packages/wwv-cli/dist/index.js create <name> --local` (Creates sandbox inside `local-plugins/`)
2. **Install**: `pnpm install` (Links dependencies via workspace)
3. **Develop**: `pnpm dev:all` (Starts frontend, data engine, and plugin watcher dynamically)
### Local vs Docker Development
The marketplace API natively supports hot-reloading for local plugins without rebuilding the core.
**Docker-Hosted Mode (Recommended for external devs):**
If a developer wants to build a plugin without cloning the 1GB+ WorldWideView monorepo, they can pull the official WWV Docker image and mount their compiled plugin code:
1. Run WWV with the development flag: `WWV_PLUGIN_DEV=true`
2. Mount the local plugin's `dist/` folder into the container's `/app/local-plugins/` directory.
3. Because the plugin is strictly code, Vite rebuilds in <100ms, and a browser refresh instantly loads the new plugin from the Docker volume.
**Core Contributor Mode:**
If working inside a cloned `worldwideview` directory, scaffolding the plugin into `local-plugins/` makes it a pnpm workspace member. Running `pnpm dev:all` automatically spawns `pnpm dev:plugins`, which continuously transpiles local plugins so the Next.js frontend can hot-reload them without complex linking.
---
DB stores **references only** — never plugin files:
```prisma
model InstalledPlugin {
pluginId String
version String
manifestUrl String // "https://marketplace.../api/plugins/aviation/manifest"
bundleUrl String? // local path or CDN URL (differs by edition)
}
```
| Edition | `bundleUrl` | How served |
|---|---|---|
| **Local** | `file:./plugins/cockpit-view/bundle.js` | Local filesystem (downloaded at install) |
| **Cloud** | `https://cdn.worldwideview.dev/plugins/cockpit-view/bundle.js` | CDN at runtime |
---
## Environment Variables & Execution Agnosticism
WWV enforces a strictly decoupled execution context, allowing plugins to function reliably whether they are loading locally (`plugins-local`) or from a public CDN (`unpkg.com`).
### Dynamic Assets (`import.meta.url`)
For file-based assets bundled alongside the plugin (such as JSON datasets or 3D GLTF models), the runtime relies on native ECMAScript relative resolution instead of hardcoded framework paths.
- Plugins securely fetch their assets using `new URL('data/data.json', import.meta.url).href`.
- **Impact:** Assets resolve properly directly from the domain they were executed on. No CORS bypasses or CDN fallbacks required.
### .env Passthrough Injection
The WorldWideView core engine functions as a "blind pipe" for developer environment variables, allowing organizations to securely route plugins to custom backend microservices (local or cloud) without modifying the platform.
1. The developer adds `NEXT_PUBLIC_WWV_PLUGIN_EXAMPLE_URL="http://localhost:5001"` to their `.env.local`.
2. The engine detects the `NEXT_PUBLIC_WWV_PLUGIN_` prefix, strips it, and dynamically hydrates the `PluginContext`.
3. The plugin natively retrieves `ctx.env.EXAMPLE_URL`.
4. *Security Clause:* Variables accessible via `ctx.env` are exposed directly to the **client-side bundle**. Core secrets/API tokens must never be used here; they must exclusively use the server-side `ctx.getSecret()` workflow.
---
## Lazy Loading
Plugins load their code **only when activated**, not at startup:
```
Startup: load manifests from DB (one query) → globe with built-in plugins only
Toggle on "Aviation" → import() aviation bundle → register → fetch data
Toggle on "Cockpit View" → import() cockpit bundle → register
```
---
## Implementation Stack
| Piece | Library |
|---|---|
| Bundle loading | Native `import()` or `@module-federation/enhanced/runtime` |
| Manifest validation | `zod` |
| Sandboxing (unverified plugins) | `comlink` + Web Workers |
| Dependency resolution | `toposort` |
| Event system | `mitt` (200 bytes) |
| Plugin registry | Custom `PluginManager` class |
---
## Built-In Plugins (Current)
| Plugin | Format | Data Source |
|---|---|---|
| Aviation | bundle | OpenSky Network (ADS-B) |
| Maritime | bundle | AIS (maritime tracking) |
| Borders | bundle | GeoJSON from public API |
| Wildfire | bundle | NASA FIRMS |
| Military Bases | bundle (compiled) | OpenStreetMap (via Overpass) |
| Nuclear Facilities | bundle (compiled) | OpenStreetMap |
| Camera/CCTV | bundle | IP camera streams |
| GeoJSON | bundle | User-supplied GeoJSON files |
Future direction: Aviation and maritime redesigned as **declarative plugins** pointing to `api.worldwideview.dev` — your own API server handles polling, persistence, and history for all tenants.
---
## Creating Static OSM Plugins
For static plugins built from OpenStreetMap data, use the scaffold script:
```bash
node scripts/scaffold-osm-plugin.mjs '{
"name": "volcanoes",
"displayName": "Volcanoes",
"description": "Active and dormant volcanoes worldwide from OSM",
"osmTag": "natural=volcano",
"icon": "Mountain",
"color": "#ef4444",
"category": "Natural Disaster"
}'
```
This queries Overpass API, converts to GeoJSON, scaffolds the package, and prints marketplace snippets.
**Common OSM tags:**
| Dataset | OSM Tag | ~Count |
|---------|---------|--------|
| Airports | `aeroway=aerodrome` | 40k |
| Seaports | `harbour=yes` | 15k |
| Nuclear facilities | `generator:source=nuclear` | 758 |
| Volcanoes | `natural=volcano` | 1.5k |
| Lighthouses | `man_made=lighthouse` | 20k |
| Embassies | `amenity=embassy` | 10k |
| Helipads | `aeroway=helipad` | 25k |
| Power plants | `power=plant` | 15k |
| Wind farms | `generator:source=wind` | 50k |
See skill: `.agents/skills/osm-static-plugin-creation.md`
---
## The Data Engine (Backend & Seeders)
When a plugin requires 24/7 background polling, long-term history (PostgreSQL), or high-frequency caching (Redis), it should leverage the **Data Engine**. The WWV architecture splits the frontend UI package and the backend data collection into two distinct tiers.
### 1. First-Party / Built-In Plugins
For plugins developed directly inside the WWV monorepo:
- **Seeder Module**: Developers write a single seeder (e.g. `seeders/aviation.ts`) inside the `wwv-data-engine` package.
- **Unified Container**: All first-party seeders run inside one singular `wwv-data-engine` Docker container. This prevents background polling from being destroyed by Next.js edge isolation or HMR.
### 2. Third-Party / Community Plugins (The VPS Problem)
Currently, **there is a high DX friction for third-party developers hosting their own heavy plugins.**
- The `wwv-data-engine` is deeply coupled to the WWV host. Third-party developers cannot push their backend seeders into WWV's actual servers.
- The `wwv-plugin-sdk` only assists with the frontend `WorldPlugin` interface.
- Consequently, a sole developer with a VPS must write their own Node.js backend entirely from scratch to serve their custom data feeds, handle their own caching, and manage multiple Node instances if they create multiple plugins.
### 3. The Future: Distributable Data Engine Image
To solve this friction and make third-party plugin hosting highly intuitive, the WWV roadmap includes publishing a **WWV Mini-Engine** as a standalone Docker image:
1. **Lightweight Deployment:** Developers run `docker run worldwideview/mini-engine` on their VPS.
2. **Backend SDK:** They write simple `.ts` fetch scripts returning `GeoEntity[]` and drop them into a mounted `/seeders` volume.
3. **Zero Configuration:** The Docker engine hot-reloads these seeders, handles the Redis caching, and exposes standardized REST or WebSocket endpoints automatically. The developer focuses only on fetching data, avoiding all backend infrastructure setup.
See skill: `.agents/skills/data-engine-seeder-creation.md`
---
## Architectural Differences: WorldWideView (WWV) vs WorldMonitor (WM)
While WorldWideView draws UI/UX and feature inspiration from WorldMonitor as a baseline, the underlying plugin architectures are fundamentally different.
### WorldMonitor (The Inspiration)
- **Monolithic Frontend**: Layers and datasets are typically hardcoded directly into the application's core logic.
- **Custom Implementations**: Each data source usually requires its own bespoke API fetching, state management, and map rendering logic (e.g., raw Mapbox/Cesium objects).
- **Coupled Backend**: Data ingestion is often tightly coupled to the monolithic application or spread across scattered, heterogeneous microservices without a unified cache.
### WorldWideView (The Reality)
- **Extensible VS Code-Style Runtime**: Plugins are wholly independent, sandboxable modules loaded dynamically via activation events. The core map knows nothing about aviation or wildfires.
- **Unified Data Contract (`GeoEntity`)**: All data layers map their bespoke formats into a strict, unified `GeoEntity` system. The core engine handles clustering, filtering, and timeline playback generically—no custom rendering trickery needed per dataset.
- **Shared Data Engine**: WWV enforces a strict separation of concerns. All 24/7 API polling, data normalizations, and history retention happen off-thread in the singular `wwv-data-engine` (utilizing Redis hot cache + PostgreSQL history retention), providing a clean, unified REST interface for the plugin frontend.
- **Cryptographic Trust**: WWV incorporates a signed marketplace registry and fine-grained capability declarations (`data:read`, `ui:sidebar`), establishing an enterprise-ready security posture for third-party extensions.
### Planned Features (Inspired by Monolithic Systems / WM)
While avoiding monolithic coupling, WWV plans to implement the following high-UX features natively or via generic plugin capabilities:
1. **Decentralized Plugin Connections (Agnostic Client)**: WorldWideView is entirely agnostic to the backend. It does NOT enforce a single unified pipeline. If a user activates 30 different plugins hosted on 30 different custom servers, the frontend will happily open 30 separate WebSocket connections. The client application knows absolutely nothing about the Data Engine—it only reads the `streamUrl` provided by each individual plugin's manifest.
2. **Cross-Layer Geofencing & Alerts**: A new `User Alert System` designed explicitly as a **plugin**. Utilizing the unified `GeoEntity` outputs, users can draw a polygon on the map and trigger webhooks/notifications if *any* entity from *any* active plugin crosses the boundary.
3. **Advanced Dead-Reckoning Hooks**: Building a `useDeadReckoning()` hook directly into the `@worldwideview/wwv-plugin-sdk`. It will automatically use standard `speed` and `heading` props to smoothly glide vehicle entities across the Cesium globe at 60 FPS between data pings, preventing plugin authors from rewriting complex math logic.
4. **Curated Workspaces / Scenarios**: The ability for users to save their current state as a "Workspace URL." This generates a shareable link that encodes the exact camera position, timeline bounds, activated plugins, and their respective layer configurations/filters.
## Local Plugins & Seeders
### 1. Local Sandbox Workflow
- **Scaffold**: `node packages/wwv-cli/dist/index.js create <name> --local` (creates inside `local-plugins/wwv-plugin-<name>`).
- **Develop**: `pnpm dev` triggers hot-reloading.
- **Promote**: `node packages/wwv-cli/dist/index.js link <name>` moves it to `packages/`.
### 2. Data Engine Seeders
Seeders in `wwv-data-engine/src/seeders/` provide the backend data.
- **Cron Seeder**: Uses `registerSeeder({ name: "plugin-id", cron: "..." })` and pushes to Redis via `setLiveSnapshot()`.
- **Init Seeder**: High-frequency or persistent websockets via `init: () => void`.
- **Constraint**: Seeder `name` MUST exactly match the frontend plugin `id`. Do not bundle workspace dependencies like `ws` or `zod` into the seeder `dist`.
-25
View File
@@ -1,25 +0,0 @@
---
trigger: model_decision
description: Server development and debugging rules for the production/homelab server using SSH and Coolify MCP.
---
# Server Management & Debugging
## 1. Coolify MCP (High-Level)
Use the `coolify` MCP server for managed actions:
- **Discovery**: `mcp_coolify_projects`, `mcp_coolify_environments`
- **Inspection**: `mcp_coolify_application_logs`, `mcp_coolify_get_application`
- **Actions**: `mcp_coolify_control` (restart, stop)
## 2. Direct SSH (Low-Level)
For filesystem or network issues, use direct SSH to the production server:
`ssh [email protected] "<command>"`
**Common Commands**:
- `docker ps`
- `docker logs --tail 100 <container_id>`
- `df -h`
- `top -b -n 1` (Never run interactive `top` or `nano` via agent commands).
## 3. Safety Precautions
Always use non-interactive flags for SSH. Communicate clearly before executing high-risk actions like volume deletion or critical infrastructure restarts.
@@ -1,378 +0,0 @@
---
trigger: model_decision
description: Human-centered design principles, stakeholder map, and decision matrix.
---
# Stakeholders & Human-Centered Design
> [!IMPORTANT]
> **This document is mandatory reading before any feature, UI, API, or architectural decision.**
> Every line of code in this project ultimately serves a human being. The AI agent MUST identify which stakeholder(s) a change affects and design the solution around their needs, abilities, and constraints — not the other way around.
---
## Why This Document Exists
Software that doesn't consider its humans fails. Not technically — socially. A plugin API that's elegant internally but hostile to external developers will die with zero third-party plugins. A monetization tier that ignores what users actually value will generate zero revenue. A contribution process that assumes PhD-level Cesium knowledge will attract zero contributors.
**The directive is simple:** Before you write code, ask *"Who is this for, what do they need, and what will they actually experience?"*
---
## The Stakeholder Universe
WorldWideView is not a single-user tool. It is a **platform ecosystem** with at least nine distinct human roles, each with different goals, different technical abilities, different emotional states when they interact with the product, and different definitions of "success."
```
┌──────────────────┐
│ Owner / Founder │
│ (silvertakana) │
└────────┬─────────┘
│ vision, architecture, deployment
┌───────────────────┼───────────────────┐
│ │ │
┌──────┴──────┐ ┌───────┴───────┐ ┌───────┴───────┐
│ Core Devs │ │ Contributors │ │ Plugin Devs │
│ (team) │ │ (community) │ │ (3rd party) │
└──────┬──────┘ └───────┬───────┘ └───────┬───────┘
│ │ │
└───────────┬───────┘ │
│ build the product │ extend the product
┌───────────┴────────────────────────────┘
│
┌──────┴──────────────────────────────────┐
│ The Application │
│ ┌────────┐ ┌────────┐ ┌───────────┐ │
│ │ Demo │ │ Local │ │ Cloud │ │
│ │Instance│ │Instance│ │ Instance │ │
│ └───┬────┘ └───┬────┘ └─────┬─────┘ │
└──────┼───────────┼─────────────┼────────┘
│ │ │
┌──────┴──┐ ┌─────┴────┐ ┌────┴─────┐
│Casual │ │Power User│ │Paying │
│Visitor │ │(self-host)│ │Customer │
└─────────┘ └──────────┘ └────┬─────┘
│
┌────────┴────────┐
│ Advertisers │
│ (demo edition) │
└─────────────────┘
```
---
## Stakeholder Profiles
### 1. Owner / Founder (silvertakana)
| Attribute | Description |
|---|---|
| **Role** | Solo founder, architect, deployer, and operator of the entire ecosystem |
| **Goal** | Build a sustainable, revenue-generating geospatial intelligence platform that rivals WorldMonitor |
| **Motivations** | Technical excellence, creative ownership, financial independence, community building |
| **Technical Ability** | Expert — TypeScript, Next.js, CesiumJS, Docker, infrastructure, Prisma, AI-assisted development |
| **Emotional State** | Ambitious but resource-constrained; every wasted hour counts double |
| **Definition of Success** | Growing user base, marketplace with active plugin developers, recurring revenue, industry recognition |
| **Pain Points** | Single point of failure for ops, deployment, and architecture decisions; time pressure; hosting cost sensitivity ($0/mo target) |
| **Interaction Surface** | `.agents/` rules, ROADMAP.md, Coolify dashboard, code reviews, AI pair-programming sessions |
**Design implications:**
- Automation is critical — CI/CD, self-healing databases, zero-config setup
- AI agent rules must preserve the owner's vision even when the owner isn't in the conversation
- Cost-efficient architecture (PostgreSQL, ThinkPad server, free-tier services) is a feature, not a compromise
- Every architectural decision must consider the owner's ability to maintain it solo
---
### 2. Core Development Team (WWV Devs)
| Attribute | Description |
|---|---|
| **Role** | Trusted developers with direct commit access to the monorepo |
| **Goal** | Ship features quickly without breaking existing functionality |
| **Motivations** | Clean code, clear architecture, fast iteration, not getting paged at 2am |
| **Technical Ability** | High — comfortable with TypeScript, Next.js App Router, Zustand, pnpm workspaces |
| **Emotional State** | Productive when patterns are clear; frustrated by undocumented gotchas |
| **Definition of Success** | PRs merged cleanly, features work on first deploy, zero regressions |
| **Pain Points** | File bloat (>150 lines), implicit state dependencies, CesiumJS rendering quirks, Docker build complexity |
| **Interaction Surface** | `src/core/`, `src/components/`, `packages/`, PR reviews, `pnpm dev` |
**Design implications:**
- 150-line file limit is sacred — enforce it in every code change
- Clear slice boundaries in Zustand store prevent cross-team conflicts
- `AGENTS.md` and `.agents/` context must always be current so that AI-augmented devs have accurate context
- Rendering rules (billboard vs. point, never mix properties) must be enforced systematically
- Docker gotchas documented in `06-dev-workflows.md` prevent time-sink debugging
---
### 3. Open Source Contributors
| Attribute | Description |
|---|---|
| **Role** | External developers who fork, fix bugs, add features, or create plugins via PRs |
| **Goal** | Contribute meaningfully without needing to understand the entire 50-package monorepo |
| **Motivations** | Portfolio building, learning CesiumJS/geospatial tech, improving a tool they use, open source goodwill |
| **Technical Ability** | **Variable** — ranges from junior devs submitting typo fixes to senior engineers contributing optimizations |
| **Emotional State** | Enthusiastic but easily discouraged by onboarding friction; will abandon contribution if setup takes >15 minutes |
| **Definition of Success** | PR accepted and merged; their name in the contributors list; feature actually ships |
| **Pain Points** | Complex monorepo, CesiumJS learning curve, Elastic License 2.0 implications, unclear "where to start" |
| **Interaction Surface** | `CONTRIBUTING.md`, `docs/SETUP.md`, `docs/PLUGIN_GUIDE.md`, GitHub Issues, PR templates |
**Design implications:**
- **One-command setup** (`pnpm run setup && pnpm dev:all`) is non-negotiable — if it breaks, contributors leave
- `CONTRIBUTING.md` must use `pnpm` (currently shows `npm` in places — this is a bug)
- "Good first issue" labels on GitHub attract newcomers
- Plugin creation is the #1 entry point for contributors — the scaffold script and Plugin Guide must be flawless
- The Elastic License 2.0 must be clearly communicated (it's NOT fully open source — contributors need to understand what they're signing up for)
---
### 4. Third-Party Plugin Developers
| Attribute | Description |
|---|---|
| **Role** | External developers who build and publish plugins to the WorldWideView Marketplace |
| **Goal** | Create a plugin that integrates their data source, potentially monetize it |
| **Motivations** | Revenue (marketplace cut), reach (installed on many WWV instances), reputation, data evangelism |
| **Technical Ability** | **Mid-to-high** — comfortable with TypeScript and npm, but NOT Cesium experts. They should never need to learn Cesium internals. |
| **Emotional State** | Evaluating whether the platform is worth investing development time in. Extremely sensitive to DX friction. |
| **Definition of Success** | Plugin published, visible in marketplace, installed by users, data renders correctly, optional revenue stream |
| **Pain Points** | (1) Backend hosting burden — they need a VPS for active plugins (the "VPS Problem"), (2) No distributable mini-engine yet, (3) SDK docs may not cover edge cases, (4) Rendering bugs with billboard/point property mixing |
| **Interaction Surface** | `@worldwideview/wwv-plugin-sdk`, `docs/PLUGIN_GUIDE.md`, `docs/creating-a-plugin.md`, marketplace submission UI, npm publishing |
**Design implications:**
- The Plugin SDK is the **#1 most important public API surface** — it must be impeccably documented, versioned, and stable
- `WorldPlugin` interface changes are breaking changes that affect real people with real published packages
- Three plugin formats (Declarative, Static, Code Bundle) exist specifically to lower the barrier:
- **Declarative** = zero code (JSON only) — anyone with a data URL can create a plugin
- **Static** = GeoJSON file + manifest — data analysts who aren't developers
- **Code Bundle** = full TypeScript — professional developers
- The future **Mini-Engine Docker image** (§3 of `03-plugin-system.md`) is critical for reducing the VPS burden
- Rendering must be abstracted — plugin devs define `renderEntity()`, and the engine handles GPU batching
- Error messages from the plugin lifecycle must be clear, not cryptic Cesium stack traces
- Marketplace analytics (install counts, usage) are what keep plugin developers motivated
---
### 5. Casual Visitors (Demo Users)
| Attribute | Description |
|---|---|
| **Role** | Anonymous visitors who land on `demo.worldwideview.dev` — no account, no setup |
| **Goal** | Explore the globe, see cool real-time data, understand what WorldWideView is |
| **Motivations** | Curiosity, news events (war, natural disasters, flight tracking), linked from social media or Reddit |
| **Technical Ability** | **Low to none** — they don't know what ADS-B or GeoJSON means. They just want to see planes on a globe. |
| **Emotional State** | Impatient. They will leave within 10 seconds if the globe doesn't load or the UI is confusing. |
| **Definition of Success** | "Whoa, this is cool" → bookmark/share → return later → potentially convert to self-host or cloud user |
| **Pain Points** | Slow boot time (Cesium asset loading), confusing UX, data not loading (API rate limits), ads disrupting experience |
| **Interaction Surface** | Boot overlay, 3D globe, layer toggle panel, entity hover/click, info cards, timeline, ad strip |
**Design implications:**
- **First 5 seconds are everything** — the boot overlay must feel cinematic, not broken
- Pre-activate compelling default layers (aviation is the universal crowd-pleaser)
- Globe interaction must work without instructions — left-click drag, scroll zoom, click entity
- The "Why ads?" banner (`DemoAdStrip`) is a trust-building honesty moment — keep it human and transparent
- Mobile layout must be flawless — many demo visitors come from social media on phones
- No login walls, no setup wizards, no configuration prompts — demo is zero-friction
- Performance on commodity hardware (integrated GPUs, 2017 laptops) must be acceptable
- Error states must be invisible — if OpenSky is down, the layer just shows fewer planes, not an error dialog
---
### 6. Self-Hosted Power Users (Local Edition)
| Attribute | Description |
|---|---|
| **Role** | Technical users who run WorldWideView on their own hardware (Raspberry Pi, home server, VPS) |
| **Goal** | Full control over their own geospatial intelligence dashboard, no cloud dependency |
| **Motivations** | Privacy, customization, data sovereignty, OSINT hobbyism, professional intelligence work |
| **Technical Ability** | **High** — comfortable with Docker, environment variables, CLI tools, possibly contributing plugins |
| **Emotional State** | Empowered but demanding. They chose self-hosting for a reason — they expect it to work without phoning home. |
| **Definition of Success** | Instance running 24/7, plugins installed and polling reliably, data persisted locally, zero external dependencies |
| **Pain Points** | Docker build complexity, PostgreSQL connection pooling issues, Cesium token setup, Prisma migration gotchas, plugin data not loading |
| **Interaction Surface** | `docker-compose.yml`, `.env.local`, Setup Wizard, Layers Panel, Config Panel, Marketplace Bridge, `pnpm run dev:all` |
**Design implications:**
- **Zero-config first run** — `pnpm run setup && pnpm dev` or `docker-compose up` must "just work"
- The Setup Wizard (Cesium token prompt) is the ONLY mandatory configuration step — minimize it
- PostgreSQL must handle concurrent reads from the app + data engine efficiently
- Self-hosted users are the most likely to become contributors — treat their issues as high-priority
- Data engine self-healing (fallback seeds) ensures restarts don't lose data
- Offline capability is a differentiator — cached entities in IndexedDB survive network outages
- `NEXT_PUBLIC_WWV_EDITION=local` must be the default, most-tested path
---
### 7. Paying Customers (Cloud Pro/Enterprise)
| Attribute | Description |
|---|---|
| **Role** | Users or organizations paying for `[user].app.worldwideview.dev` — the revenue engine |
| **Goal** | Reliable, always-on geospatial intelligence with history, higher rate limits, and team access |
| **Motivations** | Professional OSINT, military/defense analysis, logistics monitoring, journalism, corporate security |
| **Technical Ability** | **Variable** — from journalists who just need a globe to security analysts who want API access |
| **Emotional State** | **Paying customers have the lowest tolerance for bugs.** They expect enterprise-grade uptime and support. |
| **Definition of Success** | Data is live, history works, team members can access, exports work, it doesn't go down |
| **Pain Points** | Downtime, stale data, missing features compared to competitors, unclear what "Pro" actually gives them |
| **Interaction Surface** | Cloud dashboard, license key input, plugin marketplace, support channels, billing UI |
**Design implications:**
- **Uptime is king** — Uptime Kuma monitoring, error alerting, and rollback procedures exist for this reason
- Tier differentiation must be clear and valuable:
- Free: 3 users, 500MB, 24h history
- Pro: 20 users, 5GB, full history, snapshot capture
- Enterprise: unlimited, 50GB+, custom domains
- License key verification (RSA-signed JWT) prevents forgery but must not be onerous UX
- RLS tenant isolation is non-negotiable — data leaks between tenants are existential
- Response caching at the API proxy layer makes the $0/mo server viable for multiple tenants
- The cloud edition must feel **identical** to local — same UI, same plugins, same experience
---
### 8. Advertisers (Demo Edition Revenue)
| Attribute | Description |
|---|---|
| **Role** | Google AdSense (programmatic) — automated ad placement on the demo instance |
| **Goal** | Display relevant ads to engaged visitors, generate clicks/impressions |
| **Motivations** | ROI on ad spend; relevant audience targeting |
| **Technical Ability** | N/A — they interact through Google's ad platform, not with WWV directly |
| **Emotional State** | N/A — algorithmic. But their platform (AdSense) has strict technical requirements. |
| **Definition of Success** | Ads render correctly, non-intrusively, on viewable screen real estate |
| **Pain Points** | AdSense script crashes on resize, ads failing to fill on localhost/tunnels, ads overlapping interactive content |
| **Interaction Surface** | `DemoAdStrip.tsx`, `AdUnit.tsx`, Google AdSense publisher dashboard |
**Design implications:**
- Ad strip is a **structurally separate sibling container** — never overlaps the globe viewport
- Fixed-position elements (header, timeline, sidebars) use `--ad-strip-inset` CSS variable to avoid overlap
- AdSense `push()` errors on resize are handled by fully unmounting ad units below 850px height
- Ads are **demo edition only** — they never appear in local or cloud editions
- The "Why ads?" banner is a conscious UX choice that builds trust with users and is dismissible
- Ad revenue is a temporary bridge to sustainability, not the long-term business model
---
### 9. Data Source Providers (Upstream APIs)
| Attribute | Description |
|---|---|
| **Role** | External services that provide the raw data: OpenSky, AIS providers, NASA FIRMS, ACLED, CelesTrak, GPSJam, etc. |
| **Goal** | Serve data within their rate limits and terms of service |
| **Motivations** | Open data mission (NASA, CelesTrak), freemium conversion (OpenSky), research community |
| **Technical Ability** | N/A — they expose APIs; WWV consumes them |
| **Emotional State** | Hostile if rate-limited. Neutral if respected. |
| **Definition of Success** | API consumers stay within rate limits, attribute correctly, and don't DDOS their endpoints |
| **Pain Points** | Credential rotation failures (OpenSky), rate limit exhaustion, unattributed scraping |
| **Interaction Surface** | Data Engine seeders, API route proxies, `OPENSKY_CREDENTIALS` rotation, `pollingIntervalMs` settings |
**Design implications:**
- **Respect upstream rate limits** — this is an ethical and practical obligation
- Credential rotation (`OPENSKY_CREDENTIALS` comma-separated pairs) prevents per-key throttling
- The Data Engine's 5-minute Redis throttle (`setLiveSnapshot`) is specifically designed to stay within Upstash free-tier limits
- API proxy caching (1000 users → 1 upstream request) is architecturally essential
- Attribution (e.g., "Data: OpenSky Network") should be visible in entity detail panels
- Graceful degradation when APIs go down — show cached data, not error modals
---
## The Stakeholder Decision Matrix
When making any decision, use this matrix to identify impact:
| Decision Area | Owner | Core Devs | Contributors | Plugin Devs | Casual Visitors | Power Users | Paying Customers | Advertisers | Data Providers |
|---|---|---|---|---|---|---|---|---|---|
| **Plugin SDK change** | ⚠️ | ⚠️ | ⬜ | 🔴 | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| **UI/UX redesign** | ⚠️ | ⬜ | ⬜ | ⬜ | 🔴 | ⚠️ | 🔴 | ⚠️ | ⬜ |
| **Rendering pipeline change** | ⚠️ | 🔴 | ⬜ | ⚠️ | ⚠️ | ⚠️ | ⚠️ | ⬜ | ⬜ |
| **Database schema migration** | ⚠️ | ⚠️ | ⬜ | ⬜ | ⬜ | 🔴 | 🔴 | ⬜ | ⬜ |
| **New data source plugin** | ✅ | ⬜ | ✅ | ⬜ | ✅ | ✅ | ✅ | ⬜ | ⚠️ |
| **Pricing/tier change** | 🔴 | ⬜ | ⬜ | ⚠️ | ⬜ | ⚠️ | 🔴 | ⬜ | ⬜ |
| **Ad placement change** | ⚠️ | ⬜ | ⬜ | ⬜ | ⚠️ | ⬜ | ⬜ | 🔴 | ⬜ |
| **API rate limit adjustment** | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ | ⚠️ | ⬜ | 🔴 |
| **Marketplace policy** | 🔴 | ⬜ | ⬜ | 🔴 | ⬜ | ⬜ | ⚠️ | ⬜ | ⬜ |
| **Docker/deployment change** | 🔴 | ⚠️ | ⬜ | ⬜ | ⬜ | 🔴 | ⚠️ | ⬜ | ⬜ |
| **License change** | 🔴 | ⚠️ | 🔴 | 🔴 | ⬜ | ⚠️ | ⬜ | ⬜ | ⬜ |
**Legend:** 🔴 = Primary impact (design for this person) | ⚠️ = Secondary impact (consider carefully) | ✅ = Positive impact | ⬜ = Minimal/no impact
---
## The Golden Rules
### 1. Every Feature Has a Human
Before writing a single line of code, the AI agent MUST answer:
- **WHO** is this for? (Name the stakeholder)
- **WHY** do they need it? (What problem does it solve for them?)
- **HOW** will they discover/use it? (What's their journey?)
- **WHAT** happens when it breaks? (Graceful degradation plan)
### 2. Reduce Friction, Not Features
The most dangerous failure mode is building powerful systems with hostile interfaces. Apply the **Principle of Least Effort**:
- Contributors: one-command setup, not a wiki page of prerequisites
- Plugin devs: `renderEntity()` returns an object, not a Cesium Primitive
- Casual visitors: zero onboarding, zero configuration
- Power users: sensible defaults, optional configuration
- Paying customers: it just works, always
### 3. Design for the Least Technical User in Each Cohort
| Cohort | Least Technical Member | Design Target |
|---|---|---|
| Plugin Developers | Data analyst who can write JSON | Declarative plugin format |
| Contributors | Junior dev on their first open source PR | Scaffold scripts + clear CONTRIBUTING.md |
| Casual Visitors | Non-technical person who followed a Twitter link | Auto-loading layers, intuitive globe controls |
| Power Users | Sysadmin who can `docker-compose up` but doesn't know Next.js | Pre-built Docker images, `.env` only config |
| Paying Customers | Journalist tracking a conflict | Clear UI, no jargon, exportable data |
### 4. Protect the Ecosystem Trust
The trust hierarchy is cryptographically enforced (Ed25519 signed registry), but trust is also social:
- **Built-in plugins** must always work — they're the first impression
- **Verified plugins** carry WorldWideView's reputation — review them seriously
- **Unverified plugins** get a warning dialog — users make informed decisions
- **Plugin developers** who see installs and analytics stay motivated
- **Data providers** who aren't rate-limit abused will keep serving data
### 5. Revenue Serves Sustainability, Not Extraction
The business model exists to keep the project alive:
- Ads on demo = temporary hosting cost recovery (transparent "Why ads?" banner)
- Cloud Pro = value exchange (history, higher limits, team access)
- Marketplace cut = sustainable ecosystem (Stripe Connect revenue split)
- Plugin developer monetization = VS Code model (developers set their own prices)
Never design a feature that extracts value from users without returning it. The demo must always be useful, not crippled.
---
## Applying This Document
### For the AI Agent
When the agent is asked to implement a feature, modify architecture, or solve a bug, it MUST:
1. **Identify affected stakeholders** from the list above
2. **Consider the least-technical user** in each affected cohort
3. **Check the decision matrix** for impact severity
4. **Apply the golden rules** to the implementation approach
5. **Document stakeholder considerations** in PR descriptions and implementation plans
### For Human Developers
When reviewing PRs or proposing features, ask:
- "Did we consider how this affects plugin developers?"
- "Will a first-time contributor understand this?"
- "Does this work on the demo without configuration?"
- "Will this survive a Docker restart on the ThinkPad?"
---
## Version History
| Date | Change |
|---|---|
| 2026-04-05 | Initial stakeholder map created from comprehensive project research |
-43
View File
@@ -1,43 +0,0 @@
---
trigger: glob
description: Guidelines on managing the single source of truth across WorldWideView using Zustand, avoiding prop-drilling, and isolating UI rerenders.
globs: "src/core/state/**/*, src/components/**/*"
---
# Zustand State Management
## Purpose
Guidelines on managing the single source of truth across WorldWideView using Zustand, avoiding prop-drilling, and isolating UI rerenders.
## Single Centralized State
WorldWideView utilizes a monolithic Zustand store located at `src/core/state/store.ts`. It is broken down into exactly 9 semantic slices.
| Slice Name | Purpose | Example State |
|---|---|---|
| **Globe** | Cesium Viewer state | `cameraPosition`, `selectedEntity` |
| **Layers** | Loaded map tiles/overlays | `activeTileSets` |
| **Timeline** | Application current time | `currentTimeRange` |
| **UI** | Global overlay states | `isFilterMenuOpen` |
| **Filters** | Active visual filtration | `searchQuery`, `categoryFilter` |
| **Data** | Entity registry | `entitiesByPlugin` |
| **Config** | User preferences | `pollingIntervals`, `pluginSettings` |
| **Favorites**| Pinned entities | `favoriteIds` |
| **GeoJSON** | Local files loaded in | `localImports` |
## When to Apply
When passing deep state across components, or when a plugin must push state configurations onto the screen.
## Best Practices
### Avoid Prop Drilling
**Never** pass props further than two levels down. If a component deeper in the tree needs access to something (e.g., `cameraPosition`), it must `useStore(s => s.globe.cameraPosition)`.
### Selector Destructuring Avoidance
> [!WARNING]
> Do not destructure wide objects in selectors.
> Right: `const isHUD = useStore(state => state.ui.hudVisible);`
> Wrong: `const { hudVisible } = useStore(state => state.ui);`
> The wrong approach rerenders the component any time *anything* in the `ui` slice changes.
### Config and Plugins
If your plugin has user-configurable values (e.g., color settings, toggle states), its data MUST live inside `state.config.dataConfig.pluginSettings[pluginId]`. Do not spawn un-bound `useState` hooks for UI properties if they affect the global app state.
@@ -1,22 +0,0 @@
---
trigger: model_decision
description: Troubleshooting and debugging guidelines for WorldWideView, covering latency, logging, and namespace collisions.
---
# Troubleshooting and Debugging
## 1. Plugin Data Latency
If there is a noticeable delay (e.g., ~1 second) between enabling a plugin and the initial data arriving on the globe, **do not assume the UI is blocking**.
- **WebSocket Handshake**: Initial latency is often caused by the time it takes to establish a new WebSocket connection to the engine's `/stream`.
- **Upstream Cooldowns**: Plugins querying external providers (like OpenSky for `aviation`) have strict rate limits. If a snapshot returns a `404`, it might mean the seeder is in a cooldown period or hasn't successfully fetched data yet. This is expected behavior.
- **Diagnosis**: Instrument the `DataBusSubscriber` and `WsClient` with timing logs (`performance.now()`) to track exactly where the bottleneck occurs (manifest discovery vs connection establishment vs first payload).
## 2. Log Verbosity and Structured Logging
We employ a **structured logging strategy** in the frontend to reduce noise and improve readability.
- **Rule**: Do not add excessive `console.log` statements that dump raw JSON payloads or stream contents directly to the console in core files like `WsClient.ts` or `PluginManager.ts`.
- **Implementation**: If detailed logs are needed, use conditional debug levels or summarize the output (e.g., logging payload sizes or entity counts instead of the full object array). This ensures critical diagnostic information remains visible without overflowing the terminal.
## 3. Namespace Collisions
If a plugin fails to load in the Data Engine with a `404` or `Error [ERR_MODULE_NOT_FOUND]` error during startup:
- **Root Cause**: The seeder may exist simultaneously in both the `wwv-seeders-community` and `wwv-seeders-private` repositories.
- **Resolution**: Seeders must have a unique namespace. Ensure that private plugins (e.g., `aviation`, `maritime`, `military-aviation`) are explicitly removed from the community repository to prevent workspace-level module resolution failures when the V2 engine attempts to load them. Private seeders take priority, but overlapping names will cause dependency conflicts.
-17
View File
@@ -1,17 +0,0 @@
{
"env": {
"CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS": "1"
},
"permissions": {
"allow": [
"Read(src/**)",
"Bash(npm test)"
],
"deny": [
"Read(.env)",
"Write(config.json)"
],
"defaultMode": "bypassPermissions"
},
"prefersReducedMotion": false
}
@@ -1,257 +0,0 @@
---
name: migrate-legacy-plugin
description: Use when moving a WorldWideView plugin from packages/ (or ../worldwideview-plugins/packages/) to local-plugins/, converting legacy plugins to the new Engine & Payload architecture, or when a plugin hardcodes engine URLs, bundles Node.js built-ins, or uses import.meta.url for file paths inside Docker containers
---
# Migrate Legacy Plugin to Modern Architecture
## Overview
Moves a WorldWideView plugin from the old `packages/` monorepo location (or the external `worldwideview-plugins` repository) into the `local-plugins/` directory (which is itself a publishable Git repository). Simultaneously fixes two **independent** concerns that legacy plugins always have:
1. **Frontend routing** — hardcoded engine URLs must be replaced with `resolveEngineUrl()`
2. **Backend seeder build** — ESM bundling and container-aware file paths
**Core principle:** These are two separate problems in two separate codebases. Never fix one and forget the other.
## When to Use
- Moving a plugin from `packages/wwv-plugin-<name>/` (or `../worldwideview-plugins/packages/wwv-plugin-<name>/`) to `local-plugins/wwv-plugin-<name>/`
- Plugin's `fetch()` hardcodes a URL like `https://dataengine.worldwideview.dev`
- Plugin uses `this.context?.apiBaseUrl` instead of `resolveEngineUrl()`
- Backend seeder crashes with `Dynamic require` or `ERR_INVALID_FILE_URL_PATH` in Docker
- Seeder uses `fileURLToPath(import.meta.url)` for locating seed data files
- Plugin builds with `tsc` instead of `tsup`/`vite`
**Do NOT use for:**
- Creating a brand-new plugin from scratch → use `worldwideview-plugin-creation` skill
- Modifying rendering or Cesium primitives → see `cesium-rendering` rule
- Debugging state management → see `state-management` rule
## The Two Independent Concerns
> [!TIP]
> **Automated Migration Script**
> A new automation script is available! Before doing this manually, simply run:
> ```bash
> node scripts/migrate-plugin.mjs <plugin-name>
> ```
> Example: `node scripts/migrate-plugin.mjs gps-jamming`
>
> The script will automatically copy the frontend plugin, inject peerDependencies, refactor `index.tsx` routing, fix the backend seeder's build settings and `SEEDERS_DIR` paths, and register the plugin in `defaultPlugins.ts`.
> **You still need to run `pnpm install` and verify the builds manually as described below.**
> [!IMPORTANT]
> Legacy plugins always have BOTH of these problems. They are independent — you can fix one without the other, but you MUST fix both. Do not get tunnel-visioned into only fixing one.
| Concern | Where | Symptom | Fix |
|---|---|---|---|
| **Frontend routing** | `local-plugins/wwv-plugin-<name>/src/index.ts` | Plugin always hits cloud engine, ignores local dev engine | Use `context.getEngineUrl()` |
| **Backend build** | `local-seeders/<namespace>/packages/<name>/` | Container crashes on startup with ESM errors | `tsup --format esm --target es2022`, use `SEEDERS_DIR` |
---
## Step 1: Move the Plugin Directory
Copy the plugin from `packages/` (or the external plugins repo) to `local-plugins/`:
```bash
# From project root (if in local packages/)
cp -r packages/wwv-plugin-<name> local-plugins/wwv-plugin-<name>
# OR from external plugins repository:
cp -r ../worldwideview-plugins/packages/wwv-plugin-<name> local-plugins/wwv-plugin-<name>
```
The `local-plugins/` directory is:
- Part of the pnpm workspace (`pnpm-workspace.yaml` includes `local-plugins/*`)
- Gitignored from the main worldwideview repo (it IS its own Git repo — `worldwideview-plugins`)
- Auto-discovered by the `dev:plugins` watcher during `pnpm dev`
## Step 2: Fix Frontend Routing (getEngineUrl)
This is the fix that gets missed most often. Legacy plugins typically hardcode their engine URL or use messy dynamic imports.
```typescript
// ❌ LEGACY — always hits cloud, ignores local engine
const baseUrl = this.context?.apiBaseUrl || "https://dataengine.worldwideview.dev";
const res = await fetch(`${baseUrl}/api/myplugin`);
// ❌ BAD (Previous Workaround) — creates bundling chunks and causes 404s
const { resolveEngineUrl } = await import("@/core/data/resolveEngineUrl");
const wsUrl = resolveEngineUrl(this.id);
const engineBase = wsUrl.replace(/^ws/, 'http').replace(/\/stream$/, '');
const res = await fetch(`${engineBase}/api/myplugin`);
```
The correct approach uses `this.context.getEngineUrl()` provided by the Host when the plugin is initialized. The Host automatically handles checking `localhost:5000/manifest`, applying `getServerConfig().streamUrl`, reading `plugin.json`, or falling back to the cloud.
```typescript
// ✅ MODERN — resolves local-first, respects split-routing, clean code
async fetch(_timeRange: TimeRange): Promise<GeoEntity[]> {
try {
const engineBase = this.context!.getEngineUrl();
const res = await globalThis.fetch(`${engineBase}/api/${this.id}`);
if (!res.ok) throw new Error(`Backend returned ${res.status}`);
const data = await res.json();
return this.mapPayloadToEntities(data);
} catch (err) {
console.error(`[${this.id}] Fetch error:`, err);
return [];
}
}
```
If the plugin is WebSocket-only (`getPollingInterval()` returns 0 and `fetch()` returns `[]`), the frontend routing is handled automatically by `DataBusSubscriber` and `WsClient`. No changes needed to the plugin class — but verify the seeder ID matches.
### Why This Matters
Without this fix, running `pnpm dev:all` starts a local engine at `localhost:5000` with your seeder, but the frontend plugin ignores it and hits the cloud. You'll see stale or missing data and waste time debugging the wrong layer.
## Step 3: Update package.json
Ensure the plugin's `package.json` follows the modern format:
```json
{
"name": "@worldwideview/wwv-plugin-<name>",
"version": "1.0.0",
"main": "dist/frontend.mjs",
"types": "src/index.ts",
"type": "module",
"module": "dist/frontend.mjs",
"files": ["src"],
"worldwideview": {
"id": "<name>",
"icon": "IconName",
"category": "category-name",
"format": "bundle",
"capabilities": ["layer"]
},
"peerDependencies": {
"@worldwideview/wwv-plugin-sdk": "^1.4.10"
},
"scripts": {
"build": "vite build"
}
}
```
Key differences from legacy:
- `"main"` points to `dist/frontend.mjs` (not `src/index.ts`)
- Has `"type": "module"` and `"module"` field
- Uses `peerDependencies` (not `dependencies`) for SDK
- Uses `"workspace:*"` only for libraries that must be linked locally (e.g., `wwv-lib-incidents`)
## Step 4: Ensure Vite Build Config
Every `local-plugins/` plugin needs this `vite.config.ts`:
```typescript
import { defineConfig } from "vite";
import { wwvPluginGlobals } from "@worldwideview/wwv-plugin-sdk";
export default defineConfig({
plugins: [wwvPluginGlobals()],
build: {
lib: {
entry: "src/index.ts",
formats: ["es"],
fileName: () => "frontend.mjs",
},
minify: true,
sourcemap: false,
},
});
```
`wwvPluginGlobals()` externalizes React, Cesium, Resium, zustand, and the SDK so the plugin doesn't bundle its own copies.
## Step 5: Fix Backend Seeder (if applicable)
If the plugin has a corresponding seeder in `local-seeders/`, fix two things:
### 5a. ESM Build
Legacy seeders built with `tsc` or bundled Node built-ins. Fix:
```json
"scripts": {
"build": "tsup src/index.ts --format esm --target es2022 --clean"
}
```
This produces clean ESM that the data engine can `import()` without `Dynamic require` errors.
### 5b. Container-Aware File Paths
Legacy seeders used `fileURLToPath(import.meta.url)` to locate seed data. This breaks inside Docker because the transpiled path doesn't match the container filesystem.
```typescript
// ❌ LEGACY — breaks in Docker container
import { fileURLToPath } from "url";
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const dataPath = path.join(__dirname, "data", "seed.json");
// ✅ MODERN — works everywhere
const dataPath = process.env.SEEDERS_DIR
? path.join(process.env.SEEDERS_DIR, "community", "packages", "<name>", "data", "seed.json")
: path.resolve(process.cwd(), "data", "seed.json");
```
`SEEDERS_DIR` is set by the Docker container to `/app/seeders`. When running locally outside Docker, the fallback uses `process.cwd()`.
### 5c. Seeder Directory Naming
The folder name inside `local-seeders/community/packages/` or `local-seeders/private/packages/` **must exactly match** the plugin's string ID (`this.id` in `index.tsx`).
The `wwv-data-engine`'s auto-discovery logic (`seeder-loader.ts`) derives the internal plugin ID from the directory name. If the folder is named `gpsjam` but the frontend plugin expects `gps-jamming`, the frontend will not find it in the `/manifest` and will silently fall back to the cloud engine, breaking local testing.
```bash
# ❌ INCORRECT (Frontend expects "gps-jamming")
local-seeders/community/packages/gpsjam/
# ✅ CORRECT
local-seeders/community/packages/gps-jamming/
```
## Step 6: Clean Up packages/ References
After moving to `local-plugins/`, remove the old references:
- [ ] Delete `packages/wwv-plugin-<name>/` directory (or the source directory in `../worldwideview-plugins/packages/`)
- [ ] Remove from `transpilePackages` in `next.config.ts` (if present)
- [ ] Remove path alias from `tsconfig.json` `paths` (if present)
- [ ] Remove any registration in `AppShell.tsx` (now loaded dynamically)
- [ ] Run `pnpm install` from project root
## Step 7: Verify
```bash
# 1. Start everything
pnpm dev:all
# 2. Check engine discovered the seeder (if applicable)
curl http://localhost:5000/manifest
# Should include your plugin ID
# 3. Check frontend console — should show:
# [EngineManifest] Local engine detected: N seeders ["your-plugin", ...]
# NOT: fetching from https://dataengine.worldwideview.dev
# 4. Toggle the layer on the globe — entities should render
```
## Migration Checklist
- [ ] Plugin copied from `packages/` (or external repo) to `local-plugins/`
- [ ] Frontend `fetch()` uses `this.context.getEngineUrl()` (not hardcoded URLs or dynamic imports)
- [ ] `package.json` has `"type": "module"`, `"main": "dist/frontend.mjs"`
- [ ] `vite.config.ts` uses `wwvPluginGlobals()` externalization
- [ ] Backend seeder builds with `tsup --format esm --target es2022` (if applicable)
- [ ] Backend seeder uses `process.env.SEEDERS_DIR` for file paths (if applicable)
- [ ] Old source directory deleted
- [ ] Old `transpilePackages`/`tsconfig paths` entries removed
- [ ] `pnpm install` run from root
- [ ] `pnpm dev:all` → local engine detected → data renders on globe
@@ -1,64 +0,0 @@
---
name: supabase-postgres-best-practices
description: Postgres performance optimization and best practices from Supabase. Use this skill when writing, reviewing, or optimizing Postgres queries, schema designs, or database configurations.
license: MIT
metadata:
author: supabase
version: "1.1.1"
organization: Supabase
date: January 2026
abstract: Comprehensive Postgres performance optimization guide for developers using Supabase and Postgres. Contains performance rules across 8 categories, prioritized by impact from critical (query performance, connection management) to incremental (advanced features). Each rule includes detailed explanations, incorrect vs. correct SQL examples, query plan analysis, and specific performance metrics to guide automated optimization and code generation.
---
# Supabase Postgres Best Practices
Comprehensive performance optimization guide for Postgres, maintained by Supabase. Contains rules across 8 categories, prioritized by impact to guide automated query optimization and schema design.
## When to Apply
Reference these guidelines when:
- Writing SQL queries or designing schemas
- Implementing indexes or query optimization
- Reviewing database performance issues
- Configuring connection pooling or scaling
- Optimizing for Postgres-specific features
- Working with Row-Level Security (RLS)
## Rule Categories by Priority
| Priority | Category | Impact | Prefix |
|----------|----------|--------|--------|
| 1 | Query Performance | CRITICAL | `query-` |
| 2 | Connection Management | CRITICAL | `conn-` |
| 3 | Security & RLS | CRITICAL | `security-` |
| 4 | Schema Design | HIGH | `schema-` |
| 5 | Concurrency & Locking | MEDIUM-HIGH | `lock-` |
| 6 | Data Access Patterns | MEDIUM | `data-` |
| 7 | Monitoring & Diagnostics | LOW-MEDIUM | `monitor-` |
| 8 | Advanced Features | LOW | `advanced-` |
## How to Use
Read individual rule files for detailed explanations and SQL examples:
```
references/query-missing-indexes.md
references/query-partial-indexes.md
references/_sections.md
```
Each rule file contains:
- Brief explanation of why it matters
- Incorrect SQL example with explanation
- Correct SQL example with explanation
- Optional EXPLAIN output or metrics
- Additional context and references
- Supabase-specific notes (when applicable)
## References
- https://www.postgresql.org/docs/current/
- https://supabase.com/docs
- https://wiki.postgresql.org/wiki/Performance_Optimization
- https://supabase.com/docs/guides/database/overview
- https://supabase.com/docs/guides/auth/row-level-security
@@ -1,170 +0,0 @@
# Writing Guidelines for Postgres References
This document provides guidelines for creating effective Postgres best
practice references that work well with AI agents and LLMs.
## Key Principles
### 1. Concrete Transformation Patterns
Show exact SQL rewrites. Avoid philosophical advice.
**Good:** "Use `WHERE id = ANY(ARRAY[...])` instead of
`WHERE id IN (SELECT ...)`" **Bad:** "Design good schemas"
### 2. Error-First Structure
Always show the problematic pattern first, then the solution. This trains agents
to recognize anti-patterns.
```markdown
**Incorrect (sequential queries):** [bad example]
**Correct (batched query):** [good example]
```
### 3. Quantified Impact
Include specific metrics. Helps agents prioritize fixes.
**Good:** "10x faster queries", "50% smaller index", "Eliminates N+1"
**Bad:** "Faster", "Better", "More efficient"
### 4. Self-Contained Examples
Examples should be complete and runnable (or close to it). Include `CREATE TABLE`
if context is needed.
```sql
-- Include table definition when needed for clarity
CREATE TABLE users (
id bigint PRIMARY KEY,
email text NOT NULL,
deleted_at timestamptz
);
-- Now show the index
CREATE INDEX users_active_email_idx ON users(email) WHERE deleted_at IS NULL;
```
### 5. Semantic Naming
Use meaningful table/column names. Names carry intent for LLMs.
**Good:** `users`, `email`, `created_at`, `is_active`
**Bad:** `table1`, `col1`, `field`, `flag`
---
## Code Example Standards
### SQL Formatting
```sql
-- Use lowercase keywords, clear formatting
CREATE INDEX CONCURRENTLY users_email_idx
ON users(email)
WHERE deleted_at IS NULL;
-- Not cramped or ALL CAPS
CREATE INDEX CONCURRENTLY USERS_EMAIL_IDX ON USERS(EMAIL) WHERE DELETED_AT IS NULL;
```
### Comments
- Explain _why_, not _what_
- Highlight performance implications
- Point out common pitfalls
### Language Tags
- `sql` - Standard SQL queries
- `plpgsql` - Stored procedures/functions
- `typescript` - Application code (when needed)
- `python` - Application code (when needed)
---
## When to Include Application Code
**Default: SQL Only**
Most references should focus on pure SQL patterns. This keeps examples portable.
**Include Application Code When:**
- Connection pooling configuration
- Transaction management in application context
- ORM anti-patterns (N+1 in Prisma/TypeORM)
- Prepared statement usage
**Format for Mixed Examples:**
````markdown
**Incorrect (N+1 in application):**
```typescript
for (const user of users) {
const posts = await db.query("SELECT * FROM posts WHERE user_id = $1", [
user.id,
]);
}
```
````
**Correct (batch query):**
```typescript
const posts = await db.query("SELECT * FROM posts WHERE user_id = ANY($1)", [
userIds,
]);
```
---
## Impact Level Guidelines
| Level | Improvement | Use When |
|-------|-------------|----------|
| **CRITICAL** | 10-100x | Missing indexes, connection exhaustion, sequential scans on large tables |
| **HIGH** | 5-20x | Wrong index types, poor partitioning, missing covering indexes |
| **MEDIUM-HIGH** | 2-5x | N+1 queries, inefficient pagination, RLS optimization |
| **MEDIUM** | 1.5-3x | Redundant indexes, query plan instability |
| **LOW-MEDIUM** | 1.2-2x | VACUUM tuning, configuration tweaks |
| **LOW** | Incremental | Advanced patterns, edge cases |
---
## Reference Standards
**Primary Sources:**
- Official Postgres documentation
- Supabase documentation
- Postgres wiki
- Established blogs (2ndQuadrant, Crunchy Data)
**Format:**
```markdown
Reference:
[Postgres Indexes](https://www.postgresql.org/docs/current/indexes.html)
```
---
## Review Checklist
Before submitting a reference:
- [ ] Title is clear and action-oriented
- [ ] Impact level matches the performance gain
- [ ] impactDescription includes quantification
- [ ] Explanation is concise (1-2 sentences)
- [ ] Has at least 1 **Incorrect** SQL example
- [ ] Has at least 1 **Correct** SQL example
- [ ] SQL uses semantic naming
- [ ] Comments explain _why_, not _what_
- [ ] Trade-offs mentioned if applicable
- [ ] Reference links included
- [ ] `pnpm test` passes
@@ -1,39 +0,0 @@
# Section Definitions
This file defines the rule categories for Postgres best practices. Rules are automatically assigned to sections based on their filename prefix.
Take the examples below as pure demonstrative. Replace each section with the actual rule categories for Postgres best practices.
---
## 1. Query Performance (query)
**Impact:** CRITICAL
**Description:** Slow queries, missing indexes, inefficient query plans. The most common source of Postgres performance issues.
## 2. Connection Management (conn)
**Impact:** CRITICAL
**Description:** Connection pooling, limits, and serverless strategies. Critical for applications with high concurrency or serverless deployments.
## 3. Security & RLS (security)
**Impact:** CRITICAL
**Description:** Row-Level Security policies, privilege management, and authentication patterns.
## 4. Schema Design (schema)
**Impact:** HIGH
**Description:** Table design, index strategies, partitioning, and data type selection. Foundation for long-term performance.
## 5. Concurrency & Locking (lock)
**Impact:** MEDIUM-HIGH
**Description:** Transaction management, isolation levels, deadlock prevention, and lock contention patterns.
## 6. Data Access Patterns (data)
**Impact:** MEDIUM
**Description:** N+1 query elimination, batch operations, cursor-based pagination, and efficient data fetching.
## 7. Monitoring & Diagnostics (monitor)
**Impact:** LOW-MEDIUM
**Description:** Using pg_stat_statements, EXPLAIN ANALYZE, metrics collection, and performance diagnostics.
## 8. Advanced Features (advanced)
**Impact:** LOW
**Description:** Full-text search, JSONB optimization, PostGIS, extensions, and advanced Postgres features.
@@ -1,34 +0,0 @@
---
title: Clear, Action-Oriented Title (e.g., "Use Partial Indexes for Filtered Queries")
impact: MEDIUM
impactDescription: 5-20x query speedup for filtered queries
tags: indexes, query-optimization, performance
---
## [Rule Title]
[1-2 sentence explanation of the problem and why it matters. Focus on performance impact.]
**Incorrect (describe the problem):**
```sql
-- Comment explaining what makes this slow/problematic
CREATE INDEX users_email_idx ON users(email);
SELECT * FROM users WHERE email = '[email protected]' AND deleted_at IS NULL;
-- This scans deleted records unnecessarily
```
**Correct (describe the solution):**
```sql
-- Comment explaining why this is better
CREATE INDEX users_active_email_idx ON users(email) WHERE deleted_at IS NULL;
SELECT * FROM users WHERE email = '[email protected]' AND deleted_at IS NULL;
-- Only indexes active users, 10x smaller index, faster queries
```
[Optional: Additional context, edge cases, or trade-offs]
Reference: [Postgres Docs](https://www.postgresql.org/docs/current/)
@@ -1,55 +0,0 @@
---
title: Use tsvector for Full-Text Search
impact: MEDIUM
impactDescription: 100x faster than LIKE, with ranking support
tags: full-text-search, tsvector, gin, search
---
## Use tsvector for Full-Text Search
LIKE with wildcards can't use indexes. Full-text search with tsvector is orders of magnitude faster.
**Incorrect (LIKE pattern matching):**
```sql
-- Cannot use index, scans all rows
select * from articles where content like '%postgresql%';
-- Case-insensitive makes it worse
select * from articles where lower(content) like '%postgresql%';
```
**Correct (full-text search with tsvector):**
```sql
-- Add tsvector column and index
alter table articles add column search_vector tsvector
generated always as (to_tsvector('english', coalesce(title,'') || ' ' || coalesce(content,''))) stored;
create index articles_search_idx on articles using gin (search_vector);
-- Fast full-text search
select * from articles
where search_vector @@ to_tsquery('english', 'postgresql & performance');
-- With ranking
select *, ts_rank(search_vector, query) as rank
from articles, to_tsquery('english', 'postgresql') query
where search_vector @@ query
order by rank desc;
```
Search multiple terms:
```sql
-- AND: both terms required
to_tsquery('postgresql & performance')
-- OR: either term
to_tsquery('postgresql | mysql')
-- Prefix matching
to_tsquery('post:*')
```
Reference: [Full Text Search](https://supabase.com/docs/guides/database/full-text-search)
@@ -1,49 +0,0 @@
---
title: Index JSONB Columns for Efficient Querying
impact: MEDIUM
impactDescription: 10-100x faster JSONB queries with proper indexing
tags: jsonb, gin, indexes, json
---
## Index JSONB Columns for Efficient Querying
JSONB queries without indexes scan the entire table. Use GIN indexes for containment queries.
**Incorrect (no index on JSONB):**
```sql
create table products (
id bigint primary key,
attributes jsonb
);
-- Full table scan for every query
select * from products where attributes @> '{"color": "red"}';
select * from products where attributes->>'brand' = 'Nike';
```
**Correct (GIN index for JSONB):**
```sql
-- GIN index for containment operators (@>, ?, ?&, ?|)
create index products_attrs_gin on products using gin (attributes);
-- Now containment queries use the index
select * from products where attributes @> '{"color": "red"}';
-- For specific key lookups, use expression index
create index products_brand_idx on products ((attributes->>'brand'));
select * from products where attributes->>'brand' = 'Nike';
```
Choose the right operator class:
```sql
-- jsonb_ops (default): supports all operators, larger index
create index idx1 on products using gin (attributes);
-- jsonb_path_ops: only @> operator, but 2-3x smaller index
create index idx2 on products using gin (attributes jsonb_path_ops);
```
Reference: [JSONB Indexes](https://www.postgresql.org/docs/current/datatype-json.html#JSON-INDEXING)
@@ -1,46 +0,0 @@
---
title: Configure Idle Connection Timeouts
impact: HIGH
impactDescription: Reclaim 30-50% of connection slots from idle clients
tags: connections, timeout, idle, resource-management
---
## Configure Idle Connection Timeouts
Idle connections waste resources. Configure timeouts to automatically reclaim them.
**Incorrect (connections held indefinitely):**
```sql
-- No timeout configured
show idle_in_transaction_session_timeout; -- 0 (disabled)
-- Connections stay open forever, even when idle
select pid, state, state_change, query
from pg_stat_activity
where state = 'idle in transaction';
-- Shows transactions idle for hours, holding locks
```
**Correct (automatic cleanup of idle connections):**
```sql
-- Terminate connections idle in transaction after 30 seconds
alter system set idle_in_transaction_session_timeout = '30s';
-- Terminate completely idle connections after 10 minutes
alter system set idle_session_timeout = '10min';
-- Reload configuration
select pg_reload_conf();
```
For pooled connections, configure at the pooler level:
```ini
# pgbouncer.ini
server_idle_timeout = 60
client_idle_timeout = 300
```
Reference: [Connection Timeouts](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-IDLE-IN-TRANSACTION-SESSION-TIMEOUT)
@@ -1,44 +0,0 @@
---
title: Set Appropriate Connection Limits
impact: CRITICAL
impactDescription: Prevent database crashes and memory exhaustion
tags: connections, max-connections, limits, stability
---
## Set Appropriate Connection Limits
Too many connections exhaust memory and degrade performance. Set limits based on available resources.
**Incorrect (unlimited or excessive connections):**
```sql
-- Default max_connections = 100, but often increased blindly
show max_connections; -- 500 (way too high for 4GB RAM)
-- Each connection uses 1-3MB RAM
-- 500 connections * 2MB = 1GB just for connections!
-- Out of memory errors under load
```
**Correct (calculate based on resources):**
```sql
-- Formula: max_connections = (RAM in MB / 5MB per connection) - reserved
-- For 4GB RAM: (4096 / 5) - 10 = ~800 theoretical max
-- But practically, 100-200 is better for query performance
-- Recommended settings for 4GB RAM
alter system set max_connections = 100;
-- Also set work_mem appropriately
-- work_mem * max_connections should not exceed 25% of RAM
alter system set work_mem = '8MB'; -- 8MB * 100 = 800MB max
```
Monitor connection usage:
```sql
select count(*), state from pg_stat_activity group by state;
```
Reference: [Database Connections](https://supabase.com/docs/guides/platform/performance#connection-management)
@@ -1,41 +0,0 @@
---
title: Use Connection Pooling for All Applications
impact: CRITICAL
impactDescription: Handle 10-100x more concurrent users
tags: connection-pooling, pgbouncer, performance, scalability
---
## Use Connection Pooling for All Applications
Postgres connections are expensive (1-3MB RAM each). Without pooling, applications exhaust connections under load.
**Incorrect (new connection per request):**
```sql
-- Each request creates a new connection
-- Application code: db.connect() per request
-- Result: 500 concurrent users = 500 connections = crashed database
-- Check current connections
select count(*) from pg_stat_activity; -- 487 connections!
```
**Correct (connection pooling):**
```sql
-- Use a pooler like PgBouncer between app and database
-- Application connects to pooler, pooler reuses a small pool to Postgres
-- Configure pool_size based on: (CPU cores * 2) + spindle_count
-- Example for 4 cores: pool_size = 10
-- Result: 500 concurrent users share 10 actual connections
select count(*) from pg_stat_activity; -- 10 connections
```
Pool modes:
- **Transaction mode**: connection returned after each transaction (best for most apps)
- **Session mode**: connection held for entire session (needed for prepared statements, temp tables)
Reference: [Connection Pooling](https://supabase.com/docs/guides/database/connecting-to-postgres#connection-pooler)
@@ -1,46 +0,0 @@
---
title: Use Prepared Statements Correctly with Pooling
impact: HIGH
impactDescription: Avoid prepared statement conflicts in pooled environments
tags: prepared-statements, connection-pooling, transaction-mode
---
## Use Prepared Statements Correctly with Pooling
Prepared statements are tied to individual database connections. In transaction-mode pooling, connections are shared, causing conflicts.
**Incorrect (named prepared statements with transaction pooling):**
```sql
-- Named prepared statement
prepare get_user as select * from users where id = $1;
-- In transaction mode pooling, next request may get different connection
execute get_user(123);
-- ERROR: prepared statement "get_user" does not exist
```
**Correct (use unnamed statements or session mode):**
```sql
-- Option 1: Use unnamed prepared statements (most ORMs do this automatically)
-- The query is prepared and executed in a single protocol message
-- Option 2: Deallocate after use in transaction mode
prepare get_user as select * from users where id = $1;
execute get_user(123);
deallocate get_user;
-- Option 3: Use session mode pooling (port 5432 vs 6543)
-- Connection is held for entire session, prepared statements persist
```
Check your driver settings:
```sql
-- Many drivers use prepared statements by default
-- Node.js pg: { prepare: false } to disable
-- JDBC: prepareThreshold=0 to disable
```
Reference: [Prepared Statements with Pooling](https://supabase.com/docs/guides/database/connecting-to-postgres#connection-pool-modes)
@@ -1,54 +0,0 @@
---
title: Batch INSERT Statements for Bulk Data
impact: MEDIUM
impactDescription: 10-50x faster bulk inserts
tags: batch, insert, bulk, performance, copy
---
## Batch INSERT Statements for Bulk Data
Individual INSERT statements have high overhead. Batch multiple rows in single statements or use COPY.
**Incorrect (individual inserts):**
```sql
-- Each insert is a separate transaction and round trip
insert into events (user_id, action) values (1, 'click');
insert into events (user_id, action) values (1, 'view');
insert into events (user_id, action) values (2, 'click');
-- ... 1000 more individual inserts
-- 1000 inserts = 1000 round trips = slow
```
**Correct (batch insert):**
```sql
-- Multiple rows in single statement
insert into events (user_id, action) values
(1, 'click'),
(1, 'view'),
(2, 'click'),
-- ... up to ~1000 rows per batch
(999, 'view');
-- One round trip for 1000 rows
```
For large imports, use COPY:
```sql
-- COPY is fastest for bulk loading
copy events (user_id, action, created_at)
from '/path/to/data.csv'
with (format csv, header true);
-- Or from stdin in application
copy events (user_id, action) from stdin with (format csv);
1,click
1,view
2,click
\.
```
Reference: [COPY](https://www.postgresql.org/docs/current/sql-copy.html)
@@ -1,53 +0,0 @@
---
title: Eliminate N+1 Queries with Batch Loading
impact: MEDIUM-HIGH
impactDescription: 10-100x fewer database round trips
tags: n-plus-one, batch, performance, queries
---
## Eliminate N+1 Queries with Batch Loading
N+1 queries execute one query per item in a loop. Batch them into a single query using arrays or JOINs.
**Incorrect (N+1 queries):**
```sql
-- First query: get all users
select id from users where active = true; -- Returns 100 IDs
-- Then N queries, one per user
select * from orders where user_id = 1;
select * from orders where user_id = 2;
select * from orders where user_id = 3;
-- ... 97 more queries!
-- Total: 101 round trips to database
```
**Correct (single batch query):**
```sql
-- Collect IDs and query once with ANY
select * from orders where user_id = any(array[1, 2, 3, ...]);
-- Or use JOIN instead of loop
select u.id, u.name, o.*
from users u
left join orders o on o.user_id = u.id
where u.active = true;
-- Total: 1 round trip
```
Application pattern:
```sql
-- Instead of looping in application code:
-- for user in users: db.query("SELECT * FROM orders WHERE user_id = $1", user.id)
-- Pass array parameter:
select * from orders where user_id = any($1::bigint[]);
-- Application passes: [1, 2, 3, 4, 5, ...]
```
Reference: [N+1 Query Problem](https://supabase.com/docs/guides/database/query-optimization)
@@ -1,50 +0,0 @@
---
title: Use Cursor-Based Pagination Instead of OFFSET
impact: MEDIUM-HIGH
impactDescription: Consistent O(1) performance regardless of page depth
tags: pagination, cursor, keyset, offset, performance
---
## Use Cursor-Based Pagination Instead of OFFSET
OFFSET-based pagination scans all skipped rows, getting slower on deeper pages. Cursor pagination is O(1).
**Incorrect (OFFSET pagination):**
```sql
-- Page 1: scans 20 rows
select * from products order by id limit 20 offset 0;
-- Page 100: scans 2000 rows to skip 1980
select * from products order by id limit 20 offset 1980;
-- Page 10000: scans 200,000 rows!
select * from products order by id limit 20 offset 199980;
```
**Correct (cursor/keyset pagination):**
```sql
-- Page 1: get first 20
select * from products order by id limit 20;
-- Application stores last_id = 20
-- Page 2: start after last ID
select * from products where id > 20 order by id limit 20;
-- Uses index, always fast regardless of page depth
-- Page 10000: same speed as page 1
select * from products where id > 199980 order by id limit 20;
```
For multi-column sorting:
```sql
-- Cursor must include all sort columns
select * from products
where (created_at, id) > ('2024-01-15 10:00:00', 12345)
order by created_at, id
limit 20;
```
Reference: [Pagination](https://supabase.com/docs/guides/database/pagination)
@@ -1,50 +0,0 @@
---
title: Use UPSERT for Insert-or-Update Operations
impact: MEDIUM
impactDescription: Atomic operation, eliminates race conditions
tags: upsert, on-conflict, insert, update
---
## Use UPSERT for Insert-or-Update Operations
Using separate SELECT-then-INSERT/UPDATE creates race conditions. Use INSERT ... ON CONFLICT for atomic upserts.
**Incorrect (check-then-insert race condition):**
```sql
-- Race condition: two requests check simultaneously
select * from settings where user_id = 123 and key = 'theme';
-- Both find nothing
-- Both try to insert
insert into settings (user_id, key, value) values (123, 'theme', 'dark');
-- One succeeds, one fails with duplicate key error!
```
**Correct (atomic UPSERT):**
```sql
-- Single atomic operation
insert into settings (user_id, key, value)
values (123, 'theme', 'dark')
on conflict (user_id, key)
do update set value = excluded.value, updated_at = now();
-- Returns the inserted/updated row
insert into settings (user_id, key, value)
values (123, 'theme', 'dark')
on conflict (user_id, key)
do update set value = excluded.value
returning *;
```
Insert-or-ignore pattern:
```sql
-- Insert only if not exists (no update)
insert into page_views (page_id, user_id)
values (1, 123)
on conflict (page_id, user_id) do nothing;
```
Reference: [INSERT ON CONFLICT](https://www.postgresql.org/docs/current/sql-insert.html#SQL-ON-CONFLICT)
@@ -1,56 +0,0 @@
---
title: Use Advisory Locks for Application-Level Locking
impact: MEDIUM
impactDescription: Efficient coordination without row-level lock overhead
tags: advisory-locks, coordination, application-locks
---
## Use Advisory Locks for Application-Level Locking
Advisory locks provide application-level coordination without requiring database rows to lock.
**Incorrect (creating rows just for locking):**
```sql
-- Creating dummy rows to lock on
create table resource_locks (
resource_name text primary key
);
insert into resource_locks values ('report_generator');
-- Lock by selecting the row
select * from resource_locks where resource_name = 'report_generator' for update;
```
**Correct (advisory locks):**
```sql
-- Session-level advisory lock (released on disconnect or unlock)
select pg_advisory_lock(hashtext('report_generator'));
-- ... do exclusive work ...
select pg_advisory_unlock(hashtext('report_generator'));
-- Transaction-level lock (released on commit/rollback)
begin;
select pg_advisory_xact_lock(hashtext('daily_report'));
-- ... do work ...
commit; -- Lock automatically released
```
Try-lock for non-blocking operations:
```sql
-- Returns immediately with true/false instead of waiting
select pg_try_advisory_lock(hashtext('resource_name'));
-- Use in application
if (acquired) {
-- Do work
select pg_advisory_unlock(hashtext('resource_name'));
} else {
-- Skip or retry later
}
```
Reference: [Advisory Locks](https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS)
@@ -1,68 +0,0 @@
---
title: Prevent Deadlocks with Consistent Lock Ordering
impact: MEDIUM-HIGH
impactDescription: Eliminate deadlock errors, improve reliability
tags: deadlocks, locking, transactions, ordering
---
## Prevent Deadlocks with Consistent Lock Ordering
Deadlocks occur when transactions lock resources in different orders. Always
acquire locks in a consistent order.
**Incorrect (inconsistent lock ordering):**
```sql
-- Transaction A -- Transaction B
begin; begin;
update accounts update accounts
set balance = balance - 100 set balance = balance - 50
where id = 1; where id = 2; -- B locks row 2
update accounts update accounts
set balance = balance + 100 set balance = balance + 50
where id = 2; -- A waits for B where id = 1; -- B waits for A
-- DEADLOCK! Both waiting for each other
```
**Correct (lock rows in consistent order first):**
```sql
-- Explicitly acquire locks in ID order before updating
begin;
select * from accounts where id in (1, 2) order by id for update;
-- Now perform updates in any order - locks already held
update accounts set balance = balance - 100 where id = 1;
update accounts set balance = balance + 100 where id = 2;
commit;
```
Alternative: use a single statement to update atomically:
```sql
-- Single statement acquires all locks atomically
begin;
update accounts
set balance = balance + case id
when 1 then -100
when 2 then 100
end
where id in (1, 2);
commit;
```
Detect deadlocks in logs:
```sql
-- Check for recent deadlocks
select * from pg_stat_database where deadlocks > 0;
-- Enable deadlock logging
set log_lock_waits = on;
set deadlock_timeout = '1s';
```
Reference:
[Deadlocks](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-DEADLOCKS)
@@ -1,50 +0,0 @@
---
title: Keep Transactions Short to Reduce Lock Contention
impact: MEDIUM-HIGH
impactDescription: 3-5x throughput improvement, fewer deadlocks
tags: transactions, locking, contention, performance
---
## Keep Transactions Short to Reduce Lock Contention
Long-running transactions hold locks that block other queries. Keep transactions as short as possible.
**Incorrect (long transaction with external calls):**
```sql
begin;
select * from orders where id = 1 for update; -- Lock acquired
-- Application makes HTTP call to payment API (2-5 seconds)
-- Other queries on this row are blocked!
update orders set status = 'paid' where id = 1;
commit; -- Lock held for entire duration
```
**Correct (minimal transaction scope):**
```sql
-- Validate data and call APIs outside transaction
-- Application: response = await paymentAPI.charge(...)
-- Only hold lock for the actual update
begin;
update orders
set status = 'paid', payment_id = $1
where id = $2 and status = 'pending'
returning *;
commit; -- Lock held for milliseconds
```
Use `statement_timeout` to prevent runaway transactions:
```sql
-- Abort queries running longer than 30 seconds
set statement_timeout = '30s';
-- Or per-session
set local statement_timeout = '5s';
```
Reference: [Transaction Management](https://www.postgresql.org/docs/current/tutorial-transactions.html)
@@ -1,54 +0,0 @@
---
title: Use SKIP LOCKED for Non-Blocking Queue Processing
impact: MEDIUM-HIGH
impactDescription: 10x throughput for worker queues
tags: skip-locked, queue, workers, concurrency
---
## Use SKIP LOCKED for Non-Blocking Queue Processing
When multiple workers process a queue, SKIP LOCKED allows workers to process different rows without waiting.
**Incorrect (workers block each other):**
```sql
-- Worker 1 and Worker 2 both try to get next job
begin;
select * from jobs where status = 'pending' order by created_at limit 1 for update;
-- Worker 2 waits for Worker 1's lock to release!
```
**Correct (SKIP LOCKED for parallel processing):**
```sql
-- Each worker skips locked rows and gets the next available
begin;
select * from jobs
where status = 'pending'
order by created_at
limit 1
for update skip locked;
-- Worker 1 gets job 1, Worker 2 gets job 2 (no waiting)
update jobs set status = 'processing' where id = $1;
commit;
```
Complete queue pattern:
```sql
-- Atomic claim-and-update in one statement
update jobs
set status = 'processing', worker_id = $1, started_at = now()
where id = (
select id from jobs
where status = 'pending'
order by created_at
limit 1
for update skip locked
)
returning *;
```
Reference: [SELECT FOR UPDATE SKIP LOCKED](https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE)
@@ -1,45 +0,0 @@
---
title: Use EXPLAIN ANALYZE to Diagnose Slow Queries
impact: LOW-MEDIUM
impactDescription: Identify exact bottlenecks in query execution
tags: explain, analyze, diagnostics, query-plan
---
## Use EXPLAIN ANALYZE to Diagnose Slow Queries
EXPLAIN ANALYZE executes the query and shows actual timings, revealing the true performance bottlenecks.
**Incorrect (guessing at performance issues):**
```sql
-- Query is slow, but why?
select * from orders where customer_id = 123 and status = 'pending';
-- "It must be missing an index" - but which one?
```
**Correct (use EXPLAIN ANALYZE):**
```sql
explain (analyze, buffers, format text)
select * from orders where customer_id = 123 and status = 'pending';
-- Output reveals the issue:
-- Seq Scan on orders (cost=0.00..25000.00 rows=50 width=100) (actual time=0.015..450.123 rows=50 loops=1)
-- Filter: ((customer_id = 123) AND (status = 'pending'::text))
-- Rows Removed by Filter: 999950
-- Buffers: shared hit=5000 read=15000
-- Planning Time: 0.150 ms
-- Execution Time: 450.500 ms
```
Key things to look for:
```sql
-- Seq Scan on large tables = missing index
-- Rows Removed by Filter = poor selectivity or missing index
-- Buffers: read >> hit = data not cached, needs more memory
-- Nested Loop with high loops = consider different join strategy
-- Sort Method: external merge = work_mem too low
```
Reference: [EXPLAIN](https://supabase.com/docs/guides/database/inspect)
@@ -1,55 +0,0 @@
---
title: Enable pg_stat_statements for Query Analysis
impact: LOW-MEDIUM
impactDescription: Identify top resource-consuming queries
tags: pg-stat-statements, monitoring, statistics, performance
---
## Enable pg_stat_statements for Query Analysis
pg_stat_statements tracks execution statistics for all queries, helping identify slow and frequent queries.
**Incorrect (no visibility into query patterns):**
```sql
-- Database is slow, but which queries are the problem?
-- No way to know without pg_stat_statements
```
**Correct (enable and query pg_stat_statements):**
```sql
-- Enable the extension
create extension if not exists pg_stat_statements;
-- Find slowest queries by total time
select
calls,
round(total_exec_time::numeric, 2) as total_time_ms,
round(mean_exec_time::numeric, 2) as mean_time_ms,
query
from pg_stat_statements
order by total_exec_time desc
limit 10;
-- Find most frequent queries
select calls, query
from pg_stat_statements
order by calls desc
limit 10;
-- Reset statistics after optimization
select pg_stat_statements_reset();
```
Key metrics to monitor:
```sql
-- Queries with high mean time (candidates for optimization)
select query, mean_exec_time, calls
from pg_stat_statements
where mean_exec_time > 100 -- > 100ms average
order by mean_exec_time desc;
```
Reference: [pg_stat_statements](https://supabase.com/docs/guides/database/extensions/pg_stat_statements)
@@ -1,55 +0,0 @@
---
title: Maintain Table Statistics with VACUUM and ANALYZE
impact: MEDIUM
impactDescription: 2-10x better query plans with accurate statistics
tags: vacuum, analyze, statistics, maintenance, autovacuum
---
## Maintain Table Statistics with VACUUM and ANALYZE
Outdated statistics cause the query planner to make poor decisions. VACUUM reclaims space, ANALYZE updates statistics.
**Incorrect (stale statistics):**
```sql
-- Table has 1M rows but stats say 1000
-- Query planner chooses wrong strategy
explain select * from orders where status = 'pending';
-- Shows: Seq Scan (because stats show small table)
-- Actually: Index Scan would be much faster
```
**Correct (maintain fresh statistics):**
```sql
-- Manually analyze after large data changes
analyze orders;
-- Analyze specific columns used in WHERE clauses
analyze orders (status, created_at);
-- Check when tables were last analyzed
select
relname,
last_vacuum,
last_autovacuum,
last_analyze,
last_autoanalyze
from pg_stat_user_tables
order by last_analyze nulls first;
```
Autovacuum tuning for busy tables:
```sql
-- Increase frequency for high-churn tables
alter table orders set (
autovacuum_vacuum_scale_factor = 0.05, -- Vacuum at 5% dead tuples (default 20%)
autovacuum_analyze_scale_factor = 0.02 -- Analyze at 2% changes (default 10%)
);
-- Check autovacuum status
select * from pg_stat_progress_vacuum;
```
Reference: [VACUUM](https://supabase.com/docs/guides/database/database-size#vacuum-operations)
@@ -1,44 +0,0 @@
---
title: Create Composite Indexes for Multi-Column Queries
impact: HIGH
impactDescription: 5-10x faster multi-column queries
tags: indexes, composite-index, multi-column, query-optimization
---
## Create Composite Indexes for Multi-Column Queries
When queries filter on multiple columns, a composite index is more efficient than separate single-column indexes.
**Incorrect (separate indexes require bitmap scan):**
```sql
-- Two separate indexes
create index orders_status_idx on orders (status);
create index orders_created_idx on orders (created_at);
-- Query must combine both indexes (slower)
select * from orders where status = 'pending' and created_at > '2024-01-01';
```
**Correct (composite index):**
```sql
-- Single composite index (leftmost column first for equality checks)
create index orders_status_created_idx on orders (status, created_at);
-- Query uses one efficient index scan
select * from orders where status = 'pending' and created_at > '2024-01-01';
```
**Column order matters** - place equality columns first, range columns last:
```sql
-- Good: status (=) before created_at (>)
create index idx on orders (status, created_at);
-- Works for: WHERE status = 'pending'
-- Works for: WHERE status = 'pending' AND created_at > '2024-01-01'
-- Does NOT work for: WHERE created_at > '2024-01-01' (leftmost prefix rule)
```
Reference: [Multicolumn Indexes](https://www.postgresql.org/docs/current/indexes-multicolumn.html)
@@ -1,40 +0,0 @@
---
title: Use Covering Indexes to Avoid Table Lookups
impact: MEDIUM-HIGH
impactDescription: 2-5x faster queries by eliminating heap fetches
tags: indexes, covering-index, include, index-only-scan
---
## Use Covering Indexes to Avoid Table Lookups
Covering indexes include all columns needed by a query, enabling index-only scans that skip the table entirely.
**Incorrect (index scan + heap fetch):**
```sql
create index users_email_idx on users (email);
-- Must fetch name and created_at from table heap
select email, name, created_at from users where email = '[email protected]';
```
**Correct (index-only scan with INCLUDE):**
```sql
-- Include non-searchable columns in the index
create index users_email_idx on users (email) include (name, created_at);
-- All columns served from index, no table access needed
select email, name, created_at from users where email = '[email protected]';
```
Use INCLUDE for columns you SELECT but don't filter on:
```sql
-- Searching by status, but also need customer_id and total
create index orders_status_idx on orders (status) include (customer_id, total);
select status, customer_id, total from orders where status = 'shipped';
```
Reference: [Index-Only Scans](https://www.postgresql.org/docs/current/indexes-index-only-scans.html)
@@ -1,48 +0,0 @@
---
title: Choose the Right Index Type for Your Data
impact: HIGH
impactDescription: 10-100x improvement with correct index type
tags: indexes, btree, gin, gist, brin, hash, index-types
---
## Choose the Right Index Type for Your Data
Different index types excel at different query patterns. The default B-tree isn't always optimal.
**Incorrect (B-tree for JSONB containment):**
```sql
-- B-tree cannot optimize containment operators
create index products_attrs_idx on products (attributes);
select * from products where attributes @> '{"color": "red"}';
-- Full table scan - B-tree doesn't support @> operator
```
**Correct (GIN for JSONB):**
```sql
-- GIN supports @>, ?, ?&, ?| operators
create index products_attrs_idx on products using gin (attributes);
select * from products where attributes @> '{"color": "red"}';
```
Index type guide:
```sql
-- B-tree (default): =, <, >, BETWEEN, IN, IS NULL
create index users_created_idx on users (created_at);
-- GIN: arrays, JSONB, full-text search
create index posts_tags_idx on posts using gin (tags);
-- GiST: geometric data, range types, nearest-neighbor (KNN) queries
create index locations_idx on places using gist (location);
-- BRIN: large time-series tables (10-100x smaller)
create index events_time_idx on events using brin (created_at);
-- Hash: equality-only (slightly faster than B-tree for =)
create index sessions_token_idx on sessions using hash (token);
```
Reference: [Index Types](https://www.postgresql.org/docs/current/indexes-types.html)
@@ -1,43 +0,0 @@
---
title: Add Indexes on WHERE and JOIN Columns
impact: CRITICAL
impactDescription: 100-1000x faster queries on large tables
tags: indexes, performance, sequential-scan, query-optimization
---
## Add Indexes on WHERE and JOIN Columns
Queries filtering or joining on unindexed columns cause full table scans, which become exponentially slower as tables grow.
**Incorrect (sequential scan on large table):**
```sql
-- No index on customer_id causes full table scan
select * from orders where customer_id = 123;
-- EXPLAIN shows: Seq Scan on orders (cost=0.00..25000.00 rows=100 width=85)
```
**Correct (index scan):**
```sql
-- Create index on frequently filtered column
create index orders_customer_id_idx on orders (customer_id);
select * from orders where customer_id = 123;
-- EXPLAIN shows: Index Scan using orders_customer_id_idx (cost=0.42..8.44 rows=100 width=85)
```
For JOIN columns, always index the foreign key side:
```sql
-- Index the referencing column
create index orders_customer_id_idx on orders (customer_id);
select c.name, o.total
from customers c
join orders o on o.customer_id = c.id;
```
Reference: [Query Optimization](https://supabase.com/docs/guides/database/query-optimization)
@@ -1,45 +0,0 @@
---
title: Use Partial Indexes for Filtered Queries
impact: HIGH
impactDescription: 5-20x smaller indexes, faster writes and queries
tags: indexes, partial-index, query-optimization, storage
---
## Use Partial Indexes for Filtered Queries
Partial indexes only include rows matching a WHERE condition, making them smaller and faster when queries consistently filter on the same condition.
**Incorrect (full index includes irrelevant rows):**
```sql
-- Index includes all rows, even soft-deleted ones
create index users_email_idx on users (email);
-- Query always filters active users
select * from users where email = '[email protected]' and deleted_at is null;
```
**Correct (partial index matches query filter):**
```sql
-- Index only includes active users
create index users_active_email_idx on users (email)
where deleted_at is null;
-- Query uses the smaller, faster index
select * from users where email = '[email protected]' and deleted_at is null;
```
Common use cases for partial indexes:
```sql
-- Only pending orders (status rarely changes once completed)
create index orders_pending_idx on orders (created_at)
where status = 'pending';
-- Only non-null values
create index products_sku_idx on products (sku)
where sku is not null;
```
Reference: [Partial Indexes](https://www.postgresql.org/docs/current/indexes-partial.html)
@@ -1,80 +0,0 @@
---
title: Add Constraints Safely in Migrations
impact: HIGH
impactDescription: Prevents migration failures and enables idempotent schema changes
tags: constraints, migrations, schema, alter-table
---
## Add Constraints Safely in Migrations
PostgreSQL does not support `ADD CONSTRAINT IF NOT EXISTS`. Migrations using this syntax will fail.
**Incorrect (causes syntax error):**
```sql
-- ERROR: syntax error at or near "not" (SQLSTATE 42601)
alter table public.profiles
add constraint if not exists profiles_birthchart_id_unique unique (birthchart_id);
```
**Correct (idempotent constraint creation):**
```sql
-- Use DO block to check before adding
do $$
begin
if not exists (
select 1 from pg_constraint
where conname = 'profiles_birthchart_id_unique'
and conrelid = 'public.profiles'::regclass
) then
alter table public.profiles
add constraint profiles_birthchart_id_unique unique (birthchart_id);
end if;
end $$;
```
For all constraint types:
```sql
-- Check constraints
do $$
begin
if not exists (
select 1 from pg_constraint
where conname = 'check_age_positive'
) then
alter table users add constraint check_age_positive check (age > 0);
end if;
end $$;
-- Foreign keys
do $$
begin
if not exists (
select 1 from pg_constraint
where conname = 'profiles_birthchart_id_fkey'
) then
alter table profiles
add constraint profiles_birthchart_id_fkey
foreign key (birthchart_id) references birthcharts(id);
end if;
end $$;
```
Check if constraint exists:
```sql
-- Query to check constraint existence
select conname, contype, pg_get_constraintdef(oid)
from pg_constraint
where conrelid = 'public.profiles'::regclass;
-- contype values:
-- 'p' = PRIMARY KEY
-- 'f' = FOREIGN KEY
-- 'u' = UNIQUE
-- 'c' = CHECK
```
Reference: [Constraints](https://www.postgresql.org/docs/current/ddl-constraints.html)
@@ -1,46 +0,0 @@
---
title: Choose Appropriate Data Types
impact: HIGH
impactDescription: 50% storage reduction, faster comparisons
tags: data-types, schema, storage, performance
---
## Choose Appropriate Data Types
Using the right data types reduces storage, improves query performance, and prevents bugs.
**Incorrect (wrong data types):**
```sql
create table users (
id int, -- Will overflow at 2.1 billion
email varchar(255), -- Unnecessary length limit
created_at timestamp, -- Missing timezone info
is_active varchar(5), -- String for boolean
price varchar(20) -- String for numeric
);
```
**Correct (appropriate data types):**
```sql
create table users (
id bigint generated always as identity primary key, -- 9 quintillion max
email text, -- No artificial limit, same performance as varchar
created_at timestamptz, -- Always store timezone-aware timestamps
is_active boolean default true, -- 1 byte vs variable string length
price numeric(10,2) -- Exact decimal arithmetic
);
```
Key guidelines:
```sql
-- IDs: use bigint, not int (future-proofing)
-- Strings: use text, not varchar(n) unless constraint needed
-- Time: use timestamptz, not timestamp
-- Money: use numeric, not float (precision matters)
-- Enums: use text with check constraint or create enum type
```
Reference: [Data Types](https://www.postgresql.org/docs/current/datatype.html)
@@ -1,59 +0,0 @@
---
title: Index Foreign Key Columns
impact: HIGH
impactDescription: 10-100x faster JOINs and CASCADE operations
tags: foreign-key, indexes, joins, schema
---
## Index Foreign Key Columns
Postgres does not automatically index foreign key columns. Missing indexes cause slow JOINs and CASCADE operations.
**Incorrect (unindexed foreign key):**
```sql
create table orders (
id bigint generated always as identity primary key,
customer_id bigint references customers(id) on delete cascade,
total numeric(10,2)
);
-- No index on customer_id!
-- JOINs and ON DELETE CASCADE both require full table scan
select * from orders where customer_id = 123; -- Seq Scan
delete from customers where id = 123; -- Locks table, scans all orders
```
**Correct (indexed foreign key):**
```sql
create table orders (
id bigint generated always as identity primary key,
customer_id bigint references customers(id) on delete cascade,
total numeric(10,2)
);
-- Always index the FK column
create index orders_customer_id_idx on orders (customer_id);
-- Now JOINs and cascades are fast
select * from orders where customer_id = 123; -- Index Scan
delete from customers where id = 123; -- Uses index, fast cascade
```
Find missing FK indexes:
```sql
select
conrelid::regclass as table_name,
a.attname as fk_column
from pg_constraint c
join pg_attribute a on a.attrelid = c.conrelid and a.attnum = any(c.conkey)
where c.contype = 'f'
and not exists (
select 1 from pg_index i
where i.indrelid = c.conrelid and a.attnum = any(i.indkey)
);
```
Reference: [Foreign Keys](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-FK)
@@ -1,55 +0,0 @@
---
title: Use Lowercase Identifiers for Compatibility
impact: MEDIUM
impactDescription: Avoid case-sensitivity bugs with tools, ORMs, and AI assistants
tags: naming, identifiers, case-sensitivity, schema, conventions
---
## Use Lowercase Identifiers for Compatibility
PostgreSQL folds unquoted identifiers to lowercase. Quoted mixed-case identifiers require quotes forever and cause issues with tools, ORMs, and AI assistants that may not recognize them.
**Incorrect (mixed-case identifiers):**
```sql
-- Quoted identifiers preserve case but require quotes everywhere
CREATE TABLE "Users" (
"userId" bigint PRIMARY KEY,
"firstName" text,
"lastName" text
);
-- Must always quote or queries fail
SELECT "firstName" FROM "Users" WHERE "userId" = 1;
-- This fails - Users becomes users without quotes
SELECT firstName FROM Users;
-- ERROR: relation "users" does not exist
```
**Correct (lowercase snake_case):**
```sql
-- Unquoted lowercase identifiers are portable and tool-friendly
CREATE TABLE users (
user_id bigint PRIMARY KEY,
first_name text,
last_name text
);
-- Works without quotes, recognized by all tools
SELECT first_name FROM users WHERE user_id = 1;
```
Common sources of mixed-case identifiers:
```sql
-- ORMs often generate quoted camelCase - configure them to use snake_case
-- Migrations from other databases may preserve original casing
-- Some GUI tools quote identifiers by default - disable this
-- If stuck with mixed-case, create views as a compatibility layer
CREATE VIEW users AS SELECT "userId" AS user_id, "firstName" AS first_name FROM "Users";
```
Reference: [Identifiers and Key Words](https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS)
@@ -1,55 +0,0 @@
---
title: Partition Large Tables for Better Performance
impact: MEDIUM-HIGH
impactDescription: 5-20x faster queries and maintenance on large tables
tags: partitioning, large-tables, time-series, performance
---
## Partition Large Tables for Better Performance
Partitioning splits a large table into smaller pieces, improving query performance and maintenance operations.
**Incorrect (single large table):**
```sql
create table events (
id bigint generated always as identity,
created_at timestamptz,
data jsonb
);
-- 500M rows, queries scan everything
select * from events where created_at > '2024-01-01'; -- Slow
vacuum events; -- Takes hours, locks table
```
**Correct (partitioned by time range):**
```sql
create table events (
id bigint generated always as identity,
created_at timestamptz not null,
data jsonb
) partition by range (created_at);
-- Create partitions for each month
create table events_2024_01 partition of events
for values from ('2024-01-01') to ('2024-02-01');
create table events_2024_02 partition of events
for values from ('2024-02-01') to ('2024-03-01');
-- Queries only scan relevant partitions
select * from events where created_at > '2024-01-15'; -- Only scans events_2024_01+
-- Drop old data instantly
drop table events_2023_01; -- Instant vs DELETE taking hours
```
When to partition:
- Tables > 100M rows
- Time-series data with date-based queries
- Need to efficiently drop old data
Reference: [Table Partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html)
@@ -1,61 +0,0 @@
---
title: Select Optimal Primary Key Strategy
impact: HIGH
impactDescription: Better index locality, reduced fragmentation
tags: primary-key, identity, uuid, serial, schema
---
## Select Optimal Primary Key Strategy
Primary key choice affects insert performance, index size, and replication
efficiency.
**Incorrect (problematic PK choices):**
```sql
-- identity is the SQL-standard approach
create table users (
id serial primary key -- Works, but IDENTITY is recommended
);
-- Random UUIDs (v4) cause index fragmentation
create table orders (
id uuid default gen_random_uuid() primary key -- UUIDv4 = random = scattered inserts
);
```
**Correct (optimal PK strategies):**
```sql
-- Use IDENTITY for sequential IDs (SQL-standard, best for most cases)
create table users (
id bigint generated always as identity primary key
);
-- For distributed systems needing UUIDs, use UUIDv7 (time-ordered)
-- Requires pg_uuidv7 extension: create extension pg_uuidv7;
create table orders (
id uuid default uuid_generate_v7() primary key -- Time-ordered, no fragmentation
);
-- Alternative: time-prefixed IDs for sortable, distributed IDs (no extension needed)
create table events (
id text default concat(
to_char(now() at time zone 'utc', 'YYYYMMDDHH24MISSMS'),
gen_random_uuid()::text
) primary key
);
```
Guidelines:
- Single database: `bigint identity` (sequential, 8 bytes, SQL-standard)
- Distributed/exposed IDs: UUIDv7 (requires pg_uuidv7) or ULID (time-ordered, no
fragmentation)
- `serial` works but `identity` is SQL-standard and preferred for new
applications
- Avoid random UUIDs (v4) as primary keys on large tables (causes index
fragmentation)
Reference:
[Identity Columns](https://www.postgresql.org/docs/current/sql-createtable.html#SQL-CREATETABLE-PARMS-GENERATED-IDENTITY)
@@ -1,54 +0,0 @@
---
title: Apply Principle of Least Privilege
impact: MEDIUM
impactDescription: Reduced attack surface, better audit trail
tags: privileges, security, roles, permissions
---
## Apply Principle of Least Privilege
Grant only the minimum permissions required. Never use superuser for application queries.
**Incorrect (overly broad permissions):**
```sql
-- Application uses superuser connection
-- Or grants ALL to application role
grant all privileges on all tables in schema public to app_user;
grant all privileges on all sequences in schema public to app_user;
-- Any SQL injection becomes catastrophic
-- drop table users; cascades to everything
```
**Correct (minimal, specific grants):**
```sql
-- Create role with no default privileges
create role app_readonly nologin;
-- Grant only SELECT on specific tables
grant usage on schema public to app_readonly;
grant select on public.products, public.categories to app_readonly;
-- Create role for writes with limited scope
create role app_writer nologin;
grant usage on schema public to app_writer;
grant select, insert, update on public.orders to app_writer;
grant usage on sequence orders_id_seq to app_writer;
-- No DELETE permission
-- Login role inherits from these
create role app_user login password 'xxx';
grant app_writer to app_user;
```
Revoke public defaults:
```sql
-- Revoke default public access
revoke all on schema public from public;
revoke all on all tables in schema public from public;
```
Reference: [Roles and Privileges](https://supabase.com/blog/postgres-roles-and-privileges)
@@ -1,50 +0,0 @@
---
title: Enable Row Level Security for Multi-Tenant Data
impact: CRITICAL
impactDescription: Database-enforced tenant isolation, prevent data leaks
tags: rls, row-level-security, multi-tenant, security
---
## Enable Row Level Security for Multi-Tenant Data
Row Level Security (RLS) enforces data access at the database level, ensuring users only see their own data.
**Incorrect (application-level filtering only):**
```sql
-- Relying only on application to filter
select * from orders where user_id = $current_user_id;
-- Bug or bypass means all data is exposed!
select * from orders; -- Returns ALL orders
```
**Correct (database-enforced RLS):**
```sql
-- Enable RLS on the table
alter table orders enable row level security;
-- Create policy for users to see only their orders
create policy orders_user_policy on orders
for all
using (user_id = current_setting('app.current_user_id')::bigint);
-- Force RLS even for table owners
alter table orders force row level security;
-- Set user context and query
set app.current_user_id = '123';
select * from orders; -- Only returns orders for user 123
```
Policy for authenticated role:
```sql
create policy orders_user_policy on orders
for all
to authenticated
using (user_id = auth.uid());
```
Reference: [Row Level Security](https://supabase.com/docs/guides/database/postgres/row-level-security)
@@ -1,57 +0,0 @@
---
title: Optimize RLS Policies for Performance
impact: HIGH
impactDescription: 5-10x faster RLS queries with proper patterns
tags: rls, performance, security, optimization
---
## Optimize RLS Policies for Performance
Poorly written RLS policies can cause severe performance issues. Use subqueries and indexes strategically.
**Incorrect (function called for every row):**
```sql
create policy orders_policy on orders
using (auth.uid() = user_id); -- auth.uid() called per row!
-- With 1M rows, auth.uid() is called 1M times
```
**Correct (wrap functions in SELECT):**
```sql
create policy orders_policy on orders
using ((select auth.uid()) = user_id); -- Called once, cached
-- 100x+ faster on large tables
```
Use security definer functions for complex checks:
```sql
-- Create helper function (runs as definer, bypasses RLS)
create or replace function is_team_member(team_id bigint)
returns boolean
language sql
security definer
set search_path = ''
as $$
select exists (
select 1 from public.team_members
where team_id = $1 and user_id = (select auth.uid())
);
$$;
-- Use in policy (indexed lookup, not per-row check)
create policy team_orders_policy on orders
using ((select is_team_member(team_id)));
```
Always add indexes on columns used in RLS policies:
```sql
create index orders_user_id_idx on orders (user_id);
```
Reference: [RLS Performance](https://supabase.com/docs/guides/database/postgres/row-level-security#rls-performance-recommendations)
-112
View File
@@ -1,112 +0,0 @@
---
name: supabase
description: "Use when doing ANY task involving Supabase. Triggers: Supabase products (Database, Auth, Edge Functions, Realtime, Storage, Vectors, Cron, Queues); client libraries and SSR integrations (supabase-js, @supabase/ssr) in Next.js, React, SvelteKit, Astro, Remix; auth issues (login, logout, sessions, JWT, cookies, getSession, getUser, getClaims, RLS); Supabase CLI or MCP server; schema changes, migrations, security audits, Postgres extensions (pg_graphql, pg_cron, pg_vector)."
metadata:
author: supabase
version: "0.1.2"
---
# Supabase
## Core Principles
**1. Supabase changes frequently — verify against changelog and current docs before implementing.**
Do not rely on training data for Supabase features. Function signatures, config.toml settings, and API conventions change between versions.
First, fetch `https://supabase.com/changelog.md` (a lightweight summary index — not a heavy pull), scan for `breaking-change` tags relevant to your task, and follow the linked page for any that apply. Then look up the relevant topic using the documentation access methods below.
**2. Verify your work.**
After implementing any fix, run a test query to confirm the change works. A fix without verification is incomplete.
**3. Recover from errors, don't loop.**
If an approach fails after 2-3 attempts, stop and reconsider. Try a different method, check documentation, inspect the error more carefully, and review relevant logs when available. Supabase issues are not always solved by retrying the same command, and the answer is not always in the logs, but logs are often worth checking before proceeding.
**4. Exposing tables to the Data API:** Depending on the user's [Data API settings](https://supabase.com/dashboard/project/<ref>/integrations/data_api/settings), newly created tables may not be automatically exposed via the Data (REST) API. If this is the case, `anon` and `authenticated` roles will need to be explicitly granted access.
> Note that this is separate from RLS, which controls which _rows_ are visible once a table is accessible, not whether the table is accessible at all.
When a user reports a SQL-created table is unexpectedly inaccessible, check their Data API settings and whether the roles have been granted access via explicit `GRANT` SQL. When granting public (`anon`/`authenticated`) access, always enable RLS too. See [Exposing a Table to the Data API](https://supabase.com/docs/guides/api/securing-your-api.md) for the full setup workflow.
**5. RLS in exposed schemas.**
Enable RLS on every table in any exposed schema, which includes `public` by default. This is critical in Supabase because tables in exposed schemas can be reachable through the Data API when the `anon`/`authenticated` roles have access (see [Exposing a Table to the Data API](https://supabase.com/docs/guides/api/securing-your-api.md)). For private schemas, prefer RLS as defense in depth. After enabling RLS, create policies that match the actual access model rather than defaulting every table to the same `auth.uid()` pattern.
**6. Security checklist.**
When working on any Supabase task that touches auth, RLS, views, storage, or user data, run through this checklist. These are Supabase-specific security traps that silently create vulnerabilities:
- **Auth and session security**
- **Never use `user_metadata` claims in JWT-based authorization decisions.** In Supabase, `raw_user_meta_data` is user-editable and can appear in `auth.jwt()`, so it is unsafe for RLS policies or any other authorization logic. Store authorization data in `raw_app_meta_data` / `app_metadata` instead.
- **Deleting a user does not invalidate existing access tokens.** Sign out or revoke sessions first, keep JWT expiry short for sensitive apps, and for strict guarantees validate `session_id` against `auth.sessions` on sensitive operations.
- **If you use `app_metadata` or `auth.jwt()` for authorization, remember JWT claims are not always fresh until the user's token is refreshed.**
- **API key and client exposure**
- **Never expose the `service_role` or secret key in public clients.** Prefer publishable keys for frontend code. Legacy `anon` keys are only for compatibility. In Next.js, any `NEXT_PUBLIC_` env var is sent to the browser.
- **RLS, views, and privileged database code**
- **Views bypass RLS by default.** In Postgres 15 and above, use `CREATE VIEW ... WITH (security_invoker = true)`. In older versions of Postgres, protect your views by revoking access from the `anon` and `authenticated` roles, or by putting them in an unexposed schema.
- **UPDATE requires a SELECT policy.** In Postgres RLS, an UPDATE needs to first SELECT the row. Without a SELECT policy, updates silently return 0 rows — no error, just no change.
- **Do not put `security definer` functions in an exposed schema.** Keep them in a private or otherwise unexposed schema.
- **Storage access control**
- **Storage upsert requires INSERT + SELECT + UPDATE.** Granting only INSERT allows new uploads but file replacement (upsert) silently fails. You need all three.
For any security concern not covered above, fetch the Supabase product security index: `https://supabase.com/docs/guides/security/product-security.md`
## Supabase CLI
Always discover commands via `--help` — never guess. The CLI structure changes between versions.
```bash
supabase --help # All top-level commands
supabase <group> --help # Subcommands (e.g., supabase db --help)
supabase <group> <command> --help # Flags for a specific command
```
**Supabase CLI Known gotchas:**
- `supabase db query` requires **CLI v2.79.0+** → use MCP `execute_sql` or `psql` as fallback
- `supabase db advisors` requires **CLI v2.81.3+** → use MCP `get_advisors` as fallback
- When you need a new migration SQL file, **always** create it with `supabase migration new <name>` first. Never invent a migration filename or rely on memory for the expected format.
**Version check and upgrade:** Run `supabase --version` to check. For CLI changelogs and version-specific features, consult the [CLI documentation](https://supabase.com/docs/reference/cli/introduction) or [GitHub releases](https://github.com/supabase/cli/releases).
## Supabase MCP Server
For setup instructions, server URL, and configuration, see the [MCP setup guide](https://supabase.com/docs/guides/getting-started/mcp).
**Troubleshooting connection issues** — follow these steps in order:
1. **Check if the server is reachable:**
`curl -so /dev/null -w "%{http_code}" https://mcp.supabase.com/mcp`
A `401` is expected (no token) and means the server is up. Timeout or "connection refused" means it may be down.
2. **Check `.mcp.json` configuration:**
Verify the project root has a valid `.mcp.json` with the correct server URL. If missing, create one pointing to `https://mcp.supabase.com/mcp`.
3. **Authenticate the MCP server:**
If the server is reachable and `.mcp.json` is correct but tools aren't visible, the user needs to authenticate. The Supabase MCP server uses OAuth 2.1 — tell the user to trigger the auth flow in their agent, complete it in the browser, and reload the session.
## Supabase Documentation
Before implementing any Supabase feature, find the relevant documentation. Use these methods in priority order:
1. **MCP `search_docs` tool** (preferred — returns relevant snippets directly)
2. **Fetch docs pages as markdown** — any docs page can be fetched by appending `.md` to the URL path.
3. **Web search** for Supabase-specific topics when you don't know which page to look at.
## Making and Committing Schema Changes
**To make schema changes, use `execute_sql` (MCP) or `supabase db query` (CLI).** These run SQL directly on the database without creating migration history entries, so you can iterate freely and generate a clean migration when ready.
Do NOT use `apply_migration` to change a local database schema — it writes a migration history entry on every call, which means you can't iterate, and `supabase db diff` / `supabase db pull` will produce empty or conflicting diffs. If you use it, you'll be stuck with whatever SQL you passed on the first try.
**When ready to commit** your changes to a migration file:
1. **Run advisors** → `supabase db advisors` (CLI v2.81.3+) or MCP `get_advisors`. Fix any issues.
2. **Review the Security Checklist above** if your changes involve views, functions, triggers, or storage.
3. **Generate the migration** → `supabase db pull <descriptive-name> --local --yes`
4. **Verify** → `supabase migration list --local`
## Reference Guides
- **Skill Feedback** → [references/skill-feedback.md](references/skill-feedback.md)
**MUST read when** the user reports that this skill gave incorrect guidance or is missing information.
@@ -1,17 +0,0 @@
## What happened
**Task:** <!-- e.g., "Set up MFA on patient records" -->
**Skill said:** <!-- e.g., "Use auth.jwt()->'app_metadata' in the RLS policy" -->
**Expected:** <!-- e.g., "The function also needs SECURITY DEFINER + grant to supabase_auth_admin" -->
## Source
**File:** <!-- e.g., references/security-model.md -->
**Section:** <!-- e.g., "Trust Boundaries > user_metadata vs app_metadata" -->
## Fix suggestion
<!-- Leave blank if unsure -->
@@ -1,17 +0,0 @@
# Skill Feedback
Use this when the user reports that the skill gave incorrect guidance, is missing information, or could be improved. This is about the skill (agent instructions), not about Supabase the product.
## Steps
1. **Ask permission** — Ask the user if they'd like to submit feedback to the skill maintainers. If they decline, move on.
2. **Draft the issue** — Use the template at [assets/feedback-issue-template.md](../assets/feedback-issue-template.md) to structure the feedback. Fill in the fields based on the conversation. Always identify which specific reference file and section caused the problem.
3. **Submit** — Create a GitHub Issue on the `supabase/agent-skills` repository using the draft as the issue body. The title must follow this format: `user-feedback: <summary of the problem>`.
4. **Share the result** — Share the issue URL with the user after submission. If submission fails, give the user this link to create the issue manually:
```
https://github.com/supabase/agent-skills/issues/new
```
@@ -1,101 +0,0 @@
---
name: researching-plugins
description: Use when evaluating new data sources, researching APIs, or planning a new plugin structure, before writing any implementation code
---
# Researching Plugins
## Overview
This skill defines the rigorous process for researching, evaluating, and planning new data source plugins for WorldWideView. It combines data source validation (evaluating free tiers, rate limits, and comprehensiveness) with the generation of a placeholder-free, TDD-focused implementation plan.
## When to Use
- The user requests a new data source or layer (e.g., "add weather data", "track global earthquakes")
- You need to evaluate an API or dataset for continuous polling and 3D globe visualization
- You are formulating the implementation steps for a new plugin
- *Do not use* if the plugin is already fully planned and you are just executing tasks
## 1. Research & Evaluation Process
### Step 1: Take Inspiration from WorldMonitor
Before searching the web blindly, check if the `koala73/worldmonitor` repository already implements a similar feature.
- Review their data sources, API endpoints, and polling intervals.
- Analyze which data fields they map for visualization.
- Use this as a baseline for what is possible and expected.
### Step 2: Validate Data Sources
Search for APIs that meet the following strict criteria. If an API fails these, keep searching or ask the user for guidance:
- **Cost**: Prioritize highly robust free tiers or completely open data.
- **Comprehensiveness**: Does it provide global coverage? Does it return coordinates (lat/lon) and identifying metadata?
- **Rate Limits**: Can it support the required polling interval? (Real-time data needs high rate limits).
- **Format**: JSON or GeoJSON is preferred.
## 2. Draft the Implementation Plan
Once the data source and API are validated, you MUST write a comprehensive implementation plan using the standards from the `writing-plans` skill.
Save the research and plan to `.agents/context/YYYY-MM-DD-<plugin-name>-research.md`.
### Required Plan Header
```markdown
# [Plugin Name] Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development or superpowers:executing-plans to implement this plan task-by-task.
**Goal:** [One sentence describing what this plugin does]
**Data Source:** [API Name, API URL, Auth Method, Rate Limits]
**Architecture:** [Proxy route, Static Data, or Microservice?]
---
```
### Bite-Sized Task Structure
Each task in the plan MUST follow a strict TDD, action-by-action sequence.
**Required Task Format:**
```markdown
### Task N: [Component Name]
**Files:**
- Create: `packages/wwv-plugin-[name]/src/index.ts`
- Test: `packages/wwv-plugin-[name]/test/index.test.ts`
- [ ] **Step 1: Write the failing test**
[Include full test code block here]
- [ ] **Step 2: Run test to verify it fails**
Run: `pnpm test`
Expected: FAIL
- [ ] **Step 3: Write minimal implementation**
[Include full implementation code block here]
- [ ] **Step 4: Run test to verify it passes**
Run: `pnpm test`
Expected: PASS
- [ ] **Step 5: Commit**
[Include git commit command]
```
## The Iron Law of Plan Writing
1. **No Placeholders:** You MUST provide the exact file paths and complete code blocks for every step. Do not write "TODO: Add error handling" or "Add validation here."
2. **Exact File Paths Always:** Define exactly where files live (e.g., `src/app/api/...`).
3. **DRY & YAGNI:** Do not add speculative features or abstract prematurely.
4. **Define the Types First:** Ensure the API response interfaces and the plugin manifest types are defined early in the plan.
## Common Mistakes & Rationalizations
| Excuse / Rationalization | Reality / Fix |
|--------------------------|---------------|
| "I'll propose a paid API because the free ones are hard to find" | WorldWideView is a free tool; prioritize open data. Dig deeper into governmental or open-source datasets. |
| "I left TODOs in the plan so the execution agent can fill them in" | Execution agents fail when given placeholders. The plan MUST contain the exact code to be written. |
| "I skipped the test steps because it's just an API wrapper" | Every feature requires tests. Draft a test that mocks the API response. |
| "I forgot to specify the architecture type" | You must decide if it's a Static, Proxy, or Microservice plugin during the research phase. |
## Red Flags - STOP and Start Over
If you find yourself doing any of the following, delete your work and start over:
- Proposing an API with a 50 requests/day limit for real-time tracking.
- Writing a plan step that says "Implement similar to Task 2."
- Omitting exact file paths or complete code blocks in your tasks.
- Failing to verify if WorldMonitor has already solved this problem.
@@ -1,551 +0,0 @@
---
name: worldwideview-plugin-creation
description: Use when creating a new plugin, adding a new data source, or debugging missing plugin data in the WorldWideView project. Triggers on seeder creation, WebSocket streaming issues, plugin registration failures, manifest validation errors, GeoEntity rendering problems, or data engine integration
---
# WorldWideView Plugin Creation
## Overview
A WorldWideView plugin is a self-contained data source that renders geospatial entities on a 3D Cesium globe. Every plugin implements the `WorldPlugin` interface from the SDK and connects to a **data engine** (Fastify + Redis) that streams live data over WebSocket.
**Core principle:** Plugins are code bundles. The engine pushes data. The frontend renders it. There is no REST polling between frontend and engine — only WebSocket streaming.
## When to Use
- Creating a new data layer plugin (any architecture)
- Building a new seeder in the data engine
- Plugin data doesn't appear on the globe
- WebSocket subscription or split-routing issues
- Build failures after adding a plugin package
- Debugging `renderEntity` or entity clipping artifacts
- Marketplace plugin installation/validation issues
**Do NOT use for:**
- Modifying globe rendering primitives → see `cesium-rendering` rule
- Zustand state management → see `state-management` rule
- Prisma schema changes → see `database-migrations` rule
- Monorepo/pnpm workspace issues → see `monorepo-workflow` rule
## Architecture Decision
```dot
digraph plugin_type {
rankdir=TB;
node [shape=diamond];
"Does it need live\nexternal API data?" -> "Is polling\nfrequency > 30s?" [label="yes"];
"Does it need live\nexternal API data?" -> "Static GeoJSON\n(wwvStaticCompiler)" [label="no"];
"Is polling\nfrequency > 30s?" -> "Cron Seeder\n(engine)" [label="yes"];
"Is polling\nfrequency > 30s?" -> "Init Seeder\n(engine)" [label="no"];
"Static GeoJSON\n(wwvStaticCompiler)" [shape=box];
"Cron Seeder\n(engine)" [shape=box];
"Init Seeder\n(engine)" [shape=box];
}
```
---
## Quick Reference
| Piece | Location | Purpose |
|---|---|---|
| Sandbox plugin | `local-plugins/wwv-plugin-<name>/` | Local development workspace for new plugins |
| Plugin class | `packages/wwv-plugin-<name>/src/index.ts` | Implements `WorldPlugin` interface |
| Package metadata | `packages/wwv-plugin-<name>/package.json` | `worldwideview` block with id, type, format, category |
| Engine seeder | `wwv-data-engine/src/seeders/<name>.ts` | Fetches external data → Redis → WS broadcast |
| Build config | `next.config.ts` → `transpilePackages` | Required for monorepo packages |
| TS alias | `tsconfig.json` → `paths` | Required for monorepo packages |
| Registration | `src/core/plugins/PluginRegistry.ts` | Register plugin at boot |
| WS client | `src/core/data/WsClient.ts` | Client-side WebSocket subscriptions |
| URL resolution | `src/core/data/resolveEngineUrl.ts` | Per-plugin local vs cloud routing |
| Manifest validation | `src/core/plugins/validateManifest.ts` | Checks required fields for CDN bundles |
| Host globals | `src/core/plugins/hostGlobals.ts` | Injects shared deps for CDN plugins |
---
## Part 1: Frontend Plugin (WorldPlugin)
### 1.0 Development Workflow (Local Sandbox)
The standard way to develop a new plugin is using the **Local Sandbox** (`local-plugins/` directory). This keeps experimental code out of the main monorepo packages until it is ready for production.
1. **Scaffold**: Run `node packages/wwv-cli/dist/index.js create <name> --local` from the project root. This generates a boilerplate plugin in `local-plugins/wwv-plugin-<name>`.
2. **Develop**: Run `pnpm dev`. The built-in file watcher (`pnpm dev:plugins`) automatically rebuilds and syncs your local plugin to `public/plugins-local/` whenever you save a file.
3. **Promote**: When stable, run `node packages/wwv-cli/dist/index.js link <name>`. This promotes the plugin to the `packages/` directory and cleans up the sandbox artifacts.
4. **Install**: Run `pnpm install` from the root to update workspace symlinks.
### 1.1 Implement the Interface
Every plugin **must** implement `WorldPlugin` from `@worldwideview/wwv-plugin-sdk`:
```typescript
import type {
WorldPlugin, PluginContext, GeoEntity,
CesiumEntityOptions, TimeRange, LayerConfig, PluginCategory
} from "@worldwideview/wwv-plugin-sdk";
import { Flame } from "lucide-react";
import pkg from "../package.json";
export default class WildfiresPlugin implements WorldPlugin {
id = "wildfires";
name = "Wildfires";
description = "Live wildfire incidents from NASA FIRMS";
icon = Flame;
category: PluginCategory = "natural-disaster";
version = pkg.version; // NEVER hardcode — always import from package.json
async initialize(ctx: PluginContext): Promise<void> {
// Store context for later use (env vars, edition, callbacks)
}
destroy(): void {
// Cleanup timers, WebSocket connections, etc.
}
async fetch(timeRange: TimeRange): Promise<GeoEntity[]> {
// For WS-streamed plugins, return [] — data arrives via WsClient
return [];
}
getPollingInterval(): number {
return 0; // 0 = no REST polling (WS-only)
}
getLayerConfig(): LayerConfig {
return {
color: "#ef4444",
clusterEnabled: true,
clusterDistance: 50,
maxEntities: 5000,
};
}
renderEntity(entity: GeoEntity): CesiumEntityOptions {
return {
type: "point",
color: "#ef4444",
size: 6,
outlineColor: "#ffffff",
outlineWidth: 1,
};
}
}
```
### 1.2 Required vs Optional Methods
**Required:**
| Method | Purpose |
|---|---|
| `initialize(ctx)` | Receive `PluginContext` (env, edition, callbacks) |
| `destroy()` | Cleanup on shutdown |
| `fetch(timeRange)` | Return entities (REST path). Return `[]` for WS-only |
| `getPollingInterval()` | Polling frequency in ms. Return `0` for WS-only |
| `getLayerConfig()` | Layer appearance (color, clustering, max entities) |
| `renderEntity(entity)` | Per-entity Cesium rendering options |
**Optional:**
| Method | Purpose |
|---|---|
| `getServerConfig()` | REST/WS endpoint config (`apiBasePath`, `streamUrl`) |
| `getSelectionBehavior(entity)` | Trail/fly-to on entity click |
| `getFilterDefinitions()` | User-facing filter controls |
| `getLegend()` | Legend entries for layer panel |
| `getSidebarComponent()` | Custom sidebar React component |
| `getDetailComponent()` | Custom detail panel for selected entity |
| `getSettingsComponent()` | Plugin settings UI |
| `getGlobeComponent()` | React component injected into globe view |
| `mapWebsocketPayload(payload, existing)` | Transform raw WS data into `GeoEntity[]` |
| `requiresConfiguration(settings)` | Return `true` if plugin needs setup first |
### 1.3 Package Metadata
The `package.json` **must** include a `worldwideview` block:
```json
{
"name": "@worldwideview/wwv-plugin-wildfires",
"version": "1.0.0",
"main": "src/index.ts",
"worldwideview": {
"id": "wildfires",
"type": "data-layer",
"format": "bundle",
"category": "natural-disaster",
"icon": "Flame",
"capabilities": ["data:own", "globe:overlay"]
},
"dependencies": {
"@worldwideview/wwv-plugin-sdk": "workspace:*"
}
}
```
**Valid categories** (lowercase only): `aviation` · `maritime` · `military` · `conflict` · `natural-disaster` · `infrastructure` · `space` · `cyber` · `economic` · `intelligence` · `custom`
**Valid capabilities**: `data:own` · `data:read:{plugin}` · `ui:detail-panel` · `ui:sidebar` · `ui:toolbar` · `ui:settings` · `globe:overlay` · `globe:camera` · `storage:read` · `storage:write` · `network:fetch`
### 1.4 Rendering Rules
| Entity Type | Properties | Notes |
|---|---|---|
| `"point"` | `color`, `size`, `outlineColor`, `outlineWidth` | Simple colored dots |
| `"billboard"` | `iconUrl`, `color`, `iconScale` | SVG/PNG icons |
| `"model"` | `modelUrl`, `modelScale`, `modelMinPixelSize` | 3D glTF models |
| `"label"` | `labelText`, `labelFont`, `color` | Text labels |
> **CRITICAL:** NEVER mix point and billboard properties. Using `size`/`outlineWidth` on a billboard entity causes GPU clipping artifacts. This is a hard rendering constraint.
Use `createSvgIconUrl()` from the SDK for billboard icons:
```typescript
import { createSvgIconUrl } from "@worldwideview/wwv-plugin-sdk";
import { Plane } from "lucide-react";
const iconUrl = createSvgIconUrl(Plane, { color: "#3b82f6" });
```
### 1.5 Registration (3 Paths)
**A. Built-in (monorepo package):**
```typescript
import { pluginRegistry } from "@/core/plugins/PluginRegistry";
import { pluginManager } from "@/core/plugins/PluginManager";
import WildfiresPlugin from "@worldwideview/wwv-plugin-wildfires";
const plugin = new WildfiresPlugin();
pluginRegistry.register(plugin);
await pluginManager.registerPlugin(plugin);
```
`AppShell.tsx` iterates `pluginRegistry.getAll()` at startup.
**B. Marketplace (database-installed):**
`InstalledPluginsLoader` reads from the `installed_plugins` PostgreSQL table and calls `pluginManager.loadFromManifest()`. No code changes needed.
**C. Dynamic import (runtime):**
For user-imported GeoJSON layers, call `pluginManager.loadFromManifest(manifest)` directly.
### 1.6 Build Configuration (Promoted Monorepo Packages Only)
Once a plugin is promoted from the local sandbox via the `link` command, it becomes a formal monorepo package. You must manually add it to the build pipeline:
Add to `next.config.ts`:
```typescript
transpilePackages: [
"@worldwideview/wwv-plugin-sdk",
"@worldwideview/wwv-plugin-wildfires", // ADD
],
```
Add to `tsconfig.json`:
```json
"paths": {
"@worldwideview/wwv-plugin-wildfires": ["./packages/wwv-plugin-wildfires/src"]
}
```
Run `pnpm install` from project root after creating the package.
---
## Part 2: Data Engine Seeder
The data engine is a **content-agnostic runner** (`wwv-data-engine` public, deployed via Docker). It runs Fastify on port 5000 with Redis caching and WebSocket streaming. Seeders are volume-mounted from `local-seeders/` (dev) or downloaded from GitHub Releases (production).
> [!IMPORTANT]
> If you are **migrating** an existing plugin from `packages/` to `local-plugins/`, use the `migrate-legacy-plugin` skill instead. It covers both frontend routing fixes AND backend seeder build fixes as two independent concerns.
### 2.1 Engine Architecture
```
External API → Seeder (fetch + transform)
→ setLiveSnapshot(pluginId, payload, ttlSeconds)
→ broadcastPluginData() → Active WS subscribers (immediate)
→ Redis SET data:{pluginId}:live (throttled to every 5 min)
→ SQLite history (optional, via better-sqlite3)
```
**Key components:**
- `src/scheduler.ts` — Seeder registry and cron scheduling
- `src/redis.ts` — Redis client, `setLiveSnapshot()` and `getLiveSnapshot()`
- `src/websocket.ts` — WS handler, `broadcastPluginData()`
- `src/seed-utils.ts` — `withRetry()`, `fetchWithTimeout()`, `haversineKm()`
- `src/db.ts` — SQLite (better-sqlite3) for local history storage
- `src/seeders/index.ts` — Auto-discovery, dynamically requires all seeder files
### 2.2 Two Seeder Patterns
**Cron Seeder** (periodic polling — most common):
```typescript
import { db } from "../db";
import { setLiveSnapshot } from "../redis";
import { fetchWithTimeout, withRetry } from "../seed-utils";
import { registerSeeder } from "../scheduler";
// Optional: SQLite history table insert
const insertRow = db.prepare(
"INSERT OR IGNORE INTO my_data (id, payload, source_ts, fetched_at) VALUES (@id, @payload, @source_ts, @fetched_at)"
);
async function seedMyPlugin() {
console.log("[MyPlugin] Polling external API...");
const res = await withRetry(() =>
fetchWithTimeout("https://api.example.com/data")
);
const data = await res.json();
const items = data.features.map((f: any) => ({
id: f.id,
lat: f.geometry.coordinates[1],
lon: f.geometry.coordinates[0],
name: f.properties.name,
// ... transform to your schema
}));
// Optional: save to SQLite
for (const item of items) {
insertRow.run({
id: item.id,
payload: JSON.stringify(item),
source_ts: Date.now(),
fetched_at: Date.now(),
});
}
// REQUIRED: Push to Redis + broadcast to WS subscribers
await setLiveSnapshot("my-plugin", {
source: "my-plugin",
fetchedAt: new Date().toISOString(),
items,
totalCount: items.length,
}, 3600); // TTL in seconds
}
registerSeeder({
name: "my-plugin", // MUST match the plugin's id on the frontend
cron: "*/15 * * * *", // Every 15 minutes
fn: seedMyPlugin,
});
```
**Init Seeder** (persistent/high-frequency — e.g., ISS every 5s):
```typescript
import { setLiveSnapshot } from "../redis";
import { registerSeeder } from "../scheduler";
function startMySeeder() {
console.log("[MyPlugin] Starting persistent seeder...");
async function poll() {
const res = await fetch("https://api.example.com/live");
const data = await res.json();
await setLiveSnapshot("my-plugin", data, 60);
}
poll(); // Initial fetch
setInterval(poll, 5000); // Every 5 seconds
}
registerSeeder({
name: "my-plugin",
init: startMySeeder, // No cron — uses init for persistent polling
});
```
### 2.3 SeederDefinition Interface
```typescript
interface SeederDefinition {
name: string; // Plugin ID — MUST match frontend plugin id
cron?: string; // Cron expression (e.g., "0 * * * *" = hourly)
fn?: () => Promise<void>; // Function called on each cron tick
init?: () => void; // Initialization for persistent seeders (websockets, intervals)
}
```
### 2.4 Auto-Discovery
`src/seeders/index.ts` dynamically `require()`s every `.ts`/`.js` file in the seeders directory (except `index.ts` and test files). **You do not need to manually import your seeder** — just create the file and call `registerSeeder()` at module scope.
### 2.5 SQLite History (Optional)
For data that benefits from historical queries (playback mode), add a table in `src/db.ts`:
```typescript
db.exec(`
CREATE TABLE IF NOT EXISTS my_data (
id TEXT PRIMARY KEY,
payload JSON NOT NULL,
source_ts INTEGER NOT NULL,
fetched_at INTEGER NOT NULL
)
`);
```
### 2.6 Utility Functions
| Function | Purpose |
|---|---|
| `withRetry(fn, maxRetries, delayMs)` | Exponential backoff retry wrapper |
| `fetchWithTimeout(url, options, timeoutMs)` | Fetch with abort signal (default 15s) |
| `haversineKm(lat1, lon1, lat2, lon2)` | Distance between coordinates in km |
| `sleep(ms)` | Promise-based delay |
| `CHROME_UA` | Chrome User-Agent string for scraping |
---
## Part 3: Data Pipeline (Frontend ↔ Engine)
### 3.1 WebSocket Protocol
**Client → Engine:**
```json
{ "action": "subscribe", "pluginId": "earthquakes" }
{ "action": "unsubscribe", "pluginId": "earthquakes" }
```
**Engine → Client:**
```json
{ "type": "welcome", "engine": "wwv-data-engine", "plugins": ["earthquakes", "wildfires", ...] }
{ "type": "data", "pluginId": "earthquakes", "payload": { "items": [...], "totalCount": 42 } }
```
On subscribe, the engine immediately pushes the latest cached Redis snapshot — no waiting for the next cron tick.
### 3.2 Split-Routing (resolveEngineUrl)
`resolveEngineUrl(pluginId)` determines which engine serves each plugin:
1. **Local engine** (`localhost:5000`) if running and its `/manifest` includes this plugin ← **HIGHEST PRIORITY**
2. Plugin's `getServerConfig().streamUrl` (code override)
3. Plugin's `PluginManifest.dataSource.streamUrl` (manifest override)
4. `NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL` env var
5. Fallback: `wss://dataenginev2.worldwideview.dev/stream` (cloud)
### 3.3 Full Data Flow
```
Engine seeder calls setLiveSnapshot(pluginId, payload, ttl)
→ broadcastPluginData() fans out to subscribed WS connections
→ WsClient receives { type: "data", pluginId, payload }
→ WsClient calls plugin.mapWebsocketPayload() if defined
→ DataBus.emit("dataUpdated", { pluginId, entities })
→ DataBusSubscriber → Zustand store.entitiesByPlugin
→ GlobeView (memoized visible entities)
→ EntityRenderer (billboard/point/label primitives)
```
### 3.4 Host Globals (CDN Plugins)
CDN-loaded plugins share the host app's dependencies via `globalThis.__WWV_HOST__`:
- `react`, `react-dom`, `react/jsx-runtime`
- `cesium`, `resium`
- `zustand`
- `@worldwideview/wwv-plugin-sdk`
Use `wwvPluginGlobals()` in your Vite config to externalize automatically:
```typescript
import { defineConfig } from "vite";
import { wwvPluginGlobals } from "@worldwideview/wwv-plugin-sdk";
export default defineConfig({
plugins: [wwvPluginGlobals()],
build: {
lib: { entry: "./src/index.ts", formats: ["es"], fileName: "frontend" },
rollupOptions: { output: { entryFileNames: "frontend.mjs" } },
},
});
```
---
## Part 4: Static GeoJSON Plugins
For plugins with static data (no live API), use `wwvStaticCompiler`:
```typescript
import { defineConfig } from "vite";
import { wwvPluginGlobals, wwvStaticCompiler } from "@worldwideview/wwv-plugin-sdk";
export default defineConfig({
plugins: [wwvStaticCompiler(), wwvPluginGlobals()],
build: {
lib: { entry: "./src/index.ts", formats: ["es"], fileName: "frontend" },
},
});
```
Place GeoJSON in `data/data.json`. The compiler auto-generates a `WorldPlugin` class from `package.json` metadata at build time. No `src/index.ts` needed.
---
## Common Mistakes
### "Sandbox Plugin Breaking Next.js Build"
1. **Premature Configuration** — Modifying `next.config.ts` (`transpilePackages`) or `tsconfig.json` (`paths`) while a plugin is still in `local-plugins/`. The `local-plugins` sandbox relies on dynamic hot-loading in the browser via `pnpm dev:plugins`. Do **not** touch monorepo configs until the plugin is formally promoted using `wwv link <name>`.
2. **Manual Registration** — Manually adding a sandbox plugin to `AppShell.tsx`. Let the local hot-loading manifest handle sandbox plugins dynamically.
### "Missing Boilerplate / Errors in package.json"
1. **Bypassing the CLI** — Creating a plugin manually instead of using `node packages/wwv-cli/dist/index.js create <name> --local`. Always use the CLI to ensure the correct metadata, type definitions, and test scaffolding are generated.
### "No data on the globe"
1. **Seeder name ≠ plugin id** — The `name` in `registerSeeder()` MUST match the `id` property of the frontend `WorldPlugin` class
2. **Engine `/manifest` missing plugin** — Check `GET localhost:5000/manifest` → `plugins` array
3. **WsClient not subscribing** — Add `console.log` in `resolveEngineUrl()` to verify URL resolution
4. **Plugin hardcodes engine URL** — Legacy plugins use `this.context?.apiBaseUrl` instead of `resolveEngineUrl()`. See `migrate-legacy-plugin` skill
4. **`mapWebsocketPayload` missing** — If the engine sends a non-standard payload shape, the plugin must implement `mapWebsocketPayload()` or data silently drops
### "Build crash / Module not found"
1. Missing `transpilePackages` entry in `next.config.ts` → **immediate runtime crash**
2. Missing `tsconfig.json` path alias → TypeScript compilation error
3. Forgot `pnpm install` after creating package → dependency resolution failure
### "Invalid Hook Call" (CDN plugins)
Plugin bundles its own React instead of using host globals. Fix: add `wwvPluginGlobals()` to Vite config.
### "Manifest validation failed"
Required fields: `id`, `name`, `version`, `entry`, `trust`. Entry URL must be from allowed domains (CDN, localhost, `worldwideview.dev`, or relative path). `capabilities` must be non-empty array.
### "GPU clipping / entity disappearing"
Using `size`/`outlineWidth`/`outlineColor` on a `"billboard"` entity. These properties are **point-only**. Billboard uses `iconUrl`/`iconScale`/`color`.
### "Redis snapshot not updating"
`setLiveSnapshot()` throttles Redis writes to every 5 minutes to avoid exceeding Upstash request limits. WebSocket broadcasts are **not** throttled. Check WS delivery first.
### "Category type error"
Categories are **lowercase**: `"aviation"` not `"Aviation"`, `"natural-disaster"` not `"NaturalDisaster"`. Must match the `PluginCategory` union in the SDK.
---
## End-to-End Checklist
- [ ] **Development:** Scaffolded via `wwv create <name> --local` and developed in `local-plugins/`
- [ ] **Promotion:** Promoted to monorepo via `wwv link <name>` when stable
- [ ] **Frontend:** Plugin class implements `WorldPlugin` with correct `id`
- [ ] **Frontend:** `package.json` has `worldwideview` block with matching `id`
- [ ] **Frontend:** Version imported from `package.json` (never hardcoded)
- [ ] **Frontend:** Added to `transpilePackages` in `next.config.ts`
- [ ] **Frontend:** Added path alias in `tsconfig.json`
- [ ] **Frontend:** Registered via `pluginRegistry` + `pluginManager` in `AppShell.tsx`
- [ ] **Frontend:** `renderEntity()` doesn't mix point/billboard properties
- [ ] **Engine:** Seeder file in `src/seeders/<name>.ts`
- [ ] **Engine:** `registerSeeder({ name })` matches frontend plugin `id`
- [ ] **Engine:** Calls `setLiveSnapshot()` with correct plugin ID key
- [ ] **Engine:** Data shape is an object with `items` array (or plugin implements `mapWebsocketPayload`)
- [ ] **Engine:** SQLite table created in `db.ts` if historical data needed
- [ ] **Verification:** `GET localhost:5000/manifest` includes the plugin ID
- [ ] **Verification:** `GET localhost:5000/health` shows seeder last-run timestamp
- [ ] **Verification:** WebSocket subscribe returns data snapshot
- [ ] **Workspace:** Ran `pnpm install` from project root
- [ ] **Workspace:** Version bump follows `/commit` workflow
+2 -11
View File
@@ -1,20 +1,11 @@
# PostgreSQL connection string.
# For local dev: The platform will automatically spin up PostgreSQL via Docker Compose.
# Set DATABASE_URL to: postgresql://postgres:postgres@localhost:5432/worldwideview?schema=public
#
# For cloud/production: Provide your external Postgres connection string (e.g. Supabase)
# and set WWV_SKIP_LOCAL_DB=true to prevent Docker from spinning up a local database instance.
# DATABASE_URL=postgresql://postgres:postgres@localhost:5432/worldwideview?schema=public
# Set this to true in production to prevent docker-compose or pnpm run dev from starting the local database.
# WWV_SKIP_LOCAL_DB=false
DATABASE_URL=file:./data/wwv.db
# Default plugins enabled on boot for demo edition
# Comma-separated list of plugin IDs
NEXT_PUBLIC_DEMO_DEFAULT_PLUGINS=iranwarlive,satellite,gps-jamming
# Edition: "local" | "cloud" | "demo"
NEXT_PUBLIC_WWV_EDITION=demo
NEXT_PUBLIC_WWV_EDITION=local
# Auth Secret (REQUIRED for Docker — generate with: openssl rand -hex 32)
# Must be stable across container restarts to preserve sessions.
+2 -2
View File
@@ -4,8 +4,8 @@
# This file is tracked by git. Only put non-secret values here.
# Prisma reads DATABASE_URL from this file.
# Database (PostgreSQL)
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/worldwideview
# Database (SQLite for local — zero config; PostgreSQL for cloud)
DATABASE_URL=file:./data/wwv.db
# ============================================================
# .env.local — NOT committed (secrets & API keys)
-29
View File
@@ -1,29 +0,0 @@
version: 2
updates:
# 1. Maintain Node.js / pnpm dependencies
- package-ecosystem: "npm"
directory: "/"
schedule:
interval: "weekly"
day: "monday"
# Group all minor/patch updates into a single PR so it doesn't spam you
groups:
dependencies:
patterns:
- "*"
update-types:
- "minor"
- "patch"
# 2. Maintain Docker base images
- package-ecosystem: "docker"
directory: "/"
schedule:
interval: "weekly"
day: "monday"
# 3. Maintain GitHub Actions workflows
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "monthly" # Actions update less frequently, monthly is fine
+16 -10
View File
@@ -9,9 +9,11 @@ jobs:
name: Type Check
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: pnpm/action-setup@v6
- uses: actions/setup-node@v6
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
version: 9
- uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
@@ -23,9 +25,11 @@ jobs:
name: Unit Tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: pnpm/action-setup@v6
- uses: actions/setup-node@v6
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
version: 9
- uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
@@ -38,13 +42,15 @@ jobs:
runs-on: ubuntu-latest
needs: [typecheck, test]
env:
DATABASE_URL: postgresql://ci:ci@localhost:5432/ci
DATABASE_URL: file:./data/wwv.db
AUTH_SECRET: ci-placeholder-secret-not-used-in-tests
NEXT_PUBLIC_WWV_EDITION: local
steps:
- uses: actions/checkout@v6
- uses: pnpm/action-setup@v6
- uses: actions/setup-node@v6
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
version: 9
- uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
+1 -1
View File
@@ -27,7 +27,7 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v6
uses: actions/checkout@v4
with:
fetch-depth: 1
+1 -1
View File
@@ -26,7 +26,7 @@ jobs:
actions: read # Required for Claude to read CI results on PRs
steps:
- name: Checkout repository
uses: actions/checkout@v6
uses: actions/checkout@v4
with:
fetch-depth: 1
+9 -16
View File
@@ -1,22 +1,23 @@
name: Docker Image CI
name: Docker Build and Publish
on:
push:
branches: [ "main" ]
branches: [ "main", "feat/docker-ci" ]
pull_request:
branches: [ "main" ]
workflow_dispatch:
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
build-and-push-image:
build-and-push:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
@@ -39,24 +40,16 @@ jobs:
tags: |
type=raw,value=latest,enable={{is_default_branch}}
type=ref,event=branch
type=sha,format=short
type=ref,event=pr
type=semver,pattern={{version}}
type=sha
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: .
push: ${{ github.event_name != 'pull_request' }}
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
build-args: |
NEXT_PUBLIC_CESIUM_ION_TOKEN=${{ secrets.NEXT_PUBLIC_CESIUM_ION_TOKEN }}
NEXT_PUBLIC_GOOGLE_MAPS_API_KEY=${{ secrets.NEXT_PUBLIC_GOOGLE_MAPS_API_KEY }}
NEXT_PUBLIC_BING_MAPS_KEY=${{ secrets.NEXT_PUBLIC_BING_MAPS_KEY }}
NEXT_PUBLIC_WWV_EDITION=${{ vars.NEXT_PUBLIC_WWV_EDITION || 'demo' }}
NEXT_PUBLIC_WS_ENGINE_URL=${{ vars.NEXT_PUBLIC_WS_ENGINE_URL }}
NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL=${{ vars.NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL }}
NEXT_PUBLIC_ADSENSE_CLIENT_ID=${{ secrets.NEXT_PUBLIC_ADSENSE_CLIENT_ID }}
NEXT_PUBLIC_SUPABASE_URL=${{ vars.NEXT_PUBLIC_SUPABASE_URL }}
NEXT_PUBLIC_SUPABASE_ANON_KEY=${{ secrets.NEXT_PUBLIC_SUPABASE_ANON_KEY }}
+2 -2
View File
@@ -12,9 +12,9 @@ jobs:
contents: read
id-token: write # Required for OIDC trusted publishing (no NPM_TOKEN needed)
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v4
- uses: actions/setup-node@v6
- uses: actions/setup-node@v4
with:
node-version: 20
registry-url: "https://registry.npmjs.org"
+2 -30
View File
@@ -18,9 +18,7 @@ node_modules/
/out/
# agents
/.agents/plans
/.agents/context
/.agents/research/
/.agents/
# claude code IDE config
/.claude/
@@ -84,11 +82,6 @@ packages/*/dist/
/local-scripts/
/public/plugins-local/
# Local plugin development (gitignored workspace)
local-plugins/*
!local-plugins/.gitkeep
!local-plugins/README.md
# git worktrees
/.worktrees/
@@ -96,25 +89,4 @@ local-plugins/*
packages/wwv-data-engine/
# IDE workspace files
*.code-workspace
# Build outputs and generated assets
/public/cesium/
build_output*.log
build_output*.txt
# Seeder repositories
/local-seeders/
/.local-seeders-dist/
# WorldwideView Ecosystem Repositories (Development Structure)
# The following ignores represent the different repositories related to the entire setup found in C:\dev
/worldwideview-marketplace/
/worldwideview-plugins/
/worldwideview-web/
/wwv-data-engine/
/wwv-data-engine-public/
/wwv-seeders/
/wwv-seeders-private/
/wwv-test/
worldwideview.code-workspace
*.code-workspace
+152 -66
View File
@@ -19,7 +19,7 @@ Our primary design, feature-set, and operational layout goal is to mimic the str
| State | Zustand (slice-based: globe, layers, timeline, UI, filters, data, config, favorites, geojson) |
| Event Bus | Custom typed `DataBus` (pub/sub singleton) |
| Styling | Vanilla CSS — **no Tailwind** |
| Database | PostgreSQL via Prisma 7 (local and cloud) |
| Database | SQLite via Prisma (local), PostgreSQL (cloud) |
| Auth | NextAuth v5 beta (Credentials provider, JWT sessions) |
| Package Manager | pnpm (monorepo with `pnpm-workspace.yaml`) |
| Testing | Vitest + jsdom + React Testing Library |
@@ -30,8 +30,58 @@ Our primary design, feature-set, and operational layout goal is to mimic the str
## 3. Directory Structure
> [!NOTE]
> See `.agents/rules/directory-structure.md` for the full project directory map and related repositories.
```
worldwideview/
├── src/
│ ├── app/ # Next.js App Router (pages, API routes, layouts)
│ │ ├── api/ # Server-side API routes (auth, aviation, camera, etc.)
│ │ ├── login/ # Login page
│ │ ├── setup/ # First-time setup page
│ │ └── globals.css # Root stylesheet
│ ├── components/
│ │ ├── common/ # Shared UI: BootOverlay, FloatingWindow, PluginIcon
│ │ ├── layout/ # AppShell, Header, SearchBar, DataBusSubscriber
│ │ ├── panels/ # LayerPanel, EntityInfoCard, FilterPanel, GraphicsSettings
│ │ ├── timeline/ # Timeline component
│ │ ├── marketplace/ # Plugin install/unverified dialogs
│ │ ├── ui/ # Tooltip, ReloadToast
│ │ └── video/ # Floating video manager
│ ├── core/
│ │ ├── data/ # DataBus, PollingManager, CacheLayer, SmartFetcher
│ │ ├── filters/ # filterEngine (applies plugin filters to entities)
│ │ ├── globe/ # GlobeView, EntityRenderer, AnimationLoop, StackManager,
│ │ │ │ CameraController, InteractionHandler, SelectionHandler,
│ │ │ │ ModelManager, ImageryProviderFactory
│ │ │ └── hooks/ # useCameraActions, useEntityRendering, useModelRendering, etc.
│ │ ├── hooks/ # useBootSequence, useIsMobile, useMarketplaceSync
│ │ ├── plugins/ # PluginManager, PluginRegistry, PluginManifest,
│ │ │ │ loadPluginFromManifest, validateManifest, InstalledPluginsLoader
│ │ │ └── loaders/ # DeclarativePlugin, StaticDataPlugin, mapJsonToEntities
│ │ └── state/ # Zustand store + slices (config, data, globe, layers, timeline, ui, etc.)
│ ├── lib/ # auth, db, rateLimit, analytics, AIS stream, marketplace APIs
│ ├── plugins/ # GeoJSON plugin registrations
│ ├── styles/ # HUD animations CSS
│ ├── types/ # GeoJSON types, Umami types
│ └── generated/ # Prisma generated client (gitignored)
├── packages/ # pnpm monorepo workspace packages
│ ├── wwv-plugin-sdk/ # Plugin SDK: type definitions, manifest schema
│ ├── wwv-plugin-aviation/
│ ├── wwv-plugin-maritime/
│ ├── wwv-plugin-wildfire/
│ ├── wwv-plugin-borders/
│ ├── wwv-plugin-camera/
│ ├── wwv-plugin-military-aviation/
│ ├── wwv-plugin-satellite/
│ ├── wwv-plugin-iranwarlive/ # Includes standalone backend/ microservice
│ └── wwv-plugin-{airports,embassies,lighthouses,nuclear,seaports,spaceports,volcanoes}/
├── prisma/ # schema.prisma, migrations/
├── public/ # Static assets, Cesium workers, plugin GeoJSON data
├── scripts/ # Build scripts (copy-cesium, scaffold-osm-plugin, setup)
├── data/ # SQLite database (gitignored, Docker volume)
├── Dockerfile # Multi-stage production build
├── docker-compose.yml # Main app + microservice backends
└── .agents/ # Agent documentation, rules, skills, workflows
```
---
@@ -52,12 +102,12 @@ Visibility Toggle → DataBusSubscriber subscribes to layer via WsClient
```
Four plugin architectures exist (All-Bundle Model):
1. **Data Engine Seeder** — Lightweight `seeder.mjs` script executed by the dynamic `wwv-data-engine` runner (previously standalone microservices).
1. **Data Engine Seeder** — Standalone Fastify container with SQLite microservice or unified seeders (e.g., `iranwarlive-backend`).
2. **Dynamic CDN Loaded (Bundle)** — Externally developed plugins dynamically imported at runtime via ES module CDNs (e.g., `unpkg.com` version-pinned URLs).
3. **Static Compiled (Bundle)** — Static GeoJSON data wrapped into JS bundles via `wwvStaticCompiler` during build/sync (previously `StaticDataPlugin`).
4. **Active Proxied (Bundle)** — Next.js API routes bundled to provide frontend interactions (previously `DeclarativePlugin`).
3. **WWV CLI Unpacked (Dev)** — Developed externally using `@worldwideview/cli dev`, which spins up a local Vite server and hot-reloads via WebSocket injection directly into the running platform.
4. **Marketplace Package (.wwvpkg)** — Built via `@worldwideview/cli publish` and distributed through the WorldWideView Marketplace.
All plugins are now dynamically imported at runtime as ES module bundles via `loadPluginFromManifest` utilizing `import(/* webpackIgnore: true */ entry)`. The legacy `StaticDataPlugin` and `DeclarativePlugin` runtimes are fully deprecated.
All plugins are now dynamically imported at runtime as ES module bundles via `loadPluginFromManifest` utilizing `import(/* webpackIgnore: true */ entry)`. The plugin passport is defined in `wwv-manifest.json` which governs its trust tier (`developer`, `unverified`, `verified`, `built-in`) and capabilities.
Plugin types are re-exported from SDK through `src/core/plugins/PluginTypes.ts` and `PluginManifest.ts` — **source of truth is always `@worldwideview/wwv-plugin-sdk`**.
@@ -81,15 +131,17 @@ Engine push /stream → DataBusSubscriber WsClient router
→ StackManager (co-located entity grouping)
```
### 4.3.1 Engine & Seeders Architecture
### 4.3.1 Split-Routing (Cloud Fallback)
The data engine is a **content-agnostic runner** (`wwv-data-engine`, public) that discovers and executes seeder scripts from a configurable directory.
When a local engine is detected at `localhost:5001`, the frontend fetches its `/manifest` to discover available seeders. Per-plugin routing via `resolveEngineUrl`:
- Plugin's seeder is in local manifest → `ws://localhost:5001/stream`
- Plugin's seeder is NOT in local manifest → `wss://dataengine.worldwideview.dev/stream`
- **Local Dev**: Engine runs via Docker Compose on port 5000, reading seeders dynamically from `local-seeders/` (split into `community` and `private` tiers).
- **Production**: Engine container on Coolify, downloads release bundles from `wwv-seeders-community` and `wwv-seeders-private` on startup and unzips them into `/app/seeders`.
- **Split-routing**: `resolveEngineUrl` prioritizes the **Local Dev Engine (localhost:5001)** for local testing (following 12-Factor App methodology), falling back to cloud-hosted endpoints.
- **Agnostic Frontend Architecture**: WorldWideView is a completely agnostic renderer. It has absolutely no concept of a "unified" Data Engine. If 30 plugins require 30 different WebSocket servers, the application will blindly open 30 connections.
- **Self-Contained Plugins**: Each plugin is a self-contained package and **MUST explicitly declare its own `streamUrl` in its manifest or config**. Do NOT assume the frontend acts as a unified pipe. It just so happens that our default plugins share the `wwv-data-engine` backend, but the platform is 100% decentralized.
This enables core contributors running the public (community) engine locally to see local data for their seeders while proprietary sources (aviation, maritime) continue streaming from the cloud.
The data engine uses a **public upstream / private fork** model:
- `wwv-data-engine` (public) — community seeders, contributors fork and PR here
- `wwv-data-engine-internal` (private) — all seeders, deploys to Coolify via `git merge upstream/main`
### 4.4 Rendering Pipeline
@@ -143,16 +195,16 @@ Built-in plugins are instantiated in `AppShell.tsx` and registered via `PluginRe
### 5.6 Workspace Rules
- Always run `pnpm install` from project root after creating new packages or linking local plugins
- Official plugin packages go in `packages/wwv-plugin-<name>/`
- Experimental/local plugin sandboxes go in `local-plugins/wwv-plugin-<name>/`
- Globs for `packages/*`, `packages/*/backend`, and `local-plugins/*` are mapped in `pnpm-workspace.yaml` and `tsconfig.json` paths
- Add new `packages/` plugins to `transpilePackages` in `next.config.ts` if required
- Always run `pnpm install` from project root after creating new packages
- Plugin packages go in `packages/wwv-plugin-<name>/`
- Microservice backends go in `packages/wwv-plugin-<name>/backend/`
- Both globs (`packages/*` and `packages/*/backend`) are in `pnpm-workspace.yaml`
- Add new plugins to `transpilePackages` in `next.config.ts` and `paths` in `tsconfig.json`
### 5.7 AI Meta-Directives: Antigravity Standard
### 5.7 AI Meta-Directives: Antigravity Standard (Claude Code)
> [!NOTE]
> This repository is orchestrated via the **Antigravity open standard**. The entry point for the agent framework is `CLAUDE.md` / `AGENTS.md` at the project root.
> This repository is orchestrated via the **Antigravity open standard** using **Claude Code** as the active agent. The entry point for Claude Code is `CLAUDE.md` at the project root.
> [!WARNING]
> - **Always** use standard `.md` file extensions for rules, skills, and workflows.
@@ -161,30 +213,94 @@ Built-in plugins are instantiated in `AppShell.tsx` and registered via `PluginRe
> - **MUST**: You MUST update Semantic Versioning numbering inside the relevant `package.json` file prior to executing any code commits, adhering strictly to the `[/commit]` workflow rules (`feat:` -> Minor, `fix/refactor/perf:` -> Patch).
> - **MUST Detail Commit Levels & Bumps**: On description changes or release notes, you must detail the level of commit (Major/Minor/Fix) for *each* individual change. If there are multiple accumulated changes, you MUST EITHER commit them individually and bump the version each time, OR commit them all at once and bump the version multiple times.
> - **MUST Explain Complex Concepts Simply**: Whenever providing a complicated technical explanation to the user, you MUST include a simple explanation below it. Use an analogy with reference to the correct terminology, comparing the concept to something from everyday life to ensure the user easily understands it.
> - **MUST Be Transparent & Narrate Actions (Gemini 3.1 Agents)**: If you are a Gemini 3.1 agent, you MUST always be fully transparent. **Whenever you do anything, you must explicitly say what you are going to do in a visible chat message to the user, and ONLY THEN do it.** This ensures the user can actually see and verify what you are doing in real time. Do not jump to destructive actions without stating your intent first. Keep this narration conversational and natural. Avoid stiff, robotic templates. Just explicitly drop a brief, casual note about what you are checking, fixing, or deleting *before* you run the tool.
> - **MUST Wait for Explicit Authorization**: Do not take action unless explicitly told to do so. If the user highlights a piece of code, brings up a bug, or asks a question, your ONLY job is to analyze it, investigate the root cause, and explain what is wrong or answer the question. **Do not write the fix, delete files, or execute changes unless the user explicitly gives you the order to do so.** Wait for clear authorization before taking action. **Crucially, if you realize you have violated this rule by taking an unauthorized action, DO NOT automatically revert it. Reverting is itself an action that requires authorization. Just answer the question.**
### 5.8 Workspace Hygiene
Whenever agents generate temporary debugging scripts, test REST endpoints via `.mjs`, or dump traces/JSON outputs, they **MUST** save these exclusively inside `/local-scripts/`. The root directory is strictly for production configuration files.
### 5.9 Rule Adherence & Proactive Updates
> [!CAUTION]
> **Agents will trust the `.agents/rules/` files unconditionally.**
> Whenever you execute a major architectural shift or make a change that invalidates an existing rule, you **MUST** immediately update the corresponding rule file(s) and/or `AGENTS.md` to reflect the new reality. Do not leave rules untouched after a change; operating on outdated information will cause future agents to break the codebase.
---
## 6. Environment Variables
| Variable | Required | Description |
|---|---|---|
| `DATABASE_URL` | Yes | SQLite: `file:./data/wwv.db` |
| `AUTH_SECRET` | Yes | JWT signing secret (generate with `openssl rand -hex 32`) |
| `NEXT_PUBLIC_CESIUM_ION_TOKEN` | No | Cesium Ion access token |
| `NEXT_PUBLIC_BING_MAPS_KEY` | No | Bing Maps imagery |
| `NEXT_PUBLIC_GOOGLE_MAPS_API_KEY` | No | Google 3D Tiles |
| `NEXT_PUBLIC_WWV_EDITION` | No | `local` / `cloud` / `demo` (default: `local`) |
| `NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL` | No | Override engine WebSocket URL (default: cloud) |
| `OPENSKY_CREDENTIALS` | No | Comma-separated `id:secret` pairs for credential rotation |
| `WWV_BRIDGE_TOKEN` | No | Shared secret for marketplace → WWV install bridge |
| `WWV_DEMO_ADMIN_SECRET` | No | Demo edition admin password |
| `IRANWARLIVE_BACKEND_URL` | No | Override for IranWarLive microservice URL |
Secrets go in `.env.local` (gitignored). Non-secrets go in `.env` (committed).
---
## 6. Environment & Configuration
## 7. Development Commands
> [!NOTE]
> See `.agents/rules/environment-config.md` for required environment variables and secrets.
```bash
pnpm install # Install all workspace dependencies
pnpm run setup # Generate .env.local with AUTH_SECRET (first-time setup)
pnpm dev # Next.js frontend only (auto-runs prisma migrate deploy + copy-cesium)
pnpm dev:all # Frontend + wwv-data-engine concurrently (normal development)
pnpm dev:backends # Run only the data engine in dev mode
pnpm build # Production build
pnpm test # Run all Vitest tests (scoped to src/lib, src/core, src/plugins)
pnpm db:reset # Reset and re-migrate the frontend SQLite database (destructive)
pnpm start:backends # Start all plugin microservice backends in parallel
pnpm clean:backends # Wipe all plugin SQLite databases
pnpm run scaffold-osm-plugin <name> # Generate a new plugin from scaffold
```
Frontend runs at `http://localhost:3000`.
---
## 7. Development, Deployment & Testing
## 8. Deployment
> [!NOTE]
> See `.agents/rules/deployment-and-testing.md` for development commands, Docker architecture, Vitest strategy, and CSP security headers.
- **Docker**: Multi-stage Dockerfile using the Extractor Pattern (`deps` → `builder` → `runner`). The `node_modules` folders must be explicitly untracked in from `git` across the workspace to prevent corrupted BuildKit contextual cache overlaps during the `COPY . .` stage.
- **Standalone output**: `next.config.ts` uses `output: "standalone"`.
- **Cesium assets**: Copied to `public/cesium/` via `scripts/copy-cesium.mjs` at build time, excluded from output tracing.
- **Prisma Configuration**: `prisma.config.ts` must export a native javascript object instead of dynamically importing CLI wrapper binaries (`prisma/config` or `dotenv`). The standalone Next.js tracer strips CLI devDependencies during the build, which will cause fatal runtime container crashes if imported.
- **Microservices**: Separate containers defined in `docker-compose.yml`, proxied via `next.config.ts` rewrites.
- **Coolify**: Deployed via Dockerfile builder natively mapping environment variables continuously into the container shell. Persistent storage mount required at `/app/data` for the SQLite root node database.
- **Docker volumes**: Mount `/app/data` (frontend SQLite), `/app/packages/wwv-data-engine/data` (data engine SQLite), `/data` (Redis)
---
## 9. Testing Strategy
- **Framework**: Vitest with jsdom environment
- **Coverage**: `src/lib/**`, `src/core/**`, `src/plugins/**`
- **Run**: `pnpm test` (or `vitest run`)
- **Key test files**: `rateLimit.test.ts`, `edition.test.ts`, `demoAdmin.test.ts`, `DeclarativePlugin.test.ts`, `cors.test.ts`, `repository.test.ts`, `marketplaceToken.test.ts`
---
## 10. Security Headers
Configured in `next.config.ts` `headers()`:
- **CSP**: Restrictive with exceptions for CesiumJS (`unsafe-eval`, `unsafe-inline`), camera streams (`http: https:`), and analytics
- **X-Frame-Options**: DENY
- **X-Content-Type-Options**: nosniff
- **Referrer-Policy**: strict-origin-when-cross-origin
- **Permissions-Policy**: camera/microphone disabled, geolocation self-only
---
## 11. Related Repositories
| Repo | Purpose |
|---|---|
| `worldwideview` | Main application (this repo) |
| `wwv-data-engine` | Open-source data engine — community seeders (PUBLIC) |
| `wwv-data-engine-internal` | Proprietary data engine — all seeders (PRIVATE, deployed to Coolify) |
| `worldwideview-marketplace` | Plugin marketplace web app |
| `worldwideview-plugins` | Published npm plugin packages |
| `worldwideview-web` | Marketing / landing page |
---
@@ -194,25 +310,13 @@ Read the relevant rule file when working in that domain:
| Rule | When to use | Path |
|---|---|---|
| `platform-architecture` | High-level platform goals, product vision, business model, and Edition System | `.agents/rules/platform-architecture.md` |
| `application-architecture` | Next.js frontend, Zustand state management, and CesiumJS integration | `.agents/rules/application-architecture.md` |
| `plugin-architecture` | Creating/modifying plugins, lifecycle, capability declarations, and seeders | `.agents/rules/plugin-architecture.md` |
| `marketplace-architecture` | Dynamic plugin installation, DB sync, and CDN loading | `.agents/rules/marketplace-architecture.md` |
| `cloud-auth-architecture` | Cloud edition, PostgreSQL RLS, multi-tenancy, and licensing | `.agents/rules/cloud-auth-architecture.md` |
| `server-management` | Server development and debugging using SSH and Coolify MCP | `.agents/rules/server-management.md` |
| `stakeholders-and-human-centered-design` | Human-centered design principles and stakeholder map | `.agents/rules/stakeholders-and-human-centered-design.md` |
| `directory-structure` | Project structure and related repositories | `.agents/rules/directory-structure.md` |
| `ecosystem-repositories` | WorldWideView ecosystem repositories and cross-repository workflows | `.agents/rules/ecosystem-repositories.md` |
| `deployment-and-testing` | Docker build patterns, testing strategies, security headers | `.agents/rules/deployment-and-testing.md` |
| `environment-config` | Secrets and environment variables | `.agents/rules/environment-config.md` |
| `monorepo-workflow` | pnpm commands, adding packages, workspace config | `.agents/rules/monorepo-workflow.md` |
| `data-engine-architecture` | Data Engine backend seeder loading, pnpm workspace dependencies | `.agents/rules/data-engine-architecture.md` |
| `plugin-architecture` | Creating/modifying plugins, lifecycle, registration | `.agents/rules/plugin-architecture.md` |
| `cesium-rendering` | Globe rendering, entity types, primitives, LOD, culling | `.agents/rules/cesium-rendering.md` |
| `state-management` | Zustand slices, store access, plugin settings | `.agents/rules/state-management.md` |
| `database-migrations` | Prisma schema changes, migrations, PostgreSQL | `.agents/rules/database-migrations.md` |
| `database-migrations` | Prisma schema changes, migrations, SQLite/Postgres | `.agents/rules/database-migrations.md` |
| `continuous-improvement` | When to create/update rules, skills, or workflows | `.agents/rules/continuous-improvement.md` |
| `context-and-memory` | How to orient and maintain project context between sessions | `.agents/rules/context-and-memory.md` |
| `troubleshooting-and-debugging` | Resolving plugin latency, namespace collisions, or deployment issues | `.agents/rules/troubleshooting-and-debugging.md` |
---
@@ -247,7 +351,7 @@ Refer to these skill documents for specialized tasks:
| `worldwideview-plugin-creation` | **Use when creating any plugin** — strict architectural checklist |
| `plugin-creation-master-guide.md` | Decision matrix for choosing plugin architecture |
| `osm-static-plugin-creation.md` | Creating static GeoJSON plugins from OpenStreetMap |
| `microservice-plugin-creation.md` | Legacy guide for standalone Fastify microservices |
| `microservice-plugin-creation.md` | Building standalone Fastify microservice backends |
| `database-operations.md` | Prisma schema changes, migrations, database queries |
| `database-incident-recovery-procedures.md` | Authoritative protocol for safely restoring a broken production database |
| `index-documentation.md` | Maintaining project documentation index |
@@ -257,21 +361,3 @@ Refer to these skill documents for specialized tasks:
### Global Skills
52 skills are available across all projects. See `.agents/global-skills-index.md` for the full list and invocation paths.
---
## 15. Pull Request & Commit Guidelines
- **Commit Format**: We strictly enforce Conventional Commits (`feat:`, `fix:`, `refactor:`, `perf:`).
- **Workflow**: You **MUST** use the `/commit` workflow before every git commit to ensure proper semantic versioning bumps.
- **Required Checks**: Ensure `pnpm test` and `pnpm build` complete successfully before proposing a merge.
- **Review Process**: Use `/pr-review` to conduct a comprehensive multi-role review on any pull request.
---
## 16. Debugging and Troubleshooting
- **Prisma & PostgreSQL Sync Issues**: If the local database state falls out of sync with Prisma schema, do not manually drop tables. Instead, run `pnpm db:reset` to cleanly wipe and re-apply all migrations.
- **Cesium Entity Clipping**: If `billboard` entities are clipping or failing to render correctly, verify that you are not mixing `point` primitive properties (like `size` or `outlineWidth`) into the `billboard` options.
- **Build Exhaustion (Docker)**: Multi-stage pnpm builds generate massive cache layers. If a Coolify deployment fails silently or PostgreSQL crashes abruptly, check host disk space and run `docker builder prune -a -f`.
- **Next.js Typechecking Failures**: If backend-only `scripts/` fail during Next.js build, verify that the scripts directory is properly listed in the `exclude` array of `tsconfig.json`.
+14 -29
View File
@@ -6,56 +6,47 @@ RUN find . -type f \! -name 'package.json' \! -name 'pnpm-workspace.yaml' \! -na
find . -type d -empty -delete
# Stage 1: Install ALL dependencies (needed for build)
FROM node:26-alpine AS deps
RUN npm install -g [email protected]
FROM node:22-alpine AS deps
RUN corepack enable pnpm
RUN apk add --no-cache python3 make g++
WORKDIR /app
# Copy only the extracted package.jsons
COPY --from=extractor /app ./
# Install dependencies with cache mount for pnpm store
RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm install
RUN --mount=type=cache,id=pnpm,target=/root/.local/share/pnpm/store pnpm install --frozen-lockfile
# Stage 3: Build the application
FROM deps AS builder
# Copy full source code
COPY . .
RUN DATABASE_URL="file:./data/wwv.db" npx prisma generate
# Create an empty SQLite database with all tables applied
RUN mkdir -p ./data && DATABASE_URL=file:./data/wwv.db npx prisma migrate deploy
# Next.js inlines NEXT_PUBLIC_* vars at build time — must be declared as ARGs
ARG NEXT_PUBLIC_CESIUM_ION_TOKEN
ARG NEXT_PUBLIC_GOOGLE_MAPS_API_KEY
ARG NEXT_PUBLIC_BING_MAPS_KEY
ARG NEXT_PUBLIC_WWV_EDITION
ARG NEXT_PUBLIC_WS_ENGINE_URL
ARG NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL
ARG NEXT_PUBLIC_ADSENSE_CLIENT_ID
ARG NEXT_PUBLIC_SUPABASE_URL
ARG NEXT_PUBLIC_SUPABASE_ANON_KEY
# Run our pregenerate schema swap script and then generate Prisma client
RUN NEXT_PUBLIC_WWV_EDITION=$NEXT_PUBLIC_WWV_EDITION pnpm run generate
# Database migrations run at container startup via docker-entrypoint.sh
# DATABASE_URL must be set to a PostgreSQL connection string
# Run Next.js build with Webpack cache mounted
RUN --mount=type=cache,target=/app/.next/cache NODE_OPTIONS="--max_old_space_size=3072" pnpm run build
RUN node scripts/copy-cesium.mjs
# Deploy flattened production dependencies
RUN pnpm --filter worldwideview deploy --prod /app/prod
# Stage 4: Production runner
FROM node:26-alpine AS runner
FROM node:22-alpine AS runner
WORKDIR /app
RUN apk add --no-cache openssl
RUN npm install -g [email protected]
ENV NODE_ENV=production
ENV HOSTNAME=0.0.0.0
ENV PORT=3000
# DATABASE_URL must be provided via environment variable (no default)
# Example: postgresql://user:pass@host:5432/dbname
ENV DATABASE_URL=file:./data/wwv.db
ENV AUTH_TRUST_HOST=true
# Copy Prisma schema + migrations for runtime DB init
@@ -68,26 +59,20 @@ COPY --from=builder /app/src/generated ./src/generated
# Copy standalone server output
COPY --from=builder /app/.next/standalone ./
# Copy deployed production node_modules
COPY --from=builder /app/prod/node_modules ./node_modules
# We no longer copy proddeps/node_modules. Next.js standalone output
# already traces and copies all the exact node_modules needed for production.
# Copy static assets that standalone mode does NOT include
COPY --from=builder /app/.next/static ./.next/static
COPY --from=builder /app/public ./public
COPY --from=builder /app/scripts/https-proxy.mjs ./scripts/https-proxy.mjs
COPY --from=builder /app/scripts/migrate-legacy.mjs ./scripts/migrate-legacy.mjs
# Entrypoint: migrate DB on first run, then start server
COPY docker-entrypoint.sh ./docker-entrypoint.sh
RUN sed -i 's/\r$//' ./docker-entrypoint.sh
RUN chmod +x ./docker-entrypoint.sh
EXPOSE 3000 3001
# Declare /app/data as a persistent volume mount point.
VOLUME ["/app/data"]
HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1
EXPOSE 3000
ENTRYPOINT ["./docker-entrypoint.sh"]
BIN
View File
Binary file not shown.
+2 -91
View File
@@ -7,20 +7,6 @@
*A modular situational awareness platform designed to ingest live data streams and render them as interactive, cinematic layers on a high-fidelity CesiumJS 3D globe.*
[![CI Build](https://github.com/silvertakana/worldwideview/actions/workflows/ci.yml/badge.svg)](https://github.com/silvertakana/worldwideview/actions/workflows/ci.yml)
[![Codecov](https://img.shields.io/codecov/c/github/silvertakana/worldwideview.svg)](https://codecov.io/gh/silvertakana/worldwideview)
[![NPM Version](https://img.shields.io/npm/v/@worldwideview/wwv-plugin-sdk.svg)](https://www.npmjs.com/package/@worldwideview/wwv-plugin-sdk)
[![GitHub Release](https://img.shields.io/github/v/release/silvertakana/worldwideview?sort=semver)](https://github.com/silvertakana/worldwideview/releases)
[![Contributors](https://img.shields.io/github/contributors/silvertakana/worldwideview.svg)](https://github.com/silvertakana/worldwideview/graphs/contributors)
<br>
[![Next.js 16](https://img.shields.io/badge/Next.js-16-black?logo=next.js)](https://nextjs.org/)
[![CesiumJS](https://img.shields.io/badge/Cesium-JS-4272D0)](https://cesium.com/)
[![TypeScript Strict](https://img.shields.io/badge/TypeScript-Strict-3178C6?logo=typescript)](https://www.typescriptlang.org/)
[![Docker](https://img.shields.io/badge/Docker-Multi--Stage-2496ED?logo=docker)](https://www.docker.com/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Discord](https://img.shields.io/discord/1481088998750294139?color=7289da&label=discord&logo=discord)](#)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://makeapullrequest.com)
![WorldWideView Interface](docs/assets/screenshot.png)
</div>
@@ -29,56 +15,15 @@
WorldWideView is a real-time geospatial engine visualizing live global data on an interactive 3D globe. Utilizing a dynamic "All-Bundle" plugin architecture, independent data sources—like live aircraft, maritime vessels, or conflict events—are ingested and rendered decoupled from the core 3D viewer.
## Key Features
- **"All-Bundle" Plugin Architecture**: Ingest any data source dynamically without touching the core platform.
- **High-Fidelity 3D Rendering**: Google Photorealistic 3D Tiles and LOD modeling powered by CesiumJS.
- **Real-Time Data Pipeline**: High-frequency WebSocket updates managed by a custom `DataBus`.
- **Advanced Entity Management**: Automatic horizon culling, chunked primitive rendering, and 3D stacking/spiderification.
- **Marketplace Integration**: Download and sync new plugins directly from the UI.
## Core Technologies
- **Frontend:** Next.js 16 (App Router), React 19, TypeScript 5
- **3D Engine:** CesiumJS + Resium (Google Photorealistic 3D Tiles)
- **State Management:** Zustand
- **Event Bus:** Custom typed `DataBus` for high-frequency WebSocket updates
- **Database:** PostgreSQL via Prisma 7
- **Database:** SQLite via Prisma (local), PostgreSQL (cloud)
- **Deployment:** Docker multi-stage build, Coolify
## Project Architecture
WorldWideView separates the data acquisition layer from the frontend rendering loop, using a real-time event bus to bridge them.
```mermaid
flowchart TD
subgraph Data Sources
A[Public APIs] -->|Poll| B[Data Engine Seeders]
C[Microservices] -->|Stream| B
end
subgraph Frontend Pipeline
B -->|WebSocket /stream| D[DataBus & WsClient]
D -->|Hydrate| E[Zustand Store]
E -->|Memoized Entities| F[Entity Renderer]
F -->|Primitives| G[CesiumJS Globe]
end
subgraph Plugins
H[Plugin Registry] -->|Load| I[Dynamic Import CDNs]
end
H -.-> D
```
## Prerequisites
Before running the application, ensure you have the following installed:
- [Node.js](https://nodejs.org/) (v18+)
- [pnpm](https://pnpm.io/) (v9+)
- [Docker](https://www.docker.com/) (for self-hosting or full local dev)
- PostgreSQL (or rely on the `coolify-db` / local compose container)
## Quick Start (Self-Hosting)
WorldWideView uses a multi-stage Dockerfile designed for standalone output. To deploy instantly on your own server:
@@ -96,7 +41,7 @@ Invoke-WebRequest -Uri https://raw.githubusercontent.com/silvertakana/worldwidev
```
> [!NOTE]
> Ensure you connect a PostgreSQL database via the `DATABASE_URL` environment variable for production deployments.
> Ensure you mount the `/app/data` volume to persist the SQLite database across container restarts.
## Quick Start (Local Development)
@@ -111,24 +56,6 @@ pnpm run dev:all # boots the UI, cache layers, and the data engine
```
Visit `http://localhost:3000` to see the live globe.
## Project Structure
The codebase utilizes a `pnpm` monorepo configuration:
```text
worldwideview/
├── src/ # Core frontend app
│ ├── app/ # Next.js App Router (pages, API routes)
│ ├── components/ # Shared UI, Globe panels, and 3D layouts
│ ├── core/ # DataBus, Polling, PluginManager, Store
│ └── plugins/ # Built-in plugins and registry logic
├── packages/ # Monorepo packages
│ ├── wwv-plugin-sdk/ # SDK interfaces and manifest schemas
│ └── wwv-plugin-*/ # Individual plugins & their backends
├── prisma/ # Database schemas & migrations
└── .agents/ # Agent rules, workflows, and documentation
```
## Plugin Ecosystem
WorldWideView operates on an open-core philosophy. The platform itself is data-agnostic; all data sources are dynamically imported as plugins at runtime.
@@ -146,22 +73,6 @@ WorldWideView is distributed across several specialized repositories:
4. **`worldwideview-marketplace`**: The web application driving the plugin directory.
5. **`worldwideview-web`**: Marketing and landing site.
## Development & Workflow
- **Branching & Commits:** We strictly enforce [Conventional Commits](https://www.conventionalcommits.org/) (`feat:`, `fix:`, `refactor:`). Every commit should utilize our semantic versioning `[/commit]` workflow.
- **Coding Standards:** We emphasize vanilla CSS (no Tailwind), strict TypeScript 5, and file modularity (max 150 lines per file).
- **Testing:** We use Vitest with `jsdom`. All new core logic should be accompanied by tests, running via `pnpm test`.
See [Docs: Development](docs/development.md) and [Docs: Testing](docs/testing.md) for more details.
## Contributing
We welcome community contributions! Please review our coding standards and PR processes before submitting code. For detailed instructions on local development and setting up your environment, see our [CONTRIBUTING.md](CONTRIBUTING.md).
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## Documentation Index
Explore our comprehensive documentation suite for detailed engineering insights:
+3 -3
View File
@@ -26,11 +26,11 @@ Each stage is **self-contained and demo-able**. Any stage can be shipped indepen
---
## Stage 3: Prisma ORM & Local Database ✅
**Demo:** "Run `pnpm run dev` on a fresh clone with a valid DATABASE_URL. App works seamlessly."
**Demo:** "Run `pnpm run dev` on a fresh clone. SQLite auto-creates. App works with zero config."
- [x] Prisma (`prisma` + `@prisma/client`)
- [x] Schema: `InstalledPlugin`, `Setting`, `User`, `AviationHistory`, `NpmCache`
- [x] PostgreSQL default: `DATABASE_URL` required
- [x] SQLite default: `DATABASE_URL=file:./data/wwv.db`
- [x] `predev` script: `prisma migrate deploy`
- [x] DB auto-creates on first run — zero manual setup
@@ -142,7 +142,7 @@ Each stage is **self-contained and demo-able**. Any stage can be shipped indepen
| **License** | EL2.0 for WorldWideView core; Open Source for marketplace |
| **Shared types** | `@worldwideview/plugin-sdk` npm package (MIT) |
| **Hosting** | Coolify (ThinkPad T480s). Landing page on Vercel (static export) |
| **Database** | PostgreSQL, Prisma ORM |
| **Database** | Supabase PostgreSQL (cloud), SQLite (local), Prisma ORM |
| **Tenant isolation** | Row-Level Security (RLS) in shared PostgreSQL |
| **User settings** | Runtime settings store in DB, not `.env` |
| **Bot prevention** | Email verification + Cloudflare Turnstile + rate limiting |
-37
View File
@@ -1,37 +0,0 @@
# Security Policy
Security is a priority for our project. We appreciate the efforts of security researchers and the open-source community in helping us keep our software secure.
## Supported Versions
Security updates are provided for the following versions of the project. We highly recommend all users stay on the latest stable release.
| Version | Supported | Notes |
| ------- | ------------------ | ----- |
| 2.x.x | :white_check_mark: | Current main release. Receives all security updates. |
| 1.x.x | :x: | Legacy version. No longer receiving updates. |
| < 1.0 | :x: | Beta versions are not supported. |
## Reporting a Vulnerability
**Please do not report security vulnerabilities through public GitHub issues.**
If you believe you have found a security vulnerability in this project, please report it privately so we have time to fix it before the vulnerability is made public.
### How to Report
1. **Email us directly:** Send your report to **[support@worldwideview.dev]**.
2. **GitHub Security Advisories:** Alternatively, if this repository has Private Vulnerability Reporting enabled, you can click on the `Security` tab -> `Advisories` -> `Report a vulnerability`.
### What to Include
To help us quickly validate and fix the issue, please include the following in your report:
* A description of the vulnerability and its potential impact.
* The steps required to reproduce the issue (a proof of concept is highly appreciated).
* Any relevant environment details (OS, browser, version).
### What to Expect
* **Acknowledgment:** We will acknowledge receipt of your vulnerability report within **48 hours**.
* **Triage & Fix:** We will review the report, determine its severity, and work on a patch. We will keep you updated on our progress.
* **Disclosure:** Once the vulnerability is patched and a new version is released, we will publish a security advisory. We are happy to credit you for the discovery if you would like!
*Note: As an open-source project, we currently do not offer financial bug bounties, but we deeply appreciate your contributions to keeping our users safe.*
+1
View File
@@ -0,0 +1 @@
{"message":"fix(aviation): implement exponential backoff for rate limits","tree":"55c74a09dac7ce178ecf94d5cb48b2502e57e759","parents":["e25ddc0b648b757ea32b7376bab50d8e75d6d330"]}
+54
View File
@@ -0,0 +1,54 @@
const fs = require("fs");
const { execSync } = require("child_process");
const uuidsToDelete = [
"i331gbc1zzytx7via8lntdm6",
"pow71xsobdas7etf5r3hy5ug",
"l79dyd03lhtcpgi7swkoxm55",
"ry6160tpy4gh2xjfwr011no0",
"gfe78t8jm53lltentz19ze60",
"e1cg7q8khyvpti5rkp351559",
"c35cvvqxhdbslk8yc24g04wk",
"ijj9pqkavlusy2wmzsbjtc74",
"ghe6bzlr6ex6j51lgxzyhd46",
"h3v45w2u1essulzeg56i27a3",
"oxk05amog4fla0lskmkfx64x",
"yxm4mn60lfp9qah0q4yf2hij",
"ezjxo4ehs96lv0ji1n2r0auw",
"t12x7pgptsqiu5gbvv35gun5",
"urq1ll00a3vv9on6pulmpofe",
"eb5zgw5wlgvio2gggbiphpam",
"ld1h3pody4nxj3b44pbskl53",
"x111xfygqxrlbf5wqlx59f6d",
"se1weqr4dj7thp4p1l055wfc",
"cqses6urxff0n94i0u6lktzt",
"nvcir33nyuvzk5niy3s7eg9w",
"s63onga2s6v8r3k10lrdwzth",
"cf3uedqnjmwv48h7tm91d95r",
"o9124dnnieau9wxoefyxvfgu",
"kxzobc8qlvigbsk1d35dpiyi",
"eh8lzjbsy2c9kwj2emztcmf6",
"umfsos2vcmotfk2cfpqdoycc",
"yn1mdh0fngiati6r4625qvv9"
];
const token = "1|vPZPgvB8vVi8ZD5ZKS8lrCrwNlPofVv4I8Tjmqmyd01f92d2";
const baseUrl = "https://coolify.worldwideview.dev/api/v1";
const appUuid = "nmn55t4io3myfubs72worn7k";
async function run() {
for (const id of uuidsToDelete) {
console.log("Deleting", id);
try {
const res = await fetch(`${baseUrl}/applications/${appUuid}/envs/${id}`, {
method: "DELETE",
headers: { "Authorization": `Bearer ${token}` }
});
console.log(id, res.status);
} catch(e) {
console.error(e);
}
}
}
run();
-7
View File
@@ -1,7 +0,0 @@
SUPABASE_PG_URL=postgresql://postgres:[email protected]:5432/postgres
SUPABASE_URL=https://xyz.supabase.co
SUPABASE_ANON_KEY=eyJ...
SUPABASE_SERVICE_ROLE_KEY=eyJ...
ACME_EMAIL=[email protected]
STRIPE_SECRET_KEY=sk_test_...
STRIPE_WEBHOOK_SECRET=whsec_...
-47
View File
@@ -1,47 +0,0 @@
services:
db:
image: postgres:15-alpine
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: password
POSTGRES_DB: postgres
ports:
- "5432:5432"
wwv-cloud:
build:
context: ../..
dockerfile: Dockerfile
args:
- NEXT_PUBLIC_WWV_EDITION=cloud
restart: always
environment:
NEXT_PUBLIC_WWV_EDITION: cloud
DATABASE_URL: postgresql://postgres:password@db:5432/postgres
NEXT_PUBLIC_SUPABASE_URL: ${SUPABASE_URL:-http://dummy}
NEXT_PUBLIC_SUPABASE_ANON_KEY: ${SUPABASE_ANON_KEY:-dummy}
SUPABASE_SERVICE_ROLE_KEY: ${SUPABASE_SERVICE_ROLE_KEY:-dummy}
AUTH_SECRET: ${AUTH_SECRET:-dummy_secret_for_local_testing_only_1234567890}
ports:
- "3001:3000"
labels:
- "traefik.enable=true"
- "traefik.http.routers.wwv.rule=HostRegexp(`{user:[a-z0-9-]+}.app.worldwideview.dev`) || Host(`app.worldwideview.dev`) || Host(`localhost`) || HostRegexp(`{user:[a-z0-9-]+}.localhost`)"
- "traefik.http.routers.wwv.entrypoints=web,websecure"
# - "traefik.http.routers.wwv.tls.certresolver=letsencrypt"
- "traefik.http.services.wwv.loadbalancer.server.port=3000"
depends_on:
- db
traefik:
image: traefik:v3.0
command:
- "--api.insecure=true"
- "--providers.docker=true"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
ports:
- "80:80"
- "443:443"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
-44
View File
@@ -1,44 +0,0 @@
services:
wwv:
image: ghcr.io/silvertakana/worldwideview:latest
pull_policy: always
ports:
- "3100:3000"
volumes:
- wwv-data:/app/data
env_file:
- path: .env
required: false
environment:
- "AUTH_SECRET=${AUTH_SECRET:-default_secret_for_local_dev}"
- "WWV_DEMO_ADMIN_SECRET=${WWV_DEMO_ADMIN_SECRET:-}"
- "WWV_BRIDGE_TOKEN=${WWV_BRIDGE_TOKEN:-}"
- "DATABASE_URL=${DATABASE_URL:-postgresql://postgres:postgres@db:5432/worldwideview?schema=public}"
# Provide the explicit public or host IP of the data engine for the frontend to use
- "NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL=http://192.168.68.69:5002/stream"
restart: unless-stopped
depends_on:
db:
condition: service_healthy
db:
image: postgres:15-alpine
restart: unless-stopped
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: worldwideview
ports:
- "5433:5432"
volumes:
- postgres-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d worldwideview"]
interval: 5s
timeout: 5s
retries: 5
start_period: 60s
volumes:
wwv-data:
postgres-data:
@@ -1,35 +0,0 @@
services:
wwv-data-engine:
image: ghcr.io/silvertakana/wwv-data-engine:latest
pull_policy: always
ports:
- '5002:5000'
env_file:
- path: .env
required: false
environment:
- PORT=5000
- REDIS_URL=redis://wwv-redis:6379
- DOWNLOAD_SEEDERS=true
- GITHUB_PAT=${GITHUB_PAT:-}
- DB_PATH=/app/data/engine.db
depends_on:
wwv-redis:
condition: service_healthy
volumes:
- wwv-engine-data:/app/data
- wwv-seeders:/app/seeders
restart: unless-stopped
wwv-redis:
image: redis:7-alpine
restart: unless-stopped
healthcheck:
test: ['CMD', 'redis-cli', 'ping']
interval: 5s
timeout: 3s
retries: 5
volumes:
wwv-engine-data:
wwv-seeders:
-56
View File
@@ -12,65 +12,9 @@ services:
- "AUTH_SECRET=${AUTH_SECRET:-default_secret_for_local_dev}"
- "WWV_DEMO_ADMIN_SECRET=${WWV_DEMO_ADMIN_SECRET:-}"
- "WWV_BRIDGE_TOKEN=${WWV_BRIDGE_TOKEN:-}"
- "DATABASE_URL=${DATABASE_URL:-postgresql://postgres:postgres@db:5432/worldwideview?schema=public}"
depends_on:
db:
condition: service_healthy
healthcheck:
test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
db:
image: postgres:15-alpine
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: worldwideview
ports:
- "5432:5432"
volumes:
- postgres-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d worldwideview"]
interval: 5s
timeout: 5s
retries: 5
start_period: 60s
profiles:
- "${WWV_SKIP_LOCAL_DB:-}"
wwv-data-engine:
build: ../wwv-data-engine-public
ports:
- '5000:5000'
environment:
- PORT=5000
- REDIS_URL=redis://wwv-redis:6379
- SEEDERS_DIR=/mnt/host/c/dev/worldwideview/local-seeders
- SKIP_SEEDER_DOWNLOAD=true
- DB_PATH=/app/data/engine.db
depends_on:
wwv-redis:
condition: service_healthy
volumes:
- ./local-seeders:/mnt/host/c/dev/worldwideview/local-seeders
restart: unless-stopped
wwv-redis:
image: redis:7-alpine
healthcheck:
test: ['CMD', 'redis-cli', 'ping']
interval: 5s
timeout: 3s
retries: 5
volumes:
wwv-data:
postgres-data:
networks:
coolify:
+5 -38
View File
@@ -1,47 +1,14 @@
#!/bin/sh
# ─── Docker Entrypoint ───────────────────────────────────────
# Ensures the PostgreSQL database is migrated before starting
# the application. DATABASE_URL must point to a PostgreSQL
# instance (Supabase, self-hosted, etc).
# Ensures the SQLite database exists and is migrated before
# starting the application. On first run with a fresh volume
# the DB file won't exist yet, so we run prisma migrate deploy.
set -e
if [ -z "$DATABASE_URL" ]; then
echo "────────────────────────────────────────────────────────────────"
echo "🚨 CRITICAL: DATABASE_URL IS NOT SET"
echo "────────────────────────────────────────────────────────────────"
echo "WorldWideView has migrated to PostgreSQL. If you are an old user,"
echo "please update your docker-compose.yml to include a Postgres service."
echo ""
echo "To migrate your legacy data after setting up Postgres, run:"
echo "docker exec -it <container_name> node scripts/migrate-legacy.mjs"
echo "────────────────────────────────────────────────────────────────"
# If dev.db exists, we know they are upgrading. Don't crash, just wait for config.
if [ -f "prisma/dev.db" ]; then
echo "[entrypoint] Legacy data detected. Waiting for DATABASE_URL to be configured..."
# We don't exit here, we'll let the app try to start or just sleep.
# For now, let's just let it proceed so they can see the setup page error.
else
echo "[entrypoint] Exiting. Please provide a DATABASE_URL."
exit 1
fi
fi
mkdir -p ./data
echo "[entrypoint] Running database migrations..."
prisma migrate deploy
npx -y prisma migrate deploy
echo "[entrypoint] Migrations complete."
# Generate self-signed SSL certificates for local HTTPS bridging if they don't exist
if [ ! -f "./data/localhost.crt" ] || [ ! -f "./data/localhost.key" ]; then
echo "[entrypoint] Generating self-signed SSL certificates for port 3001..."
mkdir -p ./data
openssl req -nodes -new -x509 -keyout ./data/localhost.key -out ./data/localhost.crt -days 365 -subj "/CN=localhost" 2>/dev/null || echo "[entrypoint] Warning: Failed to generate SSL certs"
fi
# Start the HTTPS proxy in the background
if [ -f "./scripts/https-proxy.mjs" ]; then
node ./scripts/https-proxy.mjs &
fi
exec node server.js
+2 -2
View File
@@ -12,7 +12,7 @@ The core design philosophy is the "All-Bundle Model," where plugins are dynamica
- **State Management:** Zustand slice-based architecture (in `src/core/state/`).
- **Data Pipeline:** Custom event bus (`DataBus`) and WebSocket clients (`DataBusSubscriber`) routing live data to state (in `src/core/data/`).
- **Plugin System:** Registration, lifecycle management, and ES module loaders (in `src/core/plugins/`).
- **Database Layer:** Prisma ORM connecting to PostgreSQL (in `prisma/`).
- **Database Layer:** Prisma ORM connecting to local SQLite or cloud PostgreSQL (in `prisma/`).
## Key Files
- `src/core/plugins/PluginManager.ts` - The PluginManager class. Core registry that instantiates plugins, calls `initialize()`, and handles lifecycle events.
@@ -24,7 +24,7 @@ The core design philosophy is the "All-Bundle Model," where plugins are dynamica
## Data Flow
Real-time data follows a strict unidirectional stream optimized for sub-second updates:
1. **Engine Push:** A remote data engine pushes state over WebSocket (`/stream`) from either local seeders (`ws://localhost:5000/stream`) or the cloud fallback (`wss://dataengine.worldwideview.dev/stream`).
1. **Engine Push:** A remote data engine pushes state over WebSocket (`/stream`) from either local seeders (`ws://localhost:5001/stream`) or the cloud fallback (`wss://dataengine.worldwideview.dev/stream`).
2. **WebSocket Client Router:** `WsClient.handleMessage()` receives the payload and pipes it to the event bus.
3. **DataBus Emission:** The bus fires `DataBus.emit("websocketData", WsStreamPayload)`.
4. **State Hydration:** The `DataBusSubscriber` component listens to the bus and calls `_hydrateSnapshot()` to update the Zustand store's `entitiesByPlugin` cache.
+3 -3
View File
@@ -11,7 +11,7 @@ The core build configuration resides in `package.json` and `next.config.ts`. A c
### Standard Development
- **Install Dependencies:** `pnpm install` (Installs all workspace packages defined in `pnpm-workspace.yaml`).
- **Initial Setup:** `pnpm run setup` (Executes `scripts/setup.mjs` to generate a `.env.local` with a fresh `AUTH_SECRET`).
- **Run Development Server (All):** `pnpm dev:all` (Concurrently starts the Next.js frontend and the local `wwv-data-engine` + Redis containers via Docker Compose).
- **Run Development Server (All):** `pnpm dev:all` (Concurrently starts the Next.js frontend and the local `wwv-data-engine` backends).
### Production Build
- **Build Command:** `pnpm build` (Executes `next build --webpack`).
@@ -20,8 +20,8 @@ The core build configuration resides in `package.json` and `next.config.ts`. A c
## Platform Setup
- **Database Migrations:** Prisma schema is located at `prisma/schema.prisma`. During development, `pnpm dev` triggers a `predev` hook running `npx prisma migrate deploy` to ensure PostgreSQL tables are initialized.
- **Data Engine:** The generic data engine runner and Redis cache are started via `docker compose up wwv-data-engine wwv-redis`. Custom data seeders are placed in the `local-seeders/` sandbox and dynamically loaded by the engine.
- **Database Migrations:** Prisma schema is located at `prisma/schema.prisma`. During development, `pnpm dev` triggers a `predev` hook running `npx prisma migrate deploy` to ensure SQLite tables exist in `data/wwv.db`.
- **Microservices:** Data engine seeders and standalone Fastify backend plugins (like `wwv-plugin-iranwarlive/backend`) are started via `pnpm start:backends` and managed locally inside `packages/*/backend`.
- **Plugin Packages:** Internal plugin packages must be properly listed in `tsconfig.json` paths and `next.config.ts`'s `transpilePackages` array.
## Reference
+6 -38
View File
@@ -10,57 +10,25 @@ WorldWideView utilizes a containerized deployment strategy based on Docker's mul
- **Build Output:** Located at `.next/standalone/`.
- **Static Assets:** Cesium static workers are injected at build time via `scripts/copy-cesium.mjs`.
### Data Engine Runner
- The generic `wwv-data-engine` runner is deployed as a single Coolify service, orchestrating telemetry data by executing lightweight scripts from volume-mounted seeder directories.
### Plugin Microservices
- Standalone plugin backends (e.g., `wwv-plugin-iranwarlive/backend`) are isolated via individual `Dockerfile` configurations and orchestrated together in production using `docker-compose.yml`.
## Platform Deployment
### Coolify Integration
WorldWideView deploys optimally to Coolify using a Dockerfile builder.
- **Environment Variables:** Must be explicitly mapped in the Coolify UI (e.g., `DATABASE_URL`, `AUTH_SECRET`).
- **Persistent Storage:** PostgreSQL databases must be hosted externally or mounted via persistent volumes to ensure the frontend registry (installed plugins, user configs) survives container rebuilds.
- **Seeder Deployment:** The generic `wwv-data-engine` runner is deployed as a Coolify service, reading plugin seeders dynamically from a volume-mounted directory. Seeders are pulled from private/community repositories via GitHub releases.
- **Persistent Storage:** SQLite requires a persistent volume mounted to `/app/data` to ensure the frontend registry (installed plugins, user configs) survives container rebuilds.
- **Microservices Deployment:** The `wwv-data-engine-internal` and associated plugin seeders are deployed as separate Coolify services communicating over private internal networks.
### Docker Structure
- **Dockerfile:** Found at the project root (`Dockerfile`). Uses an Extractor Pattern (`deps` → `builder` → `runner`). The `.git` and `node_modules` folders must be explicitly untracked to prevent cache overlap.
- **Compose:** Local multi-container emulation is handled via `docker-compose.yml`.
## Reverse Proxies (Nginx Proxy Manager)
When deploying WorldWideView behind a reverse proxy like Nginx Proxy Manager (NPM), you must configure both the proxy and the container environment correctly to prevent NextAuth from hanging during login, and to allow the real-time Cesium data stream to function.
### Container Environment Variables
NextAuth v5 requires strict host trust when placed behind a proxy. Add the following to your Docker `.env` file or Coolify environment variables:
```bash
# Required: Tells NextAuth to trust the reverse proxy headers
AUTH_TRUST_HOST=true
# Required: The exact external URL your users visit
NEXTAUTH_URL=https://your-domain.com
```
### Nginx Proxy Manager Configuration
In the NPM UI for your WorldWideView Proxy Host:
1. **Details Tab:** Ensure **Websockets Support** is toggled **ON** (Required for the live `/stream` telemetry).
2. **Details Tab:** Ensure **Force SSL** is toggled **ON**. NextAuth will often refuse authentication flows over plain HTTP.
3. **Advanced Tab:** Nginx Proxy Manager sometimes fails to pass the correct host headers required by Next.js. Add the following to the Custom Nginx Configuration box:
```nginx
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header Host $host;
# Optional: Disable proxy buffering if 3D tiles load slowly
proxy_buffering off;
proxy_request_buffering off;
```
## Reference
- **Standalone Config:** `next.config.ts` (sets `output: "standalone"`).
- **Prisma Export:** `prisma.config.ts` ensures no CLI wrappers are invoked inside the standalone container, preventing fatal `MODULE_NOT_FOUND` runtime crashes.
- **Docker Mounts:**
- Main App: PostgreSQL data volume (if running in-cluster)
- Data Engine: PostgreSQL data volume (if running in-cluster)
- Main App: `/app/data` (SQLite DB)
- Data Engine: `/app/packages/wwv-data-engine/data` (SQLite DB)
- Cache: `/data` (Redis)
-4
View File
@@ -14,7 +14,6 @@ Development on WorldWideView prioritizes modularity, high performance, and stric
- **Global Styles:** Located at `src/app/globals.css`.
- **Component Styles:** Use CSS Modules (`.module.css`) or co-locate `.css` files directly next to their components.
- **HUD Animations:** Specialized interface animations are stored in `src/styles/hud-animations.css`.
- **Color Theming & Design Tokens:** Defined in `src/styles/theme-tokens.css` via CSS variables (e.g., `[data-theme='black']`).
### Import Aliases
- Application core: `@/*` → `./src/*`
@@ -64,9 +63,6 @@ const toggleLayer = useStore((state) => state.layers.toggle);
Whenever generating temporary debugging scripts, testing REST endpoints via `.mjs`, or dumping traces/JSON outputs, **save these exclusively inside `/local-scripts/`**.
- *Reference:* The root directory is strictly for production configuration files.
### Local Plugin & Seeder Testing
When developing a custom seeder for a plugin, place the seeder directory inside `local-seeders/`. The local `wwv-data-engine` Docker container automatically mounts this folder and discovers scripts on startup, providing a seamless sandbox.
### Scaffold a New Component
1. Create a new directory under `src/components/{category}/MyComponent`.
2. Add `MyComponent.tsx` and `MyComponent.module.css`.
+4 -4
View File
@@ -21,13 +21,13 @@ The system is highly modularized to separate React UI rendering from heavy Cesiu
## Build System
- `package.json` - Root monorepo configuration with `pnpm` workspace bindings.
- `next.config.ts` - Next.js compiler settings, security headers, and standalone output configuration.
- `docker-compose.yml` - Defines the orchestration of the frontend, Redis cache, and the generic data engine runner (`wwv-data-engine`) loaded with local seeders.
- `docker-compose.yml` - Defines the orchestration of the frontend, Redis cache, and all local plugin microservice backends (`wwv-data-engine`).
## Configuration
- `prisma/schema.prisma` - PostgreSQL database schema for storing user settings and installed plugin manifests.
- `prisma/schema.prisma` - Local SQLite and cloud PostgreSQL database schema for storing user settings and installed plugin manifests.
- `scripts/copy-cesium.mjs` - Crucial build-time script that extracts and copies CesiumJS static web workers to `public/cesium/`.
- `public/` - Static assets, plugin icons, and compiled Cesium workers.
## Reference
- **Plugins Directory:** All frontend plugins conform to the `@worldwideview/wwv-plugin-sdk` interface and are developed inside `packages/wwv-plugin-<name>/` or `local-plugins/<name>/`.
- **Seeders Directory:** Backend telemetry scripts are developed in the `local-seeders/community/<name>/` or `local-seeders/private/<name>/` directories and are mounted directly into the data engine at runtime.
- **Plugins Directory:** All plugins conform to the `@worldwideview/wwv-plugin-sdk` interface and are developed inside `packages/wwv-plugin-<name>/`.
- **Microservices Directory:** Standalone Fastify backends are nested within `packages/wwv-plugin-<name>/backend/`.
-116
View File
@@ -1,116 +0,0 @@
# GeoJSON Data Guide
WorldWideView supports visualizing external datasets and local static files by interpreting GeoJSON files. When using the built-in GeoJSON parser (for instance, via declarative configs, static plugins, or user-uploaded files), the data is automatically normalized into our internal `GeoEntity` format so it seamlessly displays on the globe and the UI's detail panels.
This guide outlines exactly what structures are supported, what isn't, and how you can map your data into the UI.
---
## What is Supported?
Our parser enforces a subset of the **RFC 7946** standard. To be successfully interpreted, your JSON file must meet the following criteria:
### 1. Root Level
The root of your JSON document must be a `FeatureCollection`. It must contain a `features` array.
```json
{
"type": "FeatureCollection",
"features": [ ... ]
}
```
### 2. Supported Geometry: Points
WorldWideView's declarative parser strictly supports **Point** geometries. Your coordinates must be a flat array of numbers: `[longitude, latitude]` or optionally `[longitude, latitude, altitude]`.
```json
"geometry": {
"type": "Point",
"coordinates": [-83.74088, 42.27756, 150.0]
}
```
*Note: Altitude (the 3rd coordinate) is optional. If omitted, the entity will be clamped to the ground.*
### 3. Properties and Metadata
You can include any arbitrary metadata inside the `properties` object. The parser maps these properties to the UI using a `FieldMapping` configuration.
```json
"properties": {
"stream": "http://example.com/video.mjpg",
"country": "United States",
"is_static": false,
"nested_data": {
"speed": 45,
"heading": 180
}
}
```
#### Field Mapping
When configuring your data source, you can define how your GeoJSON properties map to standard WorldWideView `GeoEntity` fields. This is done via dot-notation:
| GeoEntity Field | Description | Example Path |
| --- | --- | --- |
| `id` | Unique identifier | Automatically maps to `Feature.id` if present, otherwise uses array index. |
| `heading` | Rotation angle (degrees) | `properties.nested_data.heading` |
| `speed` | Speed value | `properties.nested_data.speed` |
| `timestamp` | Time of the event (string or epoch) | `properties.last_updated` |
| `label` | Primary text label shown on map | `properties.city` |
| `properties` | Custom fields mapped to the UI panel | `properties.stream` |
Any property extracted via mapping will appear automatically in the Entity Info Card when a user clicks the point on the map.
---
## What is NOT Supported?
To ensure extreme performance (60 FPS with 10k+ entities) and a unified UI experience, the following GeoJSON structures are **not supported** by the automatic data parser:
### 1. Complex Geometries
The declarative parser will fail to render the following geometry types:
- `Polygon`
- `MultiPolygon`
- `LineString`
- `MultiLineString`
- `GeometryCollection`
*Why?* The automatic parser maps features to single point entities (billboards or 3D models). If you supply a `Polygon`, the parser will read the nested coordinate arrays (`[[lon, lat], ...]`) as invalid numbers, resulting in a silent failure (`NaN` clamping).
If you need to render borders, heatmaps, or route lines, you must build a **Code Bundle Plugin** and interact with the Cesium primitives (`PolygonHierarchy`, `PolylineCollection`) directly via the `renderEntity` API.
### 2. Top-Level Features
A single isolated `Feature` object at the root of the document will not be parsed. The data must be wrapped in a `FeatureCollection`.
### 3. Non-Standard Coordinate Orders
Cesium and GeoJSON strictly require `[longitude, latitude]`. If your system outputs `[latitude, longitude]`, you must transform the data before passing it to WorldWideView.
---
## Example: Perfect GeoJSON Structure
Here is a fully supported and optimized example demonstrating cameras mounted globally:
```json
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"id": "cam-001",
"geometry": {
"type": "Point",
"coordinates": [-83.74088, 42.27756]
},
"properties": {
"stream": "http://173.167.10.225:80/mjpg/video.mjpg",
"country": "United States",
"city": "Ann Arbor",
"categories": ["traffic"],
"is_static": false
}
}
]
}
```
By ensuring your data matches this structure, WorldWideView can effortlessly stream, render, and display your intelligence layers on the interactive 3D globe.
+127 -67
View File
@@ -23,20 +23,29 @@ Ensure you have the following installed on your machine:
npm install -g pnpm
```
### 1.2 Clone the Repository
### 1.2 Clone the Repositories
You only need the main WorldWideView application. The data engine backend will run automatically via Docker. Open your terminal and navigate to the folder where you want to store your code (e.g., `C:\dev`).
You need both the frontend application and the backend data engine. Open your terminal and navigate to the folder where you want to store your code (e.g., `C:\dev`).
Run this command to download the code:
Run these commands to download the code:
```bash
git clone https://github.com/silvertakana/worldwideview.git
git clone https://github.com/silvertakana/wwv-data-engine.git
```
### 1.3 Install Dependencies
Next, install dependencies for the main WorldWideView application:
Next, we need to install the libraries that these projects depend on.
First, install dependencies for the Data Engine:
```bash
cd worldwideview
cd wwv-data-engine
pnpm install
```
Second, install dependencies for the main WorldWideView application:
```bash
cd ../worldwideview
pnpm install
```
@@ -57,61 +66,62 @@ Your workspace is now ready!
The ISS moves incredibly fast. If our frontend tries to ask for its location every second, it will lag. Instead, we use the **Bring Your Own Backend (BYOB)** pattern. We will write a "Seeder" script in the Data Engine that fetches the ISS location and instantly broadcasts it to the frontend via WebSockets.
### 2.1 Create the Seeder Directory and File
### 2.1 Create the Seeder File
Navigate to your `worldwideview` repository. Inside the `local-seeders` folder, create a new folder named `community/iss`. Inside that folder, create a file named `seeder.mjs`:
`c:\dev\worldwideview\local-seeders\community\iss\seeder.mjs`
> [!NOTE]
> **Dependency Management:** You do *not* need to create a `package.json` or manually install standard dependencies (like `axios`) for your seeder. The `wwv-data-engine` runner dynamically resolves these via the pnpm workspace, keeping your seeder lightweight.
Navigate to your data engine repository and create a new file named `iss.ts` in the `src/seeders` folder:
`c:\dev\wwv-data-engine\src\seeders\iss.ts`
### 2.2 Write the Polling Logic
Open `seeder.mjs` in your text editor and paste the following code. The Data Engine Runner will dynamically discover this script and execute its `fetch(ctx)` function based on the interval you define.
Open `iss.ts` in your text editor and paste the following code. This script reaches out to the free WhereTheISS.at API every 5 seconds and saves the data to the engine's memory (Redis).
```typescript
import { setLiveSnapshot } from '../core/redis';
import axios from 'axios';
```javascript
// seeder.mjs
const WTIA_URL = 'https://api.wheretheiss.at/v1/satellites/25544';
const POLLING_INTERVAL = 5000; // 5 seconds
export default {
// Required: the unique namespace used by your plugin
id: "iss",
// Define polling interval inside the seeder
intervalMs: 5000,
// The Data Engine will call this function every intervalMs
async fetch(ctx) {
const { axios, logger } = ctx;
async function pollISS() {
try {
// 1. Fetch data from the API
const response = await axios.get(WTIA_URL);
const data = response.data;
// 2. Format the data into a GeoEntity object
const stateObj = {
id: 25544,
name: "International Space Station",
latitude: data.latitude,
longitude: data.longitude,
altitude: data.altitude * 1000, // Convert kilometers to meters
velocity: data.velocity,
visibility: data.visibility,
footprint: data.footprint
};
// 3. Save to Redis under the 'iss' namespace for 60 seconds
await setLiveSnapshot('iss', { "25544": stateObj }, 60);
console.log(`[ISS] Poll OK: updated position to ${data.latitude}, ${data.longitude}`);
try {
// 1. Fetch data from the API
const response = await axios.get(WTIA_URL);
const data = response.data;
// 2. Format the data into a GeoEntity array
const entities = [{
id: "25544",
name: "International Space Station",
latitude: data.latitude,
longitude: data.longitude,
altitude: data.altitude * 1000, // Convert kilometers to meters
velocity: data.velocity,
visibility: data.visibility,
footprint: data.footprint
}];
logger.info(`[ISS] Poll OK: updated position to ${data.latitude}, ${data.longitude}`);
// 3. Return the array of entities, the engine runner will broadcast it over WebSockets
return entities;
} catch (error) {
logger.error(`[ISS] Polling error: ${error.message}`);
return [];
}
} catch (error) {
console.error(`[ISS] Polling error:`, error.message);
} finally {
// 4. Repeat the process after 5 seconds
setTimeout(pollISS, POLLING_INTERVAL);
}
};
}
// Start the loop
pollISS();
```
### 2.3 Register the Seeder
The engine needs to know your seeder exists. Open `c:\dev\wwv-data-engine\src\seeders\index.ts` and add this line at the bottom:
```typescript
import './iss';
```
---
@@ -122,11 +132,38 @@ Before we build the frontend, let's prove that our backend is actually fetching
### 3.1 Start the Engine
In your terminal, navigate to the `worldwideview` folder and start the entire stack using Docker Compose:
In your terminal, navigate to the `wwv-data-engine` folder and start the backend:
```bash
pnpm dev:all
pnpm dev:backends
```
Wait for the Data Engine Docker container to start. You should see `[ISS] Poll OK: updated position` appearing in the terminal logs every 5 seconds. Leave this terminal running.
You should see `[ISS] Poll OK: updated position` appearing in your console every 5 seconds. Leave this terminal running.
### 3.2 Write a Quick Test Script
Open a *new* terminal window. Let's write a tiny script to peek into the engine's memory. Create a file called `test_iss.mjs` in the `wwv-data-engine` folder:
```javascript
// test_iss.mjs
import Redis from 'ioredis';
const redis = new Redis();
async function verify() {
const data = await redis.get('data:iss:live');
if (data) {
console.log("Success! We found the ISS:", JSON.parse(data));
} else {
console.log("No data found yet.");
}
process.exit(0);
}
verify();
```
Run the script:
```bash
node test_iss.mjs
```
If you see a large block of data containing coordinates and altitude, your backend is working perfectly!
---
@@ -136,18 +173,22 @@ Now we will build the visual part of the plugin that connects to our backend str
### 4.1 Scaffold the Plugin
In your terminal, navigate to your main `worldwideview` folder. We will use the workspace CLI tool to generate a blank plugin template in the local sandboxes folder. Run:
In your terminal, navigate back to your main `c:\dev` folder. We will use the official CLI tool to generate a blank plugin template. Run:
```bash
node packages/wwv-cli/dist/index.js create wwv-plugin-iss --local
pnpm install
npx @worldwideview/create-plugin@latest wwv-plugin-iss
```
When prompted, name the folder `wwv-plugin-iss`.
This creates your plugin at `local-plugins/wwv-plugin-iss` and automatically registers it with the workspace.
Move into your new plugin folder and install its dependencies:
```bash
cd wwv-plugin-iss
npm install
```
### 4.2 Write the Plugin Logic
Open `c:\dev\worldwideview\local-plugins\wwv-plugin-iss\src\index.ts` in your text editor. Replace everything in the file with this code:
Open `c:\dev\wwv-plugin-iss\src\index.ts` in your text editor. Replace everything in the file with this code:
```typescript
import type { WorldPlugin, GeoEntity, PluginContext, LayerConfig, CesiumEntityOptions } from "@worldwideview/wwv-plugin-sdk";
@@ -205,14 +246,33 @@ export class IssPlugin implements WorldPlugin {
Now we will link your new plugin to your local WorldWideView application to see it in action.
### 5.1 Develop Your Plugin
### 5.1 Link the Plugin
Because your plugin is located in `local-plugins/wwv-plugin-iss`, it is automatically detected by the workspace.
You do not need to link it manually.
In your `wwv-plugin-iss` terminal, tell the CLI where your main WorldWideView folder is:
```bash
npx wwv config set wwv-path C:\dev\worldwideview
```
Since you already started the whole stack (`pnpm dev:all`) in Step 3, the `dev:plugins` watcher is already running in the background. It will automatically build your frontend plugin code whenever you save changes.
Now, link the plugin. This creates a shortcut so WorldWideView can see your code:
```bash
npm run link
```
### 5.2 View the ISS
Start the plugin watcher. This will automatically update your plugin if you make any changes to the code:
```bash
npm run dev
```
Leave this terminal running.
### 5.2 Start WorldWideView
Open a third, final terminal window. Navigate to your main `worldwideview` folder and start the frontend application:
```bash
cd C:\dev\worldwideview
pnpm dev
```
### 5.3 View the ISS
1. Open your web browser and go to `http://localhost:3000`.
2. Click the **Layers** icon on the left sidebar.
@@ -221,9 +281,9 @@ Since you already started the whole stack (`pnpm dev:all`) in Step 3, the `dev:p
> [!TIP]
> **Troubleshooting missing points:**
> - Ensure your `pnpm dev:all` terminal is still running without errors.
> - Ensure your Data Engine terminal (`pnpm dev:backends`) is still running.
> - Press `F12` in your browser to open Developer Tools. Check the `Console` tab for any red errors.
> - Ensure the `id` in your plugin (`"iss"`) exactly matches the `id` defined in `seeder.mjs`.
> - Ensure the `id` in your plugin (`"iss"`) exactly matches the name you used in the backend `setLiveSnapshot('iss', ...)`.
---
@@ -233,7 +293,7 @@ You've built it, now share it with the world!
### 6.1 Update package.json
Open `c:\dev\worldwideview\local-plugins\wwv-plugin-iss\package.json`. You must add a `"worldwideview"` metadata block so the marketplace knows how to read your plugin. Ensure your file looks like this:
Open `c:\dev\wwv-plugin-iss\package.json`. You must add a `"worldwideview"` metadata block so the marketplace knows how to read your plugin. Ensure your file looks like this:
```json
{
+11 -14
View File
@@ -18,26 +18,23 @@ WorldWideView operates on a strict **Dynamic CDN Loaded (Bundle)** architecture.
## Real-Time Data: Bring Your Own Backend (BYOB)
Relying on the frontend `fetch()` method is insufficient for high-frequency real-time tracking (like aviation or maritime). For continuous telemetry, you must build a **Data Engine Seeder** — a lightweight Javascript data polling script.
Relying on the frontend `fetch()` method is insufficient for high-frequency real-time tracking (like aviation or maritime). For continuous telemetry, you must build a **Data Engine Seeder** — a containerized backend microservice.
### Data Engine Seeder Architecture
Instead of the frontend fetching data, you write a lightweight seeder script that connects to an upstream source, normalizes the data, and is executed by the central `wwv-data-engine` runner.
### Microservice Seeder Architecture
Instead of the frontend fetching data, your backend container connects to an upstream source, normalizes the data, and publishes it to the central Redis stream managed by `wwv-data-engine`.
1. **Create a Seeder Directory:** Inside your WorldWideView project, create a folder under `local-seeders/` (e.g., `local-seeders/my-plugin/`).
2. **Write the Seeder Script:** Create a `seeder.mjs` file that exports a `fetch(ctx)` function.
3. **Engine Auto-Discovery:** The local Docker-based `wwv-data-engine` automatically mounts this directory, discovers your script, and runs it on the defined interval.
4. **WebSocket Delivery:** The central `wwv-data-engine` immediately broadcasts your returned updates over WebSockets to all connected clients.
### Dependency Management & Monorepo Hoisting
Seeders within `local-seeders/` are strictly orchestrated within the pnpm workspace. They are executed by the central runner, not as standalone applications.
- **Keep `package.json` clean**: Do not include bulky `dependencies` in your seeder's local `package.json` (unless it's an exceptional, bespoke library).
- **Workspace Resolution**: Standard packages (e.g., `axios`, `ioredis`, `@worldwideview/wwv-plugin-sdk`) are provided by the engine. At runtime, the `wwv-data-engine` leverages native Node.js module resolution to fetch the required dependencies directly from the root workspace or its own containerized runtime.
- **Lightweight by Design**: This dependency orchestration guarantees that seeders remain extremely lightweight, hot-reloading takes milliseconds, and Docker container size stays optimized.
1. **Scaffold a Backend:** Use the CLI to generate a backend template.
```bash
npx @worldwideview/cli create-backend my-plugin-backend
```
2. **Containerize:** This generates a Fastify Node.js server with a `Dockerfile` and `docker-compose.yml`.
3. **Publish to Redis:** Your backend polls the external API (e.g., ADS-B Exchange) and pushes snapshots to the WorldWideView Redis pipeline.
4. **WebSocket Delivery:** The central `wwv-data-engine` immediately broadcasts these updates over WebSockets to all connected clients.
> [!TIP]
> **Debugging WebSockets:** If your frontend isn't receiving data from your backend seeder:
> 1. Check the `wwv-data-engine` logs to ensure your seeder is publishing to Redis successfully.
> 2. Verify the frontend is connected to the correct WebSocket endpoint. Local instances default to `ws://localhost:5000/stream`, while unrecognized seeders fall back to the cloud at `wss://dataengine.worldwideview.dev/stream`.
> 2. Verify the frontend is connected to the correct WebSocket endpoint. Local instances default to `ws://localhost:5001/stream`, while unrecognized seeders fall back to the cloud at `wss://dataengine.worldwideview.dev/stream`.
## Advanced Cesium Rendering
+22 -12
View File
@@ -14,29 +14,39 @@ Before you start, ensure you have:
## 2. Scaffold Your Plugin
WorldWideView plugins are built as standard packages within the monorepo workspace. To generate a new plugin sandbox, use the official CLI from the root of your `worldwideview` repository:
WorldWideView plugins are built as standard NPM packages. They are entirely decoupled from the core application, allowing you to develop and publish them independently.
To generate a new plugin template, use the official CLI:
```bash
node packages/wwv-cli/dist/index.js create my-first-plugin --local
pnpm install
npx @worldwideview/create-plugin@latest my-first-plugin
cd my-first-plugin
npm install
```
This creates a lightweight Vite-based project pre-configured with the `@worldwideview/wwv-plugin-sdk` inside the `local-plugins/wwv-plugin-my-first-plugin/` directory.
This creates a lightweight Vite-based project pre-configured with the `@worldwideview/wwv-plugin-sdk`.
## 3. Develop Your Plugin
## 3. Link Your Plugin to WorldWideView
Because your plugin is created inside the `local-plugins/` directory, it is automatically recognized by the pnpm workspace.
To see your plugin live on the globe while you develop, simply start the standard development servers from the repository root:
To see your plugin live on the globe while you develop, you need to link it to your local WorldWideView instance.
First, tell the CLI where your WorldWideView repository is located:
```bash
pnpm dev:all
npx wwv config set wwv-path C:\path\to\your\worldwideview
```
The `dev:all` command automatically starts the frontend, the data engine, and the `dev:plugins` watcher. This watcher will automatically rebuild your plugin whenever you save a file in `local-plugins/`.
Next, link the plugin. This creates a symlink so your local WorldWideView instance can dynamically load your code:
```bash
npm run link
```
Finally, start the plugin watcher. This will automatically rebuild your plugin whenever you save a file:
```bash
npm run dev
```
> [!TIP]
> **Debugging Registration Issues:** If your plugin doesn't appear in the WorldWideView "Installed Plugins" list, check your `package.json` inside your local plugin folder. Ensure `id`, `name`, and `version` are populated correctly. Also verify that you ran `pnpm install` from the workspace root.
> **Debugging Link Issues:** If your plugin doesn't appear in the WorldWideView "Installed Plugins" list after linking, check your `package.json`. The `"worldwideview"` object block is strictly required. Ensure `id`, `name`, and `version` are populated correctly.
## 4. Explore the Code
@@ -113,4 +123,4 @@ export class MyFirstPlugin implements WorldPlugin {
Congratulations! You've successfully built and linked your first plugin.
For advanced features—like real-time WebSocket streaming, creating backend seeders using the `local-seeders` sandbox, or publishing your plugin to the global Marketplace—proceed to the **[Advanced Plugin Guide](docs/plugin-advanced.md)**.
For advanced features—like real-time WebSocket streaming, containerizing your own backend microservices, or publishing your plugin to the global Marketplace—proceed to the **[Advanced Plugin Guide](docs/plugin-advanced.md)**.
+2 -2
View File
@@ -17,8 +17,8 @@ The core value proposition of WorldWideView is mapping diverse, decoupled intell
- **3D Engine:** CesiumJS + Resium (Google Photorealistic 3D Tiles) — `src/core/globe/GlobeView.tsx`
- **State Management:** Zustand — `src/core/state/` (slice-based store)
- **Event Bus:** Custom typed `DataBus` — `src/core/data/DataBus.ts`
- **Database:** PostgreSQL via Prisma 7 — `prisma/schema.prisma`
- **Styling:** Vanilla CSS (no Tailwind) — `src/app/globals.css` and `src/styles/theme-tokens.css`
- **Database:** SQLite via Prisma (local), PostgreSQL (cloud) — `prisma/schema.prisma`
- **Styling:** Vanilla CSS (no Tailwind) — `src/app/globals.css`
## Platform Support
Built as a highly portable Next.js container, it supports multiple deployment profiles controlled by `NEXT_PUBLIC_WWV_EDITION` (default: `local`).
@@ -0,0 +1,216 @@
# Plugin Architecture Migration Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Migrate the WorldWideView plugin architecture from compile-time hardcoded imports to a dynamic, runtime-loaded ecosystem where external developers can build, host, and distribute plugins without touching the core repository.
**Architecture:** We will introduce a bundler (`tsup`) to the SDK to compile `bundle` plugins into browser-ready ECMAScript Modules (ESM). We will expose shared dependencies (`react`, `cesium`, `lucide-react`) via a global `window.__WWV_SHARED__` object. Finally, we will refactor `AppShell.tsx` to dynamically load all plugins at runtime via `import(/* webpackIgnore: true */)`, removing the hardcoded imports, and we will update `loadPluginFromManifest.ts` to seamlessly handle the global dependency injection.
**Tech Stack:** Next.js, React, CesiumJS, tsup.
---
## User Review Required
> [!WARNING]
> **Breaking Change**: This migration will decouple your built-in plugins from `AppShell.tsx`. They will now act as true "external" plugins that are loaded at runtime. During the transition, if the build steps aren't executed, the plugins will temporarily stop appearing on the globe. We will mitigate this by ensuring the build step runs before `next dev`.
> [!IMPORTANT]
> **Static GeoJSON Files**: Since static `.geojson` files currently live in `public/data`, we will leave them there during phase 1 of this migration to ensure the app doesn't break, but we will point the plugin manifests to load them via absolute or absolute-relative URLs (`/data/...`). In phase 2, we will introduce a symlink script to move them out of `public` entirely.
## Open Questions
1. **Build Tool Choice**: I am proposing `tsup` because it requires zero configuration to output multiple formats, automatically externalizes dependencies, and natively transpiles TypeScript/React. Are you comfortable adding `tsup` to the `wwv-plugin-sdk` package?
2. **Local Serving**: To test "built-in" plugins using the dynamic loading method, we need the Next.js app to serve their compiled `plugin.js` bundles. My plan adds a quick copy script to the root `package.json` that copies their `dist` folders into `public/builtin-plugins/` so they can be fetched locally via HTTP. Does this sound good?
## Proposed Changes
### Task 1: Initialize Global Shared Dependencies
**Files:**
- Modify: `c:\dev\worldwideview\src\components\layout\AppShell.tsx`
- Create: `c:\dev\worldwideview\src\core\plugins\PluginGlobals.ts`
- [ ] **Step 1: Create the PluginGlobals initializer**
```typescript
import React from "react";
import ReactDOM from "react-dom";
import * as ReactDomServer from "react-dom/server";
import { Plane, Ship, Camera, Shield, Flame, Map, Swords, Atom, Landmark, Mountain, PlaneTakeoff, Anchor, Lamp, Rocket, SunMoon, Cable, Pickaxe, ShieldAlert, Crosshair, SatelliteDish, Hand, Radar, Scale } from "lucide-react";
import * as Cesium from "cesium";
import * as WWVPluginSDK from "@worldwideview/wwv-plugin-sdk";
export function initPluginGlobals() {
if (typeof window === "undefined") return;
// Create a registry of Lucide icons commonly used by plugins to prevent them from bundling the massive icon library
const lucideIcons = {
Plane, Ship, Camera, Shield, Flame, Map, Swords, Atom, Landmark, Mountain,
PlaneTakeoff, Anchor, Lamp, Rocket, SunMoon, Cable, Pickaxe, ShieldAlert,
Crosshair, SatelliteDish, Hand, Radar, Scale
};
(window as any).__WWV_SHARED__ = {
"react": React,
"react-dom": ReactDOM,
"react-dom/server": ReactDomServer,
"cesium": Cesium,
"@worldwideview/wwv-plugin-sdk": WWVPluginSDK,
"lucide-react": lucideIcons
};
}
```
- [ ] **Step 2: Inject globals in AppShell before anything loads**
In `AppShell.tsx`, import and call `initPluginGlobals()` at the top of the file, outside of any React components.
```tsx
import { initPluginGlobals } from "@/core/plugins/PluginGlobals";
import dynamic from "next/dynamic";
// Initialize globals immediately so they exist before any dynamic imports
if (typeof window !== "undefined") {
initPluginGlobals();
}
//... rest of the file
```
### Task 2: Configure SDK Build Tool (`tsup`)
**Files:**
- Modify: `c:\dev\worldwideview\packages\wwv-plugin-sdk\package.json`
- Create: `c:\dev\worldwideview\packages\wwv-plugin-sdk\bin\wwv-plugin-build.js`
- [ ] **Step 1: Add tsup dependency**
Run: `pnpm --filter @worldwideview/wwv-plugin-sdk add -D tsup`
- [ ] **Step 2: Create the wwv-plugin-build CLI wrapper**
Add this to `wwv-plugin-sdk/bin/wwv-plugin-build.js`:
```javascript
#!/usr/bin/env node
const { build } = require("tsup");
const path = require("path");
async function run() {
const cwd = process.cwd();
await build({
entry: ["src/index.ts"],
outDir: "dist",
format: ["esm"],
target: "es2022",
clean: true,
dts: false,
minify: false,
sourcemap: true,
// Mark all these as externals so they aren't bundled
external: ["react", "react-dom", "cesium", "@worldwideview/wwv-plugin-sdk", "lucide-react"],
// Provide a shim that converts bare imports into global lookups
esbuildPlugins: [{
name: "wwv-globals",
setup(build) {
build.onResolve({ filter: /^(react|react-dom|cesium|@worldwideview\/wwv-plugin-sdk|lucide-react)$/ }, args => {
return { path: args.path, namespace: "wwv-global" };
});
build.onLoad({ filter: /.*/, namespace: "wwv-global" }, args => {
let globalKey = args.path;
if (args.path === "lucide-react") return { contents: `module.exports = window.__WWV_SHARED__["lucide-react"];` };
return { contents: `module.exports = window.__WWV_SHARED__["${globalKey}"];` };
});
}
}]
});
}
run();
```
- [ ] **Step 3: Update SDK package.json to expose the bin**
```json
"bin": {
"wwv-plugin-build": "./bin/wwv-plugin-build.js"
},
```
### Task 3: Refactor Plugins to use Build Tool and Manifests
**Files:**
- Modify: `wwv-plugin-aviation/package.json`
- Modify: `wwv-plugin-aviation/plugin.json` (Create)
- [ ] **Step 1: Add build script to aviation plugin**
```json
"scripts": {
"build": "wwv-plugin-build"
},
```
- [ ] **Step 2: Create plugin.json for aviation**
```json
{
"id": "aviation",
"name": "Aviation",
"version": "1.0.7",
"type": "data-layer",
"format": "bundle",
"trust": "built-in",
"capabilities": ["data:own", "network:fetch"],
"category": "aviation",
"icon": "Plane",
"entry": "/builtin-plugins/aviation/index.mjs"
}
```
*(Repeat for all other built-in plugins)*
### Task 4: Dynamic Initial Loader
**Files:**
- Modify: `c:\dev\worldwideview\src\components\layout\AppShell.tsx`
- Modify: `c:\dev\worldwideview\package.json`
- [ ] **Step 1: Remove Hardcoded Plugins**
In `AppShell.tsx`, delete the `builtIns` array and `import` statements at the top.
- [ ] **Step 2: Fetch and load all plugins via manifest**
Modify `AppShell.tsx` `startPlatform`:
```typescript
// Fetch the unified list of ALL installed plugins (both built-in and dynamic)
const res = await fetch("/api/marketplace/load");
if (res.ok) {
const data = await res.json();
for (const manifest of data.manifests) {
if (disabledIds.has(manifest.id)) continue;
try {
await pluginManager.loadFromManifest(manifest);
initLayer(manifest.id);
} catch (e) {
console.error("Failed to load plugin", manifest.id, e);
}
}
}
```
- [ ] **Step 3: Add predev copy-script**
Modify `package.json` to copy compiled plugin `dist` folders to `public/builtin-plugins` so the Next.js server can serve them at the `/builtin-plugins/...` path.
## Verification Plan
### Automated Tests
- Run `pnpm --filter wwv-plugin-aviation build` to ensure `tsup` generates a valid ES Module with no bundled React.
- Verify `dist/index.mjs` contains `window.__WWV_SHARED__["react"]` lookups.
### Manual Verification
- Start the Next.js server.
- Open network tab in browser and verify `GET /builtin-plugins/aviation/index.mjs` succeeds.
- Verify the Aviation plugin appears on the globe and renders planes successfully, indicating that it successfully retrieved Cesium and React from the global shared context.
@@ -0,0 +1,475 @@
# Demo Google AdSense Panel Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add a Google AdSense vertical ad strip to the right side of the demo instance, with a dismissible banner explaining it's temporary to fund hosting.
**Architecture:** Three layers: (1) load the AdSense script conditionally in the root layout only when `isDemo` is true, (2) a reusable `AdUnit` client component that calls `adsbygoogle.push({})`, (3) a `DemoAdStrip` shell component that wraps the ad unit + a "why ads?" dismissible message, positioned as a fixed vertical strip on the right edge. CSP headers in `next.config.ts` must be updated to allow the AdSense domains.
**Tech Stack:** Next.js 16 (App Router, `next/script`), Google AdSense, vanilla CSS, Vitest
---
## File Structure
| File | Action | Responsibility |
|---|---|---|
| `src/app/layout.tsx` | MODIFY | Conditionally load AdSense `<Script>` for demo |
| `next.config.ts` | MODIFY | Add AdSense domains to CSP `script-src` and `frame-src` |
| `.env` | MODIFY | Add `NEXT_PUBLIC_ADSENSE_CLIENT_ID` placeholder |
| `src/components/ads/AdUnit.tsx` | CREATE | Generic reusable AdSense unit component |
| `src/components/ads/DemoAdStrip.tsx` | CREATE | Demo-only vertical strip: ad + "why ads?" message |
| `src/components/ads/DemoAdStrip.css` | CREATE | Styles for the vertical ad strip |
| `src/components/layout/AppShell.tsx` | MODIFY | Import and render `<DemoAdStrip />` |
| `src/types/adsense.d.ts` | CREATE | TypeScript ambient declaration for `window.adsbygoogle` |
---
> [!IMPORTANT]
> **Before starting:** You need a Google AdSense publisher ID (`ca-pub-XXXXXXXXXXXXXXXX`) and an ad slot ID. These go into `.env` as `NEXT_PUBLIC_ADSENSE_CLIENT_ID` and are passed to the ad unit component. You can get these from [Google AdSense](https://www.google.com/adsense/). Without them the ads will not render, but the component framework will still work.
---
### Task 1: TypeScript Ambient Declaration for AdSense
**Files:**
- Create: `src/types/adsense.d.ts`
- [ ] **Step 1: Create the ambient type declaration**
```ts
// src/types/adsense.d.ts
interface Window {
adsbygoogle: Array<Record<string, unknown>>;
}
```
- [ ] **Step 2: Commit**
```bash
git add src/types/adsense.d.ts
git commit -m "chore: add adsense ambient type declaration"
```
---
### Task 2: Environment Variable
**Files:**
- Modify: `.env`
- [ ] **Step 1: Add the AdSense client ID placeholder**
Add the following line to the end of `.env`:
```
# Google AdSense — only used on demo edition
NEXT_PUBLIC_ADSENSE_CLIENT_ID=
```
The actual value (`ca-pub-XXXXXXXXXXXXXXXX`) should be set in `.env.local` for local testing or in the deployment environment. The ad slot IDs will be hardcoded in the component since there's only one placement.
- [ ] **Step 2: Commit**
```bash
git add .env
git commit -m "chore: add NEXT_PUBLIC_ADSENSE_CLIENT_ID env var"
```
---
### Task 3: CSP Headers Update
**Files:**
- Modify: `next.config.ts:29-52`
- [ ] **Step 1: Update CSP in next.config.ts**
In the `headers()` function, modify the `script-src` and `frame-src` directives to allow AdSense domains.
```ts
// Change this line (line 39):
"script-src 'self' 'unsafe-eval' 'unsafe-inline' blob: https://analytics.worldwideview.dev https://va.vercel-scripts.com",
// To this:
"script-src 'self' 'unsafe-eval' 'unsafe-inline' blob: https://analytics.worldwideview.dev https://va.vercel-scripts.com https://pagead2.googlesyndication.com https://adservice.google.com https://www.googletagservices.com",
```
```ts
// Change this line (line 49):
"frame-src 'self' *.youtube.com *.youtube-nocookie.com *.twitch.tv *.vimeo.com *.webcamera.pl *.ivideon.com *.rtsp.me *.bnu.tv",
// To this:
"frame-src 'self' *.youtube.com *.youtube-nocookie.com *.twitch.tv *.vimeo.com *.webcamera.pl *.ivideon.com *.rtsp.me *.bnu.tv https://googleads.g.doubleclick.net https://tpc.googlesyndication.com",
```
- [ ] **Step 2: Commit**
```bash
git add next.config.ts
git commit -m "feat: add Google AdSense domains to CSP headers"
```
---
### Task 4: AdSense Script in Root Layout
**Files:**
- Modify: `src/app/layout.tsx:24-39`
- [ ] **Step 1: Conditionally load AdSense script for demo edition**
Add the AdSense `<Script>` tag inside `<head>`, using the `NEXT_PUBLIC_` env var for the client ID and the `NEXT_PUBLIC_WWV_EDITION` env var to gate loading.
```tsx
// src/app/layout.tsx — updated <head> section
<head>
{/* Load CesiumJS base styles (optional, but helps with UI widgets if used later) */}
<link rel="stylesheet" href="/cesium/Widgets/widgets.css" />
</head>
<body suppressHydrationWarning>
{children}
<Analytics />
<Script
src="https://analytics.worldwideview.dev/script.js"
data-website-id="2c8f6c09-2651-4a2a-af99-b8cee1612b9a"
strategy="afterInteractive"
/>
{process.env.NEXT_PUBLIC_WWV_EDITION === "demo" &&
process.env.NEXT_PUBLIC_ADSENSE_CLIENT_ID && (
<Script
src={`https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=${process.env.NEXT_PUBLIC_ADSENSE_CLIENT_ID}`}
crossOrigin="anonymous"
strategy="afterInteractive"
/>
)}
</body>
```
> **Note:** We use the raw env var at the layout level (Server Component) rather than importing `isDemo` which uses `process.env.NEXT_PUBLIC_WWV_EDITION` anyway. This avoids importing the edition module into the root layout.
- [ ] **Step 2: Commit**
```bash
git add src/app/layout.tsx
git commit -m "feat: conditionally load AdSense script on demo edition"
```
---
### Task 5: Reusable AdUnit Component
**Files:**
- Create: `src/components/ads/AdUnit.tsx`
- [ ] **Step 1: Create the generic AdSense unit component**
```tsx
// src/components/ads/AdUnit.tsx
"use client";
import { useEffect, useRef } from "react";
interface AdUnitProps {
adSlot: string;
adFormat?: string;
style?: React.CSSProperties;
className?: string;
}
export function AdUnit({ adSlot, adFormat = "auto", style, className }: AdUnitProps) {
const pushed = useRef(false);
const clientId = process.env.NEXT_PUBLIC_ADSENSE_CLIENT_ID;
useEffect(() => {
if (pushed.current || !clientId) return;
try {
(window.adsbygoogle = window.adsbygoogle || []).push({});
pushed.current = true;
} catch (err) {
console.error("[AdUnit] AdSense push error:", err);
}
}, [clientId]);
if (!clientId) return null;
return (
<div className={className} style={style}>
<ins
className="adsbygoogle"
style={{ display: "block" }}
data-ad-client={clientId}
data-ad-slot={adSlot}
data-ad-format={adFormat}
data-full-width-responsive="false"
/>
</div>
);
}
```
> **Key detail:** The `pushed` ref prevents double-push on React strict mode / hot reload. `data-full-width-responsive="false"` prevents the ad from expanding beyond the container — critical for a fixed-width vertical strip.
- [ ] **Step 2: Commit**
```bash
git add src/components/ads/AdUnit.tsx
git commit -m "feat: add reusable AdUnit component"
```
---
### Task 6: DemoAdStrip Component + Styles
**Files:**
- Create: `src/components/ads/DemoAdStrip.tsx`
- Create: `src/components/ads/DemoAdStrip.css`
- [ ] **Step 1: Create the CSS for the vertical ad strip**
```css
/* src/components/ads/DemoAdStrip.css */
.demo-ad-strip {
position: fixed;
right: 0;
top: 0;
bottom: 0;
width: 160px;
z-index: 45;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
gap: var(--space-md);
padding: var(--space-sm);
pointer-events: none;
transition: transform var(--duration-slow) var(--ease-smooth),
opacity var(--duration-slow) var(--ease-smooth);
}
.demo-ad-strip--hidden {
transform: translateX(200px);
opacity: 0;
}
.demo-ad-strip__ad {
pointer-events: auto;
width: 160px;
min-height: 600px;
}
.demo-ad-strip__banner {
pointer-events: auto;
display: flex;
flex-direction: column;
align-items: center;
gap: var(--space-xs);
padding: var(--space-sm) var(--space-md);
background: var(--bg-glass);
backdrop-filter: blur(var(--glass-blur));
-webkit-backdrop-filter: blur(var(--glass-blur));
border: 1px solid var(--border-subtle);
border-radius: var(--radius-md);
max-width: 160px;
text-align: center;
cursor: pointer;
transition: all var(--duration-fast) var(--ease-smooth);
}
.demo-ad-strip__banner:hover {
border-color: rgba(var(--accent-rgb), 0.3);
box-shadow: var(--glow-cyan);
}
.demo-ad-strip__banner-label {
font-size: 9px;
font-weight: 700;
letter-spacing: 0.12em;
text-transform: uppercase;
color: var(--text-muted);
}
.demo-ad-strip__banner-text {
font-size: 11px;
line-height: 1.4;
color: var(--text-secondary);
}
.demo-ad-strip__close {
position: absolute;
top: var(--space-xs);
right: var(--space-xs);
width: 20px;
height: 20px;
border-radius: 50%;
background: transparent;
border: none;
color: var(--text-muted);
font-size: 12px;
cursor: pointer;
display: flex;
align-items: center;
justify-content: center;
transition: color var(--duration-fast) var(--ease-smooth);
}
.demo-ad-strip__close:hover {
color: var(--text-primary);
}
/* Hide on mobile — ads shouldn't overlay the full-screen globe */
@media (max-width: 768px) {
.demo-ad-strip {
display: none;
}
}
```
- [ ] **Step 2: Create the DemoAdStrip component**
```tsx
// src/components/ads/DemoAdStrip.tsx
"use client";
import { useState } from "react";
import { isDemo } from "@/core/edition";
import { AdUnit } from "./AdUnit";
import { X } from "lucide-react";
import "./DemoAdStrip.css";
/** The ad slot ID for the vertical skyscraper unit — set this in AdSense dashboard. */
const VERTICAL_AD_SLOT = "REPLACE_WITH_YOUR_AD_SLOT_ID";
export function DemoAdStrip() {
const [showMessage, setShowMessage] = useState(false);
const [closed, setClosed] = useState(false);
if (!isDemo || closed) return null;
return (
<aside className="demo-ad-strip" aria-label="Sponsored content">
<AdUnit
adSlot={VERTICAL_AD_SLOT}
adFormat="vertical"
className="demo-ad-strip__ad"
/>
<div
className="demo-ad-strip__banner"
onClick={() => setShowMessage((prev) => !prev)}
role="button"
tabIndex={0}
onKeyDown={(e) => { if (e.key === "Enter") setShowMessage((prev) => !prev); }}
>
<button
className="demo-ad-strip__close"
onClick={(e) => { e.stopPropagation(); setClosed(true); }}
aria-label="Dismiss ad panel"
>
<X size={12} />
</button>
<span className="demo-ad-strip__banner-label">Why ads?</span>
{showMessage ? (
<span className="demo-ad-strip__banner-text">
This isn&apos;t permanent — it&apos;s temporary because I&apos;m a single
dev and need to cover hosting costs. Thank you for understanding.
</span>
) : (
<span className="demo-ad-strip__banner-text">
Click to learn more
</span>
)}
</div>
</aside>
);
}
```
- [ ] **Step 3: Commit**
```bash
git add src/components/ads/DemoAdStrip.tsx src/components/ads/DemoAdStrip.css
git commit -m "feat: add DemoAdStrip with Google AdSense vertical unit"
```
---
### Task 7: Integrate into AppShell
**Files:**
- Modify: `src/components/layout/AppShell.tsx:8-9,168`
- [ ] **Step 1: Add import**
After the existing `CameraStatsPanel` import (line 8), add:
```tsx
import CameraStatsPanel from "@/components/panels/CameraStatsPanel";
import { DemoAdStrip } from "@/components/ads/DemoAdStrip"; // ADD THIS
import { Timeline } from "@/components/timeline/Timeline";
```
- [ ] **Step 2: Add the JSX**
After `<EntityInfoCard />` (line 168), add:
```tsx
<EntityInfoCard />
<DemoAdStrip />
<Timeline />
```
- [ ] **Step 3: Commit**
```bash
git add src/components/layout/AppShell.tsx
git commit -m "feat: integrate DemoAdStrip into AppShell"
```
---
### Task 8: ads.txt for AdSense Verification
**Files:**
- Create: `public/ads.txt`
- [ ] **Step 1: Create ads.txt**
```
google.com, ca-pub-XXXXXXXXXXXXXXXX, DIRECT, f08c47fec0942fa0
```
> Replace `ca-pub-XXXXXXXXXXXXXXXX` with your actual publisher ID. This file is required by AdSense for domain verification.
- [ ] **Step 2: Commit**
```bash
git add public/ads.txt
git commit -m "chore: add ads.txt for AdSense domain verification"
```
---
## Open Questions
> [!IMPORTANT]
> **Do you have a Google AdSense publisher ID yet?** If not, you'll need to sign up at https://www.google.com/adsense/ and get your `ca-pub-XXXX` ID and create a 160x600 "Wide Skyscraper" ad unit to get the slot ID. The code will render nothing gracefully until these are configured.
> [!WARNING]
> **CSP changes are required for AdSense to work.** The current CSP is restrictive and blocks `pagead2.googlesyndication.com` and `googleads.g.doubleclick.net`. Task 3 adds them. Without this, ads will be silently blocked by the browser.
---
## Verification Plan
### Manual Verification
1. Set `NEXT_PUBLIC_WWV_EDITION=demo` and `NEXT_PUBLIC_ADSENSE_CLIENT_ID=ca-pub-XXXX` in `.env.local`
2. Run `pnpm dev`
3. Verify: A 160px vertical strip appears on the right edge of the screen
4. Verify: The "Why ads?" banner is clickable and toggles the explanation message
5. Verify: Clicking the X dismisses the entire strip
6. Verify: On mobile viewport widths (< 768px), the strip is hidden
7. Verify: Set `NEXT_PUBLIC_WWV_EDITION=local` → strip does not render at all
8. Verify: Browser console shows no CSP violations related to AdSense domains
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,145 @@
# Expand OSM Chips Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Expand the available OSM search chips to include a curated list of ~30 pre-loaded OSINT-relevant tags, and add a live Taginfo API autocomplete search to dynamically find and select any of the millions of other OSM tags.
**Architecture:** We will replace the current small `COMMON_TAGS` array with a richer, categorized list inside the `OSMSidebar` file. To allow access to "all" tags without crashing the UI, we will add a debounced fetch hook that hits `https://taginfo.openstreetmap.org/api/4/tags/popular` (or the equivalent search endpoint) when the user types in the filter box. These dynamically fetched tags will temporarily populate the chip list.
**Tech Stack:** React, TypeScript, Taginfo API
---
### Task 1: Update Pre-Loaded COMMON_TAGS
**Files:**
- Modify: `c:\dev\worldwideview\packages\wwv-plugin-osm-search\src\components\OSMSidebar.tsx:6-11`
- [ ] **Step 1: Replace existing COMMON_TAGS array**
```tsx
const COMMON_TAGS = [
// Military & Security
"military=base", "military=bunker", "amenity=police", "amenity=prison",
// Aviation & Maritime
"aeroway=aerodrome", "aeroway=helipad", "aeroway=hangar", "man_made=pier", "harbour=yes",
// Infrastructure & Utilities
"power=plant", "power=substation", "telecom=antenna", "man_made=communications_tower", "man_made=water_tower",
// Transport
"highway=bus_stop", "railway=station", "amenity=fuel", "amenity=parking",
// Medical & Emergency
"amenity=hospital", "amenity=fire_station", "amenity=clinic",
// General Commercial & Industrial
"shop=supermarket", "tourism=hotel", "industrial=factory", "landuse=industrial",
// Education & Amenities
"amenity=school", "amenity=university", "amenity=cafe", "building=house"
];
```
- [ ] **Step 2: Commit**
```bash
git add packages/wwv-plugin-osm-search/src/components/OSMSidebar.tsx
git commit -m "feat(osm-search): expand quick preset tags with OSINT categories"
```
### Task 2: Implement Taginfo Dynamic Autocomplete
**Files:**
- Modify: `c:\dev\worldwideview\packages\wwv-plugin-osm-search\src\components\OSMSidebar.tsx`
- [ ] **Step 1: Add state and debounced fetch logic inside OSMSidebar component**
```tsx
const [searchText, setSearchText] = useState("");
const [activeTags, setActiveTags] = useState<string[]>([]);
const [dynamicTags, setDynamicTags] = useState<string[]>([]);
const [isSearchingApi, setIsSearchingApi] = useState(false);
// Add this useEffect below the useState hooks
React.useEffect(() => {
if (searchText.length < 3) {
setDynamicTags([]);
return;
}
const timer = setTimeout(async () => {
setIsSearchingApi(true);
try {
// Fetch tags matching the search term from OSM Taginfo
const res = await fetch(`https://taginfo.openstreetmap.org/api/4/search/by_keyword?query=${encodeURIComponent(searchText)}`);
const data = await res.json();
if (data && data.data) {
// Extract exact key=value pairs or just keys, format as "key=value"
const tags = data.data
.filter((item: any) => item.key && item.value)
.map((item: any) => `${item.key}=${item.value}`)
.slice(0, 15); // Limit to top 15 results
setDynamicTags(tags);
}
} catch (err) {
console.error("Taginfo fetch failed", err);
} finally {
setIsSearchingApi(false);
}
}, 500); // 500ms debounce
return () => clearTimeout(timer);
}, [searchText]);
```
- [ ] **Step 2: Update rendering logic to merge static and dynamic tags**
Look for the `renderedTags` definition and replace it:
```tsx
// Combine filtered common tags with newly fetched dynamic tags, removing duplicates
const filteredCommon = searchText
? COMMON_TAGS.filter(t => t.toLowerCase().includes(searchText.toLowerCase()))
: COMMON_TAGS;
const renderedTags = Array.from(new Set([...filteredCommon, ...dynamicTags]));
```
- [ ] **Step 3: Add loading indicator to the search input area**
Look for the `<input>` element for `searchText` and wrap it/add a loading indicator:
```tsx
<div style={{ position: "relative" }}>
<input
style={{
width: "100%",
padding: "8px",
paddingRight: "24px",
backgroundColor: "rgba(0,0,0,0.3)",
color: "#fff",
border: "1px solid var(--border-subtle)",
borderRadius: "4px",
fontSize: "13px"
}}
placeholder="Filter or search OSM for tags..."
value={searchText}
onChange={e => setSearchText(e.target.value)}
/>
{isSearchingApi && (
<span style={{
position: "absolute",
right: "8px",
top: "50%",
transform: "translateY(-50%)",
fontSize: "10px",
color: "var(--text-muted)"
}}>
⏳
</span>
)}
</div>
```
- [ ] **Step 4: Commit**
```bash
git add packages/wwv-plugin-osm-search/src/components/OSMSidebar.tsx
git commit -m "feat(osm-search): add live Taginfo autocomplete for unlisted tags"
```
@@ -0,0 +1,139 @@
# Plugin Versioning Streamlining Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Refactor all WorldWideView plugin classes to derive their `version` property dynamically from their local `package.json` file rather than hardcoding string literals.
**Architecture:** We will replace the hardcoded `version = "x.y.z"` strings in the core `WorldPlugin` interface implementations across all monorepo packages. We will utilize TypeScript's JSON module resolution feature to `import pkg from "../package.json"` and map `version = pkg.version`. This ensures that `pnpm version patch` is the single source of truth and eliminates the double-bookkeeping that previously made CI fixes tedious.
**Tech Stack:** TypeScript, Next.js / bundler (JSON module resolution), pnpm workspaces.
---
## Task 1: Verify TypeScript Configuration
**Files:**
- Modify: N/A (Verification step)
- [ ] **Step 1: Check `tsconfig.json` for JSON resolution**
Run: `cat tsconfig.json | grep resolveJsonModule`
Expected: `"resolveJsonModule": true`
- [ ] **Step 2: Check compiler compatibility**
Run: `pnpm exec tsc --noEmit`
Expected: PASS (exit code 0)
## Task 2: Refactor Group A Plugins (Core Infrastructure)
**Files:**
- Modify: `packages/wwv-plugin-aviation/src/index.ts`
- Modify: `packages/wwv-plugin-maritime/src/index.ts`
- Modify: `packages/wwv-plugin-military-aviation/src/index.ts`
- [ ] **Step 1: Update plugin classes**
For each file above, modify the imports and the class body:
```typescript
import pkg from "../package.json";
// ...
export class AviationPlugin implements WorldPlugin { // (or extends BasePlugin)
// ...
version = pkg.version;
```
- [ ] **Step 2: Verify compilation**
Run: `pnpm exec tsc --noEmit`
Expected: PASS
- [ ] **Step 3: Commit**
```bash
git commit -am "refactor(plugins): derive version dynamically for core infrastructure plugins [Patch]"
```
## Task 3: Refactor Group B Plugins (Facilities & Infrastructure)
**Files:**
- Modify: `packages/wwv-plugin-airports/src/index.ts`
- Modify: `packages/wwv-plugin-seaports/src/index.ts`
- Modify: `packages/wwv-plugin-spaceports/src/index.ts`
- Modify: `packages/wwv-plugin-embassies/src/index.ts`
- Modify: `packages/wwv-plugin-lighthouses/src/index.ts`
- Modify: `packages/wwv-plugin-nuclear/src/index.ts`
- Modify: `packages/wwv-plugin-undersea-cables/src/index.ts`
- Modify: `packages/wwv-plugin-mineral-mines/src/index.ts`
- [ ] **Step 1: Update plugin classes**
For each file above, replace `version = "..."` with `version = pkg.version` and add `import pkg from "../package.json";`.
- [ ] **Step 2: Verify compilation**
Run: `pnpm exec tsc --noEmit`
Expected: PASS
- [ ] **Step 3: Commit**
```bash
git commit -am "refactor(plugins): derive version dynamically for facility plugins [Patch]"
```
## Task 4: Refactor Group C Plugins (Hazards & Events)
**Files:**
- Modify: `packages/wwv-plugin-earthquakes/src/index.ts`
- Modify: `packages/wwv-plugin-volcanoes/src/index.ts`
- Modify: `packages/wwv-plugin-wildfire/src/index.ts`
- Modify: `packages/wwv-plugin-conflict-events/src/index.tsx`
- Modify: `packages/wwv-plugin-conflict-zones/src/index.tsx`
- Modify: `packages/wwv-plugin-civil-unrest/src/index.tsx`
- Modify: `packages/wwv-plugin-cyber-attacks/src/index.tsx`
- Modify: `packages/wwv-plugin-international-sanctions/src/index.tsx`
- [ ] **Step 1: Update plugin classes**
For each file above, replace `version = "..."` with `version = pkg.version` and add `import pkg from "../package.json";`. Note that some files are `.tsx`.
- [ ] **Step 2: Verify compilation**
Run: `pnpm exec tsc --noEmit`
Expected: PASS
- [ ] **Step 3: Commit**
```bash
git commit -am "refactor(plugins): derive version dynamically for event and hazard plugins [Patch]"
```
## Task 5: Refactor Group D Plugins (Misc & Utilities)
**Files:**
- Modify: `packages/wwv-plugin-satellite/src/index.ts`
- Modify: `packages/wwv-plugin-surveillance-satellites/src/index.tsx`
- Modify: `packages/wwv-plugin-osm-search/src/index.ts`
- Modify: `packages/wwv-plugin-iranwarlive/src/index.ts`
- Modify: `packages/wwv-plugin-daynight/src/index.ts`
- Modify: `packages/wwv-plugin-camera/src/index.ts`
- Modify: `packages/wwv-plugin-borders/src/index.ts`
- Modify: `packages/wwv-plugin-gps-jamming/src/index.tsx`
- Modify: `packages/wwv-plugin-air-defense/src/index.tsx`
- [ ] **Step 1: Update plugin classes**
For each file above, replace `version = "..."` with `version = pkg.version` and add `import pkg from "../package.json";`.
- [ ] **Step 2: Verify compilation and tests**
Run: `pnpm exec tsc --noEmit`
Run: `pnpm test`
Expected: PASS
- [ ] **Step 3: Commit**
```bash
git commit -am "refactor(plugins): derive version dynamically for utility plugins [Patch]"
```
@@ -0,0 +1,124 @@
# Demo Default Plugins Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Enable specific plugins by default on the demo edition via environment variables so the user is immediately impressed.
**Architecture:** Update `layersSlice.ts` to accept an optional `defaultEnabled` parameter in `initLayer`. In `AppShell.tsx`, look up `NEXT_PUBLIC_DEMO_DEFAULT_PLUGINS` when the edition is `demo`, and pass the flag true to `initLayer` for matching plugins during registration. Document the variable in `.env.example`.
**Tech Stack:** Next.js, Zustand, TypeScript.
---
### Task 1: Update layer slice definition
**Files:**
- Modify: `src/core/state/layersSlice.ts`
- [ ] **Step 1: Write the implementation**
Update `initLayer` in `src/core/state/layersSlice.ts` to accept a second parameter:
```typescript
// Add defaultEnabled parameter to the interface
export interface LayersSlice {
// ...
initLayer: (pluginId: string, defaultEnabled?: boolean) => void;
}
// ...
// Use the parameter in the state updater
initLayer: (pluginId, defaultEnabled = false) =>
set((state) => ({
layers: {
...state.layers,
[pluginId]: state.layers[pluginId] || { enabled: defaultEnabled, entityCount: 0, loading: false },
},
})),
```
- [ ] **Step 2: Commit**
```bash
git add src/core/state/layersSlice.ts
npm run commit -- "feat: add defaultEnabled param to initLayer in layers slice"
```
### Task 2: Apply default configuration in AppShell
**Files:**
- Modify: `src/components/layout/AppShell.tsx`
- [ ] **Step 1: Write the implementation**
Add `isDemo` import at the top of `src/components/layout/AppShell.tsx`:
```typescript
import { isDemo } from "@/core/edition";
```
In `AppShell.tsx`, around line 71, inside `startPlatform()`, fetch the value from the environment if the instance is demo:
```typescript
// Fetch disabled built-in plugins before registration
let disabledIds = new Set<string>();
// ... (keep existing try/catch block)
// Setup demo defaults
const demoDefaultPlugins = new Set<string>();
if (isDemo) {
const envVar = process.env.NEXT_PUBLIC_DEMO_DEFAULT_PLUGINS || "";
envVar.split(",").forEach(s => {
const clean = s.trim();
if (clean) demoDefaultPlugins.add(clean);
});
}
```
Then update the `pluginManager` registration loop to inject the boolean for `initLayer`:
```typescript
for (const plugin of pluginRegistry.getAll()) {
await pluginManager.registerPlugin(plugin);
initLayer(plugin.id, demoDefaultPlugins.has(plugin.id));
}
```
- [ ] **Step 2: Run test to verify it passes**
Run type checks to verify no compilation issues:
`npx tsc --noEmit`
- [ ] **Step 3: Commit**
```bash
git add src/components/layout/AppShell.tsx
npm run commit -- "feat: parse and apply default demo plugins via NEXT_PUBLIC_DEMO_DEFAULT_PLUGINS"
```
### Task 3: Document environment definitions
**Files:**
- Modify: `.env.example`
- Modify: `.env`
- [ ] **Step 1: Write the implementation**
Update `.env.example` and `.env` around the `NEXT_PUBLIC_WWV_EDITION=local` definition:
```dotenv
# Default plugins enabled on boot for demo edition
# Comma-separated list of plugin IDs (e.g. aviation,maritime,iranwarlive)
NEXT_PUBLIC_DEMO_DEFAULT_PLUGINS=
# Edition: "local" | "cloud" | "demo"
NEXT_PUBLIC_WWV_EDITION=local
```
- [ ] **Step 2: Commit**
```bash
git add .env.example .env
npm run commit -- "docs: add NEXT_PUBLIC_DEMO_DEFAULT_PLUGINS to env templates"
```
Loaded 100 of 706 files, more files were not shown because too many files have changed in this diff. Show more