Compare commits

...
Author SHA1 Message Date
Antigravity AI 433004598d Merge branch 'fix/lint-cleanups'
# Conflicts:
#	eslint.config.mjs
#	package.json
2026-05-17 22:08:13 +12:00
Antigravity AI 02d8355a83 feat: add inverted app icon and update lockfile dependencies 2026-05-17 22:05:48 +12:00
Silvertakana 89010b2425 feat(ui): Bottom Panel System and E2E Testing (#147)
* test(bottom-panel): add e2e resize testing and data-testid

* test(e2e): fix playwright drag events skipping React state
2026-05-17 20:14:18 +12:00
Antigravity AI 0ff1d23e4a chore(lint): clean up legacy linting errors and hallucinated rules 2026-05-17 20:04:13 +12:00
Silvertakanaandsnyk-bot fd29865e7a [Snyk] Security upgrade ws from 8.20.0 to 8.20.1 (#151)
* fix: package.json to reduce vulnerabilities

The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JS-WS-16722635

* fix: update pnpm-lock.yaml for ws dependency bump and bump version to 2.15.11

---------

Co-authored-by: snyk-bot <[email protected]>
2026-05-17 19:54:03 +12:00
Silvertakana 42564b537f ci(playwright): Add multi-browser test matrix (#150)
* ci(playwright): Add multi-browser test matrix (Patch)

* fix(cesium): Disable MSAA in Firefox to prevent readPixels crash (Patch)
2026-05-17 19:14:54 +12:00
Antigravity AI 598e00171c fix(cesium): Disable MSAA in Firefox to prevent readPixels crash (Patch) 2026-05-17 18:35:09 +12:00
devangpratap b849bb2b4d feat(api): add ISS Tracker proxy routes (#143)
* feat(api): add ISS position proxy route

Server-side proxy to the Where the ISS at? API (NORAD 25544).
Returns current latitude, longitude, altitude, velocity, visibility,
and footprint with 10-second revalidation and input validation.

Closes #8

* test(iss): add unit tests for ISS position route

Covers: valid response, field stripping, upstream failure,
invalid coordinates, and network error scenarios.

* feat(api): add ISS TLE orbit data route

Exposes the latest Two-Line Element set for NORAD 25544 via
/api/iss/tle. TLE data enables client-side orbit prediction
with satellite.js. Cached for 1 hour since TLEs update infrequently.

* test(iss): add unit tests for ISS TLE route

Covers: valid TLE response, field normalization (snake_case to
camelCase), upstream failure, empty TLE lines, and network errors.

* feat(api): add ISS historical positions route with tests

Batch endpoint at /api/iss/positions?timestamps=t1,t2,... returns
multiple past positions for orbit trail rendering. Validates input
(max 10 timestamps, numeric only), filters invalid upstream data,
and strips extraneous fields. Includes 7 test cases covering
happy path, validation, overflow, and upstream failure.

* fix(iss): harden validation and add missing edge case tests

- Remove unused TypeScript interfaces from route validators,
  align with earthquake route's bare validation style
- Add velocity validation to positions route (was missing,
  inconsistent with main route)
- Add null body tests for position and TLE routes
- Add empty string timestamps test for positions route
- Add NaN velocity rejection test for positions route
- Add network error test for positions route (was missing)
2026-05-17 17:36:51 +12:00
SilvertakanaandCopilot Autofix powered by AI fc7dd02ee2 refactor(core): harden typescript and fix linting violations (#146)
* chore(merge): resolve conflicts, fix mockWs error, and apply refactor standards

* Potential fix for pull request finding 'CodeQL / Incomplete URL substring sanitization'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Potential fix for pull request finding 'CodeQL / Incomplete URL substring sanitization'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Potential fix for pull request finding 'CodeQL / Incomplete URL substring sanitization'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix(ci): Stabilize github actions and resolve test failures

- fix(tests): update WsClient mock to use standard function for Vitest constructor compatibility

- fix(db): correct PrismaClient import path to use generated output directory

- fix(e2e): add predev database boot step to playwright CI workflow

- chore: sync pnpm-lock.yaml after merge conflict resolution

* refactor(plugins): decouple GeoJSON plugin and implement data bus lifecycle

- feat(sdk): [Minor] Added dynamicPluginCreate, dynamicPluginRemove, and pluginUnregistered to DataBusEvents.

- feat(core): [Minor] Implemented unregisterPlugin in PluginManager.ts for automated IndexedDB caching cleanup and polling teardown.

- feat(core): [Minor] Added removeLayer to layersSlice.ts to support safe removal of plugin states.

- refactor(plugins): [Patch] Refactored useGeoJsonImport.ts and ImportedLayerList.tsx to decouple from direct manager imports, utilizing DataBus for lifecycle events.

- refactor(core): [Patch] Added pluginUnregistered listener to DataBusSubscriber.tsx to handle state and cache cleanup synchronously via event.

* fix(types): resolve PluginManager type errors

* refactor(core): decouple PluginManager and fix rendering invariants

- refactor(core): Replace Zustand mutations in PluginManager with DataBus events (Patch)
- refactor(sdk): Add layerLoadingChanged, pluginError, pluginUnregistered to DataBusEvents (Patch)
- refactor(ui): Subscribe to new DataBus events in DataBusSubscriber (Patch)
- fix(plugins): Remove size from BaseAviationPlugin to fix model rendering clipping (Patch)
- fix(plugins): Migrate BaseIncidentPlugin from size to iconScale to fix billboard clipping (Patch)

* chore: resolve lockfile conflict

* fix: restore BottomPanelManager in AppShell and fix vitest config

* fix(layout): restore bottom panel positioning and transition styling

This commit corrects the missing bottom panel issue by resolving CSS aggregation errors and fixing transition logic. The version is bumped to 2.15.5.

* refactor(core): harden typescript and fix linting violations

This commit addresses the remaining any types and linting errors across the core engine.

* fix(eslint): resolve react-hook immutability and cascading render violations

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-05-17 17:36:33 +12:00
Antigravity AI caf1638fdc ci(playwright): Add multi-browser test matrix (Patch) 2026-05-17 15:51:48 +12:00
SilvertakanaandCopilot Autofix powered by AI ad3c7845a5 fix: restore bottom panel system and resolve E2E test failure (#145)
* chore(merge): resolve conflicts, fix mockWs error, and apply refactor standards

* Potential fix for pull request finding 'CodeQL / Incomplete URL substring sanitization'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Potential fix for pull request finding 'CodeQL / Incomplete URL substring sanitization'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Potential fix for pull request finding 'CodeQL / Incomplete URL substring sanitization'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix(ci): Stabilize github actions and resolve test failures

- fix(tests): update WsClient mock to use standard function for Vitest constructor compatibility

- fix(db): correct PrismaClient import path to use generated output directory

- fix(e2e): add predev database boot step to playwright CI workflow

- chore: sync pnpm-lock.yaml after merge conflict resolution

* refactor(plugins): decouple GeoJSON plugin and implement data bus lifecycle

- feat(sdk): [Minor] Added dynamicPluginCreate, dynamicPluginRemove, and pluginUnregistered to DataBusEvents.

- feat(core): [Minor] Implemented unregisterPlugin in PluginManager.ts for automated IndexedDB caching cleanup and polling teardown.

- feat(core): [Minor] Added removeLayer to layersSlice.ts to support safe removal of plugin states.

- refactor(plugins): [Patch] Refactored useGeoJsonImport.ts and ImportedLayerList.tsx to decouple from direct manager imports, utilizing DataBus for lifecycle events.

- refactor(core): [Patch] Added pluginUnregistered listener to DataBusSubscriber.tsx to handle state and cache cleanup synchronously via event.

* fix(types): resolve PluginManager type errors

* refactor(core): decouple PluginManager and fix rendering invariants

- refactor(core): Replace Zustand mutations in PluginManager with DataBus events (Patch)
- refactor(sdk): Add layerLoadingChanged, pluginError, pluginUnregistered to DataBusEvents (Patch)
- refactor(ui): Subscribe to new DataBus events in DataBusSubscriber (Patch)
- fix(plugins): Remove size from BaseAviationPlugin to fix model rendering clipping (Patch)
- fix(plugins): Migrate BaseIncidentPlugin from size to iconScale to fix billboard clipping (Patch)

* chore: resolve lockfile conflict

* fix: restore BottomPanelManager in AppShell and fix vitest config

* fix(layout): restore bottom panel positioning and transition styling

This commit corrects the missing bottom panel issue by resolving CSS aggregation errors and fixing transition logic. The version is bumped to 2.15.5.

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-05-16 15:33:43 +12:00
Silvertakana 339bf6c0f3 fix(plugins): ensure plugin fetch uses HTTP scheme (#140)
* fix(plugin-manager): downgrade non-fatal 'Failed to fetch' errors to warn, remove polling re-throw

* fix(codeql): resolve format string vulnerability in console logging

* fix(ci): restore playwright dependencies and configure tsconfig properly

* fix(plugins): ensure plugin fetch uses HTTP scheme instead of WSS
2026-05-16 13:03:50 +12:00
SilvertakanaandCopilot Autofix powered by AI 16c82c9184 refactor(lint): address React 19 hook purity and immutability violations (#136)
* chore(merge): resolve conflicts, fix mockWs error, and apply refactor standards

* Potential fix for pull request finding 'CodeQL / Incomplete URL substring sanitization'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Potential fix for pull request finding 'CodeQL / Incomplete URL substring sanitization'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Potential fix for pull request finding 'CodeQL / Incomplete URL substring sanitization'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix(ci): Stabilize github actions and resolve test failures

- fix(tests): update WsClient mock to use standard function for Vitest constructor compatibility

- fix(db): correct PrismaClient import path to use generated output directory

- fix(e2e): add predev database boot step to playwright CI workflow

- chore: sync pnpm-lock.yaml after merge conflict resolution

* refactor(plugins): decouple GeoJSON plugin and implement data bus lifecycle

- feat(sdk): [Minor] Added dynamicPluginCreate, dynamicPluginRemove, and pluginUnregistered to DataBusEvents.

- feat(core): [Minor] Implemented unregisterPlugin in PluginManager.ts for automated IndexedDB caching cleanup and polling teardown.

- feat(core): [Minor] Added removeLayer to layersSlice.ts to support safe removal of plugin states.

- refactor(plugins): [Patch] Refactored useGeoJsonImport.ts and ImportedLayerList.tsx to decouple from direct manager imports, utilizing DataBus for lifecycle events.

- refactor(core): [Patch] Added pluginUnregistered listener to DataBusSubscriber.tsx to handle state and cache cleanup synchronously via event.

* fix(types): resolve PluginManager type errors

* refactor(core): decouple PluginManager and fix rendering invariants

- refactor(core): Replace Zustand mutations in PluginManager with DataBus events (Patch)
- refactor(sdk): Add layerLoadingChanged, pluginError, pluginUnregistered to DataBusEvents (Patch)
- refactor(ui): Subscribe to new DataBus events in DataBusSubscriber (Patch)
- fix(plugins): Remove size from BaseAviationPlugin to fix model rendering clipping (Patch)
- fix(plugins): Migrate BaseIncidentPlugin from size to iconScale to fix billboard clipping (Patch)

* chore: resolve lockfile conflict

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-05-16 13:03:05 +12:00
Silvertakana ad324a6592 feat(ui): bottom panel plugin architecture testing (#141)
* test(ui): implement playwright end-to-end testing for bottom panel architecture

* docs(e2e): improve playwright testing rules and create mock plugin guide
2026-05-16 12:29:09 +12:00
devangpratap 0a51e198d7 feat(globe): add OSM 3D Buildings overlay for non-Google imagery layers (#135)
Loads Cesium OSM Buildings (Ion asset 96188) as a 3DTileset when the
active base layer is not Google Photorealistic 3D Tiles. Buildings
only render in 3D scene mode and are automatically removed when
switching to Google 3D, 2D, or Columbus View.

Adds a "3D Buildings" toggle in Graphics Settings, persisted via
the existing wwv_graphics cookie.
2026-05-16 12:06:30 +12:00
Silvertakana 1d6381182e fix(plugin-manager): downgrade non-fatal 'Failed to fetch' errors to warn (#139)
* fix(plugin-manager): downgrade non-fatal 'Failed to fetch' errors to warn, remove polling re-throw

* fix(codeql): resolve format string vulnerability in console logging

* fix(ci): restore playwright dependencies and configure tsconfig properly
2026-05-16 11:39:41 +12:00
Antigravity AI beb295445f fix(data-engine): properly respect NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL in getLocalWsUrl and align port with engineManifest 2026-05-15 22:59:47 +12:00
Silvertakana dd62116351 feat(ui): bottom panel system refinements and visual polish (#137)
- Major: Add DESIGN.md source-of-truth document

- Minor: Update Bottom Panel plugin SDK to include 'enabled' prop

- Fix: Resolve React 19 hydration error in root layout script tag

- Fix: Refine bottom panel transparency, contrast, and resize clamps

- Fix: Transition pill styling for empty panel states
2026-05-15 21:31:39 +12:00
GodisDeadxandAntigravity AI d675d91441 Fix(Plugins): Fix sync-local-plugins looking in wrong location (#102)
Co-authored-by: Antigravity AI <[email protected]>
2026-05-15 15:15:38 +12:00
SilvertakanaandCopilot Autofix powered by AI fb07195ec2 feat(testing): stabilize Playwright E2E Tests infrastructure (#134)
* feat(testing): add Playwright E2E infrastructure and stabilize test suite

- Add Playwright E2E configuration and ui-navigation / plugin-system tests
- Add GitHub Actions workflow for Playwright CI tests
- Add test helpers and fixtures (e2e-mock-plugin)
- fix(testing): resolve UnverifiedPluginBatchDialog UI blocking issue in E2E tests
- fix(testing): limit Playwright to 2 workers locally to avoid Next.js/Prisma exhaustion
- fix(api): replace deprecated processAuthCodeResponse with authorizationCodeGrant
- test(ui): add data-testid to PanelToggleArrows and AppShell

* Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix(ci): restrict GITHUB_TOKEN permissions to read contents

* docs(e2e): document Playwright infrastructure and mock plugin lifecycle

* fix(e2e): explicitly load .env in teardown and fix dev command pathing

* fix(ci): resolve playwright environment setup and database initialization

Changes:
- Patch: Added pnpm run setup step to .github/workflows/playwright.yml to generate .env in CI
- Patch: Added explicit loadEnv() helper to 	ests/global.setup.ts and 	ests/global.teardown.ts to ensure database credentials are fresh
- Patch: Hardened scripts/safe-db-push.mjs to detect missing DATABASE_URL and explicitly prompt for pnpm run setup rather than assuming production
- Patch: Updated scripts/dev.mjs to execute pnpm exec next dev instead of raw
ext dev to resolve pathing issues
- Patch: Bumped version in package.json to 2.14.4

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-05-15 14:57:37 +12:00
devangpratap 1838f1a5cd fix(setup): remove broken legacy migration banner from setup page (#130)
The setup page unconditionally displays a banner telling users to run
scripts/migrate-legacy.mjs, but that script does not exist in the repo
or the Docker image. Every user hitting /setup sees a broken command,
including fresh installs with no legacy data.

The docker-entrypoint.sh already detects prisma/dev.db and logs a
migration warning for actual upgraders.

Closes #129
2026-05-15 13:20:27 +12:00
Silvertakana 28e4e67043 feat: Multi-Worktree Database Lifecycle Automation and Deterministic Ports (#133)
* feat: implement Extensible Bottom Panels and Worktree DB automation

- feat: Implemented BottomPanelManager and circular navigation dock for plugins
- feat: Added dynamic DATABASE_URL injection with deterministic port assignments
- fix: Added 3-retry mechanism to handle file lock (EBUSY / os error 32) issues
- feat: Added telemetry for boot sequence visibility

* fix(ui): resolve TS2339 by accessing plugin.name directly instead of plugin.metadata
2026-05-15 11:54:55 +12:00
Silvertakana 594f85afb2 feat(dx): implement worktree DB orchestration and graceful dev teardown (#132)
* feat(dx): implement worktree DB orchestration and graceful dev teardown

- feat(dx): [Minor] Implement deterministic port binding per git worktree
  to mitigate Docker port conflicts during parallel development.
- feat(dx): [Minor] Implement graceful development environment teardown
  using a custom node orchestrator (scripts/dev.mjs) replacing CLI concurrently,
  enabling cross-platform tree-kill and optional DB stopping on exit.
- build(deps): Add tree-kill for process termination.
- fix(dx): Prevent rm -rf from orphaning Docker volumes by adding
  a db:prune script and tying volume cleanup to git-wt remove.

* fix: update openid-client api usage to v6 and resolve db import

- Migrated processAuthCodeResponse to authorizationCodeGrant
- Updated db import to export prisma client correctly
- Updated unit tests with v6 openid-client mocks
- Bumped version to 2.13.1

* fix(ci): setup .env from example in local dev CI test
2026-05-15 11:20:29 +12:00
Antigravity AI b29133fca0 feat: add watcher script for automatic local plugin synchronization 2026-05-14 22:23:52 +12:00
Antigravity AI ac0092c8a8 ci: add workflow to test local development environment startup and predev tasks 2026-05-14 22:19:44 +12:00
Antigravity AI f0daefd53d chore: map postgres port to host for external access in docker-compose 2026-05-14 22:09:35 +12:00
Antigravity AI 94f8a47e0d feat(auth): implement PKCE token exchange and AES credential encryption (Phase 3)
- Minor: Implement /api/marketplace/callback route with openid-client
- Minor: Add invoke-code-review.ps1 script to automate code review subagent dispatch
- Fix: Remove duplicate global skills from .agents/skills/
- Docs: Update ADR-0001 references in rule files and AGENTS.md
2026-05-14 21:03:23 +12:00
Antigravity AI 161401a0af docs(architecture): update marketplace architecture rule with PKCE token exchange flow 2026-05-14 20:51:12 +12:00
Antigravity AI 399b811c7e fix: resolve typescript typecheck errors in ssrf utility 2026-05-14 20:42:54 +12:00
Antigravity AI 786e60865a fix: resolve SSRF and encryption bugs from premortem 2026-05-14 19:23:56 +12:00
Antigravity AI ded2d587e4 feat(auth): implement decentralized plugin auth and SSRF hardening
- Add PBKDF2 + AES-256-GCM encryption for credentials
- Update Prisma schema to include MarketplaceCredential
- Implement comprehensive SSRF utility with IP pinning
- Harden /api/camera/proxy and stream proxy with safeFetch
- Fix openid-client PKCE generator usage
- Update workspace configurations and ADRs
- Bump version to 2.10.0 (Minor)
2026-05-14 19:17:09 +12:00
SilvertakanaandCopilot Autofix powered by AI b177c6987d feat(wwv-plugin-sdk): add shared auth contracts for decentralized plugin auth (#127)
* refactor(docker): separate self-host Docker compose structure from development

test(ci): add github action to verify self-host scripts

* feat(wwv-plugin-sdk): add shared auth contracts for decentralized plugin auth

* Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix(wwv-plugin-sdk): wrap auth contract type tests in vitest suite

* fix(wwv-plugin-sdk): implement PR review feedback for auth contracts

* fix(wwv-plugin-sdk): address PR comments on auth contracts

* docs(rules): codify decentralized plugin auth contracts and security rules

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-05-14 19:01:39 +12:00
SilvertakanaandCopilot Autofix powered by AI b11125fd5f refactor(docker): separate self-host Docker compose structure from development (#125)
* refactor(docker): separate self-host Docker compose structure from development

test(ci): add github action to verify self-host scripts

* Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-05-14 14:22:40 +12:00
szski db1a9ad191 Merge pull request #103 from AlsoTheZv3n/perf/cesium-hot-path-wins
perf(globe): three Cesium hot-path wins
2026-05-13 16:49:02 -04:00
szski cfd8e4b7d0 Merge pull request #104 from AlsoTheZv3n/fix/ci-triggers-and-audit
ci: trigger on push to main and add dependency audit
2026-05-13 16:42:42 -04:00
Sven WeidenmannandClaude Opus 4.7 a475a5a562 perf(globe): replace viewer mutation with WeakMap<Viewer, Collections> [Patch]
Primitive collections (points, billboards, labels, polylines) were
attached directly to the Cesium Viewer instance as `_wwvX` properties
behind `(viewer as any)` casts. This created hidden coupling, blocked
multi-viewer setups, and left ~11 `as any` casts across three files.

Replace the viewer-mutation pattern with a module-level
`WeakMap<CesiumViewer, PrimitiveCollections>`:

- `initPrimitiveCollections` now writes into the WeakMap
- `getCollections` reads from the WeakMap and returns a destructurable
  shape with optional fields, preserving the existing call pattern
- `AnimationLoop` and `useEntityRendering` use `getCollections(viewer)`
  instead of direct `(viewer as any)._wwvX` reads
- Zero `(viewer as any)._wwv*` remain; collections GC cleanly when
  the viewer is dropped

Adds `src/core/globe/EntityRenderer.test.ts` with 7 tests covering:
- WeakMap roundtrip (init populates, getCollections retrieves, per-viewer
  isolation, graceful no-scene)
- Defer-clone behaviour from the previous commit (no clone on
  iconUrl/model path; clone on auto-SVG path; original untouched)

Refs #95

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-13 16:42:12 -04:00
Sven WeidenmannandClaude Opus 4.7 8f859f4559 perf(globe): defer effectiveOptions clone to auto-SVG branch [Patch]
`renderSingleEntity` was cloning `options` for every entity via
`const effectiveOptions = { ...options }`, even when the auto-SVG
fallback below didn't need to mutate it. With 10k+ entities per
render pass this adds avoidable GC pressure in the hot loop.

The clone is now deferred into the auto-SVG branch that actually
mutates, so entities that arrive with `iconUrl` set (or `type === "model"`)
pass `options` through by reference with zero allocation.

Refs #95

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-13 16:42:12 -04:00
Sven WeidenmannandClaude Opus 4.7 8a594d28ea perf(globe): type window._wwvFrameCount via Window interface extension [Patch]
Replace `(window as any)._wwvFrameCount = frame` with a typed
`declare global { interface Window { _wwvFrameCount?: number } }`
and a plain `window._wwvFrameCount = frame` assignment.

The external plugin contract (a window-published frame counter for
throttling) is preserved; only the cast is removed.

Refs #95

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-13 16:42:12 -04:00
Sven WeidenmannandClaude Opus 4.7 382ceed2f1 ci: trigger on push to main and add non-blocking dependency audit (fixes #97) [Patch]
Previously the workflow only ran on `pull_request: branches: [main]`,
so any direct push to `main` (or any other branch) bypassed CI entirely.

Changes:
- Add `push: branches: [main]` to the trigger list so main-branch pushes
  also run typecheck / test / build / audit.
- Add a new `audit` job that runs `pnpm audit --audit-level high` and
  surfaces high/critical advisories in the CI output.

The audit job is intentionally non-blocking (`continue-on-error: true`)
on first introduction: the lockfile currently has 2 high + 1 critical
transitive advisories (sentry-webpack-plugin → webpack → schema-utils →
ajv → fast-uri chain) that need to be addressed before audit can become
a hard gate. A comment in the workflow notes that the flag should be
flipped to `false` once those are resolved.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-13 16:36:02 -04:00
szski df072feea7 Merge pull request #123 from szski/chore/pin-vulnerable-transitive-deps
chore(deps): pin form-data + fast-uri overrides to patched versions
2026-05-13 16:35:27 -04:00
SapphireandClaude Opus 4.7 943ab91564 chore(deps): pin form-data + fast-uri overrides to patched versions
Resolves three high+critical advisories surfaced by the dependency
audit added in PR #104:

  CRITICAL  form-data <2.5.4   GHSA-fjxv-7rqg-78g4  unsafe random boundary
  HIGH      fast-uri  <=3.1.0  GHSA-q3j6-qgpj-74h6  path traversal
  HIGH      fast-uri  <=3.1.1  GHSA-v39h-62p7-jpjc  host confusion

form-data was 2.3.3 transitively via insecam-api -> [email protected].2.
fast-uri was 3.1.0 via @sentry/webpack-plugin -> webpack -> schema-utils
-> ajv (15 paths total).

Uses scoped pnpm overrides keyed on the vulnerable version ranges so
each major release line is patched in place: form-data stays in 2.x
(2.3.3 -> 2.5.5) rather than jumping to 4.x and risking compatibility
with the long-abandoned `request` package. Overrides for form-data 3.x
and 4.x are pre-emptive -- no current matches in the tree, but they
future-proof against new deps pulling vulnerable form-data versions.

After install, `pnpm audit --audit-level high` returns exit 0.
Five moderate findings remain (request SSRF, tough-cookie, qs,
@hono/node-server, postcss) -- all below the CI threshold and
addressable in a follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-13 16:28:19 -04:00
szski 7a262c617c Merge pull request #79 from szski/feat/agent-bus
feat(agent): SSE-based AgentBus for external-tool UI control
2026-05-13 16:07:41 -04:00
SapphireandClaude Opus 4.7 a69b1d9aac docs(agent): how-to for wiring an MCP server to the agent bus
Adds docs/agent-bus.md covering enable-via-build-arg, a curl smoke test,
Node + Docker MCP client config snippets (Claude Code / Desktop / Cursor /
Cline), and per-user security notes. Links szski/wwv-mcp as the reference
MCP server with a note that it may relocate under the project org.

Cross-links the new doc from README Key Features, Repository Ecosystem,
and Documentation Index.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-13 16:01:02 -04:00
szski a1b3a71e6e Merge pull request #64 from szski/feat/osm-surveillance-camera-adapter
feat(api/camera): OSM surveillance-points adapter (depends on #62)
2026-05-13 15:51:06 -04:00
SapphireandClaude Opus 4.7 338cde0813 feat(api/camera): OSM surveillance-points adapter
Adds an adapter that pulls `man_made=surveillance` nodes from the
OpenStreetMap Overpass API as location-only camera markers. The geometry
alone is useful as a "where do public / government cameras exist" overlay
alongside the live DOT feeds — most OSM surveillance nodes don't have
stream URLs, but their tags (operator, camera:type, surveillance:type)
are surfaced via `properties.extra` for filtering.

The query is bounded to a list of major metros (~30km radius around SF,
LA, NYC, London, Paris, Tokyo, Raleigh) to keep the response size sane;
operators who want fuller coverage can extend the query in-source. Output
is capped at 3000 nodes per fetch.

Cache TTL is 7 days (overrides the default 24h) since OSM data changes
slowly and Overpass is shared infrastructure that politely asks consumers
not to poll aggressively.

Built on top of the CameraAdapter pattern from the camera adapter
refactor PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-13 15:43:26 -04:00
szski b7311268f8 Merge pull request #63 from szski/feat/ncdot-camera-adapter
feat(api/camera): NCDOT adapter (depends on #62)
2026-05-13 15:41:23 -04:00
szski 54078f351c Merge pull request #105 from AlsoTheZv3n/docs/architecture-migration
docs: migrate architecture deep-dive from AGENTS.md to docs/ARCHITECTURE.md
2026-05-13 15:36:23 -04:00
SapphireandClaude Opus 4.7 a149be8560 feat(api/camera): NCDOT adapter
Adds North Carolina DOT as a sixth source for `/api/camera/traffic`.
~770 cameras statewide, no API key required, direct JPEG snapshots.

NCDOT exposes a two-step API: a bulk endpoint that returns only
`{ id, latitude, longitude }` per camera, and a per-camera detail
endpoint with the rest (locationName, mileMarker, roadId, countyId,
imageURL, status). The adapter fans out the detail fetches with a
bounded concurrency of 20, lands the initial population in ~15 seconds,
and the registry's per-adapter cache holds it for 24h after that.

Cameras with `status !== "OK"` are filtered out. The adapter is built
on top of the CameraAdapter pattern introduced in the camera adapter
refactor PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-13 15:32:14 -04:00
szski 10fdafbdb6 Merge pull request #99 from szski/fix/camera-adapter-timeout
fix(api/camera): per-adapter timeout so one slow source can't stall the rest
2026-05-13 15:31:25 -04:00
szski 2280fc4d19 Merge pull request #108 from devangpratap/fix/imagery-fallback-blank-globe
fix(globe): fall back to OSM when imagery providers fail without API keys
2026-05-13 15:27:56 -04:00
szski 118106ff8f Merge pull request #121 from szski/fix/setup-page-white-on-white
fix(setup): logo + Sign In button rendering white-on-white
2026-05-13 15:18:11 -04:00
szski 6a915cca62 Merge pull request #122 from szski/fix/engine-manifest-port
fix(engineManifest): correct local engine port (5001) + honor data-engine env var
2026-05-13 15:15:15 -04:00
szski 42dadd0b40 Merge pull request #120 from szski/feat/test-bedrock-round-3
test: round 3 — consolidated test bedrock + stryker setup (supersedes #113, #114, #117, #118)
2026-05-13 15:03:34 -04:00
Sapphire 583c0e57a6 fix(engineManifest): correct local engine port + honor data-engine env var
Two related bugs in the local-engine detection path:

1. Hardcoded port 5000 — should be 5001
   wwv-data-engine's docker-compose.yaml binds the engine to port 5001
   (and has since the v2 refactor), but engineManifest.ts polls
   `localhost:5000/manifest` to detect it. The detection always
   reported "no local engine" even when one was running, falling back
   to the cloud engine on every plugin toggle.

   Bonus pain: macOS hijacks port 5000 for the AirPlay Receiver, which
   returns 403 to any GET request. On any macOS dev machine, this
   produced a misleading 403 in the browser console even when no
   engine was running at all. Famous local-dev gotcha that ate dev
   time.

2. NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL is ignored
   Every other code path in the app honors this env var as the
   override for the engine URL — `resolveEngineUrl`, plugin context
   injection, the radio-garden plugin, etc. But `engineManifest` ran
   its own detection independently against hardcoded localhost. So
   even when the operator pointed plugins at a custom engine URL,
   manifest detection failed silently.

Fix: read `NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL` first (normalizing
ws[s]:// → http[s]:// and stripping a trailing /stream), then fall
back to localhost:5001 if not set. Same approach used elsewhere in
the codebase.

Side effects:
- Symptoms in the browser console downstream of this bug (notably
  the spurious 403 on `localhost:5000/manifest` and the cascade of
  "seeder not running" 404s when plugins fall back to the cloud
  engine that doesn't have that seeder) clear up.
- Existing engineManifest unit tests still pass — they mock fetch
  and don't care about the URL shape.

Worth adding a test for the env-var precedence in a follow-up; left
out of this PR to keep the diff to the one-file behaviour fix.
2026-05-13 14:38:46 -04:00
Sapphire ae14a30168 fix(setup): use explicit brand color instead of unset theme accent
The setup page's `.logo` and `.button` rules use
`rgb(var(--accent-rgb, 200, 30, 30))`. Because CSS `var()` only treats
the value after the first comma as the *fallback*, the syntax
intends "use --accent-rgb if defined, otherwise fall back to
`200, 30, 30`."

The fallback never triggers in practice: `theme-tokens.css` defines
`--accent-rgb: 255, 255, 255` at the `:root` level, so the variable
is always set when the setup page renders. `rgb(255, 255, 255)`
combined with the rules' `color: #fff` gives white-on-white — the
"Sign In" button text and the "W" logo are invisible on the page.

The setup flow is pre-auth and pre-theme; it shouldn't be inheriting
the accent. Replace the var() lookups with an explicit brand red
(`rgb(200, 30, 30)`) so the page is readable regardless of which
theme is active in the rest of the app.

Affected rules: `.logo` background, `.button` background, `.input:focus`
border. Three identical edits.
2026-05-13 14:34:48 -04:00
Sapphireandsilvertakana 019b427442 test: round 3 — consolidated test bedrock + stryker setup
Consolidates four overlapping test-infra PRs (#113, #114, #117, #118)
into a single coherent batch. Each contributing PR has redundancies
with one or more of the others; together they ship a real ~134-test
expansion of the unit suite plus mutation-testing infrastructure.

Same shape as #110 / #111: small, focused, behavior-pinning tests
against load-bearing systems. Each test answers one question — *what
specific regression does it catch that I'd otherwise find from a user
bug report?* Tests pin observable contracts (events emitted, cache
entries written, public-API state) rather than internal mechanics, so
they survive refactors during the longer-term migration toward the
six-interface architecture.

## What's in this PR

### Stryker mutation testing (new in this batch)

- `stryker.config.json` — hybrid config: `coverageAnalysis: "perTest"`
  for the 5–10× speedup on a real codebase, `ignoreStatic: true` to
  drop false-positive mutants in module-load code, and a `mutate`
  scope of `src/core/**` + `src/lib/**` covering the logic-heavy
  layers without dragging the React component surface into noise.
- `.github/workflows/mutation-testing.yml` — runs Stryker on a weekly
  schedule (Sundays 22:00 US Eastern / 03:00 UTC Monday) plus
  `workflow_dispatch` for on-demand validation. Per-PR triggering
  would be too expensive (10–30 min per run even with `perTest`).
- `package.json` — adds `@stryker-mutator/core`, `@stryker-mutator/vitest-runner`,
  and `fast-check` as devDependencies. New `test:mutate` script.

### New test files (22 added)

Plugin lifecycle and loaders:
- `src/core/plugins/PluginManager.test.ts` — orchestrator lifecycle:
  register, enable, disable, data-flow contract, getAllEntities
  filters by enabled, updateTimeRange, destroy, togglePlugin,
  loadFromManifest. 11 tests.
- `src/core/plugins/loadPluginFromManifest.test.ts`
- `src/core/plugins/validateManifest.test.ts`
- `src/core/plugins/loaders/mapGeoJsonToEntities.test.ts`
- `src/core/plugins/loaders/mapJsonToEntities.test.ts`

Data layer:
- `src/core/data/DataUtils.test.ts`
- `src/core/data/WsClient.test.ts`

State (9 Zustand slices + the integration tests):
- `src/core/state/StateSlices.test.ts`
- `src/core/state/configSlice.test.ts`
- `src/core/state/dataSlice.test.ts`
- `src/core/state/favoritesSlice.test.ts`
- `src/core/state/filterSlice.test.ts`
- `src/core/state/geojsonSlice.test.ts`
- `src/core/state/globeSlice.test.ts`
- `src/core/state/layersSlice.test.ts`
- `src/core/state/timelineSlice.test.ts`
- `src/core/state/uiSlice.test.ts`

Filters:
- `src/core/filters/filterEngine.test.ts`

Library:
- `src/lib/AuthConfig.test.ts`
- `src/lib/analytics.test.ts`
- `src/lib/origin.test.ts`

### Test files extended (9 modified)

Property-based and edge-case additions to existing tests, drawn from
the upstream PRs:

- `src/core/plugins/loaders/getNestedValue.test.ts` (+46 lines)
- `src/lib/geojson/converter.test.ts` (+31)
- `src/lib/geojson/normalizer.test.ts` (+52)
- `src/lib/rateLimit.test.ts` (+44)
- `src/core/plugins/loaders/DeclarativePlugin.test.ts` (+147 — type
  fixes for `DataSourceConfig.method`/`pollInterval`/`format`)
- `src/lib/marketplace/cors.test.ts` (+5)
- `src/lib/marketplace/marketplaceToken.test.ts` (+21)
- `src/core/filters/filterEngine.test.ts` (type-corrected — uses
  `latitude`/`longitude` not `position.lat`/`lon`, nested
  `range: { min, max, step }` config, etc.)

## What's *not* in this PR (intentionally)

- **Duplicate tests for code already on main.** #113 originally
  included `.spec.ts` versions of `CacheLayer`, `DataBus`,
  `PollingManager`, `cors`, `marketplaceToken`, `repository`,
  `rateLimit`, `edition`, and `seedDefaultPlugins` — all of which are
  already tested via `.test.ts` files on main from #110 + prior. Vitest
  picks up both `.test.ts` and `.spec.ts`, so keeping both would double
  the maintenance burden without adding signal. Dropped.
- **Tautological tests.** A few accessors-only tests from #113
  (`getEntities` returning the cached array, `getAllPlugins().length`
  equals N) were dropped per the test-writing bar set on #110/#111:
  each test must catch a specific bug class that two different broken
  implementations would fail differently on.
- **`version` bump in `package.json`.** Three of the contributing PRs
  bumped the version line independently (2.9.5, 2.9.10, 2.9.10), each
  conflicting with the others. Reserve version bumps for explicit
  release commits — feature/test PRs should leave the version field
  alone to avoid this conflict hairball.
- **A duplicate `stryker.conf.json`.** #114 used that filename; #117
  used `stryker.config.json`. Stryker accepts both but only loads one
  (alphabetical resolution). Picked `stryker.config.json` as the
  canonical name.

## Verification

```
Test Files  38 passed (38)
Tests       271 passed (271)
```

137 tests on main before this PR; +134 from this consolidation, no
regressions. Type check clean (`pnpm exec tsc --noEmit`).

## Replaces

- PR #113 (test/improve-coverage) — most of its 18 truly-new files
  are included here; the 9 duplicate-of-main entries are not.
- PR #114 (chore/stryker-setup) — its CI workflow shape is here with
  the trigger changed to weekly+manual instead of per-PR.
- PR #117 (chore/stryker-and-fast-check) — its full deps and test
  additions are here.
- PR #118 (test/improve-project-coverage) — the type-corrected
  versions of its 6 test files are here, including the typefix commit
  pushed to that branch earlier.

Each of those PRs should be closed once this lands, with a comment
referencing this PR for credit.

Co-Authored-By: silvertakana <[email protected]>
2026-05-13 13:37:22 -04:00
Silvertakana 04beadc0e2 chore: enforce vitest coverage thresholds (#115) 2026-05-13 23:14:56 +12:00
Silvertakana cf67c5d96f docs(panels,core): complete TSDoc standardization for panels and core slices (#112)
This commit includes the work done by the Flash subagent to add comprehensive TSDoc comments to src/components/panels, src/core/state, and src/core/plugins, improving IDE intellisense and codebase maintainability.
2026-05-13 22:45:08 +12:00
devangpratap bf7e9c4bee merge: resolve package.json version conflict, take main (2.9.1) 2026-05-13 01:53:51 -04:00
devangpratap a1c4fed23b fix(globe): tiered imagery fallback — Google tiles → Ion/Bing → OSM
Replace direct Ion fallback with a 3-tier chain per maintainer feedback:
1. Google XYZ tiles (free, no API key)
2. Bing via Cesium Ion (free shared token)
3. OSM as last resort

useImageryManager catch block still falls to OSM if createImageryProvider
throws at runtime (e.g. direct Bing key failure, ArcGIS down).
2026-05-13 01:49:50 -04:00
szskiandSapphire e6c7dcf295 test: bedrock tests for the load-bearing data layer + camera registry (#110)
WorldWideView's Vitest setup is already wired and CI already runs
\`pnpm test\` on every PR. What's missing is tests for the systems
that actually break: the plugin lifecycle plumbing (data bus, cache,
polling) and the camera adapter aggregator that the UI hits on every
layer toggle.

This PR adds a focused first batch — 12 tests across 4 files — covering
the most common failure modes we've actually seen in practice. The
intent is to land the patterns alongside the tests so the next
contributor (human or agent) can copy the shape rather than improvise.

## What's tested

| File | Coverage | Tests |
| ---- | -------- | ----- |
| src/core/data/DataBus.test.ts | emit/subscribe round-trip, unsubscribe correctness, handler-failure isolation | 3 |
| src/core/data/CacheLayer.test.ts | memory-tier get/set, TTL expiry + eviction | 2 |
| src/core/data/PollingManager.test.ts | immediate-run on start(), interval scheduling, stop() cancels | 3 |
| src/app/api/camera/adapters/registry.test.ts | resolveSources for "all" / null / "none" / mixed-with-unknowns | 4 |

Each test answers one question: what regression does it catch that
we'd otherwise find by user bug report?

## Config change

vitest.config.ts now includes \`src/app/**/*.{test,spec}.*\` so API-route
tests under \`src/app/api\` are picked up by \`pnpm test\`. Previously only
\`src/lib\`, \`src/core\`, \`src/plugins\`, and \`packages\` were scanned —
api-route handlers had no test home.

## Docs

\`docs/testing.md\` gets four new sections:

- **Patterns** — plain-function, stateful-singleton, mock-with-vi.mock,
  fake-timers — with a callout for the \`runOnlyPendingTimersAsync\`
  double-counting gotcha we hit while writing PollingManager tests.
- **What NOT to Test (Yet)** — Cesium-heavy components, live DB
  round-trips, dynamic plugin imports.
- **Philosophy** — small focused suite > large expensive one, test
  pure logic, add a test when you fix a bug.

Original overview/test-types/run-commands sections are unchanged.

## Verification

Locally (Node 22 + pnpm 9.15.4):

\`\`\`
Test Files  17 passed (17)
Tests       127 passed (127)
\`\`\`

Existing 115 tests still pass; new 12 land cleanly.

## What's next (not in this PR)

The list of load-bearing systems still without coverage:

- PluginManager (the central plugin lifecycle orchestrator)
- The new plugin-backend supervisor + proxy (PR #100)
- The agent bus (PR #79)
- Plugin marketplace install/uninstall round-trip
- React component tests for LayerPanel and DataBusSubscriber

Worth a follow-up batch once the patterns from this PR are familiar.

Co-authored-by: Sapphire <sapphire@tiktokcoach>
2026-05-13 17:23:09 +12:00
Antigravity AI 9e44e91d81 chore: update docker-compose ports and document coding principles in AGENTS.md 2026-05-13 17:04:51 +12:00
Antigravity AI c6cd515588 Merge branch 'main' of https://github.com/silvertakana/worldwideview 2026-05-13 16:58:50 +12:00
Antigravity AI 0f871729c0 fix(docker): use compose profiles to prevent data engine collision on production deployments 2026-05-13 16:55:49 +12:00
e73f81d6fe chore: add ESLint baseline + CI lint job (fixes #96) [Patch] (#106)
Introduces a baseline ESLint configuration and wires it into CI.

Config:
- `eslint.config.mjs` (flat config, ESLint 9). Extends
  `eslint-config-next/core-web-vitals` and `eslint-config-next/typescript`
  directly — both ship as native flat-config arrays in `eslint-config-next` 16+.
- Ignores cover `.next`, `out`, `src/generated`, `public/cesium`,
  `packages/*/dist`, plus the gitignored sandboxes (`local-scripts`,
  `local-plugins`, `local-seeders`, `.agents`, `.claude`, `.worktrees`).

Rule policy:
- All rules that the Next.js preset surfaces as errors are demoted to
  `warn` on first introduction, so the lint job is runnable against
  the current codebase without blocking on pre-existing debt. The
  demoted set (react-hooks/* in React 19's stricter purity suite,
  no-unescaped-entities, ban-ts-comment, no-assign-module-variable,
  prefer-const) plus the originally-flagged `no-explicit-any` and
  `no-console` are all documented in the config with a recommended
  promotion order.
- Current state: 0 errors, 564 warnings.

Scripts:
- `pnpm lint` runs `eslint .`.

CI:
- New `lint` job in `.github/workflows/ci.yml`, parallel to typecheck /
  test / audit. Blocking on errors only — warnings are visible in the
  job log but don't fail the build.

Same incremental-introduction pattern as the audit job (#97): land the
infrastructure now, tighten policy via follow-up PRs as the backlog is
worked down.

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Co-authored-by: Silvertakana <[email protected]>
2026-05-13 16:49:43 +12:00
Antigravity AI f5235bf219 fix(docker): remove hardcoded host port bindings from base docker-compose to fix Coolify deployment conflicts 2026-05-13 16:30:35 +12:00
19f10d25df Fix/UI styling and buttons (#107)
* modified:   .gitignore
	modified:   AGENTS.md
	modified:   package.json
	modified:   public/cesium/ThirdParty/zip-module.wasm
	new file:   src/components/layout/ApiKeysTab.tsx
	modified:   src/components/layout/Header.tsx
	modified:   src/components/panels/DataConfig/OverlayTab.tsx
	modified:   src/components/panels/DataConfig/index.tsx
	modified:   src/components/panels/LayerPanel.tsx
	modified:   src/components/panels/PluginsTab.css
	modified:   src/components/panels/PluginsTab.tsx

* modified:   src/components/layout/Header.tsx
	new file:   src/components/layout/timeSelect.css

* UI-Testing

* UI-testing

* modified:   src/components/layout/timeSelect.css

* Fixed Duplicate Time Button

Caused by a merge conflict

* addeb back a </div>

* Fix Sync failing

* deleted:    local-plugins/wwv-plugin-iss/index.ts
	deleted:    local-plugins/wwv-plugin-iss/package.json

* fix: dark glass timeline button, sidebar expands when timeline closed, light mode button adaptation

* temp-fix

* fix blur being disabled

---------

Co-authored-by: nullptr <[email protected]>
Co-authored-by: Silvertakana <[email protected]>
2026-05-13 16:20:59 +12:00
Antigravity AI 3ddd6ec150 feat(core): plugin toggle functionality and CLI publishing enhancements
This release groups multiple feature additions and structural stability fixes:
- feat(marketplace): API routes to enable/disable local plugins via UI (Minor)
- feat(cli): interactive publishing and global org configuration persistence (Minor)
- fix(css): resolve parsing errors in PluginsTab stylesheet (Patch)
- fix(engine): fix wwv-data-engine dependency resolution and host mounts (Patch)
- fix(db): resolve P1001 database connection issues in docker loopback (Patch)
- chore(ci): revert dynamic edition runtime injection to static build args
2026-05-13 16:18:53 +12:00
devangpratap e26aee6dd2 fix(globe): fall back to OSM when imagery providers fail without API keys
Fixes #49 [Patch]
2026-05-12 21:58:21 -04:00
Antigravity AI 42f4701fda chore(cli): bump version to 1.1.2 2026-05-13 11:37:27 +12:00
Antigravity AI 4c6978ea90 feat(cli): add interactive org prompt to publish and config command 2026-05-13 11:36:25 +12:00
Antigravity AI 3c03b4e7a4 fix(cli): publish missing dist changes for default create directory 2026-05-13 11:34:17 +12:00
Antigravity AI d2bdb0cee8 feat(cli): enhance publish command and update default create directory
- feat(cli): Add --org flag to publish to allow publishing under custom NPM scopes (Minor).
- feat(cli): Allow publish command to run from project root via path resolution (Minor).
- feat(cli): Change create default scaffolding to local-plugins/ instead of packages/ (Minor).
- feat(cli): Add --core flag to create to override default and scaffold into packages/ (Minor).
- docs: Update tutorial, CLI readmes, and agent architecture workflows with new commands (Fix).
2026-05-13 10:57:33 +12:00
Sven WeidenmannandClaude Opus 4.7 335b74a486 docs: migrate architecture deep-dive from AGENTS.md to docs/ARCHITECTURE.md (fixes #98)
`docs/ARCHITECTURE.md` was a 33-line stub, despite the README pitching
a "Deep dive into the Cesium rendering pipeline, event data bus, and
performance optimizations". The substantive content was duplicated /
buried in `AGENTS.md` §4.

Migrate it to a proper human-facing doc:

- `docs/ARCHITECTURE.md` (33 → 200 lines) now contains the full
  architecture deep-dive: system overview with mermaid diagram, plugin
  system & All-Bundle Model, Zustand state management (with slice
  table), data pipeline, data engine & seeders (agnostic-renderer
  contract), rendering pipeline (primitive collections, chunked
  processing, horizon culling, LOD, stack/spiderifier), edition
  system, key source files table, and cross-refs into `.agents/rules/`.
- `AGENTS.md` §4 collapses from ~80 lines of duplicated prose to a
  short stub that points contributors to `docs/ARCHITECTURE.md` and
  lists the key invariants agents must respect (plugin SDK as source
  of truth, agnostic frontend, nine slices, primitive rendering, three
  editions). Same hub-and-spoke pattern as the maintainer already
  applied to §3, §6, §7.
- README link description updated from "DataBus event stream and
  Zustand state management" to reflect the expanded scope.

No code changes; no version bump (consistent with prior `docs:` commits
in the history that did not touch root `package.json`).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-12 21:29:11 +02:00
Sapphire 014a360d5d fix(api/camera): per-adapter timeout so one slow source can't stall the rest
The aggregator uses `Promise.allSettled`, so the request takes as long
as the slowest adapter. Today ny511 (without a real API key) returns
empty after ~5s — fast adapters (cached) come back in <50ms but the
client waits 5s+ for the aggregate response.

With a 4s per-adapter cap, the stuck source rejects fast and either
its stale cache or an empty list is served; healthy adapters return on
their own cadence.

Concrete latency before this commit, all 7 adapters in parallel:
  caltrans:        0.04s  gdot:    0.02s  tfl:        0.01s
  ny511:           5.03s  wsdot:   0.02s  ncdot:      0.02s
  osm-surveillance:0.03s
  total:           ~10s
2026-05-12 13:20:17 -04:00
Antigravity AI 21adb9f2e8 docs: add plugin development quickstart and advanced guides, and include CLI and SDK scaffolding artifacts 2026-05-13 02:42:24 +12:00
Antigravity AI b792dc0373 chore(cli): use latest sdk for public npm template 2026-05-13 02:18:14 +12:00
Antigravity AI cbbc38ecc9 feat(plugin-iss): implement ISS tracker plugin and seeder 2026-05-13 02:06:58 +12:00
Antigravity AI bbb231541b feat: implement interactive plugin generation via prompts in cli 2026-05-13 01:54:26 +12:00
Antigravity AI 70fd3e1818 build: add prompts dependency to wwv-cli 2026-05-13 01:52:47 +12:00
dependabot[bot] 0ee8ae700c build(deps-dev): bump vite from 6.4.2 to 8.0.12 (#91)
Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 6.4.2 to 8.0.12.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v8.0.12/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-version: 8.0.12
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-13 01:35:00 +12:00
Antigravity AI 2c17b1e701 chore: remove wwv-data-engine service from docker-compose override 2026-05-13 01:27:27 +12:00
Antigravity AI 2ed8da0e1c feat(ui): implement responsive timeline toggle and auto-hide on mobile 2026-05-13 01:21:43 +12:00
Antigravity AI 9006064240 fix: direct users to public submit URL instead of admin UI in CLI publish command 2026-05-13 01:18:28 +12:00
Antigravity AI fe2cfa9dac feat: implement publish command and complete WWV CLI reconstruction 2026-05-13 01:14:06 +12:00
Antigravity AI 84f3691c22 feat(wwv-cli): initialize commander entrypoint and stubs 2026-05-13 01:05:56 +12:00
Antigravity AI f9384b8244 build(wwv-cli): initialize typescript build pipeline and dependencies 2026-05-13 01:03:53 +12:00
Antigravity AI d0ae85049f refactor: move wwv-data-engine and wwv-redis service definitions from override to base docker-compose file 2026-05-13 00:23:13 +12:00
Antigravity AI 64815c114c feat: implement LayerPanel with tabbed navigation, plugin management, and search functionality alongside supporting styling and utility scripts. 2026-05-13 00:14:17 +12:00
Antigravity AI e86fdf56d0 chore: update dependency versions in pnpm-lock.yaml 2026-05-13 00:11:29 +12:00
Antigravity AI c25a1e6d58 refactor: consolidate docker compose setups using override pattern 2026-05-13 00:04:52 +12:00
dependabot[bot] f9c8bcff17 build(deps): bump vite from 5.4.21 to 6.4.2 (#88)
Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 5.4.21 to 6.4.2.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v6.4.2/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v6.4.2/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-version: 6.4.2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-12 22:18:29 +12:00
Antigravity AI 008e8cd01d chore(repo): hygiene improvements and onboarding fixes (Refs #86)
- chore: remove tracked .env and skills-lock.json
- chore: move military_bases.geojson to public/
- fix: correct README architecture doc hyperlink
- chore: add .nvmrc and enforce Node v20 via engines
- fix: implement safe-db-push.mjs wrapper to guard DATABASE_URL
- refactor: remove duplicate ApiKeysTab.tsx and fix imports
- docs: relax file size limit and remove TS aliases in AGENTS.md
2026-05-12 22:02:47 +12:00
Antigravity AI 1d1c7b270b chore: remove tracked dev artifacts, scratch files, and broken symlink (fixes #86) 2026-05-12 21:30:26 +12:00
Antigravity AI 63a5e5d8a1 docs: enforce explicit authorization and clarifying questions in AGENTS.md 2026-05-12 21:18:02 +12:00
Antigravity AI 39c1172b0b docs: enforce explicit authorization and clarifying questions in AGENTS.md 2026-05-12 21:18:02 +12:00
Antigravity AI 09ac165420 fix(license): restore full Elastic License 2.0 and update README (fixes #87) 2026-05-12 21:18:02 +12:00
dependabot[bot] 1daa8a6979 build(deps): bump lucide-react from 0.576.0 to 1.14.0 (#90)
Bumps [lucide-react](https://github.com/lucide-icons/lucide/tree/HEAD/packages/lucide-react) from 0.576.0 to 1.14.0.
- [Release notes](https://github.com/lucide-icons/lucide/releases)
- [Commits](https://github.com/lucide-icons/lucide/commits/1.14.0/packages/lucide-react)

---
updated-dependencies:
- dependency-name: lucide-react
  dependency-version: 1.14.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-12 20:59:40 +12:00
dependabot[bot] 2c97f8e1e3 build(deps-dev): bump the dependencies group across 1 directory with 2 updates (#94)
Bumps the dependencies group with 2 updates in the / directory: [@vitest/coverage-v8](https://github.com/vitest-dev/vitest/tree/HEAD/packages/coverage-v8) and [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest).


Updates `@vitest/coverage-v8` from 4.1.5 to 4.1.6
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.6/packages/coverage-v8)

Updates `vitest` from 4.1.5 to 4.1.6
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.6/packages/vitest)

---
updated-dependencies:
- dependency-name: "@vitest/coverage-v8"
  dependency-version: 4.1.6
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: vitest
  dependency-version: 4.1.6
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-12 20:59:28 +12:00
dependabot[bot] 1fb830e962 build(deps-dev): bump @types/node from 20.19.37 to 25.7.0 (#92)
Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 20.19.37 to 25.7.0.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

---
updated-dependencies:
- dependency-name: "@types/node"
  dependency-version: 25.7.0
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-12 20:44:49 +12:00
Antigravity AI 5e542c3ca6 fix(docker): Remove obsolete copy command for deleted script 2026-05-12 19:24:40 +12:00
Antigravity AI 7e25fdfba2 Merge branch 'fix/engine-seeders-coolify' into main 2026-05-12 19:11:39 +12:00
SapphireandClaude Opus 4.7 1845a1ced9 feat(agent): SSE-based AgentBus for external-tool UI control
Adds an opt-in agent integration surface that lets a trusted external
tool (running as the same logged-in user) drive the running WWV browser
session — fly the globe, toggle layers, focus entities — by POSTing
JSON actions to the server, which fans them out over SSE to every
connected browser tab belonging to that user.

The motivating use case is an MCP server (a Model Context Protocol
adapter): an LLM agent calls a tool like `globe_fly_to({lat, lon})`,
the MCP server posts to `/api/agent/publish`, the browser subscriber
re-emits the action onto the existing client-side `dataBus`, and the
React + Cesium app handles it the same way it would handle any UI
button click. Read-only camera / engine queries already work via the
existing REST surface — this PR adds the missing write path.

What's added:

- `src/lib/agent/bus.ts` — process-local AgentBus singleton, scoped
  per user. Subscriptions live in a `Map<userId, Map<connId, sub>>`
  so user A's publish never reaches user B's stream, even though
  both run on the same Node process.

- `src/app/api/agent/stream/route.ts` — SSE channel. Auth-gated. Sends
  initial `:connected` + `retry:3000` directives and a `:hb` heartbeat
  every 25s so intermediary proxies (Caddy, Nginx, CDNs) don't tear
  down a "stalled" connection.

- `src/app/api/agent/publish/route.ts` — POST endpoint. Accepts JSON
  AgentAction shapes (`fly_to`, `face_towards`, `layer_toggle`,
  `highlight_layer`, `select_entity`, `ping`), broadcasts only to the
  publishing user's bus bucket. Returns `{delivered, subscribers}` so
  the caller knows how many tabs received the command.

- `src/components/layout/AgentBusSubscriber.tsx` — browser EventSource
  client. Re-emits received actions onto `dataBus` (`cameraGoTo`,
  `cameraFaceTowards`, `layerToggled`, `entitySelected`) — every plugin
  is already listening on those channels, so no plugin changes needed.

- `AppShell.tsx` — mounts `AgentBusSubscriber` next to the existing
  `DataBusSubscriber`. Two lines.

- `src/app/api/build/route.ts` — `/api/build` returns build id +
  timestamp + public flag state. Pairs with a one-line console.log
  the subscriber prints on mount so a stale-bundle / config-mismatch
  is visible at a glance instead of looking like a feature regression.

Default off via `NEXT_PUBLIC_WWV_AGENT_BUS_ENABLED=true`. An unintended
deployment doesn't open a remote-control channel for users who didn't
opt in.

What's changed in the Dockerfile:

- Adds ARG declarations for the new vars (`NEXT_PUBLIC_WWV_AGENT_BUS_ENABLED`,
  `NEXT_PUBLIC_WWV_BUILD_ID`, `NEXT_PUBLIC_WWV_BUILD_AT`).
- Stamps the build with millisecond-precision id + ISO timestamp
  (`/app/.build-id`, `/app/.build-at`); both copied into the runner
  stage for `/api/build` to read.
- Writes `/app/.env.production.local` from declared ARGs *before*
  `next build` runs. Docker doesn't reliably promote ARGs into the
  Node child-process environment for the build command, so
  `process.env.NEXT_PUBLIC_*` references that previously inlined
  through Webpack's DefinePlugin sometimes fell back to runtime
  polyfill access (`tV.env.NEXT_PUBLIC_X`) that resolves to `undefined`
  in the browser. Writing the env file is the documented Next.js path
  and behaves the same on every host.

Default-off, no schema changes, no plugin changes, fully backwards-compatible
on operators who don't opt in.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-12 00:40:12 -04:00
Silvertakana e6305ca92f fix: resolve data engine deployment, optimize logs, update agent rules (#93)
* fix(docker): revert engine port mapping back to 5000:5000

* refactor(plugins): batch 2 legacy plugin migrations

- Migrated borders, camera, civil-unrest, and conflict-events to local-plugins/

- Refactored frontend routing to use this.context.getEngineUrl()

- Updated civil-unrest and conflict-events backend seeders to use strict container paths and ESM

- Renamed mismatched seeder directories (civilUnrest -> civil-unrest, conflictEvents -> conflict-events)

- Enhanced scripts/migrate-plugin.mjs to automatically fix mismatched seeder directory names

- Bumped workspace version to 2.7.4 (Patch)

* fix(plugins): restore civil-unrest and conflict-events

- Refactored fetch() to use this.context.getEngineUrl()

- Restored file structures damaged during initial replacement

- Bumped version to 2.7.5

* feat: batch 3 plugin migrations (cyber-attacks, maritime, wildfire)

* checkout: temporary commit for worktree checkout

* fix: resolve data engine deployment, optimize logs, update agent rules

* fix: resolve typescript compilation errors for Next.js build
2026-05-12 16:29:33 +12:00
a254784c3a Feat(UI/UX) Change buttons and fix Light mode buttons (#85)
* modified:   .gitignore
	modified:   AGENTS.md
	modified:   package.json
	modified:   public/cesium/ThirdParty/zip-module.wasm
	new file:   src/components/layout/ApiKeysTab.tsx
	modified:   src/components/layout/Header.tsx
	modified:   src/components/panels/DataConfig/OverlayTab.tsx
	modified:   src/components/panels/DataConfig/index.tsx
	modified:   src/components/panels/LayerPanel.tsx
	modified:   src/components/panels/PluginsTab.css
	modified:   src/components/panels/PluginsTab.tsx

* modified:   src/components/layout/Header.tsx
	new file:   src/components/layout/timeSelect.css

* UI-Testing

* UI-testing

* modified:   src/components/layout/timeSelect.css

* Fixed Duplicate Time Button

Caused by a merge conflict

* addeb back a </div>

---------

Co-authored-by: nullptr <[email protected]>
Co-authored-by: Silvertakana <[email protected]>
2026-05-12 01:49:21 +12:00
394 changed files with 357347 additions and 8240 deletions

No files matched your search

+9
View File
@@ -36,6 +36,15 @@ Application code is **identical** across editions — always calls `auth()`, `si
---
## Decentralized Plugin Authentication (Token Exchange)
For cross-domain plugin communication (Local App connecting to Marketplace or Data Engines), WWV uses an **Asymmetric JWT Token Exchange (PKCE)** model.
- **PKCE Flow**: The Local App uses standard OAuth 2.0 PKCE to exchange a `code` for a long-lived API key from the Marketplace.
- **Token Exchange**: The Local App exchanges its long-lived API key for a short-lived, audience-bound JWT before connecting to a decentralized Data Engine.
- **WebSocket First-Message Auth**: Data Engines enforce a "First-Message Auth" pattern over WebSockets. The client MUST immediately send a `{ type: "auth", v: 1, token: "..." }` message upon opening the connection to prevent SSRF vulnerabilities and credential leakage.
---
## Registration & Auto-Provisioning
Registration: `app.worldwideview.dev/register`
+16
View File
@@ -36,8 +36,20 @@ Frontend runs at `http://localhost:3000`.
- **Framework**: Vitest with jsdom environment
- **Coverage**: `src/lib/**`, `src/core/**`, `src/plugins/**`
- **Run**: `pnpm test` (or `vitest run`)
- **Type-Level Contract Testing**: When defining shared TypeScript interfaces/contracts (e.g., in `wwv-plugin-sdk`), you MUST use `expectTypeOf` from `vitest` to assert the actual contract shapes at compile-time instead of writing literal-value identity runtime assertions. You must also enforce `@ts-expect-error` negative type tests to ensure missing or invalid fields trigger compilation errors.
- **Key test files**: `rateLimit.test.ts`, `edition.test.ts`, `demoAdmin.test.ts`, `DeclarativePlugin.test.ts`, `cors.test.ts`, `repository.test.ts`, `marketplaceToken.test.ts`
### E2E Testing (Playwright)
WorldWideView uses Playwright for End-to-End browser testing to verify the complete UI integration including NextAuth.
- **Run**: `pnpm run test:e2e` (Ensure your database is running before executing)
- **Auth Bypass**: Playwright uses a `global.setup.ts` to directly insert a test user into the database and simulate a UI login to capture a `storageState.json` cookie file. This allows all tests to run pre-authenticated without bypassing production security logic.
- **Database Access in Setup**: Because Playwright setup scripts run in a vanilla Node context, importing Next.js backend utilities like `src/lib/db.ts` will crash with `next/headers` resolution errors. You MUST instantiate `PrismaClient` directly using `@prisma/adapter-pg` in any Playwright `setup` or `teardown` scripts.
- **Component Synchronization**: Do not use arbitrary timeouts. Tests MUST wait for the application boot sequence to finish by waiting for the `[data-testid="app-ready"]` marker on the `AppShell` to be attached.
- **Stable Selectors**: Always use `data-testid` properties (e.g. `data-testid="panel-toggle-left"`) for interacting with UI components in E2E tests rather than fragile CSS classes or dynamic text.
- **Worktree Port Conflicts**: If you are testing in a git worktree, configure `playwright.config.ts` to spawn the `webServer` on an alternative port (e.g. `3001`) to avoid colliding with other running worktrees.
## Security Headers
Configured in `next.config.ts` `headers()`:
@@ -247,12 +259,16 @@ The production Coolify instance runs on a local server at `192.168.68.64`. Direc
1. Push to `silvertakana/worldwideview`
2. Coolify webhook triggers → automated BuildKit deployment on the host server.
**GitHub Actions Workflows:**
- **Minimal Permissions:** CodeQL and GitHub security best practices dictate that `GITHUB_TOKEN` permissions should always be explicitly constrained. When creating or modifying `.github/workflows/*.yml` files, you MUST always include an explicit `permissions` block (e.g., `permissions: { contents: read }` or similar required scopes) at the top level or job level to adhere to the principle of least privilege.
**Coolify Deployment Rules & Gotchas (The "Redis 500k" Rule):**
- **Coolify Pulls from GitHub:** Coolify does *not* deploy your local workspace files. It strictly clones the repository from GitHub.
- **Push Before You Deploy:** If you make a critical infrastructure or optimization fix locally (e.g., swapping a massive `HSET` pipeline for a compressed 5-min throttled `setLiveSnapshot`), you **must** `git commit` and `git push` those changes to the `main` branch before triggering a Coolify deployment.
- **The "Rollback DDOS" Risk:** If you update environmental variables (like a new Upstash Redis URL) inside Coolify and deploy *without* pushing your local optimized code to GitHub, Coolify will pull the *older, unoptimized* repository code and aim it at your new endpoints. This will immediately exhaust the 500,000 requests/month Upstash free limit.
- **Redis Connection Errors:** If the data engine crashes on startup with `MaxRetriesPerRequestError: Reached the max retries per request limit (which is 3)` and `ECONNRESET`, the `REDIS_URL` in your Coolify environment variables is incorrectly using `redis://` instead of the TLS-required `rediss://`.
- **Immediate Rollback on Failure:** Whenever a Coolify deployment fails or introduces a critical bug (such as rate-limit exhaustion), **immediately select the last known working deployment in the Coolify dashboard and redeploy it.** Do not leave a broken deployment hanging while trying to debug directly in production. Roll back to stability first, debug locally, commit the verified fix, and then deploy forward.
- **Public Port Binding:** Both the `wwv` (Next.js frontend) and `wwv-data-engine` containers are meant to be directly exposed to the internet, rather than relying on Traefik internal network routing. They MUST use `ports: - "3000:3000"` and `ports: - "5000:5000"` respectively in `docker-compose.yml`. Using `expose` hides them behind internal Docker routing. Only backend infrastructure like `db` and `wwv-redis` should use `expose`.
**Local verification:**
Before pushing infrastructure changes, run `docker build -t wwv-test .` to verify that the Next.js `standalone` isolated multi-stage build works correctly.
+46
View File
@@ -0,0 +1,46 @@
---
trigger: testing
description: Guidelines and architecture for the Playwright End-to-End (E2E) testing infrastructure.
globs:
- "tests/**"
- "public/e2e-fixtures/**"
- "playwright.config.ts"
---
# E2E Testing Infrastructure
The WorldWideView project uses **Playwright** for end-to-end (E2E) testing. This document outlines the boundaries, architectural decisions, and conventions required when authoring or modifying E2E tests.
## 1. Resource Constraints & Parallelism
- **Worker Limits:** The Playwright configuration (`playwright.config.ts`) is strictly limited to a maximum of **2 workers**.
- **Reasoning:** Running Next.js compilation alongside a heavy Prisma connection pool inside multiple parallel test workers will lead to database connection exhaustion and resource contention on CI and local environments. Do not increase the worker count beyond 2 without careful consideration of the database pool limits.
## 2. Test User Lifecycle
E2E tests require an authenticated user to access core application features. We manage this test user securely to ensure the environment stays pristine:
- **Setup (`tests/global.setup.ts`):** Before tests run, the global setup script defensively deletes any existing test user (`[email protected]`) and creates a fresh instance with the `ADMIN` role. It then performs a UI login to capture the authentication state, which is shared across all test files.
- **Teardown (`tests/global.teardown.ts`):** After the entire test suite completes (regardless of success or failure), the global teardown script runs and completely purges the test user from the database.
- **Safety:** Do not use `[email protected]` for manual testing. It is volatile and intended exclusively for the CI test runner.
## 3. UI Interaction Conventions
- **Strict Selector Usage**: ALWAYS use `data-testid` selectors (e.g., `getByTestId('panel-toggle-left')`) for DOM interactions. **NEVER rely on CSS classes** (like `.layer-item__toggle` or `.dock-btn`) or deep DOM structures, as these are brittle and break during design refactors. If a `data-testid` is missing from an element you need to test, you MUST add it to the React component source code first.
- **Dialog Interception:** WorldWideView relies on heavy frontend plugin discovery and dynamic rendering. For example, the `UnverifiedPluginBatchDialog` may unexpectedly appear and block UI interactions in a fresh browser context.
- *Mitigation:* Ensure your test suites either explicitly handle these dialogs (e.g., in a `beforeEach` block) or inject mock plugins that bypass the verification flow entirely.
## 4. Mock Plugins
- **Data Interface Strictness**: WorldWideView's `PluginManager` is extremely strict. **ALL mock plugins MUST implement `getPollingInterval`, `fetch`, and `renderEntity`**, even if they are purely UI-focused (like a bottom-panel plugin). Failure to implement these will crash the renderer.
- **Capabilities**: If your mock UI plugin needs to be manually activated by the user, ensure its manifest includes the `"ui:sidebar"` capability so it appears in the layer list.
- **Injection:** We inject safe, deterministic mock plugins into the database during the `global.setup.ts` phase to test the core rendering pipeline without relying on external APIs. Ensure your mock plugin is registered in the setup and teardown cleanup arrays.
- **Location:** Mock plugin configurations and manifests are stored in `public/e2e-fixtures/`. When modifying the rendering engine, ensure you also update the mock plugin (`public/e2e-fixtures/mock-plugin.js`) to reflect any changes in the Plugin SDK capabilities.
## 5. Security Permissions
- When integrating Playwright tests into GitHub Actions (e.g., `.github/workflows/playwright.yml`), you MUST include an explicit permissions block (e.g., `permissions: { contents: read }`) to adhere to CodeQL and least-privilege scoping rules.
> [!WARNING]
> If you encounter flaky tests or `ECONNREFUSED` errors during Playwright runs, check if the local Next.js dev server or the Docker engine stack is competing for ports (e.g., port 5432 or 3000). Ensure `pnpm run dev:backends` is running and stable before launching the test suite.
## 6. Execution Environment & Tooling Limitations
The development environment operates on **Windows PowerShell**. Agents MUST adhere to the following strict execution rules when running commands or interacting with the testing infrastructure:
- **No Double Ampersands (`&&`)**: Standard Windows PowerShell (prior to v7) does not support the `&&` operator. Use the semicolon (`;`) to chain commands, or rely on `pnpm` package.json scripts which manage cross-platform execution properly.
- **Avoid Non-existent Database CLI Commands**: Do not attempt to use raw `pg` or `psql` commands unless explicitly verified to exist in the path. Instead, use Prisma (`npx prisma studio`, `npx prisma migrate`) or Docker executions (`docker exec -it wwv-db psql ...`) for database interactions.
- **Creating Test Plugins**: You will repeatedly need to create mock test plugins for verifying different platform features (e.g. bottom-panel resizing, new UI widgets). When doing so, you MUST review this `e2e-testing.md` file to ensure compliance with the `PluginManager` strict interface rules (implementing `getPollingInterval`, `fetch`, and `renderEntity`).
+2 -2
View File
@@ -14,7 +14,7 @@ If you are asked to implement a feature or fix a bug, you **MUST** determine the
| Repository Folder | Purpose & When to Edit |
|---|---|
| **`worldwideview`** | The main application, Next.js frontend, Cesium globe, UI components, and the core plugin SDK. **Edit here** for anything related to the UI, state, map rendering, or built-in plugin frontend configurations. |
| **`wwv-data-engine-public`** | The V2 Data Engine (Host Environment) that runs the seeders. **Edit here** if you need to modify how the engine parses WebSocket streams, implements REST APIs, downloads seeder bundles, or handles Redis caching. |
| **`wwv-data-engine`** | The V2 Data Engine (Host Environment) that runs the seeders. **Edit here** if you need to modify how the engine parses WebSocket streams, implements REST APIs, downloads seeder bundles, or handles Redis caching. |
| **`wwv-seeders`** | The community (public) data seeders. **Edit here** to create or modify background polling scripts (e.g. `aviation.ts`, `wildfire.ts`) that fetch data from public APIs and normalize it into `GeoEntity` arrays. |
| **`wwv-seeders-private`** | The proprietary (private) data seeders. **Edit here** for closed-source, premium, or high-security data ingestion scripts. |
| **`worldwideview-marketplace`** | The Plugin Marketplace web application. **Edit here** if you are modifying the platform where users browse, purchase, or manage plugin installations. |
@@ -29,6 +29,6 @@ If you are asked to implement a feature or fix a bug, you **MUST** determine the
- Write the frontend plugin interface inside `C:\dev\worldwideview` (or dynamically loaded via `worldwideview-plugins`).
2. **Fixing Real-Time Streaming Issues:**
- Investigate `C:\dev\worldwideview` (for frontend `WsClient` or `DataBus`).
- Investigate `C:\dev\wwv-data-engine-public` (for backend Fastify websocket broadcasting).
- Investigate `C:\dev\wwv-data-engine` (for backend Fastify websocket broadcasting).
3. **Pnpm Workspaces:**
- The Data Engine V2 automatically combines `community` and `private` seeders at runtime using pnpm workspaces. When testing seeders locally, refer to `C:\dev\worldwideview\local-seeders`.
+21 -20
View File
@@ -39,40 +39,41 @@ Plugins from the marketplace use the `bundle` format and are imported at runtime
---
## One-Click Install Flow (Redirect-Based)
## Token Exchange Authentication (PKCE)
**No static API keys.** Auth is entirely session-based. The marketplace redirects the browser to WWV, which checks the user's existing login session.
> [!IMPORTANT]
> **See ADR-0001 (Decentralized Plugin Authentication & SSRF Mitigation)**
> The PKCE flow and API key proxying MUST adhere to ADR-0001. This mandates AES-256-GCM encryption with 100,000 PBKDF2 iterations for all credentials stored in the `MarketplaceCredential` model, and DNS IP Pinning / stream chunking on all proxy requests to mitigate SSRF.
Auth is governed by an **Asymmetric JWT Token Exchange (PKCE)** model to eliminate static, long-lived, and un-rotatable tokens. The Marketplace acts as the OAuth Authorization Server.
```
User clicks "Install" on marketplace
Local App initiates connection
↓
Marketplace: check localStorage for instance URL
↓ (if not configured)
Show "Connect Your Instance" modal → user enters http://localhost:3000
Local App generates PKCE `code_verifier` and `code_challenge` (S256)
↓
MP → GET /api/marketplace/grant-token?redirectTo=...
Local App redirects to Marketplace: /oauth/authorize?response_type=code&...
↓
WWV: check Auth.js session → issue 30-day JWT → redirect ?token=<jwt>
User authenticates on Marketplace and grants access
↓
MP stores JWT in localStorage
Marketplace redirects back to Local App: /api/marketplace/callback?code=...
↓
MP: Navigate browser to /api/marketplace/install-redirect
?pluginId=borders&manifest=<b64>&redirectTo=...
Local App exchanges `code` + `code_verifier` for a long-lived API Key from Marketplace
↓
WWV: check session → upsertPlugin() → issueMarketplaceToken()
↓
Redirect back ?installed=borders&token=<jwt>
↓
MP: Show ✓ Installed — store JWT for future Manage-page calls
Local App securely stores API Key (encrypted via AES-256-GCM)
```
### Token Flow
### Data Engine Connections
After successful install, WWV issues a **30-day HS256 JWT** (`scope: "marketplace"`, signed with `AUTH_SECRET`). The marketplace stores it in `localStorage` and sends it as `Authorization: Bearer <token>` on all subsequent API calls.
Once connected, the Local App uses the long-lived API Key to fetch short-lived entitlements for specific data engines:
1. **Request JWT**: Local App requests a short-lived JWT for a specific Data Engine audience.
2. **Issue JWT**: Marketplace signs an EdDSA JWT containing the user's entitlements and the specific audience.
3. **Connect**: Local App connects to the decentralized Data Engine via WebSocket and performs "First-Message Auth" by sending the JWT.
4. **Verify**: Data Engine verifies the EdDSA signature using the Marketplace's public JWKS. No central DB lookup is required.
`validateMarketplaceAuth` accepts auth in priority order:
1. Active Auth.js session (same-origin browser)
2. Marketplace JWT Bearer token (cross-origin Manage page)
2. EdDSA JWT Bearer token (cross-origin, audience-bound)
3. Legacy static `WWV_BRIDGE_TOKEN` (backward compat)
---
@@ -181,7 +182,7 @@ tenants ─── installed_plugins
| Manifest tampering | Base64 only — no code execution. Validated via `validateManifest()` |
| Malicious plugins | Sandboxed in Web Workers (unverified). CSP headers. Code review (verified) |
| Supply chain | Plugin bundles checksummed (SHA-256), integrity verified |
| Plugin API keys | Stored encrypted in `settings` table, never exposed to browser |
| Plugin API keys | Stored encrypted (AES-256-GCM) in `MarketplaceCredential` model, never exposed to browser |
| Trust forgery | Registry signed with Ed25519 — `verified` cannot be claimed for unlisted plugins |
| Trust stamping | WWV always overwrites `manifest.trust` server-side — client/marketplace input ignored |
| Unverified consent | Warning dialog before any unverified plugin loads; approval stored per-plugin in `localStorage` |
+14 -4
View File
@@ -84,7 +84,7 @@ To rapidly iterate on plugins without polluting the core monorepo `packages/` di
1. **Create**: Run `node packages/wwv-cli/dist/index.js create <name> --local` to scaffold a full plugin structure inside the `local-plugins/` directory.
2. **Develop**: Running `pnpm dev` or `pnpm dev:all` automatically spawns the `dev:plugins` watcher. Any changes made to plugins in `local-plugins/` will be instantly rebuilt using Vite (externalizing large globals) and synced to `public/plugins-local/` for hot-reloading.
3. **Promote**: Once the plugin is stable and ready to be committed or published, run `node packages/wwv-cli/dist/index.js link <name>` to automatically move it to `packages/` and clean up the sandbox.
3. **Publish**: The `dev:plugins` script natively supports hot-reloading local plugins within the workspace. Once the plugin is ready, you can simply use `node packages/wwv-cli/dist/index.js publish <name> [--org <your-org>]` from the project root to publish it.
## When to Apply
When writing `fetch` implementations for plugins, or tracing why an entity dropped off the map. Ensure missing data correctly triggers the cache fallback via the DataBus. Ensure all new external plugins use the `bundle` format and exist in the marketplace registry to avoid CDN 404 hydrating crashes.
@@ -191,6 +191,16 @@ Plugins declare auth type in the manifest:
**Recommended for scale:** `token-exchange` — your server is never in the hot path for data requests.
### Shared Auth Contracts & Security Branding
> [!IMPORTANT]
> **See ADR-0001 (Decentralized Plugin Authentication & SSRF Mitigation)**
> All proxy and token-exchange implementations must strictly adhere to ADR-0001. This includes enforcing DNS IP Pinning and stream size limits via `undici.Agent` on all proxy routes to prevent SSRF, and encrypting all long-lived credentials (API keys) using AES-256-GCM with 100k PBKDF2 iterations before storing them in the `MarketplaceCredential` model.
When implementing the `token-exchange` flow or communicating across domains, all plugins and endpoints MUST use the strictly defined shared auth contracts from `@worldwideview/wwv-plugin-sdk/src/auth-contracts.ts` (e.g., `TokenExchangeRequest`, `WebSocketAuthMessage`).
**Sensitive Credentials**: All long-lived secrets (like API keys and tokens) MUST use the `SensitiveString` branded type and be instantiated via the `sensitive(str)` factory function. This prevents accidental logging/serialization and creates a single grep target for security audits.
---
## Plugin Development Workflows
@@ -382,9 +392,9 @@ While avoiding monolithic coupling, WWV plans to implement the following high-UX
## Local Plugins & Seeders
### 1. Local Sandbox Workflow
- **Scaffold**: `node packages/wwv-cli/dist/index.js create <name> --local` (creates inside `local-plugins/wwv-plugin-<name>`).
- **Develop**: `pnpm dev` triggers hot-reloading.
- **Promote**: `node packages/wwv-cli/dist/index.js link <name>` moves it to `packages/`.
- **Scaffold**: `node packages/wwv-cli/dist/index.js create` (creates inside `local-plugins/wwv-plugin-<name>` by default, or `packages/` if `--core` is used).
- **Develop**: `pnpm dev` triggers hot-reloading via `dev:plugins` watch script.
- **Publish**: `node packages/wwv-cli/dist/index.js publish <name> [--org <your-org>]` publishes the plugin from the project root. (No linking is needed; the sandbox natively functions in the workspace).
### 2. Data Engine Seeders
Seeders in `wwv-data-engine/src/seeders/` provide the backend data.
@@ -20,3 +20,13 @@ We employ a **structured logging strategy** in the frontend to reduce noise and
If a plugin fails to load in the Data Engine with a `404` or `Error [ERR_MODULE_NOT_FOUND]` error during startup:
- **Root Cause**: The seeder may exist simultaneously in both the `wwv-seeders-community` and `wwv-seeders-private` repositories.
- **Resolution**: Seeders must have a unique namespace. Ensure that private plugins (e.g., `aviation`, `maritime`, `military-aviation`) are explicitly removed from the community repository to prevent workspace-level module resolution failures when the V2 engine attempts to load them. Private seeders take priority, but overlapping names will cause dependency conflicts.
## 4. Agent Execution Environments (Windows/PowerShell)
When an AI agent or developer is executing CLI commands via tools in this project on Windows:
- **Rule**: NEVER use the `&&` operator to chain commands in PowerShell. It will result in parser errors.
- **Resolution**: Use sequential execution, run commands separately, or use PowerShell-safe chaining like `; if ($?) { }`.
## 5. Next.js vs. Undici Fetch Type Mismatches
When implementing SSRF protections (e.g., DNS IP Pinning via custom `Dispatcher`), you will likely use `undici.Agent`.
- **Root Cause**: Passing global Next.js `RequestInit` options directly into `undici.fetch` will throw strict TypeScript errors because the `BodyInit` and `RequestCache` signatures differ, and `dns.lookup` returns a generic `number` instead of exactly `4 | 6` for IP families.
- **Resolution**: Strip out Next.js specific keys, explicitly cast `body` as `any`, and explicitly type `resolvedFamily: number` when interfacing between Next.js request streams and the `undici` library.
+125
View File
@@ -0,0 +1,125 @@
param (
[Parameter(Mandatory=$true, HelpMessage="Base Git SHA")]
[string]$BaseSha,
[Parameter(Mandatory=$true, HelpMessage="Head Git SHA")]
[string]$HeadSha,
[Parameter(Mandatory=$false, HelpMessage="Path to ADR or Requirements File")]
[string]$RequirementsFile = "",
[Parameter(Mandatory=$false, HelpMessage="Brief description of the implemented changes")]
[string]$Description = "Implemented recent changes as per requirements.",
[Parameter(Mandatory=$false, HelpMessage="Model to use")]
[string]$Model = "gemini-2.5-flash"
)
# Set strict error handling
$ErrorActionPreference = "Stop"
Write-Host "Gathering Code Review Context..." -ForegroundColor Cyan
# Ensure local-scripts directory exists
$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
$WorkspaceRoot = Split-Path -Parent $ScriptDir
$TempPromptFile = Join-Path $ScriptDir "pr-review-prompt.txt"
# 1. Fetch Requirements text if provided
$RequirementsText = "No specific requirements provided."
if ($RequirementsFile -ne "") {
if (Test-Path $RequirementsFile) {
$RequirementsText = Get-Content $RequirementsFile -Raw
} else {
Write-Warning "Requirements file not found at $RequirementsFile"
}
}
# 2. Extract git diff
Write-Host "Extracting diff between $BaseSha and $HeadSha..." -ForegroundColor DarkGray
try {
$DiffOutput = git diff $BaseSha..$HeadSha
if ([string]::IsNullOrWhiteSpace($DiffOutput)) {
Write-Host "No changes found between $BaseSha and $HeadSha." -ForegroundColor Yellow
exit 0
}
} catch {
Write-Error "Failed to execute git diff. Ensure you are in a git repository and the SHAs are valid."
exit 1
}
# 3. Construct the Reviewer Prompt
$ReviewerPrompt = @"
You are a Senior Code Reviewer with expertise in software architecture, design patterns, and best practices. Your job is to review completed work against its plan or requirements and identify issues before they cascade. You are a sub-agent for Antigravity. Output ONLY the review format below. DO NOT use tools.
## What Was Implemented
$Description
## Requirements / Plan
$RequirementsText
## Git Range to Review
**Base:** $BaseSha
**Head:** $HeadSha
```diff
$DiffOutput
```
## What to Check
- **Plan alignment:** Does the implementation match the requirements? Are deviations justified?
- **Code quality:** Clean separation of concerns? Proper error handling?
- **Architecture:** Sound design decisions? Security concerns?
- **Testing:** Edge cases covered? Integration tests?
- **Production readiness:** Backward compatibility? Obvious bugs?
## Output Format
### Strengths
[What's well done? Be specific.]
### Issues
#### Critical (Must Fix)
[Bugs, security issues, data loss risks, broken functionality]
#### Important (Should Fix)
[Architecture problems, missing features, poor error handling, test gaps]
#### Minor (Nice to Have)
[Code style, optimization opportunities, documentation polish]
For each issue:
- File:line reference
- What's wrong
- Why it matters
- How to fix (if not obvious)
### Recommendations
[Improvements for code quality, architecture, or process]
### Assessment
**Ready to merge?** [Yes | No | With fixes]
**Reasoning:** [1-2 sentence technical assessment]
"@
# 4. Save to temporary file
$ReviewerPrompt | Set-Content $TempPromptFile -Encoding UTF8
# 5. Dispatch to Gemini CLI
Write-Host "Dispatching to Gemini CLI ($Model)..." -ForegroundColor Cyan
Write-Host "Depending on the diff size, this may take 30-60 seconds." -ForegroundColor DarkGray
$GeminiCmd = "$env:APPDATA\npm\gemini.cmd"
if (-Not (Test-Path $GeminiCmd)) {
Write-Error "Gemini CLI not found at $GeminiCmd. Ensure it is installed globally via npm."
exit 1
}
try {
Get-Content $TempPromptFile -Raw | & $GeminiCmd -m $Model
Write-Host "`nCode Review Complete." -ForegroundColor Green
} catch {
Write-Error "Failed to execute Gemini CLI. If you received a 429 RESOURCE_EXHAUSTED error, try changing the model (e.g. -Model gemini-1.5-pro)."
} finally {
# Clean up temp file
if (Test-Path $TempPromptFile) {
Remove-Item $TempPromptFile -Force
}
}
@@ -0,0 +1,22 @@
---
name: architectural-decision-records
description: Use when making significant architectural decisions, adding new patterns, or altering core workflows. This skill ensures that all major technical choices are documented via ADRs.
---
# Architectural Decision Records (ADRs)
## When to use this skill
Trigger this skill whenever you are about to introduce a new architectural pattern, make a load-bearing contract change, or bypass an established standard for a necessary reason.
## Guidelines
1. **Never make a major architectural change without an ADR**.
2. Store ADRs in `docs/architecture/decisions/` (create this directory if it doesn't exist).
3. Name ADR files using the pattern `adr-00##-name.md` (e.g., `adr-0001-use-postgresql.md`).
4. An ADR must follow this exact structure:
- **Title**: Short, descriptive noun phrase.
- **Context**: What is the problem we are solving? What forces are at play? (Be objective).
- **Decision**: What is the specific change or pattern we are adopting?
- **Consequences**: What becomes easier because of this change? What becomes harder? (There are always trade-offs).
## Actionable Steps
When tasked with a major architecture change, pause and explicitly state: *"This requires an Architectural Decision Record."* Draft the ADR and ask the user to approve the reasoning *before* writing the implementation code.
@@ -0,0 +1,72 @@
---
name: playwright-testing
description: Use when creating or modifying Playwright end-to-end tests, especially when injecting mock plugins to test platform features.
---
# Playwright E2E Testing & Mock Plugin Guide
When you are tasked with testing a new feature or plugin in WorldWideView using Playwright, you must use this skill to ensure you follow the strict architecture rules of the platform.
## 1. Mock Plugin Requirements
When creating a mock plugin (e.g., `public/e2e-fixtures/my-mock-plugin.js`) to test a specific UI feature, **you MUST implement the full Data Plugin interface**.
WorldWideView's `PluginManager` throws fatal runtime errors if a plugin is missing core data fetching methods, even if your plugin only tests a UI component like a bottom panel.
**Required Methods for EVERY Mock Plugin:**
```javascript
export default {
id: "my-mock-plugin",
name: "My Mock Plugin",
// ...
// MUST BE INCLUDED:
fetch: async (timeRange) => { return []; },
getPollingInterval: () => { return 60000; },
getLayerConfig: () => { return { color: "#FFF", clusterEnabled: false, clusterDistance: 50 }; },
renderEntity: (entity) => { return { type: "point", color: "#FFF", size: 5 }; },
// Your UI testing component (e.g., Sidebar, Bottom Panel)
getBottomPanelComponent: () => { ... }
}
```
## 2. Mock Plugin Manifest Capabilities
If your mock plugin needs to be toggled by the user in the Layers list, you MUST include the `"ui:sidebar"` capability in its `manifest.json`.
```json
{
"capabilities": ["ui:bottom-panel", "ui:sidebar"],
"extends": ["sidebar"]
}
```
## 3. Database Injection
The frontend reads plugins from the database. You MUST inject your mock plugin during `tests/global.setup.ts` using Prisma:
```typescript
const manifestPath = path.join(process.cwd(), 'public', 'e2e-fixtures', 'my-mock-plugin-manifest.json');
const manifestStr = fs.readFileSync(manifestPath, 'utf-8');
await prisma.installedPlugin.create({
data: {
pluginId: 'my-mock-plugin',
version: '1.0.0',
config: manifestStr,
enabled: true
}
});
```
Make sure to add your new mock plugin ID to the cleanup logic in `global.setup.ts` as well:
```typescript
await prisma.installedPlugin.deleteMany({
where: { pluginId: { in: ['e2e-mock-plugin', 'my-mock-plugin'] } }
});
```
## 4. UI Selectors
**DO NOT use CSS classes for your locators.** CSS classes change frequently (e.g., `.dock-btn`, `.layer-item__toggle`).
You MUST either:
1. Add explicit `data-testid` attributes to the React components you are testing.
2. Use stable accessibility locators (e.g., `getByRole('button', { name: /Text/ })`).
## 5. Execution Environment
- Ensure `pnpm run dev:backends` is running before you execute tests locally.
- Run tests using `pnpm run test:e2e` or `npx playwright test tests/your-test.spec.ts`.
- **PowerShell Warning**: Remember you are running in Windows PowerShell. Do NOT chain commands with `&&`. Use `;` instead.
@@ -1,64 +0,0 @@
---
name: supabase-postgres-best-practices
description: Postgres performance optimization and best practices from Supabase. Use this skill when writing, reviewing, or optimizing Postgres queries, schema designs, or database configurations.
license: MIT
metadata:
author: supabase
version: "1.1.1"
organization: Supabase
date: January 2026
abstract: Comprehensive Postgres performance optimization guide for developers using Supabase and Postgres. Contains performance rules across 8 categories, prioritized by impact from critical (query performance, connection management) to incremental (advanced features). Each rule includes detailed explanations, incorrect vs. correct SQL examples, query plan analysis, and specific performance metrics to guide automated optimization and code generation.
---
# Supabase Postgres Best Practices
Comprehensive performance optimization guide for Postgres, maintained by Supabase. Contains rules across 8 categories, prioritized by impact to guide automated query optimization and schema design.
## When to Apply
Reference these guidelines when:
- Writing SQL queries or designing schemas
- Implementing indexes or query optimization
- Reviewing database performance issues
- Configuring connection pooling or scaling
- Optimizing for Postgres-specific features
- Working with Row-Level Security (RLS)
## Rule Categories by Priority
| Priority | Category | Impact | Prefix |
|----------|----------|--------|--------|
| 1 | Query Performance | CRITICAL | `query-` |
| 2 | Connection Management | CRITICAL | `conn-` |
| 3 | Security & RLS | CRITICAL | `security-` |
| 4 | Schema Design | HIGH | `schema-` |
| 5 | Concurrency & Locking | MEDIUM-HIGH | `lock-` |
| 6 | Data Access Patterns | MEDIUM | `data-` |
| 7 | Monitoring & Diagnostics | LOW-MEDIUM | `monitor-` |
| 8 | Advanced Features | LOW | `advanced-` |
## How to Use
Read individual rule files for detailed explanations and SQL examples:
```
references/query-missing-indexes.md
references/query-partial-indexes.md
references/_sections.md
```
Each rule file contains:
- Brief explanation of why it matters
- Incorrect SQL example with explanation
- Correct SQL example with explanation
- Optional EXPLAIN output or metrics
- Additional context and references
- Supabase-specific notes (when applicable)
## References
- https://www.postgresql.org/docs/current/
- https://supabase.com/docs
- https://wiki.postgresql.org/wiki/Performance_Optimization
- https://supabase.com/docs/guides/database/overview
- https://supabase.com/docs/guides/auth/row-level-security
@@ -1,170 +0,0 @@
# Writing Guidelines for Postgres References
This document provides guidelines for creating effective Postgres best
practice references that work well with AI agents and LLMs.
## Key Principles
### 1. Concrete Transformation Patterns
Show exact SQL rewrites. Avoid philosophical advice.
**Good:** "Use `WHERE id = ANY(ARRAY[...])` instead of
`WHERE id IN (SELECT ...)`" **Bad:** "Design good schemas"
### 2. Error-First Structure
Always show the problematic pattern first, then the solution. This trains agents
to recognize anti-patterns.
```markdown
**Incorrect (sequential queries):** [bad example]
**Correct (batched query):** [good example]
```
### 3. Quantified Impact
Include specific metrics. Helps agents prioritize fixes.
**Good:** "10x faster queries", "50% smaller index", "Eliminates N+1"
**Bad:** "Faster", "Better", "More efficient"
### 4. Self-Contained Examples
Examples should be complete and runnable (or close to it). Include `CREATE TABLE`
if context is needed.
```sql
-- Include table definition when needed for clarity
CREATE TABLE users (
id bigint PRIMARY KEY,
email text NOT NULL,
deleted_at timestamptz
);
-- Now show the index
CREATE INDEX users_active_email_idx ON users(email) WHERE deleted_at IS NULL;
```
### 5. Semantic Naming
Use meaningful table/column names. Names carry intent for LLMs.
**Good:** `users`, `email`, `created_at`, `is_active`
**Bad:** `table1`, `col1`, `field`, `flag`
---
## Code Example Standards
### SQL Formatting
```sql
-- Use lowercase keywords, clear formatting
CREATE INDEX CONCURRENTLY users_email_idx
ON users(email)
WHERE deleted_at IS NULL;
-- Not cramped or ALL CAPS
CREATE INDEX CONCURRENTLY USERS_EMAIL_IDX ON USERS(EMAIL) WHERE DELETED_AT IS NULL;
```
### Comments
- Explain _why_, not _what_
- Highlight performance implications
- Point out common pitfalls
### Language Tags
- `sql` - Standard SQL queries
- `plpgsql` - Stored procedures/functions
- `typescript` - Application code (when needed)
- `python` - Application code (when needed)
---
## When to Include Application Code
**Default: SQL Only**
Most references should focus on pure SQL patterns. This keeps examples portable.
**Include Application Code When:**
- Connection pooling configuration
- Transaction management in application context
- ORM anti-patterns (N+1 in Prisma/TypeORM)
- Prepared statement usage
**Format for Mixed Examples:**
````markdown
**Incorrect (N+1 in application):**
```typescript
for (const user of users) {
const posts = await db.query("SELECT * FROM posts WHERE user_id = $1", [
user.id,
]);
}
```
````
**Correct (batch query):**
```typescript
const posts = await db.query("SELECT * FROM posts WHERE user_id = ANY($1)", [
userIds,
]);
```
---
## Impact Level Guidelines
| Level | Improvement | Use When |
|-------|-------------|----------|
| **CRITICAL** | 10-100x | Missing indexes, connection exhaustion, sequential scans on large tables |
| **HIGH** | 5-20x | Wrong index types, poor partitioning, missing covering indexes |
| **MEDIUM-HIGH** | 2-5x | N+1 queries, inefficient pagination, RLS optimization |
| **MEDIUM** | 1.5-3x | Redundant indexes, query plan instability |
| **LOW-MEDIUM** | 1.2-2x | VACUUM tuning, configuration tweaks |
| **LOW** | Incremental | Advanced patterns, edge cases |
---
## Reference Standards
**Primary Sources:**
- Official Postgres documentation
- Supabase documentation
- Postgres wiki
- Established blogs (2ndQuadrant, Crunchy Data)
**Format:**
```markdown
Reference:
[Postgres Indexes](https://www.postgresql.org/docs/current/indexes.html)
```
---
## Review Checklist
Before submitting a reference:
- [ ] Title is clear and action-oriented
- [ ] Impact level matches the performance gain
- [ ] impactDescription includes quantification
- [ ] Explanation is concise (1-2 sentences)
- [ ] Has at least 1 **Incorrect** SQL example
- [ ] Has at least 1 **Correct** SQL example
- [ ] SQL uses semantic naming
- [ ] Comments explain _why_, not _what_
- [ ] Trade-offs mentioned if applicable
- [ ] Reference links included
- [ ] `pnpm test` passes
@@ -1,39 +0,0 @@
# Section Definitions
This file defines the rule categories for Postgres best practices. Rules are automatically assigned to sections based on their filename prefix.
Take the examples below as pure demonstrative. Replace each section with the actual rule categories for Postgres best practices.
---
## 1. Query Performance (query)
**Impact:** CRITICAL
**Description:** Slow queries, missing indexes, inefficient query plans. The most common source of Postgres performance issues.
## 2. Connection Management (conn)
**Impact:** CRITICAL
**Description:** Connection pooling, limits, and serverless strategies. Critical for applications with high concurrency or serverless deployments.
## 3. Security & RLS (security)
**Impact:** CRITICAL
**Description:** Row-Level Security policies, privilege management, and authentication patterns.
## 4. Schema Design (schema)
**Impact:** HIGH
**Description:** Table design, index strategies, partitioning, and data type selection. Foundation for long-term performance.
## 5. Concurrency & Locking (lock)
**Impact:** MEDIUM-HIGH
**Description:** Transaction management, isolation levels, deadlock prevention, and lock contention patterns.
## 6. Data Access Patterns (data)
**Impact:** MEDIUM
**Description:** N+1 query elimination, batch operations, cursor-based pagination, and efficient data fetching.
## 7. Monitoring & Diagnostics (monitor)
**Impact:** LOW-MEDIUM
**Description:** Using pg_stat_statements, EXPLAIN ANALYZE, metrics collection, and performance diagnostics.
## 8. Advanced Features (advanced)
**Impact:** LOW
**Description:** Full-text search, JSONB optimization, PostGIS, extensions, and advanced Postgres features.
@@ -1,34 +0,0 @@
---
title: Clear, Action-Oriented Title (e.g., "Use Partial Indexes for Filtered Queries")
impact: MEDIUM
impactDescription: 5-20x query speedup for filtered queries
tags: indexes, query-optimization, performance
---
## [Rule Title]
[1-2 sentence explanation of the problem and why it matters. Focus on performance impact.]
**Incorrect (describe the problem):**
```sql
-- Comment explaining what makes this slow/problematic
CREATE INDEX users_email_idx ON users(email);
SELECT * FROM users WHERE email = '[email protected]' AND deleted_at IS NULL;
-- This scans deleted records unnecessarily
```
**Correct (describe the solution):**
```sql
-- Comment explaining why this is better
CREATE INDEX users_active_email_idx ON users(email) WHERE deleted_at IS NULL;
SELECT * FROM users WHERE email = '[email protected]' AND deleted_at IS NULL;
-- Only indexes active users, 10x smaller index, faster queries
```
[Optional: Additional context, edge cases, or trade-offs]
Reference: [Postgres Docs](https://www.postgresql.org/docs/current/)
@@ -1,55 +0,0 @@
---
title: Use tsvector for Full-Text Search
impact: MEDIUM
impactDescription: 100x faster than LIKE, with ranking support
tags: full-text-search, tsvector, gin, search
---
## Use tsvector for Full-Text Search
LIKE with wildcards can't use indexes. Full-text search with tsvector is orders of magnitude faster.
**Incorrect (LIKE pattern matching):**
```sql
-- Cannot use index, scans all rows
select * from articles where content like '%postgresql%';
-- Case-insensitive makes it worse
select * from articles where lower(content) like '%postgresql%';
```
**Correct (full-text search with tsvector):**
```sql
-- Add tsvector column and index
alter table articles add column search_vector tsvector
generated always as (to_tsvector('english', coalesce(title,'') || ' ' || coalesce(content,''))) stored;
create index articles_search_idx on articles using gin (search_vector);
-- Fast full-text search
select * from articles
where search_vector @@ to_tsquery('english', 'postgresql & performance');
-- With ranking
select *, ts_rank(search_vector, query) as rank
from articles, to_tsquery('english', 'postgresql') query
where search_vector @@ query
order by rank desc;
```
Search multiple terms:
```sql
-- AND: both terms required
to_tsquery('postgresql & performance')
-- OR: either term
to_tsquery('postgresql | mysql')
-- Prefix matching
to_tsquery('post:*')
```
Reference: [Full Text Search](https://supabase.com/docs/guides/database/full-text-search)
@@ -1,49 +0,0 @@
---
title: Index JSONB Columns for Efficient Querying
impact: MEDIUM
impactDescription: 10-100x faster JSONB queries with proper indexing
tags: jsonb, gin, indexes, json
---
## Index JSONB Columns for Efficient Querying
JSONB queries without indexes scan the entire table. Use GIN indexes for containment queries.
**Incorrect (no index on JSONB):**
```sql
create table products (
id bigint primary key,
attributes jsonb
);
-- Full table scan for every query
select * from products where attributes @> '{"color": "red"}';
select * from products where attributes->>'brand' = 'Nike';
```
**Correct (GIN index for JSONB):**
```sql
-- GIN index for containment operators (@>, ?, ?&, ?|)
create index products_attrs_gin on products using gin (attributes);
-- Now containment queries use the index
select * from products where attributes @> '{"color": "red"}';
-- For specific key lookups, use expression index
create index products_brand_idx on products ((attributes->>'brand'));
select * from products where attributes->>'brand' = 'Nike';
```
Choose the right operator class:
```sql
-- jsonb_ops (default): supports all operators, larger index
create index idx1 on products using gin (attributes);
-- jsonb_path_ops: only @> operator, but 2-3x smaller index
create index idx2 on products using gin (attributes jsonb_path_ops);
```
Reference: [JSONB Indexes](https://www.postgresql.org/docs/current/datatype-json.html#JSON-INDEXING)
@@ -1,46 +0,0 @@
---
title: Configure Idle Connection Timeouts
impact: HIGH
impactDescription: Reclaim 30-50% of connection slots from idle clients
tags: connections, timeout, idle, resource-management
---
## Configure Idle Connection Timeouts
Idle connections waste resources. Configure timeouts to automatically reclaim them.
**Incorrect (connections held indefinitely):**
```sql
-- No timeout configured
show idle_in_transaction_session_timeout; -- 0 (disabled)
-- Connections stay open forever, even when idle
select pid, state, state_change, query
from pg_stat_activity
where state = 'idle in transaction';
-- Shows transactions idle for hours, holding locks
```
**Correct (automatic cleanup of idle connections):**
```sql
-- Terminate connections idle in transaction after 30 seconds
alter system set idle_in_transaction_session_timeout = '30s';
-- Terminate completely idle connections after 10 minutes
alter system set idle_session_timeout = '10min';
-- Reload configuration
select pg_reload_conf();
```
For pooled connections, configure at the pooler level:
```ini
# pgbouncer.ini
server_idle_timeout = 60
client_idle_timeout = 300
```
Reference: [Connection Timeouts](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-IDLE-IN-TRANSACTION-SESSION-TIMEOUT)
@@ -1,44 +0,0 @@
---
title: Set Appropriate Connection Limits
impact: CRITICAL
impactDescription: Prevent database crashes and memory exhaustion
tags: connections, max-connections, limits, stability
---
## Set Appropriate Connection Limits
Too many connections exhaust memory and degrade performance. Set limits based on available resources.
**Incorrect (unlimited or excessive connections):**
```sql
-- Default max_connections = 100, but often increased blindly
show max_connections; -- 500 (way too high for 4GB RAM)
-- Each connection uses 1-3MB RAM
-- 500 connections * 2MB = 1GB just for connections!
-- Out of memory errors under load
```
**Correct (calculate based on resources):**
```sql
-- Formula: max_connections = (RAM in MB / 5MB per connection) - reserved
-- For 4GB RAM: (4096 / 5) - 10 = ~800 theoretical max
-- But practically, 100-200 is better for query performance
-- Recommended settings for 4GB RAM
alter system set max_connections = 100;
-- Also set work_mem appropriately
-- work_mem * max_connections should not exceed 25% of RAM
alter system set work_mem = '8MB'; -- 8MB * 100 = 800MB max
```
Monitor connection usage:
```sql
select count(*), state from pg_stat_activity group by state;
```
Reference: [Database Connections](https://supabase.com/docs/guides/platform/performance#connection-management)
@@ -1,41 +0,0 @@
---
title: Use Connection Pooling for All Applications
impact: CRITICAL
impactDescription: Handle 10-100x more concurrent users
tags: connection-pooling, pgbouncer, performance, scalability
---
## Use Connection Pooling for All Applications
Postgres connections are expensive (1-3MB RAM each). Without pooling, applications exhaust connections under load.
**Incorrect (new connection per request):**
```sql
-- Each request creates a new connection
-- Application code: db.connect() per request
-- Result: 500 concurrent users = 500 connections = crashed database
-- Check current connections
select count(*) from pg_stat_activity; -- 487 connections!
```
**Correct (connection pooling):**
```sql
-- Use a pooler like PgBouncer between app and database
-- Application connects to pooler, pooler reuses a small pool to Postgres
-- Configure pool_size based on: (CPU cores * 2) + spindle_count
-- Example for 4 cores: pool_size = 10
-- Result: 500 concurrent users share 10 actual connections
select count(*) from pg_stat_activity; -- 10 connections
```
Pool modes:
- **Transaction mode**: connection returned after each transaction (best for most apps)
- **Session mode**: connection held for entire session (needed for prepared statements, temp tables)
Reference: [Connection Pooling](https://supabase.com/docs/guides/database/connecting-to-postgres#connection-pooler)
@@ -1,46 +0,0 @@
---
title: Use Prepared Statements Correctly with Pooling
impact: HIGH
impactDescription: Avoid prepared statement conflicts in pooled environments
tags: prepared-statements, connection-pooling, transaction-mode
---
## Use Prepared Statements Correctly with Pooling
Prepared statements are tied to individual database connections. In transaction-mode pooling, connections are shared, causing conflicts.
**Incorrect (named prepared statements with transaction pooling):**
```sql
-- Named prepared statement
prepare get_user as select * from users where id = $1;
-- In transaction mode pooling, next request may get different connection
execute get_user(123);
-- ERROR: prepared statement "get_user" does not exist
```
**Correct (use unnamed statements or session mode):**
```sql
-- Option 1: Use unnamed prepared statements (most ORMs do this automatically)
-- The query is prepared and executed in a single protocol message
-- Option 2: Deallocate after use in transaction mode
prepare get_user as select * from users where id = $1;
execute get_user(123);
deallocate get_user;
-- Option 3: Use session mode pooling (port 5432 vs 6543)
-- Connection is held for entire session, prepared statements persist
```
Check your driver settings:
```sql
-- Many drivers use prepared statements by default
-- Node.js pg: { prepare: false } to disable
-- JDBC: prepareThreshold=0 to disable
```
Reference: [Prepared Statements with Pooling](https://supabase.com/docs/guides/database/connecting-to-postgres#connection-pool-modes)
@@ -1,54 +0,0 @@
---
title: Batch INSERT Statements for Bulk Data
impact: MEDIUM
impactDescription: 10-50x faster bulk inserts
tags: batch, insert, bulk, performance, copy
---
## Batch INSERT Statements for Bulk Data
Individual INSERT statements have high overhead. Batch multiple rows in single statements or use COPY.
**Incorrect (individual inserts):**
```sql
-- Each insert is a separate transaction and round trip
insert into events (user_id, action) values (1, 'click');
insert into events (user_id, action) values (1, 'view');
insert into events (user_id, action) values (2, 'click');
-- ... 1000 more individual inserts
-- 1000 inserts = 1000 round trips = slow
```
**Correct (batch insert):**
```sql
-- Multiple rows in single statement
insert into events (user_id, action) values
(1, 'click'),
(1, 'view'),
(2, 'click'),
-- ... up to ~1000 rows per batch
(999, 'view');
-- One round trip for 1000 rows
```
For large imports, use COPY:
```sql
-- COPY is fastest for bulk loading
copy events (user_id, action, created_at)
from '/path/to/data.csv'
with (format csv, header true);
-- Or from stdin in application
copy events (user_id, action) from stdin with (format csv);
1,click
1,view
2,click
\.
```
Reference: [COPY](https://www.postgresql.org/docs/current/sql-copy.html)
@@ -1,53 +0,0 @@
---
title: Eliminate N+1 Queries with Batch Loading
impact: MEDIUM-HIGH
impactDescription: 10-100x fewer database round trips
tags: n-plus-one, batch, performance, queries
---
## Eliminate N+1 Queries with Batch Loading
N+1 queries execute one query per item in a loop. Batch them into a single query using arrays or JOINs.
**Incorrect (N+1 queries):**
```sql
-- First query: get all users
select id from users where active = true; -- Returns 100 IDs
-- Then N queries, one per user
select * from orders where user_id = 1;
select * from orders where user_id = 2;
select * from orders where user_id = 3;
-- ... 97 more queries!
-- Total: 101 round trips to database
```
**Correct (single batch query):**
```sql
-- Collect IDs and query once with ANY
select * from orders where user_id = any(array[1, 2, 3, ...]);
-- Or use JOIN instead of loop
select u.id, u.name, o.*
from users u
left join orders o on o.user_id = u.id
where u.active = true;
-- Total: 1 round trip
```
Application pattern:
```sql
-- Instead of looping in application code:
-- for user in users: db.query("SELECT * FROM orders WHERE user_id = $1", user.id)
-- Pass array parameter:
select * from orders where user_id = any($1::bigint[]);
-- Application passes: [1, 2, 3, 4, 5, ...]
```
Reference: [N+1 Query Problem](https://supabase.com/docs/guides/database/query-optimization)
@@ -1,50 +0,0 @@
---
title: Use Cursor-Based Pagination Instead of OFFSET
impact: MEDIUM-HIGH
impactDescription: Consistent O(1) performance regardless of page depth
tags: pagination, cursor, keyset, offset, performance
---
## Use Cursor-Based Pagination Instead of OFFSET
OFFSET-based pagination scans all skipped rows, getting slower on deeper pages. Cursor pagination is O(1).
**Incorrect (OFFSET pagination):**
```sql
-- Page 1: scans 20 rows
select * from products order by id limit 20 offset 0;
-- Page 100: scans 2000 rows to skip 1980
select * from products order by id limit 20 offset 1980;
-- Page 10000: scans 200,000 rows!
select * from products order by id limit 20 offset 199980;
```
**Correct (cursor/keyset pagination):**
```sql
-- Page 1: get first 20
select * from products order by id limit 20;
-- Application stores last_id = 20
-- Page 2: start after last ID
select * from products where id > 20 order by id limit 20;
-- Uses index, always fast regardless of page depth
-- Page 10000: same speed as page 1
select * from products where id > 199980 order by id limit 20;
```
For multi-column sorting:
```sql
-- Cursor must include all sort columns
select * from products
where (created_at, id) > ('2024-01-15 10:00:00', 12345)
order by created_at, id
limit 20;
```
Reference: [Pagination](https://supabase.com/docs/guides/database/pagination)
@@ -1,50 +0,0 @@
---
title: Use UPSERT for Insert-or-Update Operations
impact: MEDIUM
impactDescription: Atomic operation, eliminates race conditions
tags: upsert, on-conflict, insert, update
---
## Use UPSERT for Insert-or-Update Operations
Using separate SELECT-then-INSERT/UPDATE creates race conditions. Use INSERT ... ON CONFLICT for atomic upserts.
**Incorrect (check-then-insert race condition):**
```sql
-- Race condition: two requests check simultaneously
select * from settings where user_id = 123 and key = 'theme';
-- Both find nothing
-- Both try to insert
insert into settings (user_id, key, value) values (123, 'theme', 'dark');
-- One succeeds, one fails with duplicate key error!
```
**Correct (atomic UPSERT):**
```sql
-- Single atomic operation
insert into settings (user_id, key, value)
values (123, 'theme', 'dark')
on conflict (user_id, key)
do update set value = excluded.value, updated_at = now();
-- Returns the inserted/updated row
insert into settings (user_id, key, value)
values (123, 'theme', 'dark')
on conflict (user_id, key)
do update set value = excluded.value
returning *;
```
Insert-or-ignore pattern:
```sql
-- Insert only if not exists (no update)
insert into page_views (page_id, user_id)
values (1, 123)
on conflict (page_id, user_id) do nothing;
```
Reference: [INSERT ON CONFLICT](https://www.postgresql.org/docs/current/sql-insert.html#SQL-ON-CONFLICT)
@@ -1,56 +0,0 @@
---
title: Use Advisory Locks for Application-Level Locking
impact: MEDIUM
impactDescription: Efficient coordination without row-level lock overhead
tags: advisory-locks, coordination, application-locks
---
## Use Advisory Locks for Application-Level Locking
Advisory locks provide application-level coordination without requiring database rows to lock.
**Incorrect (creating rows just for locking):**
```sql
-- Creating dummy rows to lock on
create table resource_locks (
resource_name text primary key
);
insert into resource_locks values ('report_generator');
-- Lock by selecting the row
select * from resource_locks where resource_name = 'report_generator' for update;
```
**Correct (advisory locks):**
```sql
-- Session-level advisory lock (released on disconnect or unlock)
select pg_advisory_lock(hashtext('report_generator'));
-- ... do exclusive work ...
select pg_advisory_unlock(hashtext('report_generator'));
-- Transaction-level lock (released on commit/rollback)
begin;
select pg_advisory_xact_lock(hashtext('daily_report'));
-- ... do work ...
commit; -- Lock automatically released
```
Try-lock for non-blocking operations:
```sql
-- Returns immediately with true/false instead of waiting
select pg_try_advisory_lock(hashtext('resource_name'));
-- Use in application
if (acquired) {
-- Do work
select pg_advisory_unlock(hashtext('resource_name'));
} else {
-- Skip or retry later
}
```
Reference: [Advisory Locks](https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS)
@@ -1,68 +0,0 @@
---
title: Prevent Deadlocks with Consistent Lock Ordering
impact: MEDIUM-HIGH
impactDescription: Eliminate deadlock errors, improve reliability
tags: deadlocks, locking, transactions, ordering
---
## Prevent Deadlocks with Consistent Lock Ordering
Deadlocks occur when transactions lock resources in different orders. Always
acquire locks in a consistent order.
**Incorrect (inconsistent lock ordering):**
```sql
-- Transaction A -- Transaction B
begin; begin;
update accounts update accounts
set balance = balance - 100 set balance = balance - 50
where id = 1; where id = 2; -- B locks row 2
update accounts update accounts
set balance = balance + 100 set balance = balance + 50
where id = 2; -- A waits for B where id = 1; -- B waits for A
-- DEADLOCK! Both waiting for each other
```
**Correct (lock rows in consistent order first):**
```sql
-- Explicitly acquire locks in ID order before updating
begin;
select * from accounts where id in (1, 2) order by id for update;
-- Now perform updates in any order - locks already held
update accounts set balance = balance - 100 where id = 1;
update accounts set balance = balance + 100 where id = 2;
commit;
```
Alternative: use a single statement to update atomically:
```sql
-- Single statement acquires all locks atomically
begin;
update accounts
set balance = balance + case id
when 1 then -100
when 2 then 100
end
where id in (1, 2);
commit;
```
Detect deadlocks in logs:
```sql
-- Check for recent deadlocks
select * from pg_stat_database where deadlocks > 0;
-- Enable deadlock logging
set log_lock_waits = on;
set deadlock_timeout = '1s';
```
Reference:
[Deadlocks](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-DEADLOCKS)
@@ -1,50 +0,0 @@
---
title: Keep Transactions Short to Reduce Lock Contention
impact: MEDIUM-HIGH
impactDescription: 3-5x throughput improvement, fewer deadlocks
tags: transactions, locking, contention, performance
---
## Keep Transactions Short to Reduce Lock Contention
Long-running transactions hold locks that block other queries. Keep transactions as short as possible.
**Incorrect (long transaction with external calls):**
```sql
begin;
select * from orders where id = 1 for update; -- Lock acquired
-- Application makes HTTP call to payment API (2-5 seconds)
-- Other queries on this row are blocked!
update orders set status = 'paid' where id = 1;
commit; -- Lock held for entire duration
```
**Correct (minimal transaction scope):**
```sql
-- Validate data and call APIs outside transaction
-- Application: response = await paymentAPI.charge(...)
-- Only hold lock for the actual update
begin;
update orders
set status = 'paid', payment_id = $1
where id = $2 and status = 'pending'
returning *;
commit; -- Lock held for milliseconds
```
Use `statement_timeout` to prevent runaway transactions:
```sql
-- Abort queries running longer than 30 seconds
set statement_timeout = '30s';
-- Or per-session
set local statement_timeout = '5s';
```
Reference: [Transaction Management](https://www.postgresql.org/docs/current/tutorial-transactions.html)
@@ -1,54 +0,0 @@
---
title: Use SKIP LOCKED for Non-Blocking Queue Processing
impact: MEDIUM-HIGH
impactDescription: 10x throughput for worker queues
tags: skip-locked, queue, workers, concurrency
---
## Use SKIP LOCKED for Non-Blocking Queue Processing
When multiple workers process a queue, SKIP LOCKED allows workers to process different rows without waiting.
**Incorrect (workers block each other):**
```sql
-- Worker 1 and Worker 2 both try to get next job
begin;
select * from jobs where status = 'pending' order by created_at limit 1 for update;
-- Worker 2 waits for Worker 1's lock to release!
```
**Correct (SKIP LOCKED for parallel processing):**
```sql
-- Each worker skips locked rows and gets the next available
begin;
select * from jobs
where status = 'pending'
order by created_at
limit 1
for update skip locked;
-- Worker 1 gets job 1, Worker 2 gets job 2 (no waiting)
update jobs set status = 'processing' where id = $1;
commit;
```
Complete queue pattern:
```sql
-- Atomic claim-and-update in one statement
update jobs
set status = 'processing', worker_id = $1, started_at = now()
where id = (
select id from jobs
where status = 'pending'
order by created_at
limit 1
for update skip locked
)
returning *;
```
Reference: [SELECT FOR UPDATE SKIP LOCKED](https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE)
@@ -1,45 +0,0 @@
---
title: Use EXPLAIN ANALYZE to Diagnose Slow Queries
impact: LOW-MEDIUM
impactDescription: Identify exact bottlenecks in query execution
tags: explain, analyze, diagnostics, query-plan
---
## Use EXPLAIN ANALYZE to Diagnose Slow Queries
EXPLAIN ANALYZE executes the query and shows actual timings, revealing the true performance bottlenecks.
**Incorrect (guessing at performance issues):**
```sql
-- Query is slow, but why?
select * from orders where customer_id = 123 and status = 'pending';
-- "It must be missing an index" - but which one?
```
**Correct (use EXPLAIN ANALYZE):**
```sql
explain (analyze, buffers, format text)
select * from orders where customer_id = 123 and status = 'pending';
-- Output reveals the issue:
-- Seq Scan on orders (cost=0.00..25000.00 rows=50 width=100) (actual time=0.015..450.123 rows=50 loops=1)
-- Filter: ((customer_id = 123) AND (status = 'pending'::text))
-- Rows Removed by Filter: 999950
-- Buffers: shared hit=5000 read=15000
-- Planning Time: 0.150 ms
-- Execution Time: 450.500 ms
```
Key things to look for:
```sql
-- Seq Scan on large tables = missing index
-- Rows Removed by Filter = poor selectivity or missing index
-- Buffers: read >> hit = data not cached, needs more memory
-- Nested Loop with high loops = consider different join strategy
-- Sort Method: external merge = work_mem too low
```
Reference: [EXPLAIN](https://supabase.com/docs/guides/database/inspect)
@@ -1,55 +0,0 @@
---
title: Enable pg_stat_statements for Query Analysis
impact: LOW-MEDIUM
impactDescription: Identify top resource-consuming queries
tags: pg-stat-statements, monitoring, statistics, performance
---
## Enable pg_stat_statements for Query Analysis
pg_stat_statements tracks execution statistics for all queries, helping identify slow and frequent queries.
**Incorrect (no visibility into query patterns):**
```sql
-- Database is slow, but which queries are the problem?
-- No way to know without pg_stat_statements
```
**Correct (enable and query pg_stat_statements):**
```sql
-- Enable the extension
create extension if not exists pg_stat_statements;
-- Find slowest queries by total time
select
calls,
round(total_exec_time::numeric, 2) as total_time_ms,
round(mean_exec_time::numeric, 2) as mean_time_ms,
query
from pg_stat_statements
order by total_exec_time desc
limit 10;
-- Find most frequent queries
select calls, query
from pg_stat_statements
order by calls desc
limit 10;
-- Reset statistics after optimization
select pg_stat_statements_reset();
```
Key metrics to monitor:
```sql
-- Queries with high mean time (candidates for optimization)
select query, mean_exec_time, calls
from pg_stat_statements
where mean_exec_time > 100 -- > 100ms average
order by mean_exec_time desc;
```
Reference: [pg_stat_statements](https://supabase.com/docs/guides/database/extensions/pg_stat_statements)
@@ -1,55 +0,0 @@
---
title: Maintain Table Statistics with VACUUM and ANALYZE
impact: MEDIUM
impactDescription: 2-10x better query plans with accurate statistics
tags: vacuum, analyze, statistics, maintenance, autovacuum
---
## Maintain Table Statistics with VACUUM and ANALYZE
Outdated statistics cause the query planner to make poor decisions. VACUUM reclaims space, ANALYZE updates statistics.
**Incorrect (stale statistics):**
```sql
-- Table has 1M rows but stats say 1000
-- Query planner chooses wrong strategy
explain select * from orders where status = 'pending';
-- Shows: Seq Scan (because stats show small table)
-- Actually: Index Scan would be much faster
```
**Correct (maintain fresh statistics):**
```sql
-- Manually analyze after large data changes
analyze orders;
-- Analyze specific columns used in WHERE clauses
analyze orders (status, created_at);
-- Check when tables were last analyzed
select
relname,
last_vacuum,
last_autovacuum,
last_analyze,
last_autoanalyze
from pg_stat_user_tables
order by last_analyze nulls first;
```
Autovacuum tuning for busy tables:
```sql
-- Increase frequency for high-churn tables
alter table orders set (
autovacuum_vacuum_scale_factor = 0.05, -- Vacuum at 5% dead tuples (default 20%)
autovacuum_analyze_scale_factor = 0.02 -- Analyze at 2% changes (default 10%)
);
-- Check autovacuum status
select * from pg_stat_progress_vacuum;
```
Reference: [VACUUM](https://supabase.com/docs/guides/database/database-size#vacuum-operations)
@@ -1,44 +0,0 @@
---
title: Create Composite Indexes for Multi-Column Queries
impact: HIGH
impactDescription: 5-10x faster multi-column queries
tags: indexes, composite-index, multi-column, query-optimization
---
## Create Composite Indexes for Multi-Column Queries
When queries filter on multiple columns, a composite index is more efficient than separate single-column indexes.
**Incorrect (separate indexes require bitmap scan):**
```sql
-- Two separate indexes
create index orders_status_idx on orders (status);
create index orders_created_idx on orders (created_at);
-- Query must combine both indexes (slower)
select * from orders where status = 'pending' and created_at > '2024-01-01';
```
**Correct (composite index):**
```sql
-- Single composite index (leftmost column first for equality checks)
create index orders_status_created_idx on orders (status, created_at);
-- Query uses one efficient index scan
select * from orders where status = 'pending' and created_at > '2024-01-01';
```
**Column order matters** - place equality columns first, range columns last:
```sql
-- Good: status (=) before created_at (>)
create index idx on orders (status, created_at);
-- Works for: WHERE status = 'pending'
-- Works for: WHERE status = 'pending' AND created_at > '2024-01-01'
-- Does NOT work for: WHERE created_at > '2024-01-01' (leftmost prefix rule)
```
Reference: [Multicolumn Indexes](https://www.postgresql.org/docs/current/indexes-multicolumn.html)
@@ -1,40 +0,0 @@
---
title: Use Covering Indexes to Avoid Table Lookups
impact: MEDIUM-HIGH
impactDescription: 2-5x faster queries by eliminating heap fetches
tags: indexes, covering-index, include, index-only-scan
---
## Use Covering Indexes to Avoid Table Lookups
Covering indexes include all columns needed by a query, enabling index-only scans that skip the table entirely.
**Incorrect (index scan + heap fetch):**
```sql
create index users_email_idx on users (email);
-- Must fetch name and created_at from table heap
select email, name, created_at from users where email = '[email protected]';
```
**Correct (index-only scan with INCLUDE):**
```sql
-- Include non-searchable columns in the index
create index users_email_idx on users (email) include (name, created_at);
-- All columns served from index, no table access needed
select email, name, created_at from users where email = '[email protected]';
```
Use INCLUDE for columns you SELECT but don't filter on:
```sql
-- Searching by status, but also need customer_id and total
create index orders_status_idx on orders (status) include (customer_id, total);
select status, customer_id, total from orders where status = 'shipped';
```
Reference: [Index-Only Scans](https://www.postgresql.org/docs/current/indexes-index-only-scans.html)
@@ -1,48 +0,0 @@
---
title: Choose the Right Index Type for Your Data
impact: HIGH
impactDescription: 10-100x improvement with correct index type
tags: indexes, btree, gin, gist, brin, hash, index-types
---
## Choose the Right Index Type for Your Data
Different index types excel at different query patterns. The default B-tree isn't always optimal.
**Incorrect (B-tree for JSONB containment):**
```sql
-- B-tree cannot optimize containment operators
create index products_attrs_idx on products (attributes);
select * from products where attributes @> '{"color": "red"}';
-- Full table scan - B-tree doesn't support @> operator
```
**Correct (GIN for JSONB):**
```sql
-- GIN supports @>, ?, ?&, ?| operators
create index products_attrs_idx on products using gin (attributes);
select * from products where attributes @> '{"color": "red"}';
```
Index type guide:
```sql
-- B-tree (default): =, <, >, BETWEEN, IN, IS NULL
create index users_created_idx on users (created_at);
-- GIN: arrays, JSONB, full-text search
create index posts_tags_idx on posts using gin (tags);
-- GiST: geometric data, range types, nearest-neighbor (KNN) queries
create index locations_idx on places using gist (location);
-- BRIN: large time-series tables (10-100x smaller)
create index events_time_idx on events using brin (created_at);
-- Hash: equality-only (slightly faster than B-tree for =)
create index sessions_token_idx on sessions using hash (token);
```
Reference: [Index Types](https://www.postgresql.org/docs/current/indexes-types.html)
@@ -1,43 +0,0 @@
---
title: Add Indexes on WHERE and JOIN Columns
impact: CRITICAL
impactDescription: 100-1000x faster queries on large tables
tags: indexes, performance, sequential-scan, query-optimization
---
## Add Indexes on WHERE and JOIN Columns
Queries filtering or joining on unindexed columns cause full table scans, which become exponentially slower as tables grow.
**Incorrect (sequential scan on large table):**
```sql
-- No index on customer_id causes full table scan
select * from orders where customer_id = 123;
-- EXPLAIN shows: Seq Scan on orders (cost=0.00..25000.00 rows=100 width=85)
```
**Correct (index scan):**
```sql
-- Create index on frequently filtered column
create index orders_customer_id_idx on orders (customer_id);
select * from orders where customer_id = 123;
-- EXPLAIN shows: Index Scan using orders_customer_id_idx (cost=0.42..8.44 rows=100 width=85)
```
For JOIN columns, always index the foreign key side:
```sql
-- Index the referencing column
create index orders_customer_id_idx on orders (customer_id);
select c.name, o.total
from customers c
join orders o on o.customer_id = c.id;
```
Reference: [Query Optimization](https://supabase.com/docs/guides/database/query-optimization)
@@ -1,45 +0,0 @@
---
title: Use Partial Indexes for Filtered Queries
impact: HIGH
impactDescription: 5-20x smaller indexes, faster writes and queries
tags: indexes, partial-index, query-optimization, storage
---
## Use Partial Indexes for Filtered Queries
Partial indexes only include rows matching a WHERE condition, making them smaller and faster when queries consistently filter on the same condition.
**Incorrect (full index includes irrelevant rows):**
```sql
-- Index includes all rows, even soft-deleted ones
create index users_email_idx on users (email);
-- Query always filters active users
select * from users where email = '[email protected]' and deleted_at is null;
```
**Correct (partial index matches query filter):**
```sql
-- Index only includes active users
create index users_active_email_idx on users (email)
where deleted_at is null;
-- Query uses the smaller, faster index
select * from users where email = '[email protected]' and deleted_at is null;
```
Common use cases for partial indexes:
```sql
-- Only pending orders (status rarely changes once completed)
create index orders_pending_idx on orders (created_at)
where status = 'pending';
-- Only non-null values
create index products_sku_idx on products (sku)
where sku is not null;
```
Reference: [Partial Indexes](https://www.postgresql.org/docs/current/indexes-partial.html)
@@ -1,80 +0,0 @@
---
title: Add Constraints Safely in Migrations
impact: HIGH
impactDescription: Prevents migration failures and enables idempotent schema changes
tags: constraints, migrations, schema, alter-table
---
## Add Constraints Safely in Migrations
PostgreSQL does not support `ADD CONSTRAINT IF NOT EXISTS`. Migrations using this syntax will fail.
**Incorrect (causes syntax error):**
```sql
-- ERROR: syntax error at or near "not" (SQLSTATE 42601)
alter table public.profiles
add constraint if not exists profiles_birthchart_id_unique unique (birthchart_id);
```
**Correct (idempotent constraint creation):**
```sql
-- Use DO block to check before adding
do $$
begin
if not exists (
select 1 from pg_constraint
where conname = 'profiles_birthchart_id_unique'
and conrelid = 'public.profiles'::regclass
) then
alter table public.profiles
add constraint profiles_birthchart_id_unique unique (birthchart_id);
end if;
end $$;
```
For all constraint types:
```sql
-- Check constraints
do $$
begin
if not exists (
select 1 from pg_constraint
where conname = 'check_age_positive'
) then
alter table users add constraint check_age_positive check (age > 0);
end if;
end $$;
-- Foreign keys
do $$
begin
if not exists (
select 1 from pg_constraint
where conname = 'profiles_birthchart_id_fkey'
) then
alter table profiles
add constraint profiles_birthchart_id_fkey
foreign key (birthchart_id) references birthcharts(id);
end if;
end $$;
```
Check if constraint exists:
```sql
-- Query to check constraint existence
select conname, contype, pg_get_constraintdef(oid)
from pg_constraint
where conrelid = 'public.profiles'::regclass;
-- contype values:
-- 'p' = PRIMARY KEY
-- 'f' = FOREIGN KEY
-- 'u' = UNIQUE
-- 'c' = CHECK
```
Reference: [Constraints](https://www.postgresql.org/docs/current/ddl-constraints.html)
@@ -1,46 +0,0 @@
---
title: Choose Appropriate Data Types
impact: HIGH
impactDescription: 50% storage reduction, faster comparisons
tags: data-types, schema, storage, performance
---
## Choose Appropriate Data Types
Using the right data types reduces storage, improves query performance, and prevents bugs.
**Incorrect (wrong data types):**
```sql
create table users (
id int, -- Will overflow at 2.1 billion
email varchar(255), -- Unnecessary length limit
created_at timestamp, -- Missing timezone info
is_active varchar(5), -- String for boolean
price varchar(20) -- String for numeric
);
```
**Correct (appropriate data types):**
```sql
create table users (
id bigint generated always as identity primary key, -- 9 quintillion max
email text, -- No artificial limit, same performance as varchar
created_at timestamptz, -- Always store timezone-aware timestamps
is_active boolean default true, -- 1 byte vs variable string length
price numeric(10,2) -- Exact decimal arithmetic
);
```
Key guidelines:
```sql
-- IDs: use bigint, not int (future-proofing)
-- Strings: use text, not varchar(n) unless constraint needed
-- Time: use timestamptz, not timestamp
-- Money: use numeric, not float (precision matters)
-- Enums: use text with check constraint or create enum type
```
Reference: [Data Types](https://www.postgresql.org/docs/current/datatype.html)
@@ -1,59 +0,0 @@
---
title: Index Foreign Key Columns
impact: HIGH
impactDescription: 10-100x faster JOINs and CASCADE operations
tags: foreign-key, indexes, joins, schema
---
## Index Foreign Key Columns
Postgres does not automatically index foreign key columns. Missing indexes cause slow JOINs and CASCADE operations.
**Incorrect (unindexed foreign key):**
```sql
create table orders (
id bigint generated always as identity primary key,
customer_id bigint references customers(id) on delete cascade,
total numeric(10,2)
);
-- No index on customer_id!
-- JOINs and ON DELETE CASCADE both require full table scan
select * from orders where customer_id = 123; -- Seq Scan
delete from customers where id = 123; -- Locks table, scans all orders
```
**Correct (indexed foreign key):**
```sql
create table orders (
id bigint generated always as identity primary key,
customer_id bigint references customers(id) on delete cascade,
total numeric(10,2)
);
-- Always index the FK column
create index orders_customer_id_idx on orders (customer_id);
-- Now JOINs and cascades are fast
select * from orders where customer_id = 123; -- Index Scan
delete from customers where id = 123; -- Uses index, fast cascade
```
Find missing FK indexes:
```sql
select
conrelid::regclass as table_name,
a.attname as fk_column
from pg_constraint c
join pg_attribute a on a.attrelid = c.conrelid and a.attnum = any(c.conkey)
where c.contype = 'f'
and not exists (
select 1 from pg_index i
where i.indrelid = c.conrelid and a.attnum = any(i.indkey)
);
```
Reference: [Foreign Keys](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-FK)
@@ -1,55 +0,0 @@
---
title: Use Lowercase Identifiers for Compatibility
impact: MEDIUM
impactDescription: Avoid case-sensitivity bugs with tools, ORMs, and AI assistants
tags: naming, identifiers, case-sensitivity, schema, conventions
---
## Use Lowercase Identifiers for Compatibility
PostgreSQL folds unquoted identifiers to lowercase. Quoted mixed-case identifiers require quotes forever and cause issues with tools, ORMs, and AI assistants that may not recognize them.
**Incorrect (mixed-case identifiers):**
```sql
-- Quoted identifiers preserve case but require quotes everywhere
CREATE TABLE "Users" (
"userId" bigint PRIMARY KEY,
"firstName" text,
"lastName" text
);
-- Must always quote or queries fail
SELECT "firstName" FROM "Users" WHERE "userId" = 1;
-- This fails - Users becomes users without quotes
SELECT firstName FROM Users;
-- ERROR: relation "users" does not exist
```
**Correct (lowercase snake_case):**
```sql
-- Unquoted lowercase identifiers are portable and tool-friendly
CREATE TABLE users (
user_id bigint PRIMARY KEY,
first_name text,
last_name text
);
-- Works without quotes, recognized by all tools
SELECT first_name FROM users WHERE user_id = 1;
```
Common sources of mixed-case identifiers:
```sql
-- ORMs often generate quoted camelCase - configure them to use snake_case
-- Migrations from other databases may preserve original casing
-- Some GUI tools quote identifiers by default - disable this
-- If stuck with mixed-case, create views as a compatibility layer
CREATE VIEW users AS SELECT "userId" AS user_id, "firstName" AS first_name FROM "Users";
```
Reference: [Identifiers and Key Words](https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS)
@@ -1,55 +0,0 @@
---
title: Partition Large Tables for Better Performance
impact: MEDIUM-HIGH
impactDescription: 5-20x faster queries and maintenance on large tables
tags: partitioning, large-tables, time-series, performance
---
## Partition Large Tables for Better Performance
Partitioning splits a large table into smaller pieces, improving query performance and maintenance operations.
**Incorrect (single large table):**
```sql
create table events (
id bigint generated always as identity,
created_at timestamptz,
data jsonb
);
-- 500M rows, queries scan everything
select * from events where created_at > '2024-01-01'; -- Slow
vacuum events; -- Takes hours, locks table
```
**Correct (partitioned by time range):**
```sql
create table events (
id bigint generated always as identity,
created_at timestamptz not null,
data jsonb
) partition by range (created_at);
-- Create partitions for each month
create table events_2024_01 partition of events
for values from ('2024-01-01') to ('2024-02-01');
create table events_2024_02 partition of events
for values from ('2024-02-01') to ('2024-03-01');
-- Queries only scan relevant partitions
select * from events where created_at > '2024-01-15'; -- Only scans events_2024_01+
-- Drop old data instantly
drop table events_2023_01; -- Instant vs DELETE taking hours
```
When to partition:
- Tables > 100M rows
- Time-series data with date-based queries
- Need to efficiently drop old data
Reference: [Table Partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html)
@@ -1,61 +0,0 @@
---
title: Select Optimal Primary Key Strategy
impact: HIGH
impactDescription: Better index locality, reduced fragmentation
tags: primary-key, identity, uuid, serial, schema
---
## Select Optimal Primary Key Strategy
Primary key choice affects insert performance, index size, and replication
efficiency.
**Incorrect (problematic PK choices):**
```sql
-- identity is the SQL-standard approach
create table users (
id serial primary key -- Works, but IDENTITY is recommended
);
-- Random UUIDs (v4) cause index fragmentation
create table orders (
id uuid default gen_random_uuid() primary key -- UUIDv4 = random = scattered inserts
);
```
**Correct (optimal PK strategies):**
```sql
-- Use IDENTITY for sequential IDs (SQL-standard, best for most cases)
create table users (
id bigint generated always as identity primary key
);
-- For distributed systems needing UUIDs, use UUIDv7 (time-ordered)
-- Requires pg_uuidv7 extension: create extension pg_uuidv7;
create table orders (
id uuid default uuid_generate_v7() primary key -- Time-ordered, no fragmentation
);
-- Alternative: time-prefixed IDs for sortable, distributed IDs (no extension needed)
create table events (
id text default concat(
to_char(now() at time zone 'utc', 'YYYYMMDDHH24MISSMS'),
gen_random_uuid()::text
) primary key
);
```
Guidelines:
- Single database: `bigint identity` (sequential, 8 bytes, SQL-standard)
- Distributed/exposed IDs: UUIDv7 (requires pg_uuidv7) or ULID (time-ordered, no
fragmentation)
- `serial` works but `identity` is SQL-standard and preferred for new
applications
- Avoid random UUIDs (v4) as primary keys on large tables (causes index
fragmentation)
Reference:
[Identity Columns](https://www.postgresql.org/docs/current/sql-createtable.html#SQL-CREATETABLE-PARMS-GENERATED-IDENTITY)
@@ -1,54 +0,0 @@
---
title: Apply Principle of Least Privilege
impact: MEDIUM
impactDescription: Reduced attack surface, better audit trail
tags: privileges, security, roles, permissions
---
## Apply Principle of Least Privilege
Grant only the minimum permissions required. Never use superuser for application queries.
**Incorrect (overly broad permissions):**
```sql
-- Application uses superuser connection
-- Or grants ALL to application role
grant all privileges on all tables in schema public to app_user;
grant all privileges on all sequences in schema public to app_user;
-- Any SQL injection becomes catastrophic
-- drop table users; cascades to everything
```
**Correct (minimal, specific grants):**
```sql
-- Create role with no default privileges
create role app_readonly nologin;
-- Grant only SELECT on specific tables
grant usage on schema public to app_readonly;
grant select on public.products, public.categories to app_readonly;
-- Create role for writes with limited scope
create role app_writer nologin;
grant usage on schema public to app_writer;
grant select, insert, update on public.orders to app_writer;
grant usage on sequence orders_id_seq to app_writer;
-- No DELETE permission
-- Login role inherits from these
create role app_user login password 'xxx';
grant app_writer to app_user;
```
Revoke public defaults:
```sql
-- Revoke default public access
revoke all on schema public from public;
revoke all on all tables in schema public from public;
```
Reference: [Roles and Privileges](https://supabase.com/blog/postgres-roles-and-privileges)
@@ -1,50 +0,0 @@
---
title: Enable Row Level Security for Multi-Tenant Data
impact: CRITICAL
impactDescription: Database-enforced tenant isolation, prevent data leaks
tags: rls, row-level-security, multi-tenant, security
---
## Enable Row Level Security for Multi-Tenant Data
Row Level Security (RLS) enforces data access at the database level, ensuring users only see their own data.
**Incorrect (application-level filtering only):**
```sql
-- Relying only on application to filter
select * from orders where user_id = $current_user_id;
-- Bug or bypass means all data is exposed!
select * from orders; -- Returns ALL orders
```
**Correct (database-enforced RLS):**
```sql
-- Enable RLS on the table
alter table orders enable row level security;
-- Create policy for users to see only their orders
create policy orders_user_policy on orders
for all
using (user_id = current_setting('app.current_user_id')::bigint);
-- Force RLS even for table owners
alter table orders force row level security;
-- Set user context and query
set app.current_user_id = '123';
select * from orders; -- Only returns orders for user 123
```
Policy for authenticated role:
```sql
create policy orders_user_policy on orders
for all
to authenticated
using (user_id = auth.uid());
```
Reference: [Row Level Security](https://supabase.com/docs/guides/database/postgres/row-level-security)
@@ -1,57 +0,0 @@
---
title: Optimize RLS Policies for Performance
impact: HIGH
impactDescription: 5-10x faster RLS queries with proper patterns
tags: rls, performance, security, optimization
---
## Optimize RLS Policies for Performance
Poorly written RLS policies can cause severe performance issues. Use subqueries and indexes strategically.
**Incorrect (function called for every row):**
```sql
create policy orders_policy on orders
using (auth.uid() = user_id); -- auth.uid() called per row!
-- With 1M rows, auth.uid() is called 1M times
```
**Correct (wrap functions in SELECT):**
```sql
create policy orders_policy on orders
using ((select auth.uid()) = user_id); -- Called once, cached
-- 100x+ faster on large tables
```
Use security definer functions for complex checks:
```sql
-- Create helper function (runs as definer, bypasses RLS)
create or replace function is_team_member(team_id bigint)
returns boolean
language sql
security definer
set search_path = ''
as $$
select exists (
select 1 from public.team_members
where team_id = $1 and user_id = (select auth.uid())
);
$$;
-- Use in policy (indexed lookup, not per-row check)
create policy team_orders_policy on orders
using ((select is_team_member(team_id)));
```
Always add indexes on columns used in RLS policies:
```sql
create index orders_user_id_idx on orders (user_id);
```
Reference: [RLS Performance](https://supabase.com/docs/guides/database/postgres/row-level-security#rls-performance-recommendations)
-112
View File
@@ -1,112 +0,0 @@
---
name: supabase
description: "Use when doing ANY task involving Supabase. Triggers: Supabase products (Database, Auth, Edge Functions, Realtime, Storage, Vectors, Cron, Queues); client libraries and SSR integrations (supabase-js, @supabase/ssr) in Next.js, React, SvelteKit, Astro, Remix; auth issues (login, logout, sessions, JWT, cookies, getSession, getUser, getClaims, RLS); Supabase CLI or MCP server; schema changes, migrations, security audits, Postgres extensions (pg_graphql, pg_cron, pg_vector)."
metadata:
author: supabase
version: "0.1.2"
---
# Supabase
## Core Principles
**1. Supabase changes frequently — verify against changelog and current docs before implementing.**
Do not rely on training data for Supabase features. Function signatures, config.toml settings, and API conventions change between versions.
First, fetch `https://supabase.com/changelog.md` (a lightweight summary index — not a heavy pull), scan for `breaking-change` tags relevant to your task, and follow the linked page for any that apply. Then look up the relevant topic using the documentation access methods below.
**2. Verify your work.**
After implementing any fix, run a test query to confirm the change works. A fix without verification is incomplete.
**3. Recover from errors, don't loop.**
If an approach fails after 2-3 attempts, stop and reconsider. Try a different method, check documentation, inspect the error more carefully, and review relevant logs when available. Supabase issues are not always solved by retrying the same command, and the answer is not always in the logs, but logs are often worth checking before proceeding.
**4. Exposing tables to the Data API:** Depending on the user's [Data API settings](https://supabase.com/dashboard/project/<ref>/integrations/data_api/settings), newly created tables may not be automatically exposed via the Data (REST) API. If this is the case, `anon` and `authenticated` roles will need to be explicitly granted access.
> Note that this is separate from RLS, which controls which _rows_ are visible once a table is accessible, not whether the table is accessible at all.
When a user reports a SQL-created table is unexpectedly inaccessible, check their Data API settings and whether the roles have been granted access via explicit `GRANT` SQL. When granting public (`anon`/`authenticated`) access, always enable RLS too. See [Exposing a Table to the Data API](https://supabase.com/docs/guides/api/securing-your-api.md) for the full setup workflow.
**5. RLS in exposed schemas.**
Enable RLS on every table in any exposed schema, which includes `public` by default. This is critical in Supabase because tables in exposed schemas can be reachable through the Data API when the `anon`/`authenticated` roles have access (see [Exposing a Table to the Data API](https://supabase.com/docs/guides/api/securing-your-api.md)). For private schemas, prefer RLS as defense in depth. After enabling RLS, create policies that match the actual access model rather than defaulting every table to the same `auth.uid()` pattern.
**6. Security checklist.**
When working on any Supabase task that touches auth, RLS, views, storage, or user data, run through this checklist. These are Supabase-specific security traps that silently create vulnerabilities:
- **Auth and session security**
- **Never use `user_metadata` claims in JWT-based authorization decisions.** In Supabase, `raw_user_meta_data` is user-editable and can appear in `auth.jwt()`, so it is unsafe for RLS policies or any other authorization logic. Store authorization data in `raw_app_meta_data` / `app_metadata` instead.
- **Deleting a user does not invalidate existing access tokens.** Sign out or revoke sessions first, keep JWT expiry short for sensitive apps, and for strict guarantees validate `session_id` against `auth.sessions` on sensitive operations.
- **If you use `app_metadata` or `auth.jwt()` for authorization, remember JWT claims are not always fresh until the user's token is refreshed.**
- **API key and client exposure**
- **Never expose the `service_role` or secret key in public clients.** Prefer publishable keys for frontend code. Legacy `anon` keys are only for compatibility. In Next.js, any `NEXT_PUBLIC_` env var is sent to the browser.
- **RLS, views, and privileged database code**
- **Views bypass RLS by default.** In Postgres 15 and above, use `CREATE VIEW ... WITH (security_invoker = true)`. In older versions of Postgres, protect your views by revoking access from the `anon` and `authenticated` roles, or by putting them in an unexposed schema.
- **UPDATE requires a SELECT policy.** In Postgres RLS, an UPDATE needs to first SELECT the row. Without a SELECT policy, updates silently return 0 rows — no error, just no change.
- **Do not put `security definer` functions in an exposed schema.** Keep them in a private or otherwise unexposed schema.
- **Storage access control**
- **Storage upsert requires INSERT + SELECT + UPDATE.** Granting only INSERT allows new uploads but file replacement (upsert) silently fails. You need all three.
For any security concern not covered above, fetch the Supabase product security index: `https://supabase.com/docs/guides/security/product-security.md`
## Supabase CLI
Always discover commands via `--help` — never guess. The CLI structure changes between versions.
```bash
supabase --help # All top-level commands
supabase <group> --help # Subcommands (e.g., supabase db --help)
supabase <group> <command> --help # Flags for a specific command
```
**Supabase CLI Known gotchas:**
- `supabase db query` requires **CLI v2.79.0+** → use MCP `execute_sql` or `psql` as fallback
- `supabase db advisors` requires **CLI v2.81.3+** → use MCP `get_advisors` as fallback
- When you need a new migration SQL file, **always** create it with `supabase migration new <name>` first. Never invent a migration filename or rely on memory for the expected format.
**Version check and upgrade:** Run `supabase --version` to check. For CLI changelogs and version-specific features, consult the [CLI documentation](https://supabase.com/docs/reference/cli/introduction) or [GitHub releases](https://github.com/supabase/cli/releases).
## Supabase MCP Server
For setup instructions, server URL, and configuration, see the [MCP setup guide](https://supabase.com/docs/guides/getting-started/mcp).
**Troubleshooting connection issues** — follow these steps in order:
1. **Check if the server is reachable:**
`curl -so /dev/null -w "%{http_code}" https://mcp.supabase.com/mcp`
A `401` is expected (no token) and means the server is up. Timeout or "connection refused" means it may be down.
2. **Check `.mcp.json` configuration:**
Verify the project root has a valid `.mcp.json` with the correct server URL. If missing, create one pointing to `https://mcp.supabase.com/mcp`.
3. **Authenticate the MCP server:**
If the server is reachable and `.mcp.json` is correct but tools aren't visible, the user needs to authenticate. The Supabase MCP server uses OAuth 2.1 — tell the user to trigger the auth flow in their agent, complete it in the browser, and reload the session.
## Supabase Documentation
Before implementing any Supabase feature, find the relevant documentation. Use these methods in priority order:
1. **MCP `search_docs` tool** (preferred — returns relevant snippets directly)
2. **Fetch docs pages as markdown** — any docs page can be fetched by appending `.md` to the URL path.
3. **Web search** for Supabase-specific topics when you don't know which page to look at.
## Making and Committing Schema Changes
**To make schema changes, use `execute_sql` (MCP) or `supabase db query` (CLI).** These run SQL directly on the database without creating migration history entries, so you can iterate freely and generate a clean migration when ready.
Do NOT use `apply_migration` to change a local database schema — it writes a migration history entry on every call, which means you can't iterate, and `supabase db diff` / `supabase db pull` will produce empty or conflicting diffs. If you use it, you'll be stuck with whatever SQL you passed on the first try.
**When ready to commit** your changes to a migration file:
1. **Run advisors** → `supabase db advisors` (CLI v2.81.3+) or MCP `get_advisors`. Fix any issues.
2. **Review the Security Checklist above** if your changes involve views, functions, triggers, or storage.
3. **Generate the migration** → `supabase db pull <descriptive-name> --local --yes`
4. **Verify** → `supabase migration list --local`
## Reference Guides
- **Skill Feedback** → [references/skill-feedback.md](references/skill-feedback.md)
**MUST read when** the user reports that this skill gave incorrect guidance or is missing information.
@@ -1,17 +0,0 @@
## What happened
**Task:** <!-- e.g., "Set up MFA on patient records" -->
**Skill said:** <!-- e.g., "Use auth.jwt()->'app_metadata' in the RLS policy" -->
**Expected:** <!-- e.g., "The function also needs SECURITY DEFINER + grant to supabase_auth_admin" -->
## Source
**File:** <!-- e.g., references/security-model.md -->
**Section:** <!-- e.g., "Trust Boundaries > user_metadata vs app_metadata" -->
## Fix suggestion
<!-- Leave blank if unsure -->
@@ -1,17 +0,0 @@
# Skill Feedback
Use this when the user reports that the skill gave incorrect guidance, is missing information, or could be improved. This is about the skill (agent instructions), not about Supabase the product.
## Steps
1. **Ask permission** — Ask the user if they'd like to submit feedback to the skill maintainers. If they decline, move on.
2. **Draft the issue** — Use the template at [assets/feedback-issue-template.md](../assets/feedback-issue-template.md) to structure the feedback. Fill in the fields based on the conversation. Always identify which specific reference file and section caused the problem.
3. **Submit** — Create a GitHub Issue on the `supabase/agent-skills` repository using the draft as the issue body. The title must follow this format: `user-feedback: <summary of the problem>`.
4. **Share the result** — Share the issue URL with the user after submission. If submission fails, give the user this link to create the issue manually:
```
https://github.com/supabase/agent-skills/issues/new
```
@@ -0,0 +1,65 @@
---
name: using-git-worktrees
description: Use when tasked with creating a new git worktree, building features in isolation, setting up parallel branches, or provisioning a safe sandbox environment to test architectural changes.
---
# Using Git Worktrees (Worktrunk)
## Overview
Git worktrees allow you to check out multiple branches of a repository simultaneously without cloning the entire repository from scratch. However, creating worktrees in nested subdirectories breaks relative path resolutions to peer monorepo packages. We use **Worktrunk** (`git-wt`) to automatically provision sibling worktrees and handle environment setup.
## When to Use
- When asked to create an isolated environment for a feature or branch.
- When you need a safe sandbox to test destructive or complex architectural changes.
- When working on multiple GitHub issues in parallel.
## Core Directives
### 1. NEVER Nest Worktrees
If you create a worktree inside a subdirectory of the current repo (e.g., `.agents/worktrees/feature`), all relative paths going UP and OUT of the repository (like `../../wwv-data-engine`) will immediately break due to the altered directory depth.
**Worktrees MUST be siblings of the main repository root.**
### 2. ALWAYS Use Worktrunk
Do not use raw `git worktree add`. Always use the `git-wt` alias provided by Worktrunk. It is pre-configured via `.config/wt.toml` to:
- Automatically place the worktree in the correct sibling path.
- Automatically run `pnpm install`.
- Automatically copy `.env` files so the environment works immediately.
> [!IMPORTANT]
> **For AI Agents:** You MUST append the `--yes` flag when creating worktrees to bypass the interactive approval prompt for these hooks, otherwise the command will fail in your non-interactive shell.
### 3. Database Lifecycle & Port Assignment
Each worktree gets a deterministic PostgreSQL port assigned automatically based on its directory hash (to avoid collisions between multiple running worktrees).
> [!CAUTION]
> **Orphaned Volumes:** You MUST use `git-wt remove` to delete worktrees. This triggers a `pre-remove` hook that cleanly tears down the Docker database. If you manually delete the folder (`rm -rf`), the Docker volume is orphaned. If this happens, run `pnpm run db:prune` in the main repository to garbage collect orphaned database volumes.
## Quick Reference
| Action | Command | Details |
| :--- | :--- | :--- |
| **Create Worktree** | `git-wt switch --create <branch> --yes` | Provisions the sibling directory, branch, env files, and dependencies. |
| **Delete Worktree** | `git-wt remove` | Deletes the worktree and the associated branch cleanly. Run from within the worktree. |
## Implementation Steps
1. Run the create command from the main repository (always include `--yes`):
```bash
git-wt switch --create feature/my-new-idea --yes
```
2. **CRITICAL:** Change your working directory to the newly created worktree before editing files. Worktrunk creates it as a sibling.
```bash
# Example if the main repo is c:\dev\worldwideview
cd ../worldwideview.feature-my-new-idea
```
3. Do your work, run tests, and commit.
4. When completely finished and merged, clean up by running `git-wt remove` from inside the worktree directory.
## Common Mistakes & Red Flags
| Excuse / Rationalization | Reality & Fix |
| :--- | :--- |
| "I'll just put the worktree in `.agents/worktrees` to keep the project folder clean." | **Red Flag:** This breaks relative paths to other monorepo packages. Delete the worktree and use `git-wt` to create a sibling worktree instead. |
| "I'll use `git worktree add` instead of `git-wt` because it's standard." | **Red Flag:** Raw git won't copy `.env` files or run the `pre-start` hooks for `pnpm install`, leaving you with a broken environment. Delete and use `git-wt`. |
| "I ran the command, but my file edits are applying to the main branch." | You forgot to `cd` into the newly created sibling directory. |
+17
View File
@@ -0,0 +1,17 @@
# TSDoc Review Log
This document tracks the progress of the codebase-wide TSDoc update.
## Ambiguous Functions Requiring 3.1 Pro Review
If a function's purpose cannot be deduced from its signature or body, it is listed here along with its file path and the specific confusion.
| File Path | Function/Interface Name | Reason for Ambiguity |
| :--- | :--- | :--- |
<!-- Flash: Append rows here as you work -->
## Completed Directories
- [/] `src/core/plugins` (In Progress: PluginRegistry.ts documented)
- [ ] `src/core/state`
- [x] `src/components` (Completed: Layout, Panels, Marketplace, Timeline, UI)
- [x] `packages/wwv-plugin-sdk`
- [x] `scripts`
@@ -7,7 +7,7 @@ description: Use when creating a new plugin, adding a new data source, or debugg
## Overview
A WorldWideView plugin is a self-contained data source that renders geospatial entities on a 3D Cesium globe. Every plugin implements the `WorldPlugin` interface from the SDK and connects to a **data engine** (Fastify + Redis) that streams live data over WebSocket.
A WorldWideView plugin is a self-contained data source that renders geospatial entities on a 3D Cesium globe. Every plugin implements the `WorldPlugin` interface from the SDK and connects to the **V2 Data Engine** (Fastify + Redis) that streams live data over WebSocket.
**Core principle:** Plugins are code bundles. The engine pushes data. The frontend renders it. There is no REST polling between frontend and engine — only WebSocket streaming.
@@ -54,7 +54,7 @@ digraph plugin_type {
| Sandbox plugin | `local-plugins/wwv-plugin-<name>/` | Local development workspace for new plugins |
| Plugin class | `packages/wwv-plugin-<name>/src/index.ts` | Implements `WorldPlugin` interface |
| Package metadata | `packages/wwv-plugin-<name>/package.json` | `worldwideview` block with id, type, format, category |
| Engine seeder | `wwv-data-engine/src/seeders/<name>.ts` | Fetches external data → Redis → WS broadcast |
| Engine seeder | `wwv-seeders/src/seeders/<name>.ts` | Fetches external data → Redis → WS broadcast |
| Build config | `next.config.ts` → `transpilePackages` | Required for monorepo packages |
| TS alias | `tsconfig.json` → `paths` | Required for monorepo packages |
| Registration | `src/core/plugins/PluginRegistry.ts` | Register plugin at boot |
@@ -72,9 +72,8 @@ digraph plugin_type {
The standard way to develop a new plugin is using the **Local Sandbox** (`local-plugins/` directory). This keeps experimental code out of the main monorepo packages until it is ready for production.
1. **Scaffold**: Run `node packages/wwv-cli/dist/index.js create <name> --local` from the project root. This generates a boilerplate plugin in `local-plugins/wwv-plugin-<name>`.
2. **Develop**: Run `pnpm dev`. The built-in file watcher (`pnpm dev:plugins`) automatically rebuilds and syncs your local plugin to `public/plugins-local/` whenever you save a file.
3. **Promote**: When stable, run `node packages/wwv-cli/dist/index.js link <name>`. This promotes the plugin to the `packages/` directory and cleans up the sandbox artifacts.
4. **Install**: Run `pnpm install` from the root to update workspace symlinks.
2. **Develop**: Run `pnpm dev`. The built-in file watcher (`pnpm dev:plugins`) automatically rebuilds and syncs your local plugin to `public/plugins-local/` whenever you save a file for instant hot-reloading.
3. **Publish (Optional)**: When stable, you can use `node packages/wwv-cli/dist/index.js publish <name>` from the project root to publish the plugin to NPM (use the `--org <your-org>` flag to publish under your own NPM organization instead of `@worldwideview`). There is no need to 'link' plugins to the core monorepo manually, as the local sandbox runs natively inside the workspace.
### 1.1 Implement the Interface
@@ -230,9 +229,9 @@ await pluginManager.registerPlugin(plugin);
**C. Dynamic import (runtime):**
For user-imported GeoJSON layers, call `pluginManager.loadFromManifest(manifest)` directly.
### 1.6 Build Configuration (Promoted Monorepo Packages Only)
### 1.6 Build Configuration (Core Monorepo Packages Only)
Once a plugin is promoted from the local sandbox via the `link` command, it becomes a formal monorepo package. You must manually add it to the build pipeline:
If you decide to manually move a plugin from the local sandbox into the core `packages/` directory, you must manually add it to the build pipeline:
Add to `next.config.ts`:
@@ -257,7 +256,7 @@ Run `pnpm install` from project root after creating the package.
## Part 2: Data Engine Seeder
The data engine is a **content-agnostic runner** (`wwv-data-engine` public, deployed via Docker). It runs Fastify on port 5000 with Redis caching and WebSocket streaming. Seeders are volume-mounted from `local-seeders/` (dev) or downloaded from GitHub Releases (production).
The data engine is a **content-agnostic Host Environment runner** (`wwv-data-engine`, deployed via Docker). It runs Fastify on port 5000 with Redis caching and WebSocket streaming. Seeders are volume-mounted from `local-seeders/` (dev) or dynamically downloaded and linked via pnpm workspaces from GitHub Releases (`wwv-seeders` and `wwv-seeders-private`) in production.
> [!IMPORTANT]
> If you are **migrating** an existing plugin from `packages/` to `local-plugins/`, use the `migrate-legacy-plugin` skill instead. It covers both frontend routing fixes AND backend seeder build fixes as two independent concerns.
@@ -373,9 +372,12 @@ interface SeederDefinition {
}
```
### 2.4 Auto-Discovery
### 2.4 Auto-Discovery & Dependencies
`src/seeders/index.ts` dynamically `require()`s every `.ts`/`.js` file in the seeders directory (except `index.ts` and test files). **You do not need to manually import your seeder** — just create the file and call `registerSeeder()` at module scope.
The V2 engine dynamically `import()`s every compiled `.mjs` file in the extracted seeder workspaces. **You do not need to manually import your seeder** — just create the file and call `registerSeeder()` at module scope.
> [!IMPORTANT]
> **V2 Engine Dependencies:** Seeders run in the unified V2 host environment which provides common packages (`zod`, `ws`, `undici`, etc.). **You MUST NOT bundle these dependencies** in your seeder's `dist` folder. Leave them externalized in your build config.
### 2.5 SQLite History (Optional)
@@ -416,7 +418,7 @@ db.exec(`
**Engine → Client:**
```json
{ "type": "welcome", "engine": "wwv-data-engine", "plugins": ["earthquakes", "wildfires", ...] }
{ "type": "welcome", "engine": "wwv-data-engine-v2", "plugins": ["earthquakes", "wildfires", ...] }
{ "type": "data", "pluginId": "earthquakes", "payload": { "items": [...], "totalCount": 42 } }
```
@@ -490,10 +492,110 @@ Place GeoJSON in `data/data.json`. The compiler auto-generates a `WorldPlugin` c
---
## Part 5: Example - Simple ISS Tracker Plugin
Here is a minimal, complete example of a live ISS Tracker plugin utilizing both the frontend SDK and the V2 Engine.
### 1. Frontend Plugin (`local-plugins/wwv-plugin-iss/src/index.ts`)
```typescript
import type {
WorldPlugin, PluginContext, GeoEntity,
CesiumEntityOptions, TimeRange, LayerConfig, PluginCategory
} from "@worldwideview/wwv-plugin-sdk";
import { Satellite } from "lucide-react";
import pkg from "../package.json";
export default class IssPlugin implements WorldPlugin {
id = "iss";
name = "ISS Tracker";
description = "Real-time International Space Station tracking";
icon = Satellite;
category: PluginCategory = "space";
version = pkg.version;
async initialize(ctx: PluginContext): Promise<void> {}
destroy(): void {}
async fetch(timeRange: TimeRange): Promise<GeoEntity[]> {
return []; // WS-only plugin
}
getPollingInterval(): number {
return 0; // WS-only
}
getLayerConfig(): LayerConfig {
return {
color: "#ffffff",
clusterEnabled: false,
maxEntities: 1,
};
}
renderEntity(entity: GeoEntity): CesiumEntityOptions {
return {
type: "point",
color: "#ffffff",
size: 10,
outlineColor: "#3b82f6",
outlineWidth: 2,
};
}
}
```
### 2. Backend Init Seeder (`wwv-seeders/src/seeders/iss.ts`)
```typescript
import { setLiveSnapshot } from "../redis";
import { registerSeeder } from "../scheduler";
function startIssSeeder() {
console.log("[ISS] Starting live tracker...");
async function poll() {
try {
const res = await fetch("https://api.wheretheiss.at/v1/satellites/25544");
const data = await res.json();
const item = {
id: "iss",
lat: data.latitude,
lon: data.longitude,
alt: data.altitude * 1000, // Convert km to meters
name: "International Space Station",
speed: data.velocity,
};
await setLiveSnapshot("iss", {
source: "iss",
fetchedAt: new Date().toISOString(),
items: [item],
totalCount: 1,
}, 10);
} catch (err) {
console.error("[ISS] Fetch failed", err);
}
}
poll(); // Initial fetch
setInterval(poll, 5000); // Poll every 5 seconds
}
registerSeeder({
name: "iss",
init: startIssSeeder,
});
```
---
## Common Mistakes
### "Sandbox Plugin Breaking Next.js Build"
1. **Premature Configuration** — Modifying `next.config.ts` (`transpilePackages`) or `tsconfig.json` (`paths`) while a plugin is still in `local-plugins/`. The `local-plugins` sandbox relies on dynamic hot-loading in the browser via `pnpm dev:plugins`. Do **not** touch monorepo configs until the plugin is formally promoted using `wwv link <name>`.
1. **Premature Configuration** — Modifying `next.config.ts` (`transpilePackages`) or `tsconfig.json` (`paths`) while a plugin is still in `local-plugins/`. The `local-plugins` sandbox relies on dynamic hot-loading in the browser via `pnpm dev:plugins`. Do **not** touch monorepo configs for local plugins.
2. **Manual Registration** — Manually adding a sandbox plugin to `AppShell.tsx`. Let the local hot-loading manifest handle sandbox plugins dynamically.
### "Missing Boilerplate / Errors in package.json"
@@ -530,8 +632,8 @@ Categories are **lowercase**: `"aviation"` not `"Aviation"`, `"natural-disaster"
## End-to-End Checklist
- [ ] **Development:** Scaffolded via `wwv create <name> --local` and developed in `local-plugins/`
- [ ] **Promotion:** Promoted to monorepo via `wwv link <name>` when stable
- [ ] **Development:** Scaffolded via `node packages/wwv-cli/dist/index.js create` and developed in `local-plugins/`
- [ ] **Publishing:** Published to NPM via `node packages/wwv-cli/dist/index.js publish <name> [--org <your-org>]` (if making public)
- [ ] **Frontend:** Plugin class implements `WorldPlugin` with correct `id`
- [ ] **Frontend:** `package.json` has `worldwideview` block with matching `id`
- [ ] **Frontend:** Version imported from `package.json` (never hardcoded)
@@ -539,7 +641,7 @@ Categories are **lowercase**: `"aviation"` not `"Aviation"`, `"natural-disaster"
- [ ] **Frontend:** Added path alias in `tsconfig.json`
- [ ] **Frontend:** Registered via `pluginRegistry` + `pluginManager` in `AppShell.tsx`
- [ ] **Frontend:** `renderEntity()` doesn't mix point/billboard properties
- [ ] **Engine:** Seeder file in `src/seeders/<name>.ts`
- [ ] **Engine:** Seeder file in `wwv-seeders/src/seeders/<name>.ts` (or private repo)
- [ ] **Engine:** `registerSeeder({ name })` matches frontend plugin `id`
- [ ] **Engine:** Calls `setLiveSnapshot()` with correct plugin ID key
- [ ] **Engine:** Data shape is an object with `items` array (or plugin implements `mapWebsocketPayload`)
+8
View File
@@ -0,0 +1,8 @@
[[pre-start]]
install_deps = "pnpm install"
[[pre-start]]
copy_env = """powershell -Command "if (Test-Path '../worldwideview/.env') { Copy-Item '../worldwideview/.env' '.env' -Force }""""
[[pre-remove]]
docker_clean = "docker compose down -v"
+2
View File
@@ -23,3 +23,5 @@ artifacts/
performance/
tmp/
*.geojson
**/.pnpm-store
local-seeders*
-24
View File
@@ -1,24 +0,0 @@
# PostgreSQL connection string.
# For local dev: The platform will automatically spin up PostgreSQL via Docker Compose.
# Set DATABASE_URL to: postgresql://postgres:postgres@localhost:5432/worldwideview?schema=public
#
# For cloud/production: Provide your external Postgres connection string (e.g. Supabase)
# and set WWV_SKIP_LOCAL_DB=true to prevent Docker from spinning up a local database instance.
# DATABASE_URL=postgresql://postgres:postgres@localhost:5432/worldwideview?schema=public
# Set this to true in production to prevent docker-compose or pnpm run dev from starting the local database.
# WWV_SKIP_LOCAL_DB=false
# Default plugins enabled on boot for demo edition
# Comma-separated list of plugin IDs
NEXT_PUBLIC_DEMO_DEFAULT_PLUGINS=iranwarlive,satellite,gps-jamming
# Edition: "local" | "cloud" | "demo"
NEXT_PUBLIC_WWV_EDITION=demo
# Auth Secret (REQUIRED for Docker — generate with: openssl rand -hex 32)
# Must be stable across container restarts to preserve sessions.
# [WARNING] NEVER commit real secrets here. Real secrets must ONLY go into .env.local!
AUTH_SECRET=changeme_d64c1f35b447da89cf6ed5a0988da1379b21904edb91a87c3f5a63942e3d23a4_changeme
+5
View File
@@ -55,6 +55,11 @@ NEXT_PUBLIC_DEMO_DEFAULT_PLUGINS=
# Edition: "local" | "cloud" | "demo"
NEXT_PUBLIC_WWV_EDITION=local
# Optional Database Teardown
# If set to true, terminating 'pnpm run dev' (Ctrl+C) will automatically
# stop the local database container (freeing up port 5432).
WWV_TEARDOWN_DB_ON_EXIT=false
# Demo Admin Secret (demo edition only)
# Allows operator plugin management on the demo instance via x-wwv-admin-secret header.
# Generate any random string. Only set this on the demo deployment.
+32
View File
@@ -1,10 +1,29 @@
name: CI
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
# Non-blocking dependency audit. Surfaces high / critical vulnerabilities
# in the CI output without failing the build. Flip `continue-on-error` to
# `false` once the current transitive vulns (see #97) are resolved.
audit:
name: Dependency Audit
runs-on: ubuntu-latest
continue-on-error: true
steps:
- uses: actions/checkout@v6
- uses: pnpm/action-setup@v6
- uses: actions/setup-node@v6
with:
node-version: 22
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: pnpm audit --audit-level high
typecheck:
name: Type Check
runs-on: ubuntu-latest
@@ -19,6 +38,19 @@ jobs:
- run: npx prisma generate
- run: pnpm exec tsc --noEmit
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: pnpm/action-setup@v6
- uses: actions/setup-node@v6
with:
node-version: 22
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: pnpm lint
test:
name: Unit Tests
runs-on: ubuntu-latest
+1 -1
View File
@@ -54,7 +54,7 @@ jobs:
NEXT_PUBLIC_CESIUM_ION_TOKEN=${{ secrets.NEXT_PUBLIC_CESIUM_ION_TOKEN }}
NEXT_PUBLIC_GOOGLE_MAPS_API_KEY=${{ secrets.NEXT_PUBLIC_GOOGLE_MAPS_API_KEY }}
NEXT_PUBLIC_BING_MAPS_KEY=${{ secrets.NEXT_PUBLIC_BING_MAPS_KEY }}
NEXT_PUBLIC_WWV_EDITION=${{ vars.NEXT_PUBLIC_WWV_EDITION || 'demo' }}
NEXT_PUBLIC_WWV_EDITION=${{ vars.NEXT_PUBLIC_WWV_EDITION || 'local' }}
NEXT_PUBLIC_WS_ENGINE_URL=${{ vars.NEXT_PUBLIC_WS_ENGINE_URL }}
NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL=${{ vars.NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL }}
NEXT_PUBLIC_ADSENSE_CLIENT_ID=${{ secrets.NEXT_PUBLIC_ADSENSE_CLIENT_ID }}
+53
View File
@@ -0,0 +1,53 @@
name: Mutation Testing
# Mutation testing is expensive (10–30 min per full run, even with
# `coverageAnalysis: "perTest"`), so we don't gate every PR on it.
# Instead it runs weekly so we catch drift in test quality, and on
# demand via workflow_dispatch when someone wants to validate a
# specific change.
on:
schedule:
# 03:00 UTC Monday = 10 PM US Eastern Sunday night.
- cron: '0 3 * * 1'
workflow_dispatch:
permissions:
contents: read
jobs:
stryker:
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0 # Required for Stryker's incremental mode.
- name: Setup pnpm
uses: pnpm/action-setup@v3
with:
version: 9
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Generate Prisma client
run: pnpm exec prisma generate
- name: Run Stryker
run: pnpm exec stryker run
- name: Upload mutation report
if: always()
uses: actions/upload-artifact@v4
with:
name: stryker-report
path: reports/mutation/
retention-days: 30
+51
View File
@@ -0,0 +1,51 @@
name: Playwright Tests
on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]
permissions:
contents: read
jobs:
test:
timeout-minutes: 60
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
browser: [chromium, firefox, webkit]
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v3
with:
version: 9.15.4
- uses: actions/setup-node@v4
with:
node-version: 20
cache: 'pnpm'
- name: Install dependencies
run: pnpm install
- name: Install Playwright Browsers
run: pnpm exec playwright install --with-deps
- name: Run Project Setup
run: pnpm run setup
- name: Boot Database
run: |
docker network create coolify || true
pnpm run predev
- name: Run Playwright tests
run: pnpm test:e2e --project=${{ matrix.browser }}
- uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: playwright-report-${{ matrix.browser }}
path: playwright-report/
retention-days: 30
+60
View File
@@ -0,0 +1,60 @@
name: Test Local Dev Environment
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
test-dev:
name: Test Local Dev Startup
runs-on: ubuntu-latest
env:
NEXT_PUBLIC_WWV_EDITION: local
AUTH_SECRET: ci-placeholder-secret-not-used-in-tests
steps:
- uses: actions/checkout@v6
- uses: pnpm/action-setup@v6
- uses: actions/setup-node@v6
with:
node-version: 22
cache: pnpm
- name: Create coolify docker network
# Needed because docker-compose.yml references external coolify network
run: docker network create coolify || true
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Setup environment
run: cp .env.example .env
- name: Test predev setup
# Tests that Docker DB boots, port mappings work, Prisma pushes schema, and plugins sync
run: pnpm run predev
- name: Test dev server startup
run: |
# Start the dev server in the background
pnpm run dev &
DEV_PID=$!
echo "Waiting for Next.js to respond on port 3000..."
# Wait up to 120 seconds for the dev server to return a response
timeout 120 bash -c 'until curl -s http://localhost:3000 > /dev/null; do sleep 5; done'
# Check if the process is still running
if ! kill -0 $DEV_PID 2>/dev/null; then
echo "Dev server crashed or failed to start within timeout!"
exit 1
fi
echo "Dev server started successfully!"
# Graceful shutdown
kill $DEV_PID
+121
View File
@@ -0,0 +1,121 @@
name: Test Setup Scripts
on:
push:
branches: [ "main" ]
pull_request:
branches: [ "main" ]
permissions:
contents: read
jobs:
test-bash:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.x'
- name: Start local HTTP server
run: |
python3 -m http.server 8080 &
sleep 2 # Wait for server to start
- name: Patch setup.sh to use local server
run: |
sed -i 's|https://raw.githubusercontent.com/silvertakana/worldwideview/main|http://localhost:8080|g' self-host/setup.sh
- name: Run setup.sh
run: |
cd self-host
bash setup.sh
- name: Verify .env generation
run: |
cd self-host
if [ ! -f .env ]; then
echo ".env file was not created"
exit 1
fi
if ! grep -q "^AUTH_SECRET=[a-f0-9]\{64\}$" .env; then
echo "AUTH_SECRET was not generated correctly or is missing"
exit 1
fi
- name: Verify docker-compose.yml download
run: |
cd self-host
if [ ! -f docker-compose.yml ]; then
echo "docker-compose.yml was not downloaded"
exit 1
fi
- name: Verify Docker containers are running
run: |
cd self-host
docker compose ps
if ! docker compose ps | grep -q "Up"; then
echo "Docker containers failed to start"
exit 1
fi
test-powershell:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.x'
- name: Start local HTTP server
run: |
python3 -m http.server 8080 &
sleep 2 # Wait for server to start
- name: Patch setup.ps1 to use local server
run: |
sed -i 's|https://raw.githubusercontent.com/silvertakana/worldwideview/main|http://localhost:8080|g' self-host/setup.ps1
- name: Run setup.ps1
run: |
cd self-host
pwsh ./setup.ps1
- name: Verify .env generation
run: |
cd self-host
if [ ! -f .env ]; then
echo ".env file was not created"
exit 1
fi
if ! grep -q "^AUTH_SECRET=[a-f0-9]\{64\}$" .env; then
echo "AUTH_SECRET was not generated correctly or is missing"
exit 1
fi
- name: Verify docker-compose.yml download
run: |
cd self-host
if [ ! -f docker-compose.yml ]; then
echo "docker-compose.yml was not downloaded"
exit 1
fi
- name: Verify Docker containers are running
run: |
cd self-host
docker compose ps
if ! docker compose ps | grep -q "Up"; then
echo "Docker containers failed to start"
exit 1
fi
+17 -2
View File
@@ -65,7 +65,6 @@ yarn-error.log*
# env files (can opt-in for committing if needed)
.env*
!.env
# vercel
.vercel
@@ -102,6 +101,17 @@ packages/wwv-data-engine/
/public/cesium/
build_output*.log
build_output*.txt
/build*.log
# Root-level scratch scripts and agent artifacts
tmp_*.js
dump-*.js
dump_*.py
test*.js
test*.mjs
delete-dups.js
commit.json
skills-lock.json
# Seeder repositories
/local-seeders/
@@ -117,4 +127,9 @@ build_output*.txt
/wwv-seeders/
/wwv-seeders-private/
/wwv-test/
worldwideview.code-workspace
worldwideview.code-workspace
# Playwright
/playwright-report/
/test-results/
/playwright/.auth/
+1
View File
@@ -0,0 +1 @@
v20
+300 -73
View File
@@ -37,76 +37,17 @@ Our primary design, feature-set, and operational layout goal is to mimic the str
## 4. Architecture Patterns
### 4.1 Plugin System (Core Abstraction)
> [!NOTE]
> See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the human-facing architecture deep-dive — plugin system, state management, data pipeline, data engine & seeders, rendering pipeline, and edition system.
Every data source is a **plugin** implementing the `WorldPlugin` interface from `@worldwideview/wwv-plugin-sdk`. The lifecycle utilizes a real-time WebSocket Firehose pipeline:
Key invariants agents MUST respect:
```text
PluginRegistry.register() → PluginManager.registerPlugin()
→ plugin.initialize(context)
Visibility Toggle → DataBusSubscriber subscribes to layer via WsClient
→ Engine responds with instantaneous websocket snapshots over /stream
→ WsClient pipes to DataBus.emit("dataUpdated", WsStreamPayload)
→ Store → EntityRenderer → Globe
```
Four plugin architectures exist (All-Bundle Model):
1. **Data Engine Seeder** — Lightweight `seeder.mjs` script executed by the dynamic `wwv-data-engine` runner (previously standalone microservices).
2. **Dynamic CDN Loaded (Bundle)** — Externally developed plugins dynamically imported at runtime via ES module CDNs (e.g., `unpkg.com` version-pinned URLs).
3. **Static Compiled (Bundle)** — Static GeoJSON data wrapped into JS bundles via `wwvStaticCompiler` during build/sync (previously `StaticDataPlugin`).
4. **Active Proxied (Bundle)** — Next.js API routes bundled to provide frontend interactions (previously `DeclarativePlugin`).
All plugins are now dynamically imported at runtime as ES module bundles via `loadPluginFromManifest` utilizing `import(/* webpackIgnore: true */ entry)`. The legacy `StaticDataPlugin` and `DeclarativePlugin` runtimes are fully deprecated.
Plugin types are re-exported from SDK through `src/core/plugins/PluginTypes.ts` and `PluginManifest.ts` — **source of truth is always `@worldwideview/wwv-plugin-sdk`**.
### 4.2 State Management
Zustand store with **nine slices**: `globe`, `layers`, `timeline`, `ui`, `filter`, `data`, `config`, `favorites`, `geojson`. Each slice is in its own file under `src/core/state/`.
- Access via `useStore` hook or `useStore.getState()` outside React
- Plugin settings stored in `configSlice.dataConfig.pluginSettings`
- Polling intervals stored in `configSlice.dataConfig.pollingIntervals`
### 4.3 Data Pipeline
```text
Engine push /stream → DataBusSubscriber WsClient router
→ WsClient.handleMessage() → DataBus.emit("websocketData")
→ DataBusSubscriber → _hydrateSnapshot() → Store.entitiesByPlugin
→ GlobeView (memoized visible entities)
→ EntityRenderer (billboard/point primitives)
→ AnimationLoop (horizon culling, hover/selection)
→ StackManager (co-located entity grouping)
```
### 4.3.1 Engine & Seeders Architecture
The data engine is a **content-agnostic runner** (`wwv-data-engine`, public) that discovers and executes seeder scripts from a configurable directory.
- **Local Dev**: Engine runs via Docker Compose on port 5000, reading seeders dynamically from `local-seeders/` (split into `community` and `private` tiers).
- **Production**: Engine container on Coolify, downloads release bundles from `wwv-seeders-community` and `wwv-seeders-private` on startup and unzips them into `/app/seeders`.
- **Split-routing**: `resolveEngineUrl` prioritizes the **Local Dev Engine (localhost:5001)** for local testing (following 12-Factor App methodology), falling back to cloud-hosted endpoints.
- **Agnostic Frontend Architecture**: WorldWideView is a completely agnostic renderer. It has absolutely no concept of a "unified" Data Engine. If 30 plugins require 30 different WebSocket servers, the application will blindly open 30 connections.
- **Self-Contained Plugins**: Each plugin is a self-contained package and **MUST explicitly declare its own `streamUrl` in its manifest or config**. Do NOT assume the frontend acts as a unified pipe. It just so happens that our default plugins share the `wwv-data-engine` backend, but the platform is 100% decentralized.
### 4.4 Rendering Pipeline
- **Primitive-based**: Uses `PointPrimitiveCollection`, `BillboardCollection`, `LabelCollection` — NOT Cesium Entity API
- **Chunked processing**: Large datasets (10k+) rendered via `ChunkedProcessor`
- **LOD system**: Model-type entities promoted to 3D models at close range (`useModelRendering`)
- **Horizon culling**: Manual dot-product calculation against Earth radius (NOT depth testing)
- **Stack/Spiderifier**: `StackManager` groups co-located entities; `stackAnimation` handles expansion
### 4.5 Edition System
Three editions controlled by `NEXT_PUBLIC_WWV_EDITION`:
- **`local`** — Self-hosted, full features, auth enabled
- **`cloud`** — Managed cloud instance, full features
- **`demo`** — Public demo, auth disabled, optional admin via `WWV_DEMO_ADMIN_SECRET`
Feature flags derived from edition in `src/core/edition.ts`.
- **Plugin source of truth**: `@worldwideview/wwv-plugin-sdk` — never define plugin types locally.
- **All-Bundle Model**: every plugin (seeder, CDN-loaded, static-compiled, active-proxied) is dynamically imported via `loadPluginFromManifest` using `import(/* webpackIgnore: true */ entry)`. Legacy `StaticDataPlugin` / `DeclarativePlugin` runtimes are deprecated.
- **Agnostic frontend**: the renderer has no concept of a "unified" data engine. Each plugin **MUST declare its own `streamUrl`** in its manifest or config; do not assume one shared pipe.
- **Nine Zustand slices** under `src/core/state/`: `globe`, `layers`, `timeline`, `ui`, `filter`, `data`, `config`, `favorites`, `geojson`. Access via `useStore` in React, `useStore.getState()` elsewhere.
- **Primitive-based rendering** (not Cesium Entity API). Point/Billboard/Label/Polyline collections only. Never mix `size`/`outlineWidth`/`outlineColor` onto billboard entities — GPU silently clips.
- **Three editions** controlled by `NEXT_PUBLIC_WWV_EDITION` (`local` / `cloud` / `demo`); feature flags derived in `src/core/edition.ts`.
---
@@ -114,13 +55,13 @@ Feature flags derived from edition in `src/core/edition.ts`.
### 5.1 File Size
**Max 150 lines per file.** If a file grows beyond this, modularize it. Extract helpers, split components, use hooks.
**Guideline: ~300 lines per file.** If a file grows excessively large, consider modularizing it. Extract helpers, split components, or use hooks.
### 5.2 Import Aliases
- `@/*` → `./src/*`
- `@worldwideview/wwv-plugin-sdk` → `./packages/wwv-plugin-sdk/src`
- Each plugin has its own alias in `tsconfig.json`
- Plugin modules are dynamically imported and do not require individual `tsconfig.json` aliases
### 5.3 CSS Rules
@@ -162,7 +103,8 @@ Built-in plugins are instantiated in `AppShell.tsx` and registered via `PluginRe
> - **MUST Detail Commit Levels & Bumps**: On description changes or release notes, you must detail the level of commit (Major/Minor/Fix) for *each* individual change. If there are multiple accumulated changes, you MUST EITHER commit them individually and bump the version each time, OR commit them all at once and bump the version multiple times.
> - **MUST Explain Complex Concepts Simply**: Whenever providing a complicated technical explanation to the user, you MUST include a simple explanation below it. Use an analogy with reference to the correct terminology, comparing the concept to something from everyday life to ensure the user easily understands it.
> - **MUST Be Transparent & Narrate Actions (Gemini 3.1 Agents)**: If you are a Gemini 3.1 agent, you MUST always be fully transparent. **Whenever you do anything, you must explicitly say what you are going to do in a visible chat message to the user, and ONLY THEN do it.** This ensures the user can actually see and verify what you are doing in real time. Do not jump to destructive actions without stating your intent first. Keep this narration conversational and natural. Avoid stiff, robotic templates. Just explicitly drop a brief, casual note about what you are checking, fixing, or deleting *before* you run the tool.
> - **MUST Wait for Explicit Authorization**: Do not take action unless explicitly told to do so. If the user highlights a piece of code, brings up a bug, or asks a question, your ONLY job is to analyze it, investigate the root cause, and explain what is wrong or answer the question. **Do not write the fix, delete files, or execute changes unless the user explicitly gives you the order to do so.** Wait for clear authorization before taking action. **Crucially, if you realize you have violated this rule by taking an unauthorized action, DO NOT automatically revert it. Reverting is itself an action that requires authorization. Just answer the question.**
> - **MUST Require Explicit Authorization**: Do NOT execute any state-changing tools (e.g., modifying files, writing code, running scripts, executing git commands) without first obtaining clear, explicit permission from the user to proceed with that specific action, **UNLESS** the action is a simple, straightforward, and safe operation (e.g., no deletion or dangerous operations). For safe operations, you may proceed without explicit permission. Otherwise, inform the user of your proposed plan and WAIT for their approval before acting. **Crucially, if you realize you have violated this rule by taking an unauthorized action, DO NOT automatically revert it. Reverting is itself an action that requires authorization. Just answer the question.**
> - **MUST Ask Clarifying Questions**: Never assume anything. If requirements are unclear, if you encounter an unexpected roadblock, or if the user's intent could be interpreted in multiple ways, you MUST pause and ask clarifying questions. Do NOT proceed until the developer has explicitly answered your question or requested you to proceed anyway.
### 5.8 Workspace Hygiene
Whenever agents generate temporary debugging scripts, test REST endpoints via `.mjs`, or dump traces/JSON outputs, they **MUST** save these exclusively inside `/local-scripts/`. The root directory is strictly for production configuration files.
@@ -196,8 +138,8 @@ Read the relevant rule file when working in that domain:
|---|---|---|
| `platform-architecture` | High-level platform goals, product vision, business model, and Edition System | `.agents/rules/platform-architecture.md` |
| `application-architecture` | Next.js frontend, Zustand state management, and CesiumJS integration | `.agents/rules/application-architecture.md` |
| `plugin-architecture` | Creating/modifying plugins, lifecycle, capability declarations, and seeders | `.agents/rules/plugin-architecture.md` |
| `marketplace-architecture` | Dynamic plugin installation, DB sync, and CDN loading | `.agents/rules/marketplace-architecture.md` |
| `plugin-architecture` | Creating/modifying plugins, lifecycle, capability declarations, and seeders (See ADR-0001) | `.agents/rules/plugin-architecture.md` |
| `marketplace-architecture` | Dynamic plugin installation, DB sync, CDN loading, and PKCE Auth (See ADR-0001) | `.agents/rules/marketplace-architecture.md` |
| `cloud-auth-architecture` | Cloud edition, PostgreSQL RLS, multi-tenancy, and licensing | `.agents/rules/cloud-auth-architecture.md` |
| `server-management` | Server development and debugging using SSH and Coolify MCP | `.agents/rules/server-management.md` |
| `stakeholders-and-human-centered-design` | Human-centered design principles and stakeholder map | `.agents/rules/stakeholders-and-human-centered-design.md` |
@@ -213,6 +155,7 @@ Read the relevant rule file when working in that domain:
| `continuous-improvement` | When to create/update rules, skills, or workflows | `.agents/rules/continuous-improvement.md` |
| `context-and-memory` | How to orient and maintain project context between sessions | `.agents/rules/context-and-memory.md` |
| `troubleshooting-and-debugging` | Resolving plugin latency, namespace collisions, or deployment issues | `.agents/rules/troubleshooting-and-debugging.md` |
| `e2e-testing` | Guidelines and architecture for the Playwright End-to-End (E2E) testing infrastructure | `.agents/rules/e2e-testing.md` |
---
@@ -266,6 +209,9 @@ Refer to these skill documents for specialized tasks:
- **Workflow**: You **MUST** use the `/commit` workflow before every git commit to ensure proper semantic versioning bumps.
- **Required Checks**: Ensure `pnpm test` and `pnpm build` complete successfully before proposing a merge.
- **Review Process**: Use `/pr-review` to conduct a comprehensive multi-role review on any pull request.
- **Git Worktrees & Database Lifecycle**: We use [Worktrunk](https://github.com/max-sixty/worktrunk) to manage parallel branches and avoid relative path breakage. `git-wt switch --create <branch>` provisions isolated sibling worktrees configured with `.env` files and `pnpm install`. The database port is assigned deterministically based on the directory hash.
> [!CAUTION]
> You MUST use `git-wt remove` to clean up worktrees. Manually deleting a worktree folder (`rm -rf`) will orphan the associated PostgreSQL Docker volume. If this happens, you must manually run `pnpm run db:prune` in the main repository to reclaim disk space.
---
@@ -275,3 +221,284 @@ Refer to these skill documents for specialized tasks:
- **Cesium Entity Clipping**: If `billboard` entities are clipping or failing to render correctly, verify that you are not mixing `point` primitive properties (like `size` or `outlineWidth`) into the `billboard` options.
- **Build Exhaustion (Docker)**: Multi-stage pnpm builds generate massive cache layers. If a Coolify deployment fails silently or PostgreSQL crashes abruptly, check host disk space and run `docker builder prune -a -f`.
- **Next.js Typechecking Failures**: If backend-only `scripts/` fail during Next.js build, verify that the scripts directory is properly listed in the `exclude` array of `tsconfig.json`.
## 17. Coding Principles
You are about to write or modify code. **Read this file first.** It exists to prevent the failure mode where every fix makes the next harder — the kind of accumulated coupling and band-aid-on-band-aid that turns a clean codebase into a fragile one over months. This file is the constitution that keeps that from happening.
### The hard rule
> **Always interface-based, extensible, composable, modular. Never band-aids on band-aids.**
If you find yourself adding a flag, an `if`-check, or a special case to make a problem go away, you are probably band-aiding. Stop and find the seam where the abstraction broke. Fix it there.
### Core principles
#### 1. Interfaces before implementations
Define the contract first. The contract is what other components depend on; the implementation is replaceable. Public functions accept the interface, not the concrete type.
```ts
// Yes
function attachSource(source: DataSource) { ... }
// No
function attachPostgresSource(pg: PostgresInstance) { ... }
```
When a pattern repeats across two modules, refactor it into an interface immediately. Don't wait for the third copy to "earn" the abstraction — by then the costs have compounded.
#### 2. Loose coupling through explicit contracts
Components communicate through well-defined contracts: typed function signatures, events on a bus, message queues, or capability registries. Pick the form that fits, but pick *one* and make it the only path.
Components do **not** communicate through:
- Direct imports of each other's internal modules
- Shared mutable state outside the contract
- Synchronous request/response when async fits the actual data flow
- "Just this once" backdoors that bypass the contract
If you're tempted to write `import { internalThing } from '../other-module/internal'`, you are doing it wrong. If the symbol you need isn't part of the other module's public contract, either it should be — propose adding it — or you're solving the problem in the wrong place.
#### 3. Domain boundaries are sacred
Each domain (module, service, bounded context, plugin) owns its own state and its own surface. Cross-domain access is **explicit, gated, and audited** — never a shortcut.
Forbidden:
- Reading another domain's storage directly
- Calling another domain's private functions
- Hardcoded `if (tenant === 'X')` or `if (mode === 'special')` branches in shared code
Allowed:
- A documented capability or API on the other domain that you call
- A configuration field on the relevant record that drives the strategy
- Cross-domain access gated by explicit policy (default **deny**, opt-in per case), with the access itself logged
The rule of thumb: if removing a domain would require code archaeology in three other domains to figure out what depends on its internals, the boundary has already been violated. Treat that as a bug.
#### 4. Don't bend your architecture to fit a dependency
When you bring in a library or framework, use it where it fits cleanly; build natively where it doesn't.
Let the dependency handle (don't reinvent):
- The thing it's specifically good at, in the shape it expects
Your code owns (the dependency can't represent):
- Anything that's specifically your domain
- Anything that requires identity, lifecycle, or invariants the dependency doesn't model
- The integration seams between the dependency and your domain
If you find yourself bending your design to fit a library convenience, stop. Either the library has a hook for what you need, or you build it natively. Don't compromise architecture for the convenience of pulling in someone else's defaults.
#### 5. Data lifetimes are deliberate, not incidental
Different data has different lifetimes. Get this wrong and you'll either retain garbage forever or delete things users care about. Common categories:
- **Operational telemetry** (logs, metrics, internal events) — short rolling window. Useful for crash recovery and immediate investigation; not useful enough to keep forever.
- **User content** (messages, documents, records the user created or that represent their lived experience with the system) — **permanent unless the user explicitly deletes it.** Never pruned by a sweep, never deleted by a retention policy authored for telemetry.
- **Resumable session state** — bounded window appropriate to the use case; archived (state flag flipped) rather than deleted.
Name each category in your codebase. Write the retention policy for each one separately. Never propose a sweep that crosses categories — "delete events older than 30 days" is fine if events are telemetry; the same policy applied to user content is data loss masquerading as housekeeping. If storage pressure ever becomes real for content, introduce cold-tiered storage, not deletion.
#### 6. Optional features must be truly removable
Every optional feature (plugin, extension, integration, side module) declares its surface explicitly and communicates through the same contracts as everything else. No optional feature reaches into another's internals.
Test for whether something is core vs. optional: **if removing this feature should break the system, it's core. Otherwise it's optional, and it should be removable without leaving residue.**
If you can't delete an "optional" module's directory and have the system keep running, it isn't optional. Either fix the coupling or move it into core honestly.
#### 7. Put code where it belongs
When a feature spans layers (a trigger lives here, the action lives there, the data model lives somewhere else), resist the urge to fold it all into one place "for convenience." The feature lives in three places because each piece belongs to a different domain. Routing across them is the job — not collapsing them.
The general rule: if a feature needs to read/write durable state, it's a tool/extension on top of the layer that owns that state; if it's "the system should know about this concept," it's a configuration or contract change. Often it's both, with a thin handler that wires them together. That thin handler is fine — it's load-bearing — but it doesn't justify smashing the layers together.
### Antipatterns to refuse
These are common failure modes that, repeated over time, calcify a codebase. Refuse them when you catch yourself reaching for them.
| Antipattern | Better answer |
|---|---|
| Adding a new flag to an existing function instead of extracting a strategy | Extract the strategy interface; existing call sites pass the default strategy |
| Patching around a bug at the call site | Find the seam where the abstraction broke; fix it there |
| Direct database access from a feature module | Go through the layer that owns the data; if no surface exists, propose one |
| Hardcoded paths, model names, IDs, environment specifics in business logic | Configuration; or look up via registry |
| `if (tenant === 'X') { ... }` in shared code | Strategy on the record, or a per-tenant config field |
| Inheriting one layer's concerns into another (or vice versa) | Cross the boundary through the documented contract |
| "It works for now" — inline coupling between layers | If the layers are real, draw the contract; if they're not, merge them honestly |
| Adding a defensive `try/catch` to swallow an error | Find why the error happens; gate cleanly at the right boundary |
| Writing a comment to explain what code does | Rename and refactor so the code reads itself |
| Copying a snippet from one module into another "to avoid the dependency" | Either it belongs in a shared module, or the duplicate will drift — pick |
### Debugging discipline
When something breaks:
1. **Reproduce it.** Don't trust intuition about what's failing.
2. **Find the seam.** Where did the abstraction break? Which contract is being violated, and by whom?
3. **Fix at the seam.** Not at the call site. Not at the consumer. At the boundary that's wrong.
4. **If you can't find the seam, surface it.** Tell the user explicitly: "this is a design gap; the right fix needs an architectural decision, not a patch." Do not paper over.
If a fix doesn't work, **don't add another guard on top.** Find the real seam and gate cleanly. Stacked defensive layers are how clean codebases become unmaintainable.
### Lifecycle invariants and state-changing operations
Most recurring bugs cluster around the same shape: an operation crosses lifecycle state without checking, and the result is silent corruption, lost data, or orphan rows. The recipe that prevents recurrences:
1. **Name the invariant.** "Closed sessions don't accept work." "Scoped grants need a scope id." "Subscriptions past retention can't deliver swept events." Write it down before reaching for code.
2. **Throw a typed error at the seam.** The single function that crosses the invariant refuses the bad case with a typed, named error — not a boolean return, not a silent skip, not a generic `Error`.
3. **Pre-check at every call site that mutates *before* reaching the seam.** If the seam check fires after `store.append()`, you've left an orphan row. The pre-check is hygiene; the seam check is correctness — both are required. When the same pre-check appears at a second call site, extract a helper.
4. **Validate, then mutate — never the reverse.** Within a single function, resolve all inputs that could fail (key derivation, lookups, permission checks) before touching durable state. A validation throw should leave nothing half-applied. If you can't separate validation from mutation, the function is doing two jobs.
5. **A documented unsafe API is still unsafe.** A docstring that says "don't use this when X" is a foot-gun, not a warranty disclaimer. Either make the API safe in case X (drain, throw, refuse) or remove it.
If you're tempted to skip step 3 because "the seam check covers it," check whether the call site writes anything before reaching the seam. If yes, you need the pre-check.
### Definition of done — checklist for adding or updating a function
Every code change must satisfy the relevant checklist before merge. Refusal triggers below mean **stop and surface to the user** — do not ship a half-met checklist.
#### Adding a new function
All must be true:
- [ ] Belongs in this module (not somewhere else)
- [ ] Fulfills an existing interface, OR a new interface is added with it (and reasons documented in a spec doc / ADR)
- [ ] Has a doc-comment header (purpose + params + returns + thrown errors if relevant)
- [ ] Has at least one test that targets the contract (not implementation details)
- [ ] Test passes locally and in CI
- [ ] Doesn't introduce direct cross-module imports of internals — communication through public contracts only
- [ ] Doesn't touch storage owned by another component — route through that component's surface
- [ ] Matches the spec doc's contract for this component (if no spec covers it, update the spec first)
- [ ] Name is clear enough that the header comment is a summary, not an explanation
#### Updating an existing function
All must be true:
- [ ] Reason for change is clear (bug fix / new requirement / refactor / spec change)
- [ ] Doc-comment still accurate after change (update if not)
- [ ] Existing tests cover the new behavior (add new tests if not)
- [ ] Build is green (all tests passing)
- [ ] If this changes the contract: spec doc updated AND an ADR added if the change is load-bearing
- [ ] Fix is at the seam, not at the symptom (no band-aid patches)
- [ ] No new flags or special cases added in lieu of strategy extraction
#### Refusal triggers (any one means stop)
- Adding `if (tenant === 'X')` to shared code → use per-tenant config instead
- Adding a flag to make a problem go away → extract the abstraction the flag implies
- Direct DB or filesystem access from a feature module → route through the owning component
- `try`/`catch` swallowing an error → find why the error happens, gate cleanly
- Test omitted because "it's trivial" → no
- Function-header skipped because "it's obvious" → no
- "Just add another guard here" when a guard didn't work → find the real seam
- Patch shipped to "unblock the build" without root-cause fix → no, surface the design gap
If any criterion isn't met, stop. Surface to the user as a design question. Don't paper over.
### Code-style minimums
#### Comments
**Every function gets a doc-comment header** in your language's idiomatic format (TSDoc, JSDoc, Python docstrings, Rust `///`, Javadoc, etc.). Not optional. Long-running codebases get re-read after gaps; function-header docs are how re-onboarding stays cheap.
Required content:
- One-line summary at a contract level (what the function guarantees, not what the code does line-by-line)
- Each parameter and what it represents
- Return value
- Thrown errors / failure conditions if relevant
- Example usage when the call shape is non-obvious
Apply this to public, internal, exported, and private functions. Apply to methods on classes too.
**Inline comments inside function bodies** still follow the standard rule: only for non-obvious WHY (hidden constraints, workarounds, surprising behavior, references to bug reports). Don't comment WHAT the code does — well-named identifiers and the function-header doc cover that.
#### Tests
**Every function ships with at least one test. Tests passing is a build requirement.**
- Tests target the contract (function signature + documented behavior + invariants), not implementation details. Refactoring should not break tests; behavior changes should.
- A new implementation of an interface must pass the same contract test suite (write tests against the interface, not against a specific implementation).
- Test names describe the case being verified, not the function: `it('refuses resume after archival window expires')` not `it('resumeSession')`.
- For storage code: tests use a real (temporary) instance of the storage layer, created and torn down per test or per suite. Don't mock the storage engine — you're testing the wrong thing if you do.
- For integration code that depends on an external library's hooks: contract tests against that library's surface so a dependency bump catches behavior changes.
- Build pipeline (CI or local) runs all tests; non-zero exit = build fails. There is no "build succeeded but tests failed" state.
#### Logging
**Every new feature with a non-happy-path ships with a log line at each rare branch.** Use a structured logger with module-scoped child loggers. The log file is where "something's not working" turns into "what actually happened."
Level guidance:
- **info** — expected-but-noteworthy edges: a UI race that hits a closed resource, a permission bypass for a privileged principal. Correct behavior an operator might want to confirm fired.
- **warn** — should be rare and worth investigating: race past a pre-check, lost events past retention, programmer/operator errors (calling a method after stop, a key that should have resolved didn't).
- **error** — actual failures the operator must respond to: handler crashes, dispatch loop crashes, integration failures.
- **debug** — high-volume internal state changes; always pair with a `module` field so it can be filtered.
Pass structured fields, not interpolated strings — fields are what makes a log queryable:
```ts
// Yes
logger.warn({ session_id, state }, "rejected append against non-live session");
// No
logger.warn(`session ${session_id} (${state}) rejected append`);
```
If you add a new typed error or a new event topic, also log it. The typed error tells the immediate caller; the event tells subscribers; the log tells the operator. The three are independent surfaces for the same fact, and all three are needed — events don't show up in the log, logs don't show up on the bus, errors don't survive the catch.
#### Structure
- Keep modules small. If a file is over ~500 lines, ask whether it has more than one responsibility.
- Prefer explicit composition (caller wires things together) over implicit dependency-injection magic.
- One public concept per module. If a file exports both `Foo` and `unrelatedHelperUsedNowhereElse`, the helper has the wrong home.
#### Dependencies
**Always check the registry for the current version before adding or updating a dependency.** Don't pick versions from memory.
The simple test: did you actually run `<package-manager> view <pkg> version` (or its equivalent) before pinning? If not, you're guessing — and the guess is almost always behind, sometimes by a major version with active deprecation.
Why this matters:
- **Long-running codebases.** Stale dependency anchors compound. A v3 you picked because that's what was on your mind will still be v3 a year later, and the v4 migration gets harder every month it waits.
- **Active deprecations are real.** Common libraries are mid-deprecation cycles for older majors. Pinning to a soon-deprecated version on day one means the eventual upgrade is forced rather than chosen.
- **"Latest" doesn't mean "use without thinking."** Read the changelog for breaking changes; decide whether to take them now or pin behind explicitly. But the *default* posture should be: pick latest, and justify in writing if pinning behind.
If you must pin behind latest, leave a comment in the manifest (or a CHANGELOG note) explaining why — a known incompatibility with another dep, an in-progress migration, a tested-and-rejected major bump. *"I haven't checked"* is not an acceptable reason.
Apply to: adding a new dep, updating an existing one, scaffolding a new package. *Not* required for routine no-op installs that don't touch versions.
#### Doc-vs-spec discipline
Specification documents (the deeper ones that describe how a component must behave) are **functional specifications**, not narrative essays. The discipline:
- **Concrete data structures** — type definitions, schemas, DDL, JSON examples
- **Concrete function signatures** — input/output types, error types
- **Concrete event/message payloads** — name + schema + example
- **Concrete API signatures** — input/output schemas, side effects, permission requirements
- **Flow pseudocode** — when orchestration is non-obvious
- **Invariants** — explicitly listed: "after X happens, Y is always true"
- **Failure modes** — enumerated: "if X fails, the system does Y"
- **Acceptance tests** — statements another engineer can implement: "given input X, output Y"
Avoid in spec docs:
- Long prose explaining motivation (that goes in ADRs / decision records)
- Discussion of alternatives considered (also ADRs)
- Background context (README handles this)
- Marketing/aspirational tone
Spec docs are for implementation. Read one and you should know what to build.
### When in doubt
Ask the human user. Specifically:
- "I don't see a clean seam for this; the options I see are A or B. Which fits the architecture you have in mind?"
- "This requires a new contract / capability / interface. Should I draft it before implementing?"
- "I'm about to add a special case here; that smells like a missing abstraction. Do you want me to extract the abstraction or note the smell and defer?"
Don't ship band-aids and ask forgiveness later. The user has explicitly said: prevent that pattern.
+51
View File
@@ -0,0 +1,51 @@
# Design System: WorldWideView
**Project ID:** local-worldwideview-app
## 1. Visual Theme & Atmosphere
The WorldWideView platform employs an **"Airy, High-Tech, and Geospatial-Focused"** aesthetic. The design philosophy centers around "Glassmorphism Pro Max"—keeping the user's primary focus on the rich, underlying 3D globe visualization while providing highly legible, structured, and floating Heads-Up Display (HUD) elements.
The mood is utilitarian yet premium, heavily relying on dense background blurring, delicate inner borders, and vibrant glowing accent states to establish depth, interactivity, and a sense of "live" intelligence.
## 2. Color Palette & Roles
WorldWideView operates using a multi-theme architecture (Default Dark, Pure Black, Legacy OLED, and Light). The core semantic roles for the **Default Dark Theme** are outlined below:
* **Obsidian Canvas** (`#171717`) - `--bg-primary`: The foundational layer behind the application shell and globe.
* **Translucent Slate Glass** (`rgba(38, 38, 38, 0.85)`) - `--bg-glass`: The primary background for floating HUD panels, sidebars, and dialogs. Provides a frosted glass effect that allows the map to bleed through.
* **Frost White** (`#ffffff`) - `--text-primary`: Primary typography for headers, active states, and high-emphasis data.
* **Muted Ash** (`#a3a3a3`) - `--text-muted`: Secondary typography used for labels, subtitles, and inactive states to establish visual hierarchy.
* **Gossamer Edge** (`rgba(255, 255, 255, 0.15)`) - `--border-subtle`: Defines the subtle, crisp edges of glass panels, dividers, and default button states.
* **Starlight Cyan** (`#ffffff` / *Legacy:* `#22d3ee`) - `--accent-cyan`: Used for primary actions, active tabs, timeline scrubbers, and play buttons. Often paired with a diffuse glow (`0 0 24px rgba(255, 255, 255, 0.15)`) for emphasis and technical flair.
* **Telemetry Green** (`#22c55e`) - `--accent-green`: Used for live status indicators, pulsing connectivity badges, and positive data deltas.
* **Warning Amber** (`#f59e0b`) - `--accent-amber`: Used for warnings, "Alpha" badges, or unavailable data states.
## 3. Typography Rules
The system uses a dual-font strategy to balance clean user interfaces with technical precision.
* **UI Font (Inter):** Used for all general interface elements, headers, and buttons.
* Headers and brand elements use a bold `600/700` weight with tight letter-spacing (`-0.02em`) for a sleek, modern feel.
* Section titles, keys, and subtitles use a very small size (`11px`), a medium weight, uppercase transformation, and wide letter-spacing (`0.05em` to `0.1em`) to mimic aerospace or military HUD interfaces.
* **Monospace Font (JetBrains Mono):** Used strictly for data values, coordinates, telemetry, and exact timestamps (`11px` to `12px`). Ensures tabular alignment, clear distinction between similar characters (e.g., 0 and O), and technical precision.
## 4. Component Stylings
* **Glass Panels (Cards/Containers):**
* **Shape:** Generously rounded corners (`16px` / `var(--radius-lg)`).
* **Surface:** Dense frosted background blur (`24px`).
* **Depth:** Elevated significantly above the map using a dual-layered drop shadow (`0 8px 32px rgba(0, 0, 0, 0.4)` combined with `0 2px 8px rgba(0, 0, 0, 0.2)`). Features a crisp inner border (`inset 0 1px 1px rgba(255, 255, 255, 0.1)`) to simulate physical glass thickness.
* **Buttons:**
* **Shape:** Subtly rounded corners (`12px` / `var(--radius-md)`) with a standard height of `36px`.
* **Surface:** Translucent glass background with a subtle border.
* **Behavior:** Smooth, spring-based transitions. On hover, the background brightens, the element scales up/translates slightly (`translateY(-1px)`), and primary buttons emit a vivid neon glow. On click (active), buttons physically depress (`scale(0.97)`).
* **Tabs & Selectors:**
* **Shape:** Softly rounded (`8px` / `var(--radius-sm)`).
* **Surface:** Clean, flat, and transparent by default.
* **Behavior:** Transitions to a soft glass hover or active state. Active tabs use a highlighted background (`rgba(var(--accent-rgb), 0.1)`) and bright cyan text with increased font weight.
* **Timeline Scrubber:**
* **Track:** Slender `4px` tall track with softly rounded edges.
* **Handle:** A circular `14px` handle that emits a cyan glow, expanding dynamically on hover (`scale(1.2)`).
## 5. Layout Principles
* **Absolute Overlays:** The UI does not push the map; instead, it floats over a full-screen, `z-index: 0` globe canvas. Elements are anchored to the edges (Header to top, Sidebar to sides, Timeline to bottom).
* **Strict 4/8dp Rhythm:** All spacing adheres to an 8-point grid base (`4px`, `8px`, `12px`, `16px`, `24px`, `32px`, `48px`).
* **Dynamic Containment:** Sidebars and bottom panels push each other dynamically to prevent overlap and optimize map visibility (e.g., when the bottom panel opens, sidebars slide upward).
* **Whitespace & Density:** Panels use a dense `16px` padding globally to maximize the amount of intelligence data visible while maintaining clear breathing room between individual data rows. Scrollable areas hide scrollbars until hovered to maintain the clean glass aesthetic.
+42 -1
View File
@@ -30,6 +30,9 @@ ARG NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL
ARG NEXT_PUBLIC_ADSENSE_CLIENT_ID
ARG NEXT_PUBLIC_SUPABASE_URL
ARG NEXT_PUBLIC_SUPABASE_ANON_KEY
ARG NEXT_PUBLIC_WWV_AGENT_BUS_ENABLED
ARG NEXT_PUBLIC_WWV_BUILD_ID
ARG NEXT_PUBLIC_WWV_BUILD_AT
# Run our pregenerate schema swap script and then generate Prisma client
RUN NEXT_PUBLIC_WWV_EDITION=$NEXT_PUBLIC_WWV_EDITION pnpm run generate
@@ -37,6 +40,41 @@ RUN NEXT_PUBLIC_WWV_EDITION=$NEXT_PUBLIC_WWV_EDITION pnpm run generate
# Database migrations run at container startup via docker-entrypoint.sh
# DATABASE_URL must be set to a PostgreSQL connection string
# Stamp the build with a millisecond-precision id + iso timestamp. If the
# operator hasn't passed NEXT_PUBLIC_WWV_BUILD_ID via build args, generate
# one here so /api/build and the client console.log both carry a unique
# value per `docker build`.
RUN if [ -z "$NEXT_PUBLIC_WWV_BUILD_ID" ]; then \
NEXT_PUBLIC_WWV_BUILD_ID="$(date +%s%3N 2>/dev/null || node -e 'process.stdout.write(String(Date.now()))')" ; \
fi && \
if [ -z "$NEXT_PUBLIC_WWV_BUILD_AT" ]; then \
NEXT_PUBLIC_WWV_BUILD_AT="$(node -e 'process.stdout.write(new Date().toISOString())')" ; \
fi && \
echo "$NEXT_PUBLIC_WWV_BUILD_ID" > /app/.build-id && \
echo "$NEXT_PUBLIC_WWV_BUILD_AT" > /app/.build-at
# Write .env.production.local so Next.js sees every NEXT_PUBLIC_* var
# whose ARG was passed in. Docker doesn't reliably promote ARGs into the
# build process's environment for `next build` to inline references with
# Webpack's DefinePlugin — `process.env.NEXT_PUBLIC_X` either gets the
# inlined literal value (when set) or stays as a runtime polyfill access
# that resolves to undefined in the browser. Writing the file is the
# documented Next.js path and behaves the same on every host.
RUN set +e ; { \
echo "NEXT_PUBLIC_WWV_BUILD_ID=$(cat /app/.build-id)" ; \
echo "NEXT_PUBLIC_WWV_BUILD_AT=$(cat /app/.build-at)" ; \
if [ -n "$NEXT_PUBLIC_WWV_EDITION" ]; then echo "NEXT_PUBLIC_WWV_EDITION=$NEXT_PUBLIC_WWV_EDITION" ; fi ; \
if [ -n "$NEXT_PUBLIC_WWV_AGENT_BUS_ENABLED" ]; then echo "NEXT_PUBLIC_WWV_AGENT_BUS_ENABLED=$NEXT_PUBLIC_WWV_AGENT_BUS_ENABLED" ; fi ; \
if [ -n "$NEXT_PUBLIC_CESIUM_ION_TOKEN" ]; then echo "NEXT_PUBLIC_CESIUM_ION_TOKEN=$NEXT_PUBLIC_CESIUM_ION_TOKEN" ; fi ; \
if [ -n "$NEXT_PUBLIC_GOOGLE_MAPS_API_KEY" ]; then echo "NEXT_PUBLIC_GOOGLE_MAPS_API_KEY=$NEXT_PUBLIC_GOOGLE_MAPS_API_KEY" ; fi ; \
if [ -n "$NEXT_PUBLIC_BING_MAPS_KEY" ]; then echo "NEXT_PUBLIC_BING_MAPS_KEY=$NEXT_PUBLIC_BING_MAPS_KEY" ; fi ; \
if [ -n "$NEXT_PUBLIC_WS_ENGINE_URL" ]; then echo "NEXT_PUBLIC_WS_ENGINE_URL=$NEXT_PUBLIC_WS_ENGINE_URL" ; fi ; \
if [ -n "$NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL" ]; then echo "NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL=$NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL" ; fi ; \
if [ -n "$NEXT_PUBLIC_ADSENSE_CLIENT_ID" ]; then echo "NEXT_PUBLIC_ADSENSE_CLIENT_ID=$NEXT_PUBLIC_ADSENSE_CLIENT_ID" ; fi ; \
if [ -n "$NEXT_PUBLIC_SUPABASE_URL" ]; then echo "NEXT_PUBLIC_SUPABASE_URL=$NEXT_PUBLIC_SUPABASE_URL" ; fi ; \
if [ -n "$NEXT_PUBLIC_SUPABASE_ANON_KEY" ]; then echo "NEXT_PUBLIC_SUPABASE_ANON_KEY=$NEXT_PUBLIC_SUPABASE_ANON_KEY" ; fi ; \
} > /app/.env.production.local
# Run Next.js build with Webpack cache mounted
RUN --mount=type=cache,target=/app/.next/cache NODE_OPTIONS="--max_old_space_size=3072" pnpm run build
RUN node scripts/copy-cesium.mjs
@@ -78,7 +116,10 @@ COPY --from=builder /app/prod/node_modules ./node_modules
COPY --from=builder /app/.next/static ./.next/static
COPY --from=builder /app/public ./public
COPY --from=builder /app/scripts/https-proxy.mjs ./scripts/https-proxy.mjs
COPY --from=builder /app/scripts/migrate-legacy.mjs ./scripts/migrate-legacy.mjs
# Build stamp — read by /api/build at runtime.
COPY --from=builder /app/.build-id ./.build-id
COPY --from=builder /app/.build-at ./.build-at
# Entrypoint: migrate DB on first run, then start server
COPY docker-entrypoint.sh ./docker-entrypoint.sh
+75 -15
View File
@@ -1,33 +1,93 @@
Elastic License 2.0
Acceptance
URL: https://www.elastic.co/licensing/elastic-license
## Acceptance
By using the software, you agree to all of the terms and conditions below.
Copyright License
## Copyright License
The licensor grants you a non-exclusive, royalty-free, worldwide, non-transferable, non-sublicensable license to use, copy, distribute, make available, and prepare derivative works of the software.
The licensor grants you a non-exclusive, royalty-free, worldwide,
non-sublicensable, non-transferable license to use, copy, distribute, make
available, and prepare derivative works of the software, in each case subject to
the limitations and conditions below.
Limitations
## Limitations
You may not provide the software to third parties as a hosted or managed service, where the service provides users with access to any substantial set of the features or functionality of the software.
You may not provide the software to third parties as a hosted or managed
service, where the service provides users with access to any substantial set of
the features or functionality of the software.
You may not move, alter, disable, or circumvent any of the licensing features contained in the software, or remove or obscure any functionality in the software that is protected by the licensing features.
You may not move, change, disable, or circumvent the license key functionality
in the software, and you may not remove or obscure any functionality in the
software that is protected by the license key.
You may not alter, remove, or obscure any licensing, copyright, or other notices of the licensor in the software. Any use of the licensor's trademarks is subject to applicable law.
You may not alter, remove, or obscure any licensing, copyright, or other notices
of the licensor in the software. Any use of the licensor’s trademarks is subject
to applicable law.
Patents
## Patents
The licensor grants you a license, under any patent claims the licensor can license, or becomes able to license, to make, have made, use, sell, offer for sale, import, and have imported, the software. This license is subject to the conditions of this license, and does not cover patent claims infringed by your modifications or your additions to the software. If you or your company make any written claim that the software infringes or contributes to an infringement of any patent, your patent license for the software granted under these terms ends immediately. If your company makes such a claim, your patent license ends immediately for work on behalf of your company.
The licensor grants you a license, under any patent claims the licensor can
license, or becomes able to license, to make, have made, use, sell, offer for
sale, import and have imported the software, in each case subject to the
limitations and conditions in this license. This license does not cover any
patent claims that you cause to be infringed by modifications or additions to
the software. If you or your company make any written claim that the software
infringes or contributes to infringement of any patent, your patent license for
the software granted under these terms ends immediately. If your company makes
such a claim, your patent license ends immediately for work on behalf of your
company.
Notices
## Notices
You must ensure that anyone who gets a copy of any part of the software from you also gets a copy of these terms. If you modify the software, you must include prominent notices in modified copies stating that you have modified it.
You must ensure that anyone who gets a copy of any part of the software from you
also gets a copy of these terms.
No Other Rights
If you modify the software, you must include in any modified copies of the
software prominent notices stating that you have modified the software.
The terms do not imply any licenses other than those expressly granted under these terms.
## No Other Rights
Termination
These terms do not imply any licenses other than those expressly granted in
these terms.
If you use the software in violation of these terms, such use is not licensed, and your licenses will automatically terminate. If the licensor provides you with a notice of your violation, and you cease all violations of this license no later than 30 days after you receive that notice, your licenses will be reinstated retroactively. However, if you violate these terms after such reinstatement, any additional violation of these terms will cause your licenses to terminate automatically and permanently.
## Termination
If you use the software in violation of these terms, such use is not licensed,
and your licenses will automatically terminate. If the licensor provides you
with a notice of your violation, and you cease all violation of this license no
later than 30 days after you receive that notice, your licenses will be
reinstated retroactively. However, if you violate these terms after such
reinstatement, any additional violation of these terms will cause your licenses
to terminate automatically and permanently.
## No Liability
*As far as the law allows, the software comes as is, without any warranty or
condition, and the licensor will not be liable to you for any damages arising
out of these terms or the use or nature of the software, under any kind of
legal claim.*
## Definitions
The **licensor** is the entity offering these terms, and the **software** is the
software the licensor makes available under these terms, including any portion
of it.
**you** refers to the individual or entity agreeing to these terms.
**your company** is any legal entity, sole proprietorship, or other kind of
organization that you work for, plus all organizations that have control over,
are under the control of, or are under common control with that
organization. **control** means ownership of substantially all the assets of an
entity, or the power to direct its management and policies by vote, contract, or
otherwise. Control can be direct or indirect.
**your licenses** are all the licenses granted to you for the software under
these terms.
**use** means anything you do with the software requiring one of your licenses.
**trademark** means trademarks, service marks, and similar rights.
+8 -5
View File
@@ -17,7 +17,7 @@
[![CesiumJS](https://img.shields.io/badge/Cesium-JS-4272D0)](https://cesium.com/)
[![TypeScript Strict](https://img.shields.io/badge/TypeScript-Strict-3178C6?logo=typescript)](https://www.typescriptlang.org/)
[![Docker](https://img.shields.io/badge/Docker-Multi--Stage-2496ED?logo=docker)](https://www.docker.com/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![License: Elastic 2.0](https://img.shields.io/badge/License-Elastic_2.0-blue.svg)](LICENSE)
[![Discord](https://img.shields.io/discord/1481088998750294139?color=7289da&label=discord&logo=discord)](#)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://makeapullrequest.com)
@@ -36,6 +36,7 @@ WorldWideView is a real-time geospatial engine visualizing live global data on a
- **Real-Time Data Pipeline**: High-frequency WebSocket updates managed by a custom `DataBus`.
- **Advanced Entity Management**: Automatic horizon culling, chunked primitive rendering, and 3D stacking/spiderification.
- **Marketplace Integration**: Download and sync new plugins directly from the UI.
- **Agent Bus (opt-in)**: HTTP+SSE control surface that lets an external tool — typically an MCP server fronting an LLM — fly the globe, toggle layers, and select entities in the running browser session. Default off; see [Agent Bus docs](docs/agent-bus.md).
## Core Technologies
@@ -86,13 +87,13 @@ WorldWideView uses a multi-stage Dockerfile designed for standalone output. To d
**Mac/Linux:**
```bash
mkdir worldwideview && cd worldwideview
curl -fsSL https://raw.githubusercontent.com/silvertakana/worldwideview/main/setup.sh | bash
curl -fsSL https://raw.githubusercontent.com/silvertakana/worldwideview/main/self-host/setup.sh | bash
```
**Windows (PowerShell):**
```powershell
mkdir worldwideview; cd worldwideview
Invoke-WebRequest -Uri https://raw.githubusercontent.com/silvertakana/worldwideview/main/setup.ps1 -UseBasicParsing | Invoke-Expression
Invoke-WebRequest -Uri https://raw.githubusercontent.com/silvertakana/worldwideview/main/self-host/setup.ps1 -UseBasicParsing | Invoke-Expression
```
> [!NOTE]
@@ -145,6 +146,7 @@ WorldWideView is distributed across several specialized repositories:
3. **`worldwideview-plugins`**: First-party maintained plugins.
4. **`worldwideview-marketplace`**: The web application driving the plugin directory.
5. **`worldwideview-web`**: Marketing and landing site.
6. **[`szski/wwv-mcp`](https://github.com/szski/wwv-mcp)**: Reference MCP server for driving the globe from an LLM agent (Claude Code, Claude Desktop, Cursor, Cline, …) via the [Agent Bus](docs/agent-bus.md). Contributor-hosted today; may move under the project org.
## Development & Workflow
@@ -160,18 +162,19 @@ We welcome community contributions! Please review our coding standards and PR pr
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
This project is licensed under the Elastic License 2.0 - see the [LICENSE](LICENSE) file for details.
## Documentation Index
Explore our comprehensive documentation suite for detailed engineering insights:
- **[Project Overview](docs/project-overview.md)**: High-level value proposition and technology stack.
- **[Architecture](docs/architecture.md)**: DataBus event stream and Zustand state management.
- **[Architecture](docs/ARCHITECTURE.md)**: Plugin system, state management, data pipeline, and Cesium rendering deep-dive.
- **[Build System](docs/build-system.md)**: Monorepo structure, Next.js standalone output, and Docker builds.
- **[Development](docs/development.md)**: Coding conventions and common implementation patterns.
- **[Testing](docs/testing.md)**: Vitest setup and coverage targets.
- **[Deployment](docs/deployment.md)**: Coolify integration and persistent volumes.
- **[Agent Bus](docs/agent-bus.md)**: Wiring an MCP server (or any external tool) to drive the running globe.
- **[Files Catalog](docs/files.md)**: Comprehensive mapping of core source files.
> [!IMPORTANT]
-7
View File
@@ -1,7 +0,0 @@
# This file lists tools managed by Aftman, a cross-platform toolchain manager.
# For more information, see https://github.com/LPGhatguy/aftman
# To add a new tool, add an entry to this table.
[tools]
rojo = "rojo-rbx/[email protected]"
# rojo = "rojo-rbx/[email protected]"
-1
View File
@@ -1 +0,0 @@
C:/dev/worldwideview/agents.md
+33 -16
View File
@@ -1,10 +1,21 @@
services:
wwv:
build: .
build:
context: .
args:
- NEXT_PUBLIC_WWV_EDITION=${NEXT_PUBLIC_WWV_EDITION:-local}
- NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL=${NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL:-}
- NEXT_PUBLIC_SUPABASE_URL=${NEXT_PUBLIC_SUPABASE_URL:-}
- NEXT_PUBLIC_SUPABASE_ANON_KEY=${NEXT_PUBLIC_SUPABASE_ANON_KEY:-}
- NEXT_PUBLIC_GOOGLE_MAPS_API_KEY=${NEXT_PUBLIC_GOOGLE_MAPS_API_KEY:-}
- NEXT_PUBLIC_BING_MAPS_KEY=${NEXT_PUBLIC_BING_MAPS_KEY:-}
- NEXT_PUBLIC_CESIUM_ION_TOKEN=${NEXT_PUBLIC_CESIUM_ION_TOKEN:-}
- NEXT_PUBLIC_ADSENSE_CLIENT_ID=${NEXT_PUBLIC_ADSENSE_CLIENT_ID:-}
ports:
- "3000:3000"
volumes:
- wwv-data:/app/data
- ./public/plugins-local:/app/public/plugins-local:ro
env_file:
- path: .env
required: false
@@ -12,7 +23,7 @@ services:
- "AUTH_SECRET=${AUTH_SECRET:-default_secret_for_local_dev}"
- "WWV_DEMO_ADMIN_SECRET=${WWV_DEMO_ADMIN_SECRET:-}"
- "WWV_BRIDGE_TOKEN=${WWV_BRIDGE_TOKEN:-}"
- "DATABASE_URL=${DATABASE_URL:-postgresql://postgres:postgres@db:5432/worldwideview?schema=public}"
- "DATABASE_URL=postgresql://postgres:postgres@db:5432/worldwideview?schema=public"
depends_on:
db:
condition: service_healthy
@@ -22,16 +33,17 @@ services:
timeout: 10s
retries: 3
start_period: 30s
restart: unless-stopped
db:
image: postgres:15-alpine
restart: always
restart: unless-stopped
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: worldwideview
ports:
- "5432:5432"
- "${WWV_DB_PORT:-5432}:5432"
volumes:
- postgres-data:/var/lib/postgresql/data
healthcheck:
@@ -40,25 +52,28 @@ services:
timeout: 5s
retries: 5
start_period: 60s
profiles:
- "${WWV_SKIP_LOCAL_DB:-}"
wwv-data-engine:
build: ../wwv-data-engine-public
image: ghcr.io/silvertakana/wwv-data-engine:latest
command: >
sh -c "
echo 'Syncing workspace dependencies...' &&
cd $$SEEDERS_DIR/community && CI=true pnpm install --ignore-scripts && pnpm -r rebuild better-sqlite3 esbuild &&
if [ -d \"$$SEEDERS_DIR/private\" ]; then cd $$SEEDERS_DIR/private && CI=true pnpm install --ignore-scripts && pnpm -r rebuild better-sqlite3 esbuild; fi &&
cd /app && node dist/server.js"
ports:
- '5000:5000'
- "5000:5000"
environment:
- PORT=5000
- REDIS_URL=redis://wwv-redis:6379
- SEEDERS_DIR=/mnt/host/c/dev/worldwideview/local-seeders
- SKIP_SEEDER_DOWNLOAD=true
- SEEDERS_DIR=/app/seeders
- DB_PATH=/app/data/engine.db
depends_on:
wwv-redis:
condition: service_healthy
volumes:
- ./local-seeders:/mnt/host/c/dev/worldwideview/local-seeders
restart: unless-stopped
profiles:
- engine
wwv-redis:
image: redis:7-alpine
@@ -67,11 +82,13 @@ services:
interval: 5s
timeout: 3s
retries: 5
volumes:
wwv-data:
postgres-data:
profiles:
- engine
networks:
coolify:
external: true
volumes:
wwv-data:
postgres-data:
+193 -26
View File
@@ -1,33 +1,200 @@
<!-- Generated: 2026-04-23 06:11:00 UTC -->
# Architecture
## Overview
WorldWideView operates on a decoupled, plugin-based architecture designed for real-time visualization of geospatial data. The system is split between a Next.js 16 frontend that handles UI and CesiumJS 3D rendering, and a network of plugin data seeders that stream live updates via WebSockets.
WorldWideView is a **real-time geospatial intelligence engine**: a Next.js 16 frontend that renders live global data on a CesiumJS 3D globe, fed by a network of independent data seeders streaming over WebSockets. The whole platform is built on an "All-Bundle" plugin architecture, so data sources are decoupled from the core viewer and can be added, removed, or hot-swapped without touching rendering code.
The core design philosophy is the "All-Bundle Model," where plugins are dynamically imported at runtime as ES module bundles (`Dynamic CDN Loaded`). The frontend maintains a modular state architecture using Zustand, routing high-frequency data updates through a custom `DataBus` singleton to avoid React render cycle bottlenecks.
This document is the deep-dive companion to the [README](../README.md). It is intended for human contributors who want to understand how the pieces fit before opening their first PR. For agent-facing rules and conventions, see [`AGENTS.md`](../AGENTS.md) at the repo root and the documents under [`.agents/rules/`](../.agents/rules/).
## Component Map
- **Frontend App Shell:** Next.js App Router providing layout and base API routes (in `src/app/`).
- **Globe Rendering Engine:** CesiumJS wrapped in Resium, responsible for 3D primitive and model rendering (in `src/core/globe/`).
- **State Management:** Zustand slice-based architecture (in `src/core/state/`).
- **Data Pipeline:** Custom event bus (`DataBus`) and WebSocket clients (`DataBusSubscriber`) routing live data to state (in `src/core/data/`).
- **Plugin System:** Registration, lifecycle management, and ES module loaders (in `src/core/plugins/`).
- **Database Layer:** Prisma ORM connecting to PostgreSQL (in `prisma/`).
## System Overview
## Key Files
- `src/core/plugins/PluginManager.ts` - The PluginManager class. Core registry that instantiates plugins, calls `initialize()`, and handles lifecycle events.
- `src/core/plugins/loaders/InstalledPluginsLoader.ts` - Dynamically fetches and imports `import(/* webpackIgnore: true */ ...)` marketplace plugins into the environment.
- `src/core/data/DataBus.ts` - Custom typed event bus singleton that routes high-frequency data around the application.
- `src/core/globe/GlobeView.tsx` - The main Cesium viewer container, handling imagery layers, cameras, and primitive collections.
- `src/core/state/store.ts` - The main Zustand store registry linking the nine state slices (globe, layers, timeline, ui, filter, data, config, favorites, geojson).
```mermaid
flowchart TD
subgraph "Data Sources"
A[Public APIs] -->|Poll| B[Data Engine Seeders]
C[Custom Microservices] -->|Stream| B
end
## Data Flow
Real-time data follows a strict unidirectional stream optimized for sub-second updates:
subgraph "Engine Tier"
B -->|WebSocket /stream| D[WsClient]
end
1. **Engine Push:** A remote data engine pushes state over WebSocket (`/stream`) from either local seeders (`ws://localhost:5000/stream`) or the cloud fallback (`wss://dataengine.worldwideview.dev/stream`).
2. **WebSocket Client Router:** `WsClient.handleMessage()` receives the payload and pipes it to the event bus.
3. **DataBus Emission:** The bus fires `DataBus.emit("websocketData", WsStreamPayload)`.
4. **State Hydration:** The `DataBusSubscriber` component listens to the bus and calls `_hydrateSnapshot()` to update the Zustand store's `entitiesByPlugin` cache.
5. **Memoized Selection:** `GlobeView` selects the visible entities from the store.
6. **Primitive Rendering:** The `EntityRenderer` translates store entities into Cesium `PointPrimitiveCollection` and `BillboardCollection` arrays for GPU rendering.
7. **Animation & LOD:** `AnimationLoop` handles horizon culling and hover effects, while `useModelRendering` promotes close-range entities to 3D glTF models.
subgraph "Frontend Pipeline"
D --> E[DataBus<br/>typed pub/sub]
E -->|"_hydrateSnapshot()"| F[Zustand Store<br/>9 slices]
F -->|Memoized visible entities| G[EntityRenderer]
G -->|Primitive collections| H[CesiumJS Globe]
H --> I[AnimationLoop<br/>horizon culling + LOD]
end
subgraph "Plugin Layer"
J[PluginRegistry] -->|"loadPluginFromManifest"| K[Dynamic ES module imports]
J -.->|register| E
end
```
The strict unidirectional flow — engine → bus → store → renderer — is what lets the system push sub-second updates for thousands of entities without React render-cycle bottlenecks.
## Plugin System (Core Abstraction)
Every data source in WorldWideView is a **plugin** implementing the `WorldPlugin` interface from [`@worldwideview/wwv-plugin-sdk`](../packages/wwv-plugin-sdk/). The plugin's lifecycle plugs into a real-time WebSocket firehose:
```text
PluginRegistry.register() → PluginManager.registerPlugin()
→ plugin.initialize(context)
Visibility Toggle → DataBusSubscriber subscribes to layer via WsClient
→ Engine responds with instantaneous WebSocket snapshots over /stream
→ WsClient pipes to DataBus.emit("dataUpdated", WsStreamPayload)
→ Store → EntityRenderer → Globe
```
Four plugin architectures coexist under the "All-Bundle Model":
1. **Data Engine Seeder** — A lightweight `seeder.mjs` script run by the dynamic `wwv-data-engine` runner. (Previously a standalone microservice container; the runner replaces that.)
2. **Dynamic CDN Loaded (Bundle)** — Externally developed plugins, dynamically imported at runtime via ES module CDNs (e.g. version-pinned `unpkg.com` URLs).
3. **Static Compiled (Bundle)** — Static GeoJSON data wrapped into JS bundles via `wwvStaticCompiler` during build/sync. (Previously `StaticDataPlugin`.)
4. **Active Proxied (Bundle)** — Next.js API routes bundled to mediate frontend interactions. (Previously `DeclarativePlugin`.)
All four are now dynamically imported at runtime as ES module bundles via `loadPluginFromManifest`, which uses `import(/* webpackIgnore: true */ entry)`. The legacy `StaticDataPlugin` and `DeclarativePlugin` runtimes are fully deprecated.
> [!IMPORTANT]
> Plugin types are re-exported from the SDK through `src/core/plugins/PluginTypes.ts` and `src/core/plugins/PluginManifest.ts`. The **source of truth is always `@worldwideview/wwv-plugin-sdk`** — never define plugin types locally.
For the contributor checklist when authoring a plugin, see [`.agents/skills/worldwideview-plugin-creation/`](../.agents/skills/) and the [Plugin Quickstart](plugin-quickstart.md).
## State Management
WorldWideView uses a Zustand store split into **nine slices**, each in its own file under [`src/core/state/`](../src/core/state/):
| Slice | Responsibility |
|---|---|
| `globe` | Camera position, imagery layers, scene settings |
| `layers` | Per-plugin visibility toggles |
| `timeline` | Playback range, current time, play/pause |
| `ui` | Panel open/closed states, selected entity |
| `filter` | Plugin filter expressions, applied via `filterEngine` |
| `data` | Entity cache (`entitiesByPlugin`), websocket connection state |
| `config` | Plugin settings, polling intervals, API keys |
| `favorites` | User-bookmarked entities |
| `geojson` | Custom GeoJSON overlays |
Access patterns:
- React components: `const camera = useStore((s) => s.camera);`
- Outside React (animation loop, hooks setup): `const state = useStore.getState();`
- Plugin settings: `configSlice.dataConfig.pluginSettings[pluginId]`
- Polling intervals: `configSlice.dataConfig.pollingIntervals[pluginId]`
The animation loop deliberately snapshots state once per frame via `getState()` rather than subscribing — `requestAnimationFrame` is the loop driver, so adding a Zustand subscription would just duplicate ticks.
## Data Pipeline
Real-time data follows a strict unidirectional path optimised for sub-second updates:
```text
Engine push /stream → DataBusSubscriber WsClient router
→ WsClient.handleMessage() → DataBus.emit("websocketData", WsStreamPayload)
→ DataBusSubscriber → _hydrateSnapshot() → Store.entitiesByPlugin
→ GlobeView (memoised visible entities)
→ EntityRenderer (billboard/point primitives)
→ AnimationLoop (horizon culling, hover/selection)
→ StackManager (co-located entity grouping)
```
The `DataBus` is a custom typed pub/sub singleton (see [`src/core/data/DataBus.ts`](../src/core/data/DataBus.ts)). It exists specifically to bypass React's render cycle for high-frequency events — store updates are batched into the Zustand store, but raw stream events ride the bus directly.
## Data Engine & Seeders
The data engine ([`wwv-data-engine`](https://github.com/silvertakana/wwv-data-engine), public) is a **content-agnostic runner**. It discovers and executes seeder scripts from a configurable directory; the engine itself knows nothing about specific data sources.
- **Local development**: The engine runs via Docker Compose on port 5001, reading seeders dynamically from `local-seeders/` (split into `community` and `private` tiers).
- **Production**: The engine container on Coolify downloads release bundles from `wwv-seeders-community` and `wwv-seeders-private` on startup, unzipping them into `/app/seeders`.
- **Split-routing**: `resolveEngineUrl` prioritises the local dev engine (`ws://localhost:5001/stream`) for local testing — 12-Factor App methodology — and falls back to cloud-hosted endpoints (e.g. `wss://dataenginev2.worldwideview.dev/stream`) when nothing local is available.
### Agnostic Frontend
WorldWideView's frontend is a **completely agnostic renderer**. It has no concept of a "unified" data engine. If 30 plugins each declare a different `streamUrl`, the application opens 30 WebSocket connections; it just so happens that the default plugin set shares `wwv-data-engine`, but the platform is 100% decentralised.
> [!IMPORTANT]
> Each plugin is a self-contained package and **MUST explicitly declare its own `streamUrl` in its manifest or config.** Do not assume the frontend acts as a unified pipe.
### Dual-Output Engine
Each seeder automatically exposes both a WebSocket stream (`/stream`) and a REST API endpoint (`/api/[plugin-id]`). Plugins can consume either depending on their data shape — bursty / live → stream, slow-moving / on-demand → REST.
### Scope Boundary (the 99 / 1 Rule)
The engine handles standard caching and broadcasting — the 99% case. Plugins that require complex on-demand compute (the 1%) must host their own custom backend; the engine intentionally does not grow features for them.
## Rendering Pipeline
The renderer is where most of WorldWideView's engineering depth lives. Three design choices drive the performance envelope:
### Primitive-based rendering
We use [`PointPrimitiveCollection`](https://cesium.com/learn/cesiumjs/ref-doc/PointPrimitiveCollection.html), `BillboardCollection`, `LabelCollection`, and `PolylineCollection` directly — **not** the Cesium Entity API. The Entity API is convenient but allocates per-entity ECS-style state that gets expensive past a few thousand entities. Primitive collections batch their state into typed GPU buffers, so 10k+ live entities still hold 60 fps on a moderate laptop.
Collections are tracked per-viewer via a module-level `WeakMap<CesiumViewer, PrimitiveCollections>` in [`src/core/globe/EntityRenderer.ts`](../src/core/globe/EntityRenderer.ts), keeping the renderer multi-viewer-capable and ensuring collections GC cleanly when a viewer is torn down.
### Chunked processing
Large datasets (10k+ entities) are rendered via [`ChunkedProcessor`](../src/core/globe/ChunkedProcessor.ts) in 500-entity chunks across multiple microtasks, so adding a new plugin layer never blocks the main thread long enough to drop a frame.
### Horizon culling and LOD
The animation loop performs **manual horizon culling** in [`AnimationLoop.ts`](../src/core/globe/AnimationLoop.ts) using a single dot-product against the Earth radius — much cheaper than depth-testing thousands of primitives every frame, and tunable for "entities just below the horizon should still be visible" semantics that depth-testing can't express.
Entities of `type: "model"` are promoted to full 3D glTF models at close range via the LOD system in [`useModelRendering.ts`](../src/core/globe/hooks/useModelRendering.ts) — so an aircraft at 50 nm range is a tinted billboard, but at 5 nm it's a textured plane model with the right wing geometry.
### Stack / Spiderifier
When multiple entities share a screen-space cluster (busy ports, busy airspaces), [`StackManager`](../src/core/globe/StackManager.ts) groups them and [`stackAnimation`](../src/core/globe/stackAnimation.ts) handles the spiderification expansion when a stack is clicked.
### Entity-type rules
> [!WARNING]
> When returning `CesiumEntityOptions` from a plugin's `renderEntity()`:
>
> - **Points**: Use `type: "point"` with `color`, `size`, `outlineColor`, `outlineWidth`.
> - **Billboards**: Use `type: "billboard"` with `iconUrl`, `color`, `iconScale`.
> - **NEVER mix**: Do not set `size` / `outlineWidth` / `outlineColor` on a billboard — Cesium silently clips the GPU upload and the icon disappears.
## Edition System
WorldWideView ships in three editions, controlled by the `NEXT_PUBLIC_WWV_EDITION` environment variable. Feature flags are derived from this in [`src/core/edition.ts`](../src/core/edition.ts).
| Edition | Auth | Use case |
|---|---|---|
| `local` | Enabled (NextAuth credentials) | Self-hosted single-user / small-team deployments |
| `cloud` | Enabled | Managed cloud instance with multi-tenancy |
| `demo` | Disabled (optional admin via `WWV_DEMO_ADMIN_SECRET`) | Public demo deployments |
Edition isn't just an auth toggle — it also gates which marketplace endpoints are queried, which default plugins boot, and whether destructive admin actions are reachable.
## Key Source Files
Quick map of the load-bearing modules:
| Path | Role |
|---|---|
| [`src/core/plugins/PluginManager.ts`](../src/core/plugins/PluginManager.ts) | Core plugin registry. Instantiates plugins, calls `initialize()`, manages lifecycle. |
| [`src/core/plugins/loaders/InstalledPluginsLoader.ts`](../src/core/plugins/loaders/InstalledPluginsLoader.ts) | Dynamic ES module loader for marketplace plugins (`import(/* webpackIgnore: true */ entry)`). |
| [`src/core/data/DataBus.ts`](../src/core/data/DataBus.ts) | Typed pub/sub singleton — the high-frequency event channel. |
| [`src/core/data/WsClient.ts`](../src/core/data/WsClient.ts) | WebSocket router. Pipes engine `/stream` messages onto the DataBus. |
| [`src/core/globe/GlobeView.tsx`](../src/core/globe/GlobeView.tsx) | The Cesium viewer container. Imagery layers, camera setup, primitive collection wiring. |
| [`src/core/globe/EntityRenderer.ts`](../src/core/globe/EntityRenderer.ts) | Translates store entities into Cesium primitives. Hot path for 10k+ entities. |
| [`src/core/globe/AnimationLoop.ts`](../src/core/globe/AnimationLoop.ts) | Per-frame update: horizon culling, hover/selection, model LOD promotion. |
| [`src/core/state/store.ts`](../src/core/state/store.ts) | Zustand store registry linking the nine slices. |
## Further Reading
For deeper coverage of specific subsystems, see the rule files under [`.agents/rules/`](../.agents/rules/):
- [`platform-architecture.md`](../.agents/rules/platform-architecture.md) — high-level platform goals, product vision, Edition System
- [`application-architecture.md`](../.agents/rules/application-architecture.md) — Next.js frontend, Zustand, CesiumJS integration
- [`plugin-architecture.md`](../.agents/rules/plugin-architecture.md) — plugin lifecycle, capability declarations, seeders
- [`marketplace-architecture.md`](../.agents/rules/marketplace-architecture.md) — dynamic plugin installation, DB sync, CDN loading
- [`cesium-rendering.md`](../.agents/rules/cesium-rendering.md) — globe rendering, entity types, primitives, LOD, culling
- [`state-management.md`](../.agents/rules/state-management.md) — slice access patterns, plugin settings
- [`data-engine-architecture.md`](../.agents/rules/data-engine-architecture.md) — engine internals, seeder loading, workspace dependencies
For getting started as a contributor, see [Development](development.md), [Plugin Quickstart](plugin-quickstart.md), and [Plugin Advanced](plugin-advanced.md).
+166
View File
@@ -0,0 +1,166 @@
# Agent Bus — driving WWV from external tools
## Overview
The Agent Bus is an opt-in HTTP+SSE surface that lets a trusted external process — most commonly a [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server fronting an LLM agent — drive the running WWV browser session. The agent can fly the globe, toggle layers, focus entities, and select things by posting JSON actions to a server endpoint, which fans them out over SSE to every browser tab belonging to the same logged-in user.
Read-only camera and engine queries already work via the existing REST API. The Agent Bus is the missing **write path**.
```
LLM client ──MCP──▶ szski/wwv-mcp ──HTTP──▶ /api/agent/publish ──SSE──▶ browser
(Claude, Cursor, …) (separate repo) (this PR adds these) (Cesium flies)
```
## Enabling the bus
The bus is **off by default**. The subscriber early-returns and no SSE connection is opened. You explicitly opt in via a build-time environment variable.
### In Docker
Pass the ARG at build time:
```sh
docker build \
--build-arg NEXT_PUBLIC_WWV_AGENT_BUS_ENABLED=true \
-t worldwideview .
```
### In a Coolify / hosted build
Set `NEXT_PUBLIC_WWV_AGENT_BUS_ENABLED=true` in the build environment. Because it's a `NEXT_PUBLIC_*` variable, it is inlined into the JavaScript bundle at `next build` time — runtime-only env changes will not enable it.
### Verifying it's on
On a fresh page load, open the browser DevTools console. You should see a one-line build banner:
```
[wwv build] id=1714572934123 built_at=2026-05-13T18:35:34Z agent_bus=on
```
If `agent_bus=off` after enabling the env var, the build did not pick up the variable — re-check that it was passed as `--build-arg` (Docker) or set before `next build` ran (other hosts).
You can also hit `/api/build` to confirm the served bundle's build id and flag state.
## Using it — quick smoke test with curl
To verify the bus end-to-end without an MCP server, log into WWV in your browser, grab your `__Secure-authjs.session-token` cookie value from DevTools, and POST an action:
```sh
curl -X POST https://your-wwv-host/api/agent/publish \
-H "Content-Type: application/json" \
-H "Cookie: __Secure-authjs.session-token=PASTE_TOKEN_HERE" \
-d '{"action":"fly_to","lat":35.7796,"lon":-78.6382,"distance":50000}'
```
If the bus is enabled and the cookie is valid, the open tab will fly to Raleigh and the response will be:
```json
{ "ok": true, "delivered": 1, "subscribers": 1 }
```
`subscribers: 0` means the bus is off in the build, the cookie's user has no open tabs, or the env var wasn't applied — see *Verifying* above.
## Using it — wiring an MCP server
The reference MCP server for WWV is published at **[szski/wwv-mcp](https://github.com/szski/wwv-mcp)**. It exposes 15 tools (`globe_fly_to`, `layer_toggle`, `engine_query`, `geocode`, `system_status`, etc.) and works with any MCP-aware client (Claude Code, Claude Desktop, Cursor, Continue, Cline).
> [!NOTE]
> `szski/wwv-mcp` lives in a contributor's namespace today and may move under the project's own organization later. Update the install URL accordingly when that happens.
### Install (Node)
```sh
git clone https://github.com/szski/wwv-mcp
cd wwv-mcp
npm install
npm run build
```
Then store credentials so the MCP server can auto-login at startup:
```sh
mkdir -p ~/.config/wwv-mcp
cat > ~/.config/wwv-mcp/credentials <<'EOF'
[email protected]
WWV_PASSWORD=your-password
EOF
chmod 600 ~/.config/wwv-mcp/credentials
```
### Install (Docker)
If a published image exists at `szski/wwv-mcp` (or wherever the maintainer mirrors it), no install step is needed — the MCP client config below uses `docker run` to launch the server on demand.
### Client config — Claude Code
Edit `~/.claude/settings.json`:
```json
{
"mcpServers": {
"wwv": {
"command": "node",
"args": ["/absolute/path/to/wwv-mcp/dist/index.js"],
"env": {
"WWV_BASE_URL": "https://your-wwv-host.example",
"WWV_ENGINE_URL": "http://localhost:5001"
}
}
}
}
```
Or, with Docker:
```json
{
"mcpServers": {
"wwv": {
"command": "docker",
"args": [
"run", "-i", "--rm",
"-e", "WWV_BASE_URL=http://host.docker.internal:3000",
"-e", "WWV_ENGINE_URL=http://host.docker.internal:5001",
"szski/wwv-mcp:latest"
]
}
}
}
```
Then `/mcp` in Claude Code should list `wwv` with all 15 tools.
### Client config — Claude Desktop
Edit `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows). The `mcpServers.wwv` block is identical to Claude Code above.
### Client config — Cursor, Continue, Cline
Each of these reads an MCP server list in its own settings UI but the shape of the entry — `command` + `args` + `env` — is the same as Claude Code's. See your client's MCP docs for the exact location of its config file.
### Verifying the wiring
Once configured, ask the agent something like *"What's the status of the WWV system?"* — it should call `system_status` and report back. If you see `agent_bus_enabled: false`, the bus is off in the WWV build; fix that before trying write tools like `globe_fly_to`.
## Security and multi-tenancy
- **Default off.** Without the env var, the subscriber doesn't open a connection, the publish endpoint exists but has no subscribers, and the bundle's dead-code elimination removes the `EventSource` constructor entirely.
- **Auth gated.** Both `/api/agent/publish` and `/api/agent/stream` use the same `auth()` helper as the rest of the marketplace API. No tokens — just the existing Auth.js session cookie.
- **Per-user scoping.** Subscriptions live in `Map<userId, …>`, so a publish from user A's session never reaches user B's stream, even though both sit on the same Node process. Multi-tenant deployments are safe.
- **Same trust model as a browser tab.** An MCP server "acts as the user" by holding the user's session cookie. If you wouldn't trust a browser tab on the same machine, don't give an MCP server the cookie.
- **Single-process only.** The `AgentBus` is in-memory and process-local. Multi-instance deployments behind a load balancer would need a Redis pub/sub adapter — about 30 lines of additional code, not included here.
## Endpoint reference
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/agent/stream` | SSE channel. Auth-gated. Streams `AgentAction` JSON events for the connected user. Heartbeat every 25s. |
| `POST` | `/api/agent/publish` | Accept an `AgentAction` JSON body, broadcast to the publishing user's subscribers, return `{ok, delivered, subscribers}`. |
| `GET` | `/api/build` | Build id + ISO timestamp + `NEXT_PUBLIC_*` flag state. Used by `AgentBusSubscriber` to print the build banner. |
`AgentAction` is currently a six-verb union: `fly_to`, `face_towards`, `layer_toggle`, `highlight_layer`, `select_entity`, `ping`. The shape is `{action: string, …payload}` JSON, so additive changes are backwards-compatible.
## Companion repos
- **[szski/wwv-mcp](https://github.com/szski/wwv-mcp)** — reference MCP server. 15 tools. Required for any LLM-driven use of the bus.
- **[silvertakana/wwv-data-engine](https://github.com/silvertakana/wwv-data-engine)** — separate repo, supplies the read-only `engine_query` data that the MCP server's read tools wrap.
@@ -0,0 +1,152 @@
# ADR-001: Decentralized Plugin Authentication & SSRF Mitigation
## Status
Proposed *(Pending Review)*
## Date
2026-05-14
## Related
- **Dependencies:** `docs/ARCHITECTURE.md` (Agnostic Client principle), `prisma/schema.prisma` (existing Local App Postgres integration).
---
## Context
The WorldWideView ecosystem faces four interlocking security and scalability problems:
1. **The SSRF Liability:** The Local App uses generic proxy routes (`/api/camera/proxy`) to bypass browser CORS restrictions. These lack robust SSRF hardening (no DNS rebinding defense, no private IP blocking), exposing the host LAN.
2. **The CORS Barrier:** Browser-enforced Cross-Origin Resource Sharing (CORS) prevents the agnostic frontend from connecting directly to independent Data Engines via standard HTTP.
3. **The Centralization Bottleneck:** If every Data Engine queries the Marketplace database to verify a user's subscription on every frame, the Marketplace becomes a catastrophic bottleneck.
4. **The Credential Interception Threat:** Passing credentials via simple URL redirects during setup exposes them to browser history theft and MITM attacks.
To resolve these, we are adopting an umbrella "Token Exchange" architecture, broken down into four specific decisions.
---
## Decisions
### ADR-001A: Direct WebSocket Data Plane
We mandate that heavy data streams utilize WebSockets (`wss://`). Browsers do not apply CORS preflight to WebSockets, allowing direct connections.
* **Security Constraint:** Because CORS is bypassed, Data Engines MUST explicitly validate the `Origin`, `Host`, TLS state, and expected tenant/plugin metadata upon connection.
### ADR-001B: Marketplace-Issued JWT Tickets
To enforce monetization without centralized database queries, we implement a stateless Token Exchange via Asymmetric Cryptography (Ed25519).
* **The JWT Contract:** The token MUST contain the following claims:
* `iss`: `https://marketplace.worldwideview.dev`
* `sub`: The user's unique ID.
* `aud`: **Crucial.** The specific Data Engine or plugin ID. Tokens MUST be audience-bound to prevent a valid token for Engine A being replayed against Engine B.
* `exp`: Expiration (exactly 5 minutes from issuance).
* `iat`: Issued at.
* `jti`: Unique token identifier.
* `tier`: Subscription level (e.g., "pro").
* `scope`: Specific entitlements.
### ADR-001C: PKCE Bootstrap (The API Key)
The Local App acts as the credential holder. It uses the OAuth 2.0 Authorization Code flow with PKCE to securely acquire a **long-lived refresh credential** (The API Key).
* **API Key Lifetime:** The API Key does not expire automatically but is bound to the specific device/installation. It rotates manually via the Marketplace UI or automatically upon suspicion of compromise.
### ADR-001D: SSRF Proxy Hardening
For legacy plugins (e.g., Webcams) that *cannot* use WebSockets, the local proxy must be hardened.
* **Constraints:** We will remove generic proxying where possible. Where proxying remains, we enforce:
1. Exact HTTPS origin allowlists.
2. Blocking of private / link-local / loopback IPs *after* DNS resolution.
3. Re-resolving or pinning DNS to defend against DNS rebinding.
4. Rejecting redirects to non-allowlisted hosts.
5. Strict payload size and time limits.
---
## Threat Model & Non-Goals
**In Scope:**
* Defending against external attackers on the user's LAN (via SSRF hardening).
* Preventing interception of the long-lived API key during setup (via PKCE).
* Preventing URL/Header credential leakage (via First-Message Auth).
**Non-Goals (Accepted Residual Risks):**
* Defending against compromised Local App hosts (root OS access).
* Defending against XSS in the Local App frontend (token theft via JS).
* Defending against malicious browser extensions (full session access).
* Real-time subscription revocation for *already issued* 5-minute tickets.
---
## Operational Implementation Details
### 1. Library Selection
* **`jose`:** The industry-standard library for modern JS. Used by the Marketplace to generate Ed25519 keys, sign JWTs, and expose the JWKS endpoint.
* **`openid-client`:** The RFC-compliant library used by the Local App to execute the PKCE Authorization Code flow securely.
* **`@fastify/jwt` + `get-jwks`:** Used by Data Engines. `@fastify/jwt` provides the middleware, while `get-jwks` (built by the Fastify core team) handles remote fetching, local LRU caching, and OKP-to-PEM conversion for Ed25519 keys.
### 2. Hardened First-Message Auth
Data Engines MUST enforce strict WebSocket handshake rules:
1. **Silence until Auth:** The server MUST accept no subscription or data messages before auth succeeds.
2. **Timeout:** Auth MUST complete within a short timeout (e.g., 3 seconds of connection).
3. **Drop on Failure:** A failed auth immediately closes the socket with a `4003` code.
4. **No Re-Auth:** Re-authentication on the same socket is forbidden.
5. **Log Redaction:** All server logs MUST redact the JWT payload from capture.
### 3. API Key Encryption at Rest (Local App)
The Local App stores the API Key in PostgreSQL. It MUST NOT be stored in plaintext.
* **Algorithm:** AES-256-GCM.
* **Key Derivation (KDF):** The encryption key is derived from `AUTH_SECRET` using PBKDF2 (100,000+ iterations) and a unique salt per installation.
* **Nonce/Salt Storage:** The 96-bit nonce and the KDF salt are stored alongside the encrypted payload in the DB.
* **Versioning:** The DB column must support key versioning (e.g., `v1:base64_nonce:base64_ciphertext`) to allow future `AUTH_SECRET` rotation without permanently bricking the DB.
---
## Failure-Mode Behavior
| Scenario | System Behavior |
| :--- | :--- |
| **Marketplace is down** | Local App cannot get new JWTs. Existing streams continue until the 5-min JWT expires. |
| **Data Engine Cold Start (Marketplace down)** | Engine cannot fetch `jwks.json`. Engine **fails closed** and rejects all connections. |
| **JWKS Cache expires (Marketplace down)** | Data Engine retains stale cache for a grace period, but eventually **fails closed**. |
| **Redis is down (Marketplace)** | Ticket issuance falls back to PostgreSQL (slower, but functional). |
| **Unknown `kid` arrives** | Data Engine immediately triggers a background re-fetch of `jwks.json`. If still unknown, rejects token. |
| **Clocks drift beyond leeway** | Token is rejected (Next.js and Fastify default to 60s leeway). NTP sync is a hard operational requirement. |
---
## Revocation Latency
| Event | Time to take effect |
| :--- | :--- |
| **User cancels subscription** | Up to 5 minutes (bounded strictly by active JWT lifetime). |
| **Admin rotates Ed25519 key** | Zero downtime (handled via key overlap in JWKS array). |
| **User revokes API Key** | Immediate for *new* issuances, up to 5 minutes for active streams. |
---
## The End-to-End Auth Flow
```mermaid
sequenceDiagram
participant U as User Browser
participant L as Local App
participant M as Marketplace
participant E as Data Engine
Note over U,M: ADR-001C: PKCE bootstrap (once)
U->>L: Click "Connect to Marketplace"
L->>L: Generate code_verifier + challenge
L->>U: Redirect to Marketplace with challenge
U->>M: Authorize
M->>U: Redirect with temp code
U->>L: Forward temp code
L->>M: POST temp code + code_verifier
M->>L: API Key (Long-lived)
L->>L: PBKDF2 + AES-GCM Encrypt & Store
Note over L,E: ADR-001B & A: Ongoing Data Plane
loop every 4.5 minutes
L->>M: POST API Key (Request audience: Engine A)
M->>L: 5-min JWT (aud: Engine A)
L->>U: JWT
U->>E: wss://engine (Origin validation)
U->>E: { type: "auth", jwt: "..." }
E->>E: Verify with get-jwks cache
E->>U: Stream
end
```
+2 -2
View File
@@ -11,7 +11,7 @@ The core build configuration resides in `package.json` and `next.config.ts`. A c
### Standard Development
- **Install Dependencies:** `pnpm install` (Installs all workspace packages defined in `pnpm-workspace.yaml`).
- **Initial Setup:** `pnpm run setup` (Executes `scripts/setup.mjs` to generate a `.env.local` with a fresh `AUTH_SECRET`).
- **Run Development Server (All):** `pnpm dev:all` (Concurrently starts the Next.js frontend and the local `wwv-data-engine` + Redis containers via Docker Compose).
- **Run Development Server (All):** `pnpm dev:all` (Concurrently starts the Next.js frontend and the local `wwv-data-engine-v2` + Redis containers via Docker Compose).
### Production Build
- **Build Command:** `pnpm build` (Executes `next build --webpack`).
@@ -21,7 +21,7 @@ The core build configuration resides in `package.json` and `next.config.ts`. A c
## Platform Setup
- **Database Migrations:** Prisma schema is located at `prisma/schema.prisma`. During development, `pnpm dev` triggers a `predev` hook running `npx prisma migrate deploy` to ensure PostgreSQL tables are initialized.
- **Data Engine:** The generic data engine runner and Redis cache are started via `docker compose up wwv-data-engine wwv-redis`. Custom data seeders are placed in the `local-seeders/` sandbox and dynamically loaded by the engine.
- **Data Engine:** The generic data engine runner and Redis cache are started via `docker compose up wwv-data-engine-v2 wwv-redis`. Custom data seeders are placed in the `local-seeders/community/` or `local-seeders/private/` sandboxes and dynamically loaded by the engine.
- **Plugin Packages:** Internal plugin packages must be properly listed in `tsconfig.json` paths and `next.config.ts`'s `transpilePackages` array.
## Reference
+2 -2
View File
@@ -11,7 +11,7 @@ WorldWideView utilizes a containerized deployment strategy based on Docker's mul
- **Static Assets:** Cesium static workers are injected at build time via `scripts/copy-cesium.mjs`.
### Data Engine Runner
- The generic `wwv-data-engine` runner is deployed as a single Coolify service, orchestrating telemetry data by executing lightweight scripts from volume-mounted seeder directories.
- The generic `wwv-data-engine-v2` runner is deployed as a single Coolify service, orchestrating telemetry data by executing lightweight scripts from volume-mounted seeder directories. It features a dual-output architecture, exposing both WebSocket streams and REST APIs natively.
## Platform Deployment
@@ -19,7 +19,7 @@ WorldWideView utilizes a containerized deployment strategy based on Docker's mul
WorldWideView deploys optimally to Coolify using a Dockerfile builder.
- **Environment Variables:** Must be explicitly mapped in the Coolify UI (e.g., `DATABASE_URL`, `AUTH_SECRET`).
- **Persistent Storage:** PostgreSQL databases must be hosted externally or mounted via persistent volumes to ensure the frontend registry (installed plugins, user configs) survives container rebuilds.
- **Seeder Deployment:** The generic `wwv-data-engine` runner is deployed as a Coolify service, reading plugin seeders dynamically from a volume-mounted directory. Seeders are pulled from private/community repositories via GitHub releases.
- **Seeder Deployment:** The generic `wwv-data-engine-v2` runner is deployed as a Coolify service, reading plugin seeders dynamically from a volume-mounted directory. Seeders are pulled from private/community repositories via GitHub releases, maintaining strict namespace separation between community and private plugins.
### Docker Structure
- **Dockerfile:** Found at the project root (`Dockerfile`). Uses an Extractor Pattern (`deps` → `builder` → `runner`). The `.git` and `node_modules` folders must be explicitly untracked to prevent cache overlap.
+1 -1
View File
@@ -65,7 +65,7 @@ Whenever generating temporary debugging scripts, testing REST endpoints via `.mj
- *Reference:* The root directory is strictly for production configuration files.
### Local Plugin & Seeder Testing
When developing a custom seeder for a plugin, place the seeder directory inside `local-seeders/`. The local `wwv-data-engine` Docker container automatically mounts this folder and discovers scripts on startup, providing a seamless sandbox.
When developing a custom seeder for a plugin, place the seeder directory inside `local-seeders/community/` or `local-seeders/private/`. The local `wwv-data-engine-v2` Docker container automatically mounts this folder and discovers scripts on startup, providing a seamless sandbox.
### Scaffold a New Component
1. Create a new directory under `src/components/{category}/MyComponent`.
+1 -1
View File
@@ -21,7 +21,7 @@ The system is highly modularized to separate React UI rendering from heavy Cesiu
## Build System
- `package.json` - Root monorepo configuration with `pnpm` workspace bindings.
- `next.config.ts` - Next.js compiler settings, security headers, and standalone output configuration.
- `docker-compose.yml` - Defines the orchestration of the frontend, Redis cache, and the generic data engine runner (`wwv-data-engine`) loaded with local seeders.
- `docker-compose.yml` - Defines the orchestration of the frontend, Redis cache, and the DataEngineV2 runner (`wwv-data-engine-v2`) loaded with local seeders.
## Configuration
- `prisma/schema.prisma` - PostgreSQL database schema for storing user settings and installed plugin manifests.
+48 -35
View File
@@ -53,21 +53,44 @@ Your workspace is now ready!
---
## Step 2: Creating the Data Engine Seeder (Backend)
## Step 2: Scaffold the Plugin and Backend Seeder
The ISS moves incredibly fast. If our frontend tries to ask for its location every second, it will lag. Instead, we use the **Bring Your Own Backend (BYOB)** pattern. We will write a "Seeder" script in the Data Engine that fetches the ISS location and instantly broadcasts it to the frontend via WebSockets.
### 2.1 Create the Seeder Directory and File
We will use the workspace CLI tool to generate a blank plugin template and its associated backend seeder in one go. In your terminal, navigate to your main `worldwideview` folder and run:
Navigate to your `worldwideview` repository. Inside the `local-seeders` folder, create a new folder named `community/iss`. Inside that folder, create a file named `seeder.mjs`:
`c:\dev\worldwideview\local-seeders\community\iss\seeder.mjs`
```bash
node packages/wwv-cli/dist/index.js create
```
The CLI will launch an interactive survey. Answer as follows:
1. **Unique ID**: `iss`
2. **Display Name**: `ISS Live Tracker`
3. **Description**: `Real-time International Space Station tracking.`
4. **Category**: `Space`
5. **Data Architecture**: `Real-Time WebSockets (Generates backend seeder)`
6. **Globe Rendering**: `2D Billboard`
7. **Community Sharing**: `Yes, public community seeder`
This automatically creates:
- Your frontend plugin at `local-plugins/wwv-plugin-iss`
- Your backend seeder at `local-seeders/community/iss`
Now install the dependencies to finalize the setup:
```bash
pnpm install
```
> [!NOTE]
> **Dependency Management:** You do *not* need to create a `package.json` or manually install standard dependencies (like `axios`) for your seeder. The `wwv-data-engine` runner dynamically resolves these via the pnpm workspace, keeping your seeder lightweight.
> **Dependency Management:** You do *not* need to create a `package.json` or manually install standard dependencies (like `axios`) for your backend seeder. The `wwv-data-engine-v2` runner dynamically resolves these via the pnpm workspace, keeping your seeder lightweight.
### 2.2 Write the Polling Logic
---
Open `seeder.mjs` in your text editor and paste the following code. The Data Engine Runner will dynamically discover this script and execute its `fetch(ctx)` function based on the interval you define.
## Step 3: Write the Polling Logic (Backend Seeder)
Open the newly generated seeder file in your text editor: `local-seeders/community/iss/seeder.mjs`.
The Data Engine Runner will dynamically discover this script and execute its `fetch(ctx)` function based on the interval you define. Replace its contents with this code:
```javascript
// seeder.mjs
@@ -116,11 +139,11 @@ export default {
---
## Step 3: Testing the Data Engine
## Step 4: Testing the Data Engine
Before we build the frontend, let's prove that our backend is actually fetching the ISS data.
Before we modify the frontend, let's prove that our backend is actually fetching the ISS data.
### 3.1 Start the Engine
### 4.1 Start the Engine
In your terminal, navigate to the `worldwideview` folder and start the entire stack using Docker Compose:
```bash
@@ -130,24 +153,13 @@ Wait for the Data Engine Docker container to start. You should see `[ISS] Poll O
---
## Step 4: Creating the WorldWideView Plugin (Frontend)
## Step 5: Write the Plugin Logic (Frontend)
Now we will build the visual part of the plugin that connects to our backend stream and draws the ISS on the 3D globe.
Now we will finalize the visual part of the plugin that connects to our backend stream and draws the ISS on the 3D globe.
### 4.1 Scaffold the Plugin
### 5.1 Update the Plugin Code
In your terminal, navigate to your main `worldwideview` folder. We will use the workspace CLI tool to generate a blank plugin template in the local sandboxes folder. Run:
```bash
node packages/wwv-cli/dist/index.js create wwv-plugin-iss --local
pnpm install
```
This creates your plugin at `local-plugins/wwv-plugin-iss` and automatically registers it with the workspace.
### 4.2 Write the Plugin Logic
Open `c:\dev\worldwideview\local-plugins\wwv-plugin-iss\src\index.ts` in your text editor. Replace everything in the file with this code:
Open `local-plugins/wwv-plugin-iss/src/index.ts` in your text editor. The CLI generated a boilerplate for you, but we want to customize the rendering with a 3D model. Replace everything in the file with this code:
```typescript
import type { WorldPlugin, GeoEntity, PluginContext, LayerConfig, CesiumEntityOptions } from "@worldwideview/wwv-plugin-sdk";
@@ -201,18 +213,18 @@ export class IssPlugin implements WorldPlugin {
---
## Step 5: Connecting and Testing Everything
## Step 6: Connecting and Testing Everything
Now we will link your new plugin to your local WorldWideView application to see it in action.
### 5.1 Develop Your Plugin
### 6.1 Develop Your Plugin
Because your plugin is located in `local-plugins/wwv-plugin-iss`, it is automatically detected by the workspace.
You do not need to link it manually.
Since you already started the whole stack (`pnpm dev:all`) in Step 3, the `dev:plugins` watcher is already running in the background. It will automatically build your frontend plugin code whenever you save changes.
Since you already started the whole stack (`pnpm dev:all`) in Step 4, the `dev:plugins` watcher is already running in the background. It will automatically build your frontend plugin code whenever you save changes.
### 5.2 View the ISS
### 6.2 View the ISS
1. Open your web browser and go to `http://localhost:3000`.
2. Click the **Layers** icon on the left sidebar.
@@ -227,11 +239,11 @@ Since you already started the whole stack (`pnpm dev:all`) in Step 3, the `dev:p
---
## Step 6: Publishing Your Plugin
## Step 7: Publishing Your Plugin
You've built it, now share it with the world!
### 6.1 Update package.json
### 7.1 Update package.json
Open `c:\dev\worldwideview\local-plugins\wwv-plugin-iss\package.json`. You must add a `"worldwideview"` metadata block so the marketplace knows how to read your plugin. Ensure your file looks like this:
@@ -251,15 +263,16 @@ Open `c:\dev\worldwideview\local-plugins\wwv-plugin-iss\package.json`. You must
}
```
### 6.2 Publish to NPM
### 7.2 Publish to NPM
In your plugin terminal, log in to NPM and publish your package:
Log in to NPM and publish your package using the WWV CLI from the project root (you can also use the `--org` flag to publish under your own NPM organization):
```bash
npm login
npm publish --access public
node packages/wwv-cli/dist/index.js publish iss [--org <your-org>]
```
*(Or `npx wwv publish iss [--org <your-org>]` if installed globally)*
### 6.3 Submit to the Marketplace
### 7.3 Submit to the Marketplace
1. Go to the official marketplace at `https://marketplace.worldwideview.dev/submit`.
2. Type in your package name (`wwv-plugin-iss`) and click submit.
+16 -9
View File
@@ -20,24 +20,28 @@ WorldWideView operates on a strict **Dynamic CDN Loaded (Bundle)** architecture.
Relying on the frontend `fetch()` method is insufficient for high-frequency real-time tracking (like aviation or maritime). For continuous telemetry, you must build a **Data Engine Seeder** — a lightweight Javascript data polling script.
### Data Engine Seeder Architecture
Instead of the frontend fetching data, you write a lightweight seeder script that connects to an upstream source, normalizes the data, and is executed by the central `wwv-data-engine` runner.
WorldWideView is a completely agnostic renderer. It has absolutely no concept of a "unified" Data Engine. If 30 plugins require 30 different WebSocket servers, the application will blindly open 30 connections. Each plugin is a self-contained package and **MUST explicitly declare its own `streamUrl` in its manifest or config**. Do NOT assume the frontend acts as a unified pipe.
1. **Create a Seeder Directory:** Inside your WorldWideView project, create a folder under `local-seeders/` (e.g., `local-seeders/my-plugin/`).
While you can host your own backend, we provide the `DataEngineV2` runner as a standardized environment for seeders.
### Data Engine V2 Seeder Architecture
Instead of the frontend fetching data, you write a lightweight seeder script that connects to an upstream source, normalizes the data, and is executed by the central `wwv-data-engine-v2` host runner.
1. **Create a Seeder Directory:** Inside your WorldWideView project, create a folder under `local-seeders/` (e.g., `local-seeders/community/my-plugin/`). Note that seeders are split into `community` and `private` tiers to prevent namespace collisions.
2. **Write the Seeder Script:** Create a `seeder.mjs` file that exports a `fetch(ctx)` function.
3. **Engine Auto-Discovery:** The local Docker-based `wwv-data-engine` automatically mounts this directory, discovers your script, and runs it on the defined interval.
4. **WebSocket Delivery:** The central `wwv-data-engine` immediately broadcasts your returned updates over WebSockets to all connected clients.
3. **Engine Auto-Discovery:** The local Docker-based `wwv-data-engine-v2` automatically mounts this directory, discovers your script, and runs it on the defined interval.
4. **WebSocket & REST Delivery:** Seeders in V2 expose both a WebSocket stream (`/stream`) for real-time instantaneous updates, and a REST API endpoint (`/api/:id`) for fetching live data snapshots directly from Redis.
### Dependency Management & Monorepo Hoisting
Seeders within `local-seeders/` are strictly orchestrated within the pnpm workspace. They are executed by the central runner, not as standalone applications.
- **Keep `package.json` clean**: Do not include bulky `dependencies` in your seeder's local `package.json` (unless it's an exceptional, bespoke library).
- **Workspace Resolution**: Standard packages (e.g., `axios`, `ioredis`, `@worldwideview/wwv-plugin-sdk`) are provided by the engine. At runtime, the `wwv-data-engine` leverages native Node.js module resolution to fetch the required dependencies directly from the root workspace or its own containerized runtime.
- **Workspace Resolution**: Standard packages (e.g., `zod`, `ws`, `node-cron`, `undici`) are provided by the engine. At runtime, `wwv-data-engine-v2` leverages native Node.js module resolution to fetch the required dependencies directly from the root workspace or its own containerized runtime. Seeders MUST NOT bundle these dependencies.
- **Lightweight by Design**: This dependency orchestration guarantees that seeders remain extremely lightweight, hot-reloading takes milliseconds, and Docker container size stays optimized.
> [!TIP]
> **Debugging WebSockets:** If your frontend isn't receiving data from your backend seeder:
> 1. Check the `wwv-data-engine` logs to ensure your seeder is publishing to Redis successfully.
> 2. Verify the frontend is connected to the correct WebSocket endpoint. Local instances default to `ws://localhost:5000/stream`, while unrecognized seeders fall back to the cloud at `wss://dataengine.worldwideview.dev/stream`.
> 1. Check the `wwv-data-engine-v2` logs to ensure your seeder is publishing to Redis successfully.
> 2. Verify the frontend is connected to the correct WebSocket endpoint. Local instances default to `ws://localhost:5001/stream`, while unrecognized plugins should explicitly define their own `streamUrl`, or fallback to the cloud at `wss://dataenginev2.worldwideview.dev/stream`.
## Advanced Cesium Rendering
@@ -71,9 +75,12 @@ renderEntity(entity: GeoEntity): CesiumEntityOptions {
To distribute your plugin globally:
1. **Publish to NPM:**
Log in to NPM and publish your package using the WWV CLI from the project root (you can also use the `--org` flag to publish under your own NPM organization):
```bash
npm publish --access public
npm login
node packages/wwv-cli/dist/index.js publish <plugin-name> [--org <your-org>]
```
*(Or `npx wwv publish <plugin-name> [--org <your-org>]` if installed globally)*
2. **Submit:** Navigate to `https://marketplace.worldwideview.dev/submit`.
3. **Register:** Enter your NPM package name. The marketplace automatically scrapes your `package.json` for the required `"worldwideview"` object block (containing your `id`, `icon`, and `category`).
4. **Review:** Once approved, your plugin's ES Module bundle will be served via CDN to all WorldWideView instances worldwide.
+16 -4
View File
@@ -14,14 +14,26 @@ Before you start, ensure you have:
## 2. Scaffold Your Plugin
WorldWideView plugins are built as standard packages within the monorepo workspace. To generate a new plugin sandbox, use the official CLI from the root of your `worldwideview` repository:
WorldWideView plugins are built as standard packages within the monorepo workspace. To generate a new plugin sandbox, use the official interactive CLI from the root of your `worldwideview` repository:
```bash
node packages/wwv-cli/dist/index.js create my-first-plugin --local
pnpm install
node packages/wwv-cli/dist/index.js create --local
```
This creates a lightweight Vite-based project pre-configured with the `@worldwideview/wwv-plugin-sdk` inside the `local-plugins/wwv-plugin-my-first-plugin/` directory.
The CLI will launch an interactive survey to configure your plugin:
1. **Unique ID**: `my-first-plugin`
2. **Display Name**: `My First Data Layer`
3. **Description**: `A simple example plugin.`
4. **Category**: `Custom`
5. **Data Architecture**: `REST Polling / Static Data (Frontend only)`
6. **Globe Rendering**: `Map Point`
This automatically creates a lightweight Vite-based project pre-configured with the `@worldwideview/wwv-plugin-sdk` inside the `local-plugins/wwv-plugin-my-first-plugin/` directory.
Once the scaffolding completes, install the dependencies:
```bash
pnpm install
```
## 3. Develop Your Plugin
+1 -1
View File
@@ -3,7 +3,7 @@
WorldWideView is a real-time geospatial intelligence engine that visualizes live global data on an interactive 3D globe. It leverages a modern frontend stack to render high-density datasets synchronously, including aircraft, maritime vessels, conflict events, satellites, and environmental data.
The core value proposition of WorldWideView is mapping diverse, decoupled intelligence streams onto a central geographic interface using a dynamically loaded plugin architecture. Data sources operate completely decoupled from the display platform, streaming observations via WebSockets for sub-second visual updates. The ecosystem consists of multiple repositories, including the main app (`worldwideview`), the plugin marketplace (`worldwideview-marketplace`), and the WebSocket push server (`wwv-data-engine`).
The core value proposition of WorldWideView is mapping diverse, decoupled intelligence streams onto a central geographic interface using a dynamically loaded plugin architecture. Data sources operate completely decoupled from the display platform, streaming observations via WebSockets for sub-second visual updates. The ecosystem consists of multiple repositories, including the main app (`worldwideview`), the plugin marketplace (`worldwideview-marketplace`), and our default data provider runners like the `DataEngineV2` (`wwv-data-engine-v2`). Note that WorldWideView itself acts as an agnostic client, and any plugin can seamlessly point to its own independent backend WebSocket stream.
## Key Files
- `package.json` (Lines 1-76) - Core dependencies and project metadata for the monorepo workspace.
+41 -1
View File
@@ -38,5 +38,45 @@ Execute the following commands from the root `worldwideview` directory:
```
## Reference
- **Coverage Scope:** The Vitest configuration primarily targets tests located inside `src/lib/**`, `src/core/**`, and `src/plugins/**`.
- **Coverage Scope:** The Vitest configuration targets tests located inside `src/lib/**`, `src/core/**`, `src/plugins/**`, `src/app/**`, and `packages/**`.
- **Environment:** The test environment is configured as `jsdom` via `vite.config.ts` or `vitest.config.ts`, ensuring that DOM-specific API calls function outside of a real browser.
## Patterns
### Plain function — no mocks needed
Pull in the function, call it with representative inputs, assert the output. See `src/lib/marketplace/cors.test.ts` or `src/app/api/camera/adapters/registry.test.ts` for examples.
### Stateful singleton — reset in `afterEach`
Most of the data layer (`dataBus`, `cacheLayer`, `pollingManager`) is exported as a singleton instance. Tests pass when run alone but fail when run together if state leaks. `DataBus.test.ts` shows the canonical cleanup:
```typescript
import { dataBus } from "./DataBus";
afterEach(() => {
dataBus.removeAllListeners();
});
```
### Module-level dependency — mock with `vi.mock`
When the module under test pulls in something heavy (the store, the database, Cesium), mock that module before the import. `vi.mock` calls are hoisted, so they take effect even when written below the import in source order. See `src/lib/marketplace/repository.test.ts` (Prisma client) or `src/core/data/PollingManager.test.ts` (Zustand store).
### Timers — fake them per test
`vi.useFakeTimers()` in `beforeEach`, `vi.useRealTimers()` in `afterEach`. Advance time explicitly with `vi.advanceTimersByTimeAsync(ms)`.
> [!CAUTION]
> `vi.runOnlyPendingTimersAsync()` ticks `setInterval`s scheduled on the same tick as the call. For tests where a function runs once synchronously and then schedules a recurring interval, this double-counts the first call. To drain microtasks without advancing the clock, use `await Promise.resolve()` a few times — see `PollingManager.test.ts` for the `flushMicrotasks()` helper.
## What NOT to Test (Yet)
- **Cesium-heavy components.** Anything that mounts a `<Viewer>` or renders entities to the globe needs WebGL, which jsdom doesn't provide. These belong in Playwright-style E2E tests rather than unit tests.
- **Live database round-trips.** Mock the Prisma client at the module boundary — `repository.test.ts` is the reference pattern. Spinning up a real Postgres in CI is expensive and unreliable.
- **Dynamic-import plugin bundles.** The loader uses `import(/* webpackIgnore: true */ entry)` at runtime against a real URL. Test the loader's pre-import logic with a fake module object, not a real dynamic import.
## Philosophy
A small focused suite is more valuable than a large one that's expensive to maintain. Each test should answer: *what specific regression does this catch that I'd otherwise discover from a user bug report?*
Two practical rules:
1. **Prefer testing pure logic over orchestration.** If a function takes inputs and returns outputs, test it directly. If it coordinates a half-dozen other things (the globe, the store, Cesium primitives), either refactor the pure parts out or use an integration test.
2. **Add a test when you fix a bug.** A one-line test that fails today and passes after the fix is enough to prevent the bug from coming back. Land the test in the same PR as the fix.
+50
View File
@@ -0,0 +1,50 @@
import nextCoreWebVitals from "eslint-config-next/core-web-vitals";
import nextTypeScript from "eslint-config-next/typescript";
const config = [
{
ignores: [
"node_modules/**",
".next/**",
"out/**",
"src/generated/**",
"public/cesium/**",
"packages/*/dist/**",
".worktrees/**",
"local-scripts/**",
"local-plugins/**",
"local-seeders/**",
".agents/**",
".claude/**",
".gemini/**",
"*.cjs",
],
},
...nextCoreWebVitals,
...nextTypeScript,
{
rules: {
// ─── Demoted on first introduction ─────────────────────────────
// The existing codebase has hundreds of violations across these
// rules. To make the lint job runnable in CI without blocking on
// pre-existing debt, every rule that currently fires errors is
// demoted to `warn`. Promote back to `error` once the backlog
// is addressed (recommended in this order: prefer-const,
// no-unescaped-entities, ban-ts-comment, then the react-hooks
// family once the team agrees on the React 19 hook-purity rules).
// Pre-existing — flagged in the original code review (#96):
"@typescript-eslint/no-explicit-any": "warn",
"no-console": "off",
// React 19 hook-purity rules from eslint-plugin-react-hooks 7+:
// (These were previously hallucinated or require extra plugins)
// Misc one-offs across the codebase:
"@typescript-eslint/ban-ts-comment": "warn",
"@next/next/no-assign-module-variable": "warn",
"prefer-const": "warn",
},
},
];
export default config;
+4 -6
View File
@@ -6,7 +6,7 @@ This directory is a **pnpm workspace** for developing plugins locally without co
```bash
# Scaffold a new plugin
pnpm wwv create my-plugin --local
node packages/wwv-cli/dist/index.js create
# Start dev server (auto-discovers local plugins)
pnpm dev
@@ -20,12 +20,10 @@ pnpm dev
4. The marketplace load API auto-discovers them in dev mode
5. Everything in this directory is gitignored
## Promoting to a Package
## Publishing to NPM
When your plugin is ready to share:
When your plugin is ready to share, you can publish it directly to NPM from the root directory:
```bash
pnpm wwv link my-plugin
node packages/wwv-cli/dist/index.js publish <plugin-name> [--org <your-org>]
```
This moves it to `packages/` and adds it to version control.
View File
Whitespace-only changes.
-2
View File
@@ -93,6 +93,4 @@ const nextConfig: NextConfig = {
},
};
import { withSentryConfig } from "@sentry/nextjs";
export default nextConfig;
+45 -12
View File
@@ -1,24 +1,28 @@
{
"name": "worldwideview",
"version": "2.7.8",
"private": true,
"version": "2.15.12",
"private": true,
"type": "module",
"packageManager": "[email protected]",
"scripts": {
"setup": "node scripts/setup.mjs",
"test-worktree": "node scripts/test-worktree.mjs",
"clean": "node -e \"const fs=require('fs');fs.rmSync('.next',{recursive:true,force:true})\"",
"dev": "concurrently \"pnpm run dev:plugins\" \"next dev\"",
"dev": "node scripts/dev.mjs",
"dev:all": "concurrently -n \"dev,backends\" -c \"magenta,blue\" \"pnpm dev\" \"pnpm dev:backends\"",
"dev:backends": "docker compose up wwv-data-engine wwv-redis",
"dev:backends": "docker compose --profile engine up wwv-data-engine wwv-redis",
"dev:plugins": "node scripts/watch-local-plugins.mjs",
"predev": "node scripts/boot-db.mjs && npx dotenv-cli -e .env.local -c -- npx prisma db push --accept-data-loss && npx prisma generate && node scripts/copy-cesium.mjs && node scripts/sync-local-plugins.mjs",
"predev": "node scripts/boot-db.mjs && npx dotenv-cli -c -- node scripts/safe-db-push.mjs && npx prisma generate && node scripts/copy-cesium.mjs && node scripts/sync-local-plugins.mjs",
"copy-cesium": "node scripts/copy-cesium.mjs",
"prebuild": "node scripts/check-build-env.mjs",
"build": "next build --webpack",
"start": "next start",
"test": "vitest run",
"test:e2e": "playwright test",
"test:mutate": "stryker run",
"lint": "eslint .",
"db:reset": "npx prisma migrate reset --force",
"db:prune": "node scripts/db-prune.mjs",
"generate": "prisma generate"
},
"dependencies": {
@@ -38,9 +42,10 @@
"hls.js": "^1.6.16",
"insecam-api": "^0.0.3",
"jose": "^6.2.3",
"lucide-react": "^0.576.0",
"lucide-react": "^1.14.0",
"next": "16.2.6",
"next-auth": "^5.0.0-beta.30",
"openid-client": "^6.8.4",
"pg": "^8.20.0",
"prisma": "^7.8.0",
"react": "19.2.6",
@@ -50,32 +55,55 @@
"satellite.js": "^7.0.0",
"stripe": "^22.1.1",
"tslib": "^2.8.1",
"undici": "^8.2.0",
"ws": "^8.14.2",
"zod": "^4.4.3",
"zustand": "^5.0.13"
},
"devDependencies": {
"@eslint/compat": "^2.1.0",
"@eslint/eslintrc": "^3.3.5",
"@playwright/test": "^1.60.0",
"@stryker-mutator/core": "^9.6.1",
"@stryker-mutator/vitest-runner": "^9.6.1",
"@testing-library/dom": "^10.4.1",
"@testing-library/react": "^16.3.2",
"@types/bcryptjs": "^3.0.0",
"@types/cesium": "^1.70.4",
"@types/node": "^20",
"@types/node": "^25",
"@types/react": "^19",
"@types/react-dom": "^19",
"@types/ws": "^8.18.1",
"@typescript-eslint/eslint-plugin": "^8.59.3",
"@typescript-eslint/parser": "^8.59.3",
"@vitejs/plugin-react": "^5.1.4",
"@vitest/coverage-v8": "4.1.5",
"@vitest/coverage-v8": "4.1.6",
"concurrently": "^9.2.1",
"copy-webpack-plugin": "^14.0.0",
"eslint": "^9",
"eslint-config-airbnb": "^19.0.4",
"eslint-config-airbnb-typescript": "^18.0.0",
"eslint-config-next": "^16.2.6",
"eslint-plugin-import": "^2.32.0",
"eslint-plugin-jsx-a11y": "^6.10.2",
"eslint-plugin-react": "^7.37.5",
"eslint-plugin-react-hooks": "^7.1.1",
"fast-check": "^4.8.0",
"jsdom": "^28.1.0",
"rollup-plugin-external-globals": "^0.13.0",
"tree-kill": "^1.2.2",
"tsx": "^4.21.0",
"typescript": "^5",
"vite": "^8.0.11",
"vitest": "^4.1.5"
"vite": "^8.0.12",
"vitest": "^4.1.6"
},
"pnpm": {
"overrides": {
"@spz-loader/core": "0.3.0"
"@spz-loader/core": "0.3.0",
"form-data@<2.5.4": "^2.5.5",
"form-data@>=3.0.0 <3.0.4": "^3.0.4",
"form-data@>=4.0.0 <4.0.4": "^4.0.5",
"fast-uri@<3.1.2": "^3.1.2"
},
"onlyBuiltDependencies": [
"@prisma/engines",
@@ -83,7 +111,12 @@
"prisma",
"protobufjs",
"sharp",
"esbuild"
"esbuild",
"better-sqlite3"
]
},
"engines": {
"node": ">=20.0.0",
"pnpm": ">=9.0.0"
}
}
+47
View File
@@ -0,0 +1,47 @@
# @worldwideview/wwv-cli
The official Command Line Interface for scaffolding and publishing WorldWideView plugins.
## Usage
This CLI is designed to be run from within the `worldwideview` monorepo.
### 1. Scaffold a new plugin
```bash
node packages/wwv-cli/dist/index.js create
```
This command generates a new plugin directory with a boilerplate `index.ts` and `package.json` configured for the WorldWideView plugin architecture inside the `local-plugins/` directory.
*(Note: To create a built-in plugin inside `packages/` instead, use the `--core` flag).*
### 2. Publish a plugin
Once your plugin is ready for release, you can publish it to NPM. You can do this from within the plugin directory or directly from the project root.
**From the plugin directory:**
```bash
cd local-plugins/<plugin-name>
npm login
node ../../packages/wwv-cli/dist/index.js publish
```
**From the project root:**
You can also run the CLI from the root by providing the plugin name:
```bash
node packages/wwv-cli/dist/index.js publish <plugin-name>
```
**Custom NPM Organizations:**
By default, plugins are scaffolded under the `@worldwideview` NPM scope. To publish to your own organization, use the `--org` flag:
```bash
node packages/wwv-cli/dist/index.js publish <plugin-name> --org my-username
```
This will permanently update your `package.json` to use `@my-username` and publish the package.
The publish command verifies that your `package.json` contains the required `"worldwideview"` manifest block and invokes `npm publish --access public`.
## Development
To re-compile the CLI during development:
```bash
pnpm --filter @worldwideview/wwv-cli run build
```
+17 -8
View File
@@ -1,15 +1,24 @@
{
"name": "@worldwideview/wwv-cli",
"version": "1.0.0",
"type": "commonjs",
"version": "1.2.0",
"type": "module",
"description": "WorldWideView CLI",
"main": "dist/index.js",
"bin": {
"wwv": "./dist/index.js"
"wwv": "dist/index.js"
},
"scripts": {
"build": "tsc",
"dev": "tsc --watch",
"clean": "rm -rf dist"
},
"dependencies": {
"commander": "^11.1.0",
"chalk": "^4.1.2",
"vite": "^5.0.0",
"archiver": "^7.0.1",
"ws": "^8.16.0"
"commander": "^12.1.0",
"prompts": "^2.4.2"
},
"devDependencies": {
"@types/node": "^20.0.0",
"@types/prompts": "^2.4.9",
"typescript": "^5.0.0"
}
}
+23
View File
@@ -0,0 +1,23 @@
/* eslint-disable no-console */
import { Command } from 'commander';
import { saveConfig } from '../utils/config.js';
export const configCommand = new Command('config')
.description('Manage WWV CLI configuration')
.argument('<action>', 'Action to perform (e.g., set)')
.argument('<key>', 'Config key (e.g., org)')
.argument('<value>', 'Config value (e.g., your-username)')
.action((action, key, value) => {
if (action !== 'set') {
console.error(`[wwv-cli] Unknown config action: ${action}`);
process.exit(1);
}
if (key === 'org') {
saveConfig({ org: value });
console.log(`[wwv-cli] Successfully set default organization to @${value}`);
} else {
console.error(`[wwv-cli] Unknown config key: ${key}. Valid keys: org`);
process.exit(1);
}
});
+235
View File
@@ -0,0 +1,235 @@
/* eslint-disable no-console */
import { Command } from 'commander';
import fs from 'fs';
import path from 'path';
import prompts from 'prompts';
export const createCommand = new Command('create')
.description('Interactively scaffold a new WorldWideView plugin')
.option('-c, --core', 'Create inside packages instead of local-plugins (for core contributors)')
.action(async (options) => {
const response = await prompts([
{
type: 'text',
name: 'pluginId',
message: 'What is the unique ID for your plugin? (e.g. my-tracker)',
validate: value => value.length > 0 ? true : 'Plugin ID is required'
},
{
type: 'text',
name: 'displayName',
message: 'What is the display name? (e.g. My Live Tracker)',
initial: (prev: string) => prev.replace(/-/g, ' ').replace(/\b\w/g, (l: string) => l.toUpperCase())
},
{
type: 'text',
name: 'description',
message: 'Enter a short description:'
},
{
type: 'select',
name: 'category',
message: 'Which category does this plugin belong to?',
choices: [
{ title: 'Aviation (Plane icon)', value: 'aviation' },
{ title: 'Maritime (Ship icon)', value: 'maritime' },
{ title: 'Space (Satellite icon)', value: 'space' },
{ title: 'Weather (Cloud icon)', value: 'weather' },
{ title: 'Custom (Box icon)', value: 'custom' }
]
},
{
type: 'select',
name: 'architecture',
message: 'How will your plugin receive data?',
choices: [
{ title: 'REST Polling / Static Data (Frontend only)', value: 'polling' },
{ title: 'Real-Time WebSockets (Generates backend seeder)', value: 'websocket' }
]
},
{
type: (prev: string) => prev === 'websocket' ? 'select' : null,
name: 'seederTier',
message: 'Which tier should your backend seeder belong to?',
choices: [
{ title: 'Community (Open data, shared)', value: 'community' },
{ title: 'Private (Requires API keys, restricted)', value: 'private' }
]
},
{
type: 'select',
name: 'renderStyle',
message: 'How should entities be rendered on the 3D globe?',
choices: [
{ title: '2D Billboard (Great for static icons)', value: 'billboard' },
{ title: '3D Model with LOD (Transitions 2D to 3D)', value: 'model' },
{ title: 'Simple Point (High performance dots)', value: 'point' }
]
}
]);
if (!response.pluginId) {
console.log('Plugin creation cancelled.');
return;
}
const { pluginId, displayName, description, category, architecture, seederTier, renderStyle } = response;
const targetBaseDir = options.core ? 'packages' : 'local-plugins';
const pluginDir = path.join(process.cwd(), targetBaseDir, `wwv-plugin-${pluginId}`);
if (fs.existsSync(pluginDir)) {
console.error(`Error: Directory ${pluginDir} already exists.`);
process.exit(1);
}
fs.mkdirSync(pluginDir, { recursive: true });
// Determine category icon
const iconMap: Record<string, string> = {
aviation: 'Plane',
maritime: 'Ship',
space: 'Satellite',
weather: 'Cloud',
custom: 'Box'
};
const defaultIcon = iconMap[category] || 'Box';
// Package.json boilerplate
const streamUrlField = architecture === 'websocket'
? `\n "streamUrl": "wss://dataenginev2.worldwideview.dev/stream",`
: '';
const packageJsonContent = `{
"name": "@worldwideview/wwv-plugin-${pluginId}",
"version": "1.0.0",
"description": "${description}",
"main": "dist/frontend.mjs",
"module": "dist/frontend.mjs",
"exports": {
".": {
"import": "./dist/frontend.mjs",
"require": "./dist/frontend.mjs"
}
},
"scripts": {
"build": "tsc"
},
"dependencies": {
"@worldwideview/wwv-plugin-sdk": "latest"
},
"worldwideview": {
"pluginId": "${pluginId}",
"name": "${displayName}",
"description": "${description}",
"icon": "${defaultIcon}",
"category": "${category}",${streamUrlField}
"author": "Your Name",
"dev_entry": "src/index.ts",
"format": "bundle"
}
}
`;
// index.ts boilerplate
const renderBoilerplate = {
billboard: `return {
type: 'billboard',
iconUrl: '/icons/default.png',
iconScale: 1.0,
color: '#ffffff'
};`,
model: `return {
type: 'billboard', // Placeholder for LOD logic, update to 3D model path when zoomed
iconUrl: '/icons/default.png',
iconScale: 1.0,
modelUrl: '/models/default.glb', // Advanced LOD
color: '#ffffff'
};`,
point: `return {
type: 'point',
color: '#ff0000',
size: 8,
outlineColor: '#ffffff',
outlineWidth: 2
};`
}[renderStyle as string] || `return {};`;
const fetchBoilerplate = architecture === 'polling'
? `\n async fetch(): Promise<GeoEntity[]> {
// Implement REST polling logic here
return [];
}\n`
: '';
const indexTsContent = `import { WorldPlugin, GeoEntity, CesiumEntityOptions } from '@worldwideview/wwv-plugin-sdk';
export default class ${pluginId.replace(/-/g, '')}Plugin implements WorldPlugin {
id = '${pluginId}';
name = '${displayName}';
async initialize(context: any): Promise<void> {
console.log('${displayName} initialized.');
}
getPollingInterval(): number {
return ${architecture === 'polling' ? '10000' : '0'}; // ${architecture === 'polling' ? '10 seconds' : '0 for WebSockets'}
}
${fetchBoilerplate}
renderEntity(entity: GeoEntity): CesiumEntityOptions {
${renderBoilerplate}
}
renderHUD(entity: GeoEntity): string {
return \`<div>
<h3>\${entity.id}</h3>
<p>Data provided by ${displayName}</p>
</div>\`;
}
}
`;
fs.writeFileSync(path.join(pluginDir, 'package.json'), packageJsonContent);
fs.mkdirSync(path.join(pluginDir, 'src'), { recursive: true });
fs.writeFileSync(path.join(pluginDir, 'src', 'index.ts'), indexTsContent);
console.log(`\n✅ Successfully created plugin frontend at ${pluginDir}`);
// Generate seeder backend if requested
if (architecture === 'websocket' && seederTier) {
const seederDir = path.join(process.cwd(), 'local-seeders', seederTier, pluginId);
if (!fs.existsSync(seederDir)) {
fs.mkdirSync(seederDir, { recursive: true });
const seederContent = `// WWV Data Engine Seeder for ${pluginId}
export default async function run(context) {
console.log('Starting ${displayName} seeder...');
// Example: Emit a mock entity to the engine
setInterval(() => {
context.emit({
id: '${pluginId}-mock-1',
lat: 0,
lng: 0,
alt: 10000,
heading: 0,
velocity: 250,
properties: {
status: 'active'
}
});
}, 1000);
}
`;
fs.writeFileSync(path.join(seederDir, 'seeder.mjs'), seederContent);
console.log(`✅ Successfully created plugin seeder at ${seederDir}`);
}
}
console.log(`\nNext steps:
1. Run \`pnpm install\` from the project root.
2. Edit \`index.ts\` in your plugin directory.
${architecture === 'websocket' ? '3. Edit `seeder.mjs` in your local-seeders directory.' : ''}
`);
});
+121
View File
@@ -0,0 +1,121 @@
/* eslint-disable no-console */
import { Command } from 'commander';
import { execSync } from 'child_process';
import path from 'path';
import fs from 'fs';
import prompts from 'prompts';
import { getConfig, saveConfig } from '../utils/config.js';
export const publishCommand = new Command('publish')
.description('Publish the plugin to NPM and notify the WWV Marketplace')
.argument('[pluginName]', 'Name of the plugin to publish (if running from root)')
.option('--org <orgName>', 'NPM organization to publish to (e.g., your-username)')
.action(async (pluginName, options) => {
console.log('[wwv-cli] Preparing to publish plugin...');
let cwd = process.cwd();
// If pluginName is provided, try to find the plugin directory
if (pluginName) {
const rootLocalPath = path.join(cwd, 'local-plugins', pluginName);
const rootWwvLocalPath = path.join(cwd, 'local-plugins', `wwv-plugin-${pluginName}`);
const packagesPath = path.join(cwd, 'packages', pluginName);
const packagesWwvPath = path.join(cwd, 'packages', `wwv-plugin-${pluginName}`);
if (fs.existsSync(rootWwvLocalPath)) {
cwd = rootWwvLocalPath;
} else if (fs.existsSync(rootLocalPath)) {
cwd = rootLocalPath;
} else if (fs.existsSync(packagesWwvPath)) {
cwd = packagesWwvPath;
} else if (fs.existsSync(packagesPath)) {
cwd = packagesPath;
} else {
console.error(`[wwv-cli] Error: Could not find plugin '${pluginName}' in local-plugins or packages.`);
process.exit(1);
}
console.log(`[wwv-cli] Found plugin at: ${cwd}`);
}
const pkgPath = path.join(cwd, 'package.json');
if (!fs.existsSync(pkgPath)) {
console.error('[wwv-cli] Error: No package.json found in directory: ' + cwd);
process.exit(1);
}
try {
const pkgContent = JSON.parse(fs.readFileSync(pkgPath, 'utf8'));
if (!pkgContent.worldwideview) {
console.error('[wwv-cli] Error: package.json is missing the "worldwideview" manifest block. Is this a WWV plugin?');
process.exit(1);
}
const originalName = pkgContent.name;
let publishedName = originalName;
// Determine organization
let targetOrg = options.org;
const config = getConfig();
if (!targetOrg) {
if (config.org) {
targetOrg = config.org;
console.log(`[wwv-cli] Using default organization from config: @${targetOrg}`);
} else {
// Fallback to NPM whoami if possible
try {
const whoami = execSync('npm whoami', { stdio: 'pipe' }).toString().trim();
targetOrg = whoami;
} catch (_e) {
// Not logged in or errored, ignore
}
const response = await prompts({
type: 'text',
name: 'org',
message: 'What NPM organization/username do you want to publish under?',
initial: targetOrg || '',
validate: value => value.length > 0 ? true : 'Organization is required'
});
if (!response.org) {
console.log('Publish cancelled.');
return;
}
targetOrg = response.org;
saveConfig({ org: targetOrg });
console.log(`[wwv-cli] Saved @${targetOrg} as your default organization.`);
}
} else {
// Save the explicitly passed org as the new default
saveConfig({ org: targetOrg });
}
if (targetOrg) {
const orgPrefix = targetOrg.startsWith('@') ? targetOrg : `@${targetOrg}`;
const baseName = originalName.startsWith('@') ? originalName.split('/')[1] : originalName;
publishedName = `${orgPrefix}/${baseName}`;
if (originalName !== publishedName) {
pkgContent.name = publishedName;
fs.writeFileSync(pkgPath, JSON.stringify(pkgContent, null, 2) + '\n');
console.log(`[wwv-cli] Updated package name to ${publishedName}`);
}
}
console.log(`[wwv-cli] Publishing ${publishedName}@${pkgContent.version} to NPM...`);
// Execute npm publish
execSync('npm publish --access public', { stdio: 'inherit', cwd });
console.log('[wwv-cli] Successfully published to NPM!');
console.log('[wwv-cli] To submit this plugin to the WorldWideView Marketplace, please visit: https://marketplace.worldwideview.dev/submit');
console.log(`[wwv-cli] Package Name: ${publishedName}`);
} catch (err: any) {
console.error('[wwv-cli] Error during publish:', err.message);
process.exit(1);
}
});
+18
View File
@@ -0,0 +1,18 @@
#!/usr/bin/env node
import { Command } from 'commander';
import { createCommand } from './commands/create.js';
import { publishCommand } from './commands/publish.js';
import { configCommand } from './commands/config.js';
const program = new Command();
program
.name('wwv')
.description('WorldWideView Plugin CLI')
.version('1.1.2');
program.addCommand(createCommand);
program.addCommand(publishCommand);
program.addCommand(configCommand);
program.parse(process.argv);
+27
View File
@@ -0,0 +1,27 @@
import fs from 'fs';
import path from 'path';
import os from 'os';
const CONFIG_PATH = path.join(os.homedir(), '.wwvrc');
export interface WwvConfig {
org?: string;
}
export function getConfig(): WwvConfig {
if (!fs.existsSync(CONFIG_PATH)) {
return {};
}
try {
const raw = fs.readFileSync(CONFIG_PATH, 'utf8');
return JSON.parse(raw);
} catch (e) {
return {};
}
}
export function saveConfig(config: WwvConfig): void {
const current = getConfig();
const next = { ...current, ...config };
fs.writeFileSync(CONFIG_PATH, JSON.stringify(next, null, 2), 'utf8');
}
+16
View File
@@ -0,0 +1,16 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"target": "ES2022",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"outDir": "./dist",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"noEmit": false
},
"include": ["src/**/*"]
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@worldwideview/wwv-lib-aviation",
"version": "1.0.1",
"version": "1.0.2",
"description": "Shared abstract base classes for WorldWideView aviation plugins (commercial, military, specialized)",
"main": "src/index.ts",
"types": "src/index.ts",
-1
View File
@@ -69,7 +69,6 @@ export abstract class BaseAviationPlugin implements WorldPlugin {
return {
type: "model",
iconUrl: this.iconUrl,
size: isAirborne ? 8 : 5,
modelUrl: this.modelUrl,
modelScale: 2.56,
modelMinPixelSize: 16,
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@worldwideview/wwv-lib-incidents",
"version": "1.0.1",
"version": "1.0.2",
"description": "Shared abstract base classes for WorldWideView incident plugins (earthquakes, wildfires, conflicts)",
"main": "src/index.ts",
"types": "src/index.ts",
+1 -3
View File
@@ -75,9 +75,7 @@ export abstract class BaseIncidentPlugin implements WorldPlugin {
type: "billboard",
iconUrl: this.iconUrls[cacheKey],
color,
size,
outlineColor: "#000000",
outlineWidth: 1,
iconScale: size / 30,
labelText: entity.label || undefined,
labelFont: "11px JetBrains Mono, monospace" // Adds optional label standard
};
Loaded 100 of 394 files, more files were not shown because too many files have changed in this diff. Show more