Compare commits

..
Author SHA1 Message Date
Antigravity AI aff9bfafd7 fix(ci): stub seeder interval+fetch + drop redundant compose mount
Companion to wwv-data-engine fix/ci-stub-interval-fetch. The source-of-truth
stub (docker/ci/stub-seeder) used cron+fn and called ctx.setLiveSnapshot,
which the seeder ctx ({ redis }) does not expose — the snapshot never landed.
Switch to interval+fetch (return the array; scheduler auto-wraps + stores it).

Also remove the plugin-CI compose's ./stub-seeder volume mount and the stale
'Known foundation gap (deferred)' comment: the stub is now baked into the :ci
image at /app/seeders/stub, so the compose needs no host path and a consumer
repo can run it with only this YAML.
2026-06-04 23:50:39 +12:00
14 changed files with 23 additions and 199 deletions

No files matched your search

-57
View File
@@ -89,60 +89,3 @@ This means:
> [!CAUTION]
> The most common silent bug: seeder sends `{ items: [...] }` (an object), frontend plugin omits `mapWebsocketPayload`. Data arrives but WsClient drops it with a console warning. The globe stays empty. Always implement `mapWebsocketPayload` for any plugin whose seeder sends an object payload.
## 8. Seeder Authoring Contract — the module shape
§7 covers what a seeder **sends**. This covers how you **write** one. The engine
auto-discovers a seeder by scanning `SEEDERS_DIR` for any folder with
`dist/index.mjs` and importing its **default export**. The seeder's `id` is the
**folder name**, not the `name` field. You never call a register function — the
authoritative reference is `wwv-data-engine/README.md` → "Authoring a Seeder".
The default export is `{ name }` plus **one** of these shapes:
| Shape | You provide | How data is published |
|---|---|---|
| `interval` + `fetch(ctx)` | `fetch` **returns** the array | scheduler wraps + stores + broadcasts it for you |
| `cron` + `fn(ctx)` | `fn` publishes itself | `import { setLiveSnapshot } from '@worldwideview/seeder-sdk'` and call it |
| `init(ctx)` | a persistent listener | publish via the SDK as push data arrives |
**The trap (this caused a real, silent CI failure):** `ctx` is **only `{ redis }`**.
There is no `ctx.setLiveSnapshot`. Reaching for it throws at runtime and the
snapshot never lands. Publish helpers come from `@worldwideview/seeder-sdk`, not
from `ctx`. The engine now types `ctx` as `SeederContext { redis }`, so this is a
compile error for any typed seeder — but `.mjs` seeders without type-checking get
no such safety net, so know the contract.
<bad-example>
```ts
// cron + fn reaching for a method that does not exist on ctx
export default {
name: "earthquakes",
cron: "0 * * * *",
fn: async (ctx) => {
await ctx.setLiveSnapshot("earthquakes", items, 3600); // ❌ not a function — dies silently
},
};
```
</bad-example>
<good-example>
```ts
// Option A — cron + fn: publish via the SDK (dominant idiom)
import { setLiveSnapshot } from "@worldwideview/seeder-sdk";
export default {
name: "earthquakes",
cron: "0 * * * *",
fn: async () => {
await setLiveSnapshot("earthquakes", { source: "earthquakes", fetchedAt: new Date().toISOString(), items, totalCount: items.length }, 3600);
},
};
// Option B — interval + fetch: just return the array, scheduler does the rest
export default {
name: "earthquakes",
interval: 60_000,
fetch: async () => items, // ✅ no publishing call needed
};
```
</good-example>
-16
View File
@@ -12,22 +12,6 @@ paths:
## Purpose
The standard operating procedure for instantiating, modifying, and registering a new data source plugin within the engine.
## Data Engine Agnosticism (Separation of Concern)
> [!IMPORTANT]
> **WWV is agnostic to which data engine a plugin uses.** Each plugin MUST declare its own `streamUrl` in `getServerConfig()`. The WWV application never controls or overrides plugin stream URLs.
This is intentional architectural separation: a user can have 30 plugins pointing to 30 different custom backend servers, and WWV opens 30 separate WebSocket connections without caring. The WWV app does not have a "the data engine" — each plugin has its own.
**Practical rules:**
- Plugins hardcode their cloud engine URL (e.g. `wss://dataenginev2.worldwideview.dev/stream`) — this is correct and by design.
- Do not add env vars to let WWV override a plugin's `streamUrl` globally — that breaks the separation.
- When editing a plugin's fallback fetch URL, use `https://dataenginev2.worldwideview.dev` (note the `v2`).
- `NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL` is a local dev only var (`.env.local` only, never production) that routes locally-developed plugin seeders to a local engine instance. It is NOT injected into plugin stream URLs.
- `WWV_DATA_ENGINE_URL` is the production server-side var (docker-compose / Coolify) for the WWV app's own REST calls (MCP tools, entity search). Never set `NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL` in production.
---
## The WorldPlugin Contract
All data ingest flows through `WorldPlugin` (defined entirely within `@worldwideview/wwv-plugin-sdk`).
-13
View File
@@ -96,19 +96,6 @@ PROXY_HOST_ALLOWLIST=*
# Comma-separated list of plugin IDs
NEXT_PUBLIC_DEMO_DEFAULT_PLUGINS=
# Local Engine URL (local dev only — set in .env.local, NEVER in production)
# Points at your local data engine instance for per-plugin WS stream routing.
# Each worktree should use a different port so their engines don't collide.
# Local dev (engine on host): http://localhost:5000
NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL=
# Data Engine REST URL (production / Docker Compose — server-side only)
# Used by MCP data tools and server-side entity queries to reach the engine REST API.
# NOT read by the browser. Set this in docker-compose or Coolify, not in .env.local.
# Docker Compose (same network): http://wwv-data-engine:5000
# External engine: http://<engine-host-ip>:5002
# WWV_DATA_ENGINE_URL=
# Edition: "local" | "cloud" | "demo"
NEXT_PUBLIC_WWV_EDITION=local
-1
View File
@@ -60,7 +60,6 @@ Agents MUST respect these at all times:
> - **MUST** require explicit user authorization before any state-changing action that isn't simple/safe.
> - **MUST** ask clarifying questions rather than assume when requirements are unclear.
> - **MUST** update `.agents/rules/` files immediately whenever an architectural shift invalidates them.
> - **MUST** close context gaps: when missing/wrong/undocumented context made you err, struggle, or go slow, fix the relevant doc/context (or add a type/guardrail) at its source — or ask the user to update it when it depends on their knowledge. In autonomous mode, log the gap to revisit later. Never fix only the symptom and leave the trap armed for the next contributor.
---
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "worldwideview",
"version": "2.48.24",
"version": "2.48.21",
"private": true,
"type": "module",
"packageManager": "[email protected]",
+2 -2
View File
@@ -15,7 +15,7 @@
import type { PluginDataSnapshot } from "@/lib/data-query/types";
import type { RegionOptions } from "@/lib/data-query/types";
import type { FilterDefinition } from "@/core/plugins/PluginTypes";
import { getAllPluginSnapshots, getEngineUrl } from "@/lib/data-query/service";
import { getAllPluginSnapshots } from "@/lib/data-query/service";
import { getLocalSourceIds } from "@/lib/data-query/localSources";
import { readActiveSessions, readGlobeState } from "@/lib/globeStateStore";
import { readSessionCatalog } from "@/lib/mcpSessionCatalog";
@@ -112,7 +112,7 @@ export interface ListStreamingPluginsResult {
* are not blocked waiting for a slow engine.
*/
async function probeEngineReachable(): Promise<boolean> {
const engineUrl = getEngineUrl() + "/manifest";
const engineUrl = (process.env.WWV_DATA_ENGINE_URL ?? "http://localhost:5001") + "/manifest";
try {
const res = await fetch(engineUrl, { signal: AbortSignal.timeout(2000) });
return res.ok;
+1 -1
View File
@@ -64,7 +64,7 @@ describe("resolveEngineUrl", () => {
await fetchLocalEngineManifest();
const url = resolveEngineUrl("plugin-local");
expect(url).toMatch(/ws:\/\/localhost:\d+\/stream/);
expect(url).toContain("localhost:5000/stream");
});
it("should fall back to plugin's custom streamUrl if not local", () => {
@@ -1,46 +0,0 @@
// src/core/data/__tests__/engineManifest.test.ts
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { fetchLocalEngineManifest, resetManifestCache } from '../engineManifest';
const mockFetch = vi.fn();
describe('fetchLocalEngineManifest', () => {
beforeEach(() => {
resetManifestCache();
mockFetch.mockReset();
vi.stubGlobal('fetch', mockFetch);
});
afterEach(() => {
vi.unstubAllGlobals();
delete process.env.NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL;
});
it('probes port 5000 when NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL is not set', async () => {
mockFetch.mockResolvedValue({ ok: false } as Response);
await fetchLocalEngineManifest();
expect(mockFetch).toHaveBeenCalledWith(
expect.stringContaining(':5000/manifest'),
expect.any(Object)
);
});
it('probes the env-configured URL when NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL is set', async () => {
process.env.NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL = 'http://localhost:5003';
mockFetch.mockResolvedValue({ ok: false } as Response);
await fetchLocalEngineManifest();
expect(mockFetch).toHaveBeenCalledWith(
expect.stringContaining(':5003/manifest'),
expect.any(Object)
);
});
it('returns plugin list when engine responds successfully', async () => {
mockFetch.mockResolvedValue({
ok: true,
json: async () => ({ plugins: ['flight-tracker', 'ais-tracker'] }),
} as Response);
const result = await fetchLocalEngineManifest();
expect(result).toEqual(['flight-tracker', 'ais-tracker']);
});
});
@@ -1,42 +0,0 @@
// src/core/data/__tests__/resolveEngineUrl.test.ts
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
vi.mock('../engineManifest', () => ({
localEngineHasPlugin: vi.fn(),
}));
vi.mock('@/core/plugins/PluginManager', () => ({
pluginManager: {
getPlugin: vi.fn().mockReturnValue(null),
getManifest: vi.fn().mockReturnValue(null),
},
}));
import { resolveEngineUrl } from '../resolveEngineUrl';
import { localEngineHasPlugin } from '../engineManifest';
describe('resolveEngineUrl', () => {
beforeEach(() => {
vi.mocked(localEngineHasPlugin).mockReset();
});
afterEach(() => {
delete process.env.NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL;
});
it('uses port 5000 by default when no env var is set', () => {
vi.mocked(localEngineHasPlugin).mockReturnValue(true);
expect(resolveEngineUrl('test-plugin')).toContain(':5000/stream');
});
it('uses the configured URL when NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL is set', () => {
process.env.NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL = 'http://localhost:5003';
vi.mocked(localEngineHasPlugin).mockReturnValue(true);
expect(resolveEngineUrl('test-plugin')).toContain(':5003/stream');
});
it('falls back to cloud URL when local engine does not have the plugin', () => {
vi.mocked(localEngineHasPlugin).mockReturnValue(false);
expect(resolveEngineUrl('test-plugin')).toContain('worldwideview.dev');
});
});
+8 -5
View File
@@ -8,13 +8,16 @@ let manifestFetched = false;
/**
* Resolve the base URL of the local data engine.
*
* Reads NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL (default http://localhost:5000).
* Only set this in .env.local for local development — never in production.
* Each plugin owns its own streamUrl; this variable is only for routing
* locally-developed plugin seeders to a local engine instance.
* Always checks localhost:5000 — the port docker-compose.yml binds for
* wwv-data-engine. NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL is intentionally
* NOT used here: that variable belongs to each plugin's own declared engine
* URL (production, third-party, etc.) and must not poison local detection.
* Mixing the two caused the production engine to be reported as "local".
*/
function getLocalEngineBase() {
return process.env.NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL ?? 'http://localhost:5000';
const port = process.env.NEXT_PUBLIC_WWV_LOCAL_ENGINE_PORT || '5000';
if (typeof window === "undefined") return `http://localhost:${port}`;
return `${window.location.protocol}//${window.location.hostname}:${port}`;
}
/**
+5 -5
View File
@@ -20,19 +20,19 @@ function toWsStreamUrl(url: string): string {
const DEFAULT_ENGINE_URL = toWsStreamUrl(RAW_ENGINE_URL);
function getLocalWsUrl() {
const base = process.env.NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL ?? 'http://localhost:5000';
const ws = base.replace(/^https:\/\//, 'wss://').replace(/^http:\/\//, 'ws://');
return ws.replace(/\/+$/, '') + '/stream';
const port = process.env.NEXT_PUBLIC_WWV_LOCAL_ENGINE_PORT || '5000';
if (typeof window === "undefined") return `ws://localhost:${port}/stream`;
return `${window.location.protocol === "https:" ? "wss:" : "ws:"}//${window.location.hostname}:${port}/stream`;
}
/**
* Resolves the WebSocket engine URL for a given plugin.
*
* Resolution order:
* 1. Local engine (if running at NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL and has this plugin's seeder)
* 1. Local engine (if running at localhost:5000 and has this plugin's seeder)
* 2. Plugin's ServerPluginConfig.streamUrl (code-based plugins)
* 3. Plugin's PluginManifest.dataSource.streamUrl (manifest-based plugins)
* 4. NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL env var
* 5. Fallback: wss://dataenginev2.worldwideview.dev/stream (cloud)
* 5. Fallback: wss://dataengine.worldwideview.dev/stream (cloud)
*/
export function resolveEngineUrl(pluginId: string): string {
// 1. Local engine (split-routing) - PRIORITY #1
+4 -4
View File
@@ -70,18 +70,18 @@ export async function injectHostGlobals(): Promise<void> {
};
// REST Engine URL (Fallback)
// Note: Local engine interception (via NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL) happens dynamically
// inside resolveEngineUrl.ts during plugin routing. These variables act as global fallbacks.
// Note: Local Docker-based engine interception (localhost:5000) happens dynamically inside
// resolveEngineUrl.ts during plugin routing. These variables act as global fallbacks.
const envDataEngine = process.env.NEXT_PUBLIC_WWV_PLUGIN_DATA_ENGINE_URL;
if (envDataEngine) {
(globalThis as Record<string, unknown>).__WWV_ENGINE_URL__ = envDataEngine;
} else {
// ALWAYS default to the cloud engine unless explicitly told otherwise via env var
(globalThis as Record<string, unknown>).__WWV_ENGINE_URL__ = 'https://dataenginev2.worldwideview.dev';
(globalThis as Record<string, unknown>).__WWV_ENGINE_URL__ = 'https://dataengine.worldwideview.dev';
}
// WebSocket Engine URL
const fallbackWs = envDataEngine ? `${envDataEngine.replace(/^http/, "ws")}/stream` : 'wss://dataenginev2.worldwideview.dev/stream';
const fallbackWs = envDataEngine ? `${envDataEngine.replace(/^http/, "ws")}/stream` : 'wss://dataengine.worldwideview.dev/stream';
(globalThis as Record<string, unknown>).__WWV_WS_ENGINE_URL__ = fallbackWs;
console.log("[HostGlobals] React and SDK injected for dynamic plugins");
@@ -123,7 +123,7 @@ describe("DeclarativePlugin", () => {
getCurrentTime: () => new Date(),
env: {},
edition: "local",
getEngineUrl: () => "http://test-engine/",
getEngineUrl: () => "http://localhost:5001",
});
const entities = await plugin.fetch({ start: new Date(), end: new Date() });
+1 -5
View File
@@ -12,11 +12,7 @@ import type { FilterValue } from "@/core/plugins/PluginTypes";
import { hasLocalSource, resolveLocalSnapshot, getLocalSourceIds } from "./localSources";
export function getEngineUrl(): string {
// WWV_DATA_ENGINE_URL = production (docker-compose, server-side only)
// NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL = local dev override
return process.env.WWV_DATA_ENGINE_URL
?? process.env.NEXT_PUBLIC_WWV_LOCAL_ENGINE_URL
?? "http://localhost:5000";
return process.env.WWV_DATA_ENGINE_URL ?? "http://localhost:5001";
}
function normalizeEntity(raw: unknown): GeoEntity | null {