drobek / docs

drobek — architecture

docs/ARCHITECTURE.md at v0.11.3 · View on GitHub · Markdown

On this page
  1. 1. One process, one image
  2. 2. Workspaces, apps and versions
  3. 3. The compile step
  4. 4. Origins and hosts
  5. 5. Serving
  6. 6. Platform modules
  7. 7. TLS
  8. 8. Background jobs
  9. 9. Agents, the dashboard and abuse

drobek is a cloud workspace for web apps that people build with their own AI agent. The agent connects over MCP and works directly in drobek: it writes files, drobek compiles them in-process with esbuild and returns the compile result in the same response, every write becomes an immutable version with an instant preview host, and a version goes live when the user asks for it. An app's backend is never code the agent writes: it is a set of platform modules (TypeScript, installed by the operator) that the app calls through a small browser SDK. The dashboard is for what does not belong in a chat with an LLM: secrets, confirmations, domains, data, users, logs.

This document is the map of how that works. The neighbours: SELF-HOSTING.md (running it), MODULES.md (the module contract and the built-in modules), AGENT.md (the agent-facing contract), SECURITY.md (the threat model), LICENSING.md (AGPL and the hosted instance boundary).

1. One process, one image

                     dashboard host (PUBLIC_APP_URL)                  apps origin (*.APPS_DOMAIN + custom domains)
                     ────────────────────────────────                 ─────────────────────────────────────────────
  MCP client  ──►  /mcp  (Streamable HTTP, Bearer)                    <slug>.<APPS_DOMAIN>            published version
  browser     ──►  /     (React Router 7 SSR dashboard)               <slug>--preview.<APPS_DOMAIN>   newest version that compiled
  MCP client  ──►  /oauth/*, /.well-known/*  (OAuth 2.1 AS)           <slug>--v<N>.<APPS_DOMAIN>      exactly version N
  browser     ──►  /api/*    (dashboard JSON API)                     shop.example.org (verified)     published version
                                                                      /__drobek/sdk.js, /__drobek/v1/<module>/…
 ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
 │ caddy: TLS for the dashboard host, *.APPS_DOMAIN and verified custom domains  ──►  drobek:3000                │
 ├──────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
 │ drobek — one Node 22 process (apps/server, Express)                                                          │
 │   1. apps-host middleware (@drobek/serving): an app host is answered here and never reaches the dashboard    │
 │   2. origin check (@drobek/auth): mutating dashboard requests from app / null / foreign origins → 403        │
 │   3. /health, /version, /api/internal/tls/ask (internal address only)                                        │
 │   4. /mcp: @drobek/oauth resource server (Bearer → user, scopes, audience) + @drobek/mcp tool bodies          │
 │   5. everything else: React Router (@drobek/dashboard routes, OAuth AS routes, /llms.txt, /report …)         │
 │      behind the body cap: DASHBOARD_MAX_BODY_BYTES → 413 (the CSV import keeps its own limit)                │
 │   in-process jobs (apps/server/server/jobs.ts): blob GC, slug release, domain re-check, audit, files sweep   │
 └──────────────┬───────────────────────────────────────────┬────────────────────────────────┬──────────────────┘
                │ postgres-js + drizzle                     │ ioredis                        │ SMTP (nodemailer) or Resend
          Postgres 17: users, workspaces, apps,       Redis 7: sessions, rate limits,     any SMTP server
          versions + blobs, module data, OAuth,       OTP counters, leases, caches,       (EMAIL_TRANSPORT; Mailpit in dev)
          API keys, domains, abuse, audit             serve-cache + live data pub/sub

2. Workspaces, apps and versions

3. The compile step

write_files (1–20 changes) → apply them to the latest version (whole files, deletions, and exact-string edits; a call with edits is stored only on top of the version it was applied to and re-applied when the same user's other session stored one in between) → validate the paths and limits → scan for secrets (a hit refuses the write and stores nothing) → compile → store the new version (also when the compile failed, so no work is lost) → notify the serve cache. The compile result is part of the tool response.

4. Origins and hosts

Origin Serves Never
dashboard host (PUBLIC_APP_URL) dashboard, dashboard API, OAuth AS, /mcp, /llms.txt, /report any app file or app JavaScript
<slug>.<APPS_DOMAIN> the published version (indexable) the dashboard session cookie is never read here
<slug>--preview.<APPS_DOMAIN> the newest version that compiled (noindex)
<slug>--v<N>.<APPS_DOMAIN> exactly version N (noindex)
a verified custom domain the published version (indexable)

5. Serving

@drobek/serving answers every app-host request in a fixed order, each step before any byte of the app is touched:

  1. method: GET/HEAD (plus the password-unlock POST); else 405;
  2. /.well-known/drobek-report → the report pointer (works for any host);
  3. the unknown-host limiter: a client IP past APPS_UNKNOWN_HOST_LIMIT "no app here" answers per window gets 429 (without a lookup for hosts the cache does not know as live apps — for a version host, that very version);
  4. the app lookup — a miss is a counted 404 page; a version host whose version does not exist or did not compile counts against the same budget (its 404 follows at step 10). Misses (unknown slugs and missing versions) are kept in a separate negative cache for 30 s, hits in the positive cache for 60 s; both are capped (LRU over every host of every app) and drop expired entries, so no range of --v<N> numbers grows the process;
  5. X-Drobek-App: <slug> on every response from here on;
  6. a taken-down app: 451 on every host and path (JSON 451 on platform paths), before the redirect, the password gate and the modules;
  7. the primary-domain 302 (production host, GET/HEAD page requests);
  8. visibility: a password app shows the password page (401) until the host-only __Host-drobek_app_access cookie (HMAC, key derived from DROBEK_MASTER_KEY) is set;
  9. /__drobek/*: the SDK, the module routes and the beacon — handed to the module runtime, never to the app's files; on the preview and version hosts /__drobek/feedback.js is the feedback widget, served by the handler;
  10. the version the host serves (404 "not published" / "nothing compiled"), then the file: built output wins over sources, .ts/.tsx/.jsx sources and drobek.json are never served, extension-less paths fall back to index.html, ETag = sha256 → 304; an HTML response names its version in Server-Timing: drobek-version;desc="<N>" (the beacon reads it). Every JS/CSS bundle is stored with an inline source map; the preview and version hosts serve it as stored, while the production host and custom domains serve the code without it (ending in sourceMappingURL=<file>.map) and answer /<file>.map with the map, split at serve time and cached — browsers fetch it only when devtools opens;
  11. no such file: the app's asset at that path, if any (see below).

Assets (video, audio, images, fonts) share the app's URL space: the asset img/s1.jpg answers /img/s1.jpg, so a page keeps its own relative paths (<video src="film.mp4" poster="poster.jpg">). The app's own file at the same path wins; an asset path always has a media extension, so the SPA fallback never swallows one. Assets honour publish: app_assets is the draft — uploads, replacements and deletes change only it, and the preview host serves it; publish freezes a set for the version it puts live (app_version_assets, app_versions.assets_frozen_at) and the production host and custom domains serve only the live version's set; a version host serves its version's set, or the draft when it was never published. Publishing the newest version that compiled freezes the draft; publishing an older one (the rollback) keeps the set it had when it was last live; restore_version of a published version resets the draft to its set. The bytes live on disk under ASSETS_DIR (/data/assets/<app_id>/<sha256> — content-addressed and never rewritten, so the draft and any number of sets share a file; files from before this keep a random key; the assets_data volume). APP_ASSETS_QUOTA counts unique files: what the draft and the live set need must fit; the sets of up to ten earlier publishes are kept for a rollback while they fit besides, the oldest dropped first. Serving: the sniffed Content-Type, Accept-Ranges: bytes, one byte range → 206 (Content-Range) or 416, ETag (sha256) / Last-Modified → 304, If-Range, HEAD; public, max-age=300, must-revalidate on the published and custom hosts, revalidate-always on preview and version hosts, private for a password app; SVG as an attachment with a second CSP sandbox. Takedown, the password gate and "not published" answer first, like for any file.

Uploading never goes through MCP or the model: create_asset_upload (or the dashboard's Assets tab) checks the path, the declared size and type, the quota and the hourly budget, then mints a single-use upload URL on the dashboard host — PUT /api/assets/upload/<token>, 32 random bytes, only its sha256 stored in Redis for 30 minutes, bound to the app, the path, the size, the type family and the user who asked (the upload is audited as theirs). The PUT takes the token before reading a byte, streams the body to a temp file while it counts (over APP_ASSET_MAX_BYTES or past the declared size → stop), hashes and sniffs it (png, jpeg, gif, webp, avif, ico, svg, mp4, webm, m4a, mp3, ogg, wav, woff, woff2 — the bytes decide, never the name), renames it into place under its sha256, and writes the draft row under a per-app advisory lock that re-checks APP_ASSETS_QUOTA — and that the user the URL was issued for is still an editor of the app. A browser GET on the URL shows a small upload page (strict CSP, the token never in the page). A delete or replace removes a file nothing references any more; an hourly sweep removes the assets of apps deleted 24 h ago, stale temp files and files neither the draft nor a kept set references.

Every response carries the app CSP (default-src 'self', scripts from the app and https://esm.sh, connect-src 'self' https://esm.sh, images, fonts, styles and media-src (<video>, <audio>) from the app, blob: or any https URL, frame-src only the curated embeds — YouTube (www.youtube-nocookie.com, www.youtube.com), player.vimeo.com, drive.google.com — plus the operator's APP_FRAME_SRC_EXTRA, frame-ancestors = the dashboard origin only, plus the origins the owner set in apps.frame_ancestors, plus — on the production host and custom domains of an app the public gallery shows, never on preview or version hosts — the operator's GALLERY_FRAME_ANCESTORS; the dashboard frames an app solely for the app-list thumbnail, the gallery for its live preview, see SECURITY.md), nosniff and Referrer-Policy: strict-origin-when-cross-origin (only the origin reaches a third party); preview and version hosts add X-Robots-Tag: noindex. Bytes come from a 256 MiB in-memory LRU; the host and manifest caches are busted through a local event emitter first and Redis pub/sub (drobek:app-changed) for other replicas, so a write is visible on its preview host at once.

6. Platform modules

A module is an npm package whose default export comes from defineModule() (@drobek/modules, contract 1.3.0; a module states the versions it works with in contract, e.g. '^1.1', and one this server does not satisfy refuses the start). The operator enables modules with DROBEK_MODULES; a short name x loads drobek-module-x (which must export the module x), a full package name may replace a built-in. Each entry is looked up first in DROBEK_MODULES_DIR (the modules_data volume: modules installed without a new image, each listed with its integrity in modules.lock.json, given the server's own @drobek/* / zod / drizzle-orm through a node:module resolve hook, migrations linted to mod_<name>[_*]), then among the server's dependencies; /healthz and /api/version list the active modules with their source (dir | builtin). A module contributes routes under /__drobek/v1/<name>/… on every app host, a slice of the browser SDK (drobek.<name>), a zod per-app config schema (its defaults overridable per server with DROBEK_MODULE_<NAME>_DEFAULTS), access rules, secrets (names only), env-named limits, its own error codes, its own tables and migrations, and a skill the agent reads with skill_info — left out only by an operator-only module, of which nothing reaches apps (it contributes to operatorOnly slots such as errors.reporter and email.transport): agents and app owners never see it, /api/version marks it operatorOnly. Modules extend each other through typed slots: a host module declares one with a zod schema, other modules contribute values, checked at start and read with contributions(slot) (a host may compose its config schema, confirm rules and secrets from the contributions at start). Core itself hosts one slot, errors.reporter: where server errors go besides the log (an incident webhook, a log service, …), chosen with ERROR_REPORTER=<id> and fed by reportError (@drobek/core) from the central error points — a 5xx, a module route throw, a failed module job or e-mail send, a start-up failure, an error nothing caught. Built in: auth (end-user sign-in by e-mailed code, plus the sign-in providers other modules contribute to its auth.provider slot, and the auth.signedIn observers told of every sign-in), email (notifications to the app's owners, and the email.transport slot for e-mail providers other modules contribute, chosen with EMAIL_TRANSPORT=<id>), forms, data (collections with per-operation rules), proxy (external APIs with the secret injected server-side), files (end-user uploads), sync (a data collection filled from an external API on a schedule) and oidc (company sign-in with any OpenID Connect provider, an auth.provider). The examples hello (a dependency of apps/server) and the opt-in acmecrm (installed into DROBEK_MODULES_DIR) are loaded exactly as third-party modules are. The contract is MODULES.md.

7. TLS

Caddy runs next to drobek (docker-compose.production.yaml) and terminates TLS; drobek speaks plain HTTP on the internal network and trusts only Caddy's X-Real-IP (TRUST_PROXY=x-real-ip). The Caddyfile is generated from the environment (@drobek/core caddy.ts, task selfhost:init / task caddy:config) and contains no secrets. The dashboard host gets a normal ACME certificate; the app hosts use exactly one of three paths:

Verified custom domains get their certificates from an on-demand catch-all behind the same ask (on by default in mode (c), TLS_CUSTOM_DOMAINS). tls internal (Caddy's local CA) serves a test box and task dev:tls. Caddy also compresses text responses (encode zstd gzip, 200 only, an explicit type list without text/event-stream, so /mcp SSE is not buffered); drobek itself never compresses. On the dashboard site Caddy refuses a request body over DASHBOARD_MAX_BODY_BYTES (request_body, 413) except on the paths drobek caps with their own limits (/mcp, the asset upload URLs, the Data tab's CSV import, the feedback note page with its screenshot); drobek applies the same cap itself in front of React Router (@drobek/core withBodyLimit), counting a chunked body as it arrives. Details: SELF-HOSTING.md → TLS.

8. Background jobs

All in-process (apps/server/server/jobs.ts), started with the server. They query through a Postgres pool of their own (runAsJob in @drobek/db) without the requests' DB_STATEMENT_TIMEOUT_MS, so a long sweep never takes a connection a request waits for and is not cut off like a request's query:

Job Interval What
version retention hourly, Redis lease deletes the versions of each app past its workspace's APP_VERSIONS_KEEP (200) — never the published one, the one the preview serves, a kept one, a rollback set or the last hour's; skips a workspace whose limits provider does not answer (@drobek/apps)
blob GC hourly, Redis lease drops the screenshots of feedback notes resolved FEEDBACK_SCREENSHOT_RETENTION_DAYS (30) ago and the screenshots for the agent older than SCREENSHOTS_RETENTION_DAYS (7), then deletes blobs no version, note or screenshot references, after 7 days
slug release hourly, Redis lease a soft-deleted app's slug is free again after 30 days
app purge APP_PURGE_INTERVAL_MS (1 h), Redis lease deletes an app deleted APP_PURGE_AFTER_DAYS (30) ago for good, one app per transaction: the apps row and through ON DELETE CASCADE its versions (their blobs go with the blob GC), module configs and secrets, domains, asset rows, feedback notes and screenshots, gallery likes and opens, logs and statistics, and the module tables (records, form submissions, end users and identities, uploads, sync state); abuse reports and duplicates keep their rows without the reference; its id leaves upstreams.allowed_app_ids, its asset directory and end-user sessions go; audited app.purge. An app a foreign key without ON DELETE holds is logged and retried every run (@drobek/apps)
domain re-check DOMAINS_RECHECK_INTERVAL_MS (1 h), Redis lease re-verifies domains checked more than 24 h ago; unverifies + mails on a definitive failure
files sweep (only with the files module) FILES_SWEEP_INTERVAL_MS (1 h), Redis lease removes the uploads of apps deleted FILES_SWEEP_RETENTION_MS (24 h) ago, stale temp uploads and blobs no mod_files row references (drobek-module-files)
assets sweep hourly, Redis lease removes the asset files and rows of apps deleted 24 h ago, stale temp uploads and files neither the draft (app_assets) nor a kept published set (app_version_assets) references (@drobek/apps)
logs prune LOGS_PRUNE_INTERVAL_MS (1 h), Redis lease removes get_logs rows past their retention for every app: browser errors older than 30 days or past the newest 500 per app, compiles and daily request stats older than 30 days (@drobek/insights)
analytics rollup a minute after start, then hourly; Redis lease moves the app traffic counters of the last 7 days from Redis into app_traffic_daily / app_traffic_top (a count never goes down) and deletes days older than ANALYTICS_RETENTION_DAYS (90) (@drobek/insights)
module jobs (only when an active module declares jobs) each job's own interval (checked every 15 s), a Redis lease per run the modules' scheduled work, for the server or for each app that configured the module; at most MODULE_JOBS_CONCURRENCY (4) runs per process, each cut off at MODULE_JOBS_TIMEOUT_MS (5 min); a failed run retries with backoff and an app's failure shows in its get_logs runtime; MODULE_JOBS_ENABLED=0 = none on this process (@drobek/modules, MODULES.md)
audit retention at start, then daily deletes audit rows older than AUDIT_RETENTION_DAYS (365) — the only deletion of audit rows anywhere
retention prune at start, then daily, Redis lease deletes OAuth access and refresh tokens 7 days after they expired (a rotated refresh token stays until then, so reuse detection still sees it) and authorization codes 37 days after (when the lineage a code minted has expired as well, so a replayed code revokes it as long as it exists) (@drobek/oauth); gallery open counts of days older than the 30-day opens window plus 7 days; abuse reports resolved more than ABUSE_REPORTS_RETENTION_DAYS (365) ago — open reports stay (@drobek/apps)

Request counters for get_logs('requests') accumulate in Redis and are flushed into Postgres on read (the whole window in one pipelined round trip) and at most once a minute per app and day; reads never delete. A flush never lowers a stored day: each count, and each path's 404 count, keeps the larger of the stored and the flushed value, so counters that restart after a Redis flush cannot shrink a day already in Postgres. The paths of failing requests are counted per app and UTC day, path only (no query, at most 100 distinct per class, the rest as __other__): a missing file's path is stored with the daily stats, a platform 4xx's or a 5xx's stays in Redis for the 30-day window.

App traffic analytics (the Analytics tab, get_analytics, get_app traffic; ANALYTICS_ENABLED) counts what the serving handler answered with a 200 or 304 HTML page to a GET on the production host or a custom domain — never a preview or version host, a file, a platform route, a prefetch, a drobek-… user agent or a frame sent without a referrer or from the dashboard or the gallery website (the app thumbnail, the gallery's live preview). Per app and UTC day Redis holds page views, bot views (user-agent list), a HyperLogLog of sha256(daily salt, app id, client IP, user agent) for the visitor estimate, and hashes of page paths and external referrer hosts (at most 200 each, the rest __other__). The salt is random per day and lives only in Redis, so the hashes cannot be linked across days; Postgres gets counts only. The rollup (a minute after start, then hourly) stores the last 7 days; a read merges today and yesterday from Redis (the larger count wins), so the tab is live.

9. Agents, the dashboard and abuse