System map
Machines
Section titled “Machines”| Where | What runs there | Reach it |
|---|---|---|
| Cloudflare | DNS, TLS, page cache, rate limit for *.mohanad.xyz |
dashboard |
Hangar box 188.245.169.96 (admin.mohanad.xyz) |
kalelm (Astro site), kalelm-api (Payload + Next admin/API) and kalelm-id (the account hub, Astro + Better Auth) as Hangar app sites |
Hangar API with a token; ssh root |
Search VPS 49.12.78.95 |
Caddy; Meilisearch 1.53.1 (25 GB, 4 indexes); Ollama + bge-m3 (query embeddings); Postgres 17 (the archive DB, TLS) | ssh root; /srv/kalelm |
| This Mac | dev backend (3001), local Postgres + Meilisearch in Docker, the transcription worker (GPU) | — |
Public URLs (test): https://kalelm.mohanad.xyz, https://kalelm-api.mohanad.xyz/admin.
Internal: search.49.12.78.95.sslip.io (Meilisearch), embed.49.12.78.95.sslip.io (Ollama), 49.12.78.95:5432 (Postgres) — all reachable only from the Hangar server.
| Name | What | TLS |
|---|---|---|
kalelm.mohanad.xyz |
the site (kalelm) |
Cloudflare |
kalelm-api.mohanad.xyz |
API + admin (kalelm-api) |
Cloudflare |
kalelm-docs.mohanad.xyz |
this documentation | Cloudflare |
kalelm-id.mohanad.xyz |
the account hub (kalelm-id): one sign-in for every site, see Learning platform and the account hub |
Cloudflare |
admin.mohanad.xyz |
Hangar API; deploys go over ssh to the box and hit it on 127.0.0.1 | Cloudflare / self-signed inside |
search.49.12.78.95.sslip.io |
Meilisearch | Caddy, Let’s Encrypt |
embed.49.12.78.95.sslip.io |
Ollama (bge-m3 query embeddings) | Caddy, Let’s Encrypt |
49.12.78.95:5432 |
Postgres, Hangar server only | self-signed /srv/kalelm/certs/server.crt with IP:49.12.78.95 as alt-name — hand it to the backend base64-encoded as PG_CA_CERT (verify-full since 2026-09-10) |
files.kalelm.com |
audio and video, outside this project | — |
A page view. Browser → Cloudflare (cache: pages 10 min fresh, 1 h stale) → Hangar/Caddy → kalelm (Astro SSR) → kalelm-api REST → Postgres on the search VPS. Facets and the browse tree are cached in the API for 5 min; the sitemap is built hourly.
A search. Site → POST /api/search with mode keyword | hybrid | semantic. Keyword hits the caption_segments index. Hybrid/semantic embed the query through Ollama on the VPS and search caption_sentences with vectors. If the vector path fails, or SEARCH_VECTORS=off is set, the API answers with keyword results and degraded: <mode>.
A transcription. Admin panel /admin/transcription enqueues eligible materials (published, audio-only, no transcript) into transcription-jobs. A worker (any machine, outbound HTTPS only, CRON_SECRET bearer) claims a job → downloads the mp3 → Cohere ASR → cues built from the word timings (ten words or more each) → uploads cues. The API creates the transcript; its hooks flag the material and push it into Meilisearch.
Screens
Section titled “Screens”The indexing page /admin/indexing shows the four indexes as tiles (held versus expected), Meilisearch’s queue, any vector build in progress, and the twenty most recently transcribed materials with a dot per index and a repair button. Every material’s edit page carries the same four dots in its sidebar («حالة الفهرسة في البحث») with a re-index button, so an editor can see whether what they just saved is searchable.
The transcription control panel at /admin/transcription: backlog, queue, workers, failures, the pause switch and the local worker card.
The queue itself as a Payload collection: one row per material with status, stage, worker and attempts.
The admin home: the archive dashboard above the collection list.
The public search page in hybrid mode, results grouped by material with cue-level marks.
An archive listing page, served from the Cloudflare cache after the first visitor.
The site home page on the test domain.
Where things live (repo: backend/)
Section titled “Where things live (repo: backend/)”| Thing | Path |
|---|---|
| Queue collection, settings global | src/transcription/jobs.ts, src/transcription/settings.ts |
| Pipeline API (stats, control, claim, heartbeat, complete) | src/transcription/api.ts |
| Control panel | src/transcription/panel/ |
| Local worker start/stop | src/transcription/localWorker.ts |
| Search API + fallback + switch | src/search/api.ts |
| Meilisearch sync hooks | src/search/sync.ts |
| Endpoint cache | src/utilities/ttlCache.ts |
| One eligibility rule for the queue; job states | src/transcription/eligible.ts, top of src/transcription/jobs.ts |
| Detached admin jobs (backfills, local worker) | src/admin/jobs.ts; the admin nav and home card in src/admin/ |
Frontend features (dark mode, player bar, library, feeds…): folders under ../frontend/src/features/, see Frontend features |
src/lib/features.ts in the frontend |
| Durations: probe, backfill, save hook, panel (the panel’s backfill buttons and the local-worker card need the repository and pnpm, so they work only on the dev backend, against its own database) | src/durations/ |
| Search index builders (materials, sentences, passages) | src/search/build/ |
WordPress import and its stage runner (pnpm migrate:wp) |
src/wordpress/ |
| Content collections, access rules, shared fields | src/collections/, src/access/, src/fields/ |
| Learning platform: courses, attempts, certificates, the grading and verify endpoints, the hub token check | src/learn/; the pages are the frontend feature src/features/learn/ |
| The account hub (its own app: accounts, OpenID Connect, per-site reader data) | hub/ (README.md; the page: Learning platform and the account hub) |
| Logs of detached jobs | logs/ (gitignored, not deployed) |
| Worker (Python) | worker/ (README.md has VPS/Mac setup) |
| Search VPS config | infra/search-vps/ (compose, Caddyfile, top-queries.sql) |
| Deploy to Hangar | scripts/deploy-hangar.sh backend|frontend|hub|docs — docs publish to https://kalelm-docs.mohanad.xyz |
| Reindex one material | scripts/reindex-material.ts <id> |
| The gate | pnpm verify (73 checks) |
Secrets and env
Section titled “Secrets and env”Never in git. Backend env is held by Hangar (GET/PUT /api/v1/sites/kalelm-api/env): DATABASE_URL (VPS Postgres), PG_CA_CERT (its self-signed certificate, so the pool runs verify-full instead of sslmode=no-verify; see Hosts), PAYLOAD_SECRET, CRON_SECRET (also the worker’s bearer), MEILI_HOST, MEILI_MASTER_KEY, OLLAMA_URL, NEXT_PUBLIC_SERVER_URL, FRONTEND_URL, optional SEARCH_VECTORS=off, PIPELINE_WORKER_PYTHON, HUB_URL (the account hub, for /api/learn/*). The hub’s and the site’s own keys: Learning platform and the account hub → Env.
VPS secrets: /srv/kalelm/.env (MEILI_MASTER_KEY, POSTGRES_PASSWORD). Tokens for Hangar, Cloudflare and Neon are yours; rotate the ones pasted into chat.
Runbook
Section titled “Runbook”| Do | How |
|---|---|
| Deploy backend / frontend / hub | HANGAR_TOKEN=… scripts/deploy-hangar.sh backend (or frontend, hub) |
| Connect a site to the account hub, author a course | Learning platform and the account hub → Runbook |
| Start a transcription worker on this Mac | worker/README.md → The owner’s Mac: one nohup … worker.py line with CRON_SECRET from Hangar env; the panel at /admin/transcription shows it by name |
| Set up a worker on a machine that never ran one | Run a worker on a new machine, written for a Claude session on that device |
| Change backend env | PUT …/sites/kalelm-api/env with the full set (it replaces); app restarts |
| Start/stop a worker on the backend host | panel → عامل على هذا الخادم; elsewhere: systemd/launchd unit in worker/README.md |
| Pause all workers | panel → إيقاف مؤقت |
| Turn semantic search off/on | set/remove SEARCH_VECTORS=off in Hangar env |
| Rebuild search indexes | pnpm migrate:wp index (keyword), sentences / passages (vectors) with MEILI_HOST pointing at the target; or copy data.ms between same-version instances |
| Fix one material’s search entry | scripts/reindex-material.ts <id> |
| See what hits the database | on the VPS: docker compose exec -T postgres psql -U kalelm -d kalelm < top-queries.sql |
| See traffic | Cloudflare dashboard → mohanad.xyz → Analytics |
| Prove the repo is sound | pnpm verify |
What runs on a schedule
Section titled “What runs on a schedule”Everything periodic, so nobody adds a second cron for something that already happens:
| What | Where | When |
|---|---|---|
| Database dump, copy to the Hangar box, 14-day rotation | root crontab on the search VPS, infra/search-vps/backup.sh |
03:00 UTC nightly (no failure ping yet, see Backups) |
| Kaggle worker run (four hours of free GPU) | root crontab on the search VPS, infra/search-vps/kaggle-run.sh |
03:00 UTC daily |
| Scheduled publishing of articles and pages | Payload’s job runner inside the backend (jobs.autoRun in payload.config.ts) |
every 5 minutes |
| Transcription queue feed and stale-job reclaim | on every worker claim (src/transcription/api.ts) |
whenever a worker asks for work |
| Duration on a new material | save hook (src/durations/fillDuration.ts) |
on save |
| Search index update | collection hooks (src/search/sync.ts) |
on save and delete |
| TLS certificates | Caddy renews the public ones; the Postgres one is self-signed and valid to 2036 | automatic |
Not scheduled, on purpose: search-log purge (979 rows, 448 kB after ten days; revisit at a million), Meilisearch snapshots (rebuildable from Postgres), log rotation (each detached job rewrites its log on start).
Known limits (test deployment)
Section titled “Known limits (test deployment)”- Search VPS has 3.7 GB RAM for a 25 GB search database: keyword fast, semantic ~1.5 s.
- Hostnames use sslip.io and the test domain; robots.txt disallows all crawlers until launch (launch block inside the file).
- Bot Fight Mode still needs one toggle in Cloudflare → Security → Bots.
- ESLint config is broken repo-wide;
tscandpnpm verifyare the checks that work.
Maximizing organization — the plan
Section titled “Maximizing organization — the plan”- Commit everything, now. Done 2026-09-10: everything is committed and pushed to two private repositories, kalelm-backend (with
worker/,infra/,docs/) and kalelm-frontend. Merging them into one is still open; the deploy script expects../frontendbesidebackend/. - One way to deploy each thing, checked in:
scripts/deploy-hangar.sh,infra/search-vps/README.md. Nothing deployed by hand-typed commands again. - Config in files, secrets in one place. Every non-secret setting in git; secrets in Hangar env /
/srv/kalelm/.envonly, with.env.examplefiles listing every key. A password manager entry per token. - Docs next to code, one page per audience: this file (map + runbook),
worker/README.md(operators of workers),infra/search-vps/README.md(the box). Update the page in the same PR as the change. - Real names. Custom domains via Cloudflare instead of sslip.io;
search./embed./db.hostnames on your domain. - The gate stays the definition of done.
pnpm verifybefore every deploy; add a check with every bug fix (the pipeline section is the pattern). - Observe three things weekly: Cloudflare analytics,
top-queries.sql, the transcription panel’s failures list.