Production audit

Five independent read-only reviews on 2026-09-10 (security, correctness, frontend, engineering practice, editor experience and live search quality). Every item was verified against the code or the live deployment. Effort: S under an hour, M a day, L several days. File paths are relative to backend/ unless prefixed frontend/.

Summary

Area	Critical	High	Medium	Low
Backend security	0	2	8	10
Backend correctness and data	2	5	9	4
Frontend	2	8	14	6
Engineering and operations	1	7	9	7
Admin UX and search quality	1	4	9	5

Fixed during the audit: the users-table SQL dump moved out of the repo, .gitignore corrected, worker state files stripped from the deploy bundle and the backend redeployed.

P0 status (2026-09-10)

#	Item	Status
1–3, 7, 8	Search sync: purge all indexes on unpublish, reads inside the transaction, shared projection, per-language ids, vectors after the read	Fixed in src/search/sync.ts, src/search/project.ts; gate audit section round-trips unpublish/republish
4, 6	Sort ranking rule, maxTotalHits in code	Fixed and applied to both instances (scripts/configure-search.ts)
5	Search input allowlists	Fixed; crafted filters return 200 with the filter ignored
9	Search hits link to the material at the cue time	Fixed; the material page seeks the audio to ?t=
10	Sheikh cards 404	Fixed; facets return the slug
11	XSS via CMS names; javascript: links	Fixed
12, 13	Fetch timeout; 500 page; header fails soft	Fixed
14	Site URL and sitemap API base	Fixed; SITE_URL is set by the deploy script
15, 16	Locale-only search; article index 404	Fixed
17	Rate limit on the API host	Partly: in-app per-IP bucket on /search with the site server exempt. The free Cloudflare plan allows one rate-limit rule per zone and no ip.src exemption, so the edge rule stays on the site host only.
18	Stale jobs re-queued forever	Fixed; fail after three claims
19, 20	Transcript FK cascade; hasTranscript backfill	Fixed in migration 20260910_070000; applied locally. Pending on production: needs a deliberate run of pnpm payload migrate against the VPS database.
22	ESLint crash	Fixed; lint runs (0 errors, 3 warnings)
23	sharp advisories	Fixed; 0.35.4
24	fillDuration overwrote manual durations	Fixed
25	Media accepts SVG	Fixed; raster images only
21	Git	Done: backend f315bf1, frontend 52180dd (its first commit). Both apps redeployed with the fixes: releases 20260910-064919 (api) and 20260910-065023 (site).

P1 status (2026-09-10, later the same day)

All P1 items were implemented by three parallel passes (backend and admin, frontend, engineering) and verified by the gate, now 106 checks.

Area	Done	Notes
Backend	cache keys and cap; facets throw; worker-bound heartbeat/complete; one transaction for complete; cue/stats caps; timing-safe bearer; search-only Meilisearch key (MEILI_SEARCH_KEY); file URLs pinned to files.kalelm.com; Gemini path deleted; WP DB connected only for WP stages; cue validation shared with the admin; language required, status auto/reviewed on transcripts; orphan tables dropped; sort honoured, longest removed; keyword facets count materials; addContext for keyword hits; tree tiebreaks; hidden sheikhs invisible; search logs sampled; worker env allowlist	42 migrated materials still carry off-origin audio URLs; they save until the URL field itself is edited. Hidden sheikhs’ materials remain in parent counts.
Admin	headings, lists, quotes, images in the editor; transcript and job joins on materials; editors may change job status; Arabic labels incl. the redirects plugin; media alt required, three sizes; scheduled publishing runs (jobs autoRun every 5 min); live preview removed	wpId not searchable in the list (Payload turns it into a null match)
Frontend	locale-safe links and paging; degraded notice; OG/Twitter/JSON-LD; per-locale caches; filter panel 266 KB → 82 KB with on-demand loading; sitemap index in three files without /search/; same-origin download route; YouTube transcript sync and ?t=; main landmark, skip link, heading levels; /sheikhs/ and /series/ pages; dead links removed; security headers; astro check at 0 errors	No CSP (inline scripts and styles everywhere; needs nonces)
Engineering	service-free gate sections and a unit section; CI workflow; env assertion at startup; TLS verification against the server certificate (PG_CA_CERT); deploy script keeps the token out of argv and handles Cloudflare timeouts; backup pings a health check and lists the dump monthly; README; .env.example complete; Dockerfile and workspace file removed	.nvmrc stays at 22.17 (Hangar runs 22.23; not installed locally)

Still open from the audit: P2 items, the unbuilt features (quizzes, live, RSS, analytics, fonts, CSP), token rotation, and one repository for both apps.

UX pass (2026-09-10, evening)

A walk through the site as a reader, not a crash hunt. Fixed, each with a gate check where the page can prove it:

• Series list pages in place. The playlist sheet on a lesson fetched ?pl=N as a whole page, stopping the lesson and closing the sheet. It now swaps the rows from /partials/playlist (src/components/PlaylistItems.astro renders both), keeps the address bar in step, and the arrows always belong to the lesson, not to the list page shown.
• Home hero scope toggle actually reaches /search (radios in the form); the “latest” thumbnails open the lesson instead of being dead buttons; “full archive” links go to the tree and the lessons, not an empty search.
• Search keeps its facets when the query, scope, mode or sort changes (hidden inputs); the title carries the query; an empty result without filters no longer blames filters.
• Browse tree search matches open like any other branch (row ids are level keys); the status line restores after clearing; a mode change clears a search; a failed fetch says so and offers a retry; the tree dims while loading; “expand all” is labelled for what it does (open sections).
• Language switcher keeps the query string on path-preserving pages; the language menu closes on outside click and Escape.
• Series are oldest-first everywhere a link opens one (series index, lesson byline and breadcrumb, /lesson-page/ redirects, card series titles now link).
• Pager has a page-number box past seven pages; a page past the end redirects to the last page instead of a 404, on every listing.
• Sort control shows the current order beside the icon.
• Header has a search entry on desktop; the Sheikhs and Articles entries highlight on their own pages; the sheikhs breadcrumb links to the index; dead middle crumbs on chapter and tag pages removed.
• Mobile filter sheet stays open across a filter tap; sheets move keyboard focus inside when opened.
• Untranslated lesson under /en/ or /ur/ explains itself and links to the Arabic page (still a 404 for crawlers).
• Share copies the page’s own URL, not a ?t= deep link, and dismissing the native sheet no longer claims “copied”; prev/next chevrons mirror under dir=ltr; the chapter chip keeps the locale.

The features first left for later landed the same evening (frontend sheikh page, MaterialDetail, search, browse):

• Player. A prompt card asks to continue where the reader stopped (position remembered per material in the browser, dropped near the end); when a lesson ends it offers the next one, and arriving with autoplay=1 starts it. One speed control drives the mp3 and the embed, remembered across lessons. Each transcript line has a copy-link button and the toolbar copies a link to the player’s current moment.
• Transcript follow was scrolling to the wrong place. offsetTop was measured against the page, not the list. It is measured from the list’s box now, and following resumes on its own once the spoken line is back in view and the reader has paused.
• Tags are chips on the lesson page, linking to /tag/…/.
• Sheikh page is a full archive: type chips with counts, series/year/transcript filters, sort, and the page-past-the-end redirect.
• Search results link the sheikh’s name (the API now returns sheikhId/seriesId); every facet value the engine returned is offered, ten at a time with a search box and “show all”.
• Browse tree remembers mode, query and open branches within the tab, so coming back from a lesson lands where the reader left.

Regroup by domain (2026-09-10, night)

The backend was organised by kind of file (collections, endpoints, components, lib, globals); one feature was spread across six folders. It is organised by what it does now, no code rewritten:

Folder	Holds
src/transcription/	queue collection, settings global, the pipeline API, the one eligibility rule (eligible.ts), the local worker, the panel
src/search/	Meilisearch config and sync, the search API, indexing stats and panel, the index builders under build/
src/durations/	probe, backfill, the save hook, the panel
src/wordpress/	the import stages and their runner
src/admin/	the detached job runner (jobs.ts, one for backfills and the worker), the nav links, the home card

Also: the repo root holds only config; logs go to logs/, the duration cross-check lives in scripts/, the eval set in scripts/kaggle/. The transcription queue now takes every material with an mp3 (they all have one; some also have a video) and a new switch, off by default, lets it redo the migrated YouTube caption transcripts from the mp3. The new-material form is in tabs, with a YouTube field that accepts a pasted address.

P0 — must fix before launch

Search returns or hides the wrong things

1. Unpublished material stays in semantic search. src/search/sync.ts:216 clears only the materials and segments indexes on unpublish or delete; sentences and passages keep the text, and enrich() returns the material with overrideAccess. Republishing never re-adds segments because status is not in the denormalized list. Fix: purge all four indexes on unpublish/delete; reindex on publish. S.
2. Admin-saved materials are indexed stale or not at all. src/search/sync.ts:37 reads the material without req, outside the saving transaction: a new material is not found and never indexed; an edited title pushes the old title. Fix: pass req, as reindexMaterialSegments already does. S.
3. Edited materials lose language, year, titleFolded in the index. sync.ts:53 projects fewer fields than the bulk indexer (src/search/build/materials.ts:129) and addDocuments replaces the document, so an edited material drops out of title search and the year facet. Fix: one shared projection function. S.
4. Sorting by newest/oldest returns HTTP 500. src/search/meili.ts:135 sets ranking rules for segments without sort, so Meilisearch rejects the sort parameter. Fix: add 'sort'. S.
5. Search filters are built from unvalidated request fields. src/search/api.ts:248 interpolates type, sheikhId, seriesId, since, language into Meilisearch filter strings; a crafted array rewrites the filter, and transcript scope has no published clause. Fix: allowlist enums, Number.isInteger on ids. S.
6. maxTotalHits is not in code. The live index was raised by hand; a rebuild from configureIndexes silently caps counts at 1,000. Fix: set it in src/search/meili.ts. S.
7. Saving an English transcript deletes the Arabic one from search. sync.ts:92 looks up one transcript per material regardless of language and segment ids are material_index. Fix: key by language, index every transcript. M.
8. The semantic reindex runs unawaited before commit. sync.ts:147 fires in the background while the transaction is open and usually finds nothing. Fix: run after commit, or pass req and await. M.

Site features that are broken today

9. Search hits play nothing. frontend/src/pages/search.astro:304 renders hits as play buttons, but no player exists and the handler only highlights. Fix: link each hit to the material with ?t=start and honour it in the detail page. S.
10. Every homepage sheikh card is a 404. frontend/src/pages/index.astro:300 links by id where the route resolves by slug. Fix: return the slug from facets. S.
11. Stored XSS from CMS content. frontend/src/pages/index.astro:201 renders sheikh and series names with set:html. Fix: plain interpolation. S. Related: frontend/src/lib/lexical.ts:57 allows javascript: links; allow only http(s), mailto and relative. S.
12. No fetch timeout anywhere. frontend/src/lib/api.ts:64: a stalled API hangs every server render. Fix: AbortSignal.timeout(8000). S.
13. No 500 page and no error boundary. Any API error, even on the 404 page, becomes bare “Internal Server Error” because the header awaits section counts. Fix: 500.astro; cached fallback in the header. S.
14. Sitemap points at localhost and the wrong domain. frontend/src/pages/sitemap.xml.ts:28 reads an env var that is never set; :18 hardcodes https://kalelm.com; no site in astro.config.mjs, so canonicals trust the Host header, which is cache-poisonable under the new edge rule. Fix: site from env, PUBLIC_API_URL. S.
15. English pages search only English material. frontend/src/pages/search.astro:47 passes the locale as the language filter, so /en/search sees 9 recordings and the English homepage chips return nothing. Fix: filter by language only when asked. S.
16. Article index 404s while every header and footer links to it. frontend/src/pages/article/index.astro:16. Fix: empty state. S.

Safety and operations

17. No rate limit on the API host. Semantic search costs 1.5 s of the embedder per call, login lockout is 5 tries, search logs are written per anonymous query. Fix: a Cloudflare rate-limit rule on kalelm-api like the frontend’s, plus a small per-IP bucket on /search. M.
18. Stale jobs are re-queued forever. src/transcription/api.ts:259 ignores attempts on reclaim; a job that kills its worker burns GPU every 30 minutes. Fix: fail after three claims. S.
19. Deleting a material with a transcript fails. transcripts.material_id is NOT NULL with ON DELETE SET NULL. Fix: migration to cascade, plus an index purge. S.
20. hasTranscript is never backfilled by the migration despite the field’s comment; bulk imports skip the hook. Fix: one UPDATE … SET has_transcript = EXISTS(…) at the end of the transcripts stage. S.
21. Git. Backend has one commit and 230 untracked files; the frontend has no repository. Fix: commit now. S.
22. ESLint has never run. eslint.config.mjs:13 wraps an already-flat config in FlatCompat and crashes. The verified fix: import eslint-config-next/core-web-vitals and /typescript directly, drop @eslint/eslintrc. S.
23. pnpm audit: sharp 0.34.2 has two high advisories. Bump to 0.35.4 or later. S.
24. fillDuration overwrites manual durations. src/durations/fillDuration.ts:24 checks the partial data only, so any partial update, including the transcript hook’s flag refresh, refetches and overwrites. Fix: consult originalDoc. S.
25. Media accepts SVG. src/collections/Media.ts:50 has no mime allowlist; an editor’s SVG with a script runs on the admin origin. Fix: allow jpeg, png, webp. S.

P1 — should fix soon

Backend

• src/utilities/ttlCache.ts keys on raw query strings with no size cap; random parameters can grow it without bound. Key on validated params, cap entries. S.
• src/endpoints/facets.ts:56 caches a returned {error} for five minutes. Throw instead. S.
• pipeline.ts:344: complete and heartbeat never check the job’s worker matches the caller; one leaked CRON_SECRET can write any running job’s transcript. Add and worker = $n. S.
• pipeline.ts:381: transcript create and job update are not one transaction. Wrap them. M.
• pipeline.ts:322: cue text length and stats size are unbounded. Cap. S.
• Public search runs with the Meilisearch master key. Create a search-only key. S.
• Bearer comparisons use ===; use timingSafeEqual. S.
• materials.audioUrl is free text fetched server-side and by workers. Validate https://files.kalelm.com/. S.
• Two auto-transcription paths race: the Gemini hook and the pipeline queue both target the same materials; Gemini also hardcodes Arabic. Delete the Gemini path or gate it. S.
• The durations panel button spawns the WordPress migration entrypoint, which needs nvm, tsx, mysql2 and the WP database; dead on production. Move it to a Payload job. M.
• Transcripts.cues has no validation on admin edit; reuse the pipeline’s validCues. S.
• transcripts.language is nullable; make it required. S.
• groupByMaterial discards sort=newest|oldest; longest is a no-op. S.
• Tree paging has no tiebreak; pages overlap on equal dates. S.
• Facet counts are cues, not materials, so counts read 3 to 40 times too high; single-word queries surface one-word cues with no context; paging stops at about 200 materials while claiming thousands. Facet with distinct, run addContext for keyword, page by material. M.
• Sheikhs marked hidden are readable through the API and listed by facets and tree. S.
• search-logs written per anonymous query; sample or throttle. S.
• lib/pipelineWorker.ts:91 passes the entire server environment to the spawned worker. Allowlist. S.
• Stale drizzle snapshot; orphan transcription, transcription_models tables. Regenerate, drop. M.

Frontend

• Locale dropped on pagination and several link builders; page 2 of an English list lands in Arabic. Route through localePath. S.
• degraded search responses never shown to the reader. Add the field and a notice. S.
• No Open Graph, Twitter or JSON-LD tags; shared links have no preview. M.
• hreflang advertises article pages that 404 in en and ur. S.
• Homepage labels claim “added today”, “most played this week” over all-time data. Reword or filter. S.
• Homepage series preview shows unrelated lessons. S.
• sheikhs(500) and four facet queries on nearly every request; the section cache is single-locale. Memoize per locale. S.
• Filter panel renders 708 sheikhs and every series hidden in HTML: listing pages are 266 KB. Cap and fetch on demand. M.
• Sitemap lists /search/ and stub pages and nears the 50k URL limit in one file. Split by collection. M.
• Download button on the cross-origin mp3 opens inline. Proxy or serve with attachment headers. M.
• Transcript sync only for the <audio> element, never for YouTube, which is 12,866 of the transcripts; no ?t= deep link. M.
• No <main> landmark on most pages, no skip link, heading levels jump. S.
• Dead href="#" in header and footer: series, quizzes, live, about, feedback, socials. S.
• No CSP or frame headers from the app. M.

Engineering

• scripts/verify.ts connects to Postgres at the top, so even the service-free sections cannot run in CI. Move the client into the sections that need it, then run tsc, build, verify exports i18n and worker/test_worker.py in CI. M.
• importMap.js is stale and not regenerated on build. Add payload generate:importmap to the build script. S.
• No startup env validation; a deploy without MEILI_HOST passes health and 500s on search. S.
• deploy-hangar.sh: token visible in ps, missing healthy treated as success, hand-maintained prune list; build from git archive instead. S.
• Backups: no failure alert, no restore rehearsal. Add a ping on success and a monthly pg_restore --list. S.
• No uptime probe beyond Hangar’s health; no error tracking; log location and retention undocumented. S.
• README.md is the Payload template. Replace with ten lines. S.
• Frontend has no lint, no astro check, no tests; install @astrojs/check. S.
• Node pins differ between .nvmrc (22.17) and the host (22.23). S.

Admin

• Rich text editor lacks headings, lists, blockquote and images; editors cannot write a structured article. S.
• Pipeline transcripts go straight into public search with no review state. Add a status and a filter preset. M.
• No link from a material to its transcript or job; re-queue is admin-only. Add join fields and editor access on job status. S.
• English leaking into the Arabic panel in a dozen labels and in the redirects and nested-docs plugins. M.
• listSearchableFields on materials is title only; add YouTube id, slug, WP id. S.
• schedulePublish is enabled on articles and pages but no job runner exists; scheduled publishes never fire. S.
• Media: alt not required; seven template image sizes. S.

P2 — done 2026-09-10

• Done: secure session cookie in production (sameSite: Lax); Dockerfile, .dockerignore and the workspace file were already gone, the --ignore-workspace note is; .env.example lists every variable the code reads (MEILI_SEARCH_KEY added, Gemini removed); worker treats a 409 on complete as accepted and reports one line, not a traceback; the passage count is memoised only once it is a number; fonts are self-hosted (25 woff2 files, Arabic and Latin subsets, body face preloaded); sheikh URLs come from one encoder; the GraphQL routes are deleted along with the disabled feature; the gate’s docs section checks that every pnpm script and repo path the docs name exists.
• Left as is: the IPs and sslip hostnames on the index, access and backups pages. They sit inside commands a reader copies, and a single hosts table would make those commands incomplete; the index page’s hosts table is the reference.

Unbuilt features

• Audio and video player with transcript-hit playback, cue highlighting and ?t= deep links, including for YouTube embeds.
• Quizzes: two “coming soon” stubs.
• Sheikhs directory page and series index page.
• Live stream page; about and feedback pages; social links; RSS feed.
• Transcript review workflow for editors, and a cue editor instead of raw JSON.
• WordPress leftovers not redirected: /page/N/, /?p=, /author/*, /feed/.
• Analytics and consent; Open Graph and JSON-LD; custom 500 page.
• CI pipeline; error tracking; uptime checks.

Feature ideas (2026-09-11, not scheduled)

Grounded in what exists: 12,870 Arabic transcripts, a passage vector index (bge-m3, multilingual), the persistent player, the local library, the offline worker, and the feature-folder system (frontend-features.md). Each fits a feature folder with at most a new extension point. Roughly in order of value for effort within each group.

Browser-only, no account (like offline saving)

• Playback queue. Add recordings to a queue from any list, play through them in the bar, reorder, clear. localStorage plus a “next” handoff on player:ended. Half a day.
• Listening stats. Minutes this week, per sheikh, per series, streak of days; all from the player ticks the library already stores.
• Highlights and notes. Select transcript text, keep it with its timestamp and a note; a “my highlights” page; export as text. Syncable later if accounts come.
• Speed per sheikh. Remember 1.5× for a fast talker; tiny.
• Bookmark a moment. “Save this position” from the bar, separate from the automatic resume; lands on the library page.
• Sleep timer, done properly. Removed 2026-09-11 for its look, not its use: one tap on the bar’s time display cycling 15/30/60.
• Offline for a whole series. “Save all” on a series page through the save queue and strip; total-size warning first.
• Share a clip. Pick start and end on the transcript → a link like ?t=120&end=180 that plays the span, plus a copyable quote. Later, a real audio clip export from the same span.
• Reading mode. Full-width transcript, larger type, no player, audio following. CSS plus one toggle.

Server-backed, still without accounts

• Chapters inside a recording. Segment each transcript into topics with timestamps from the embeddings already computed; a table of contents under the player. The biggest usability gain the data can already give; runs once in the backfill.
• Better titles. Thousands of items are “الدرس 12”. A generated one-line description from the transcript under the generic title, and in search. The dormant LLM pass in the pipeline has this one good use.
• Question answering over the archive. Cited passages with timestamps from the sentence index; the answer synthesis is the new part and needs a model.
• Related series and sheikhs from the same vectors as related lessons (/api/related/:id), on series and sheikh pages.
• Weekly digest, as a page and an RSS feed: what was added, per section; feeds the podcast feature.
• Scoped transcript search from a series or sheikh page; the API filters exist.
• Structured data and sitemap audit for search engines.

Multilingual (45,599 Arabic, 60 Urdu, 8 English materials; every transcript is Arabic)

• Cross-language search, nearly free. bge-m3 is multilingual: an English or Urdu query already lands on the right Arabic passages semantically. Turn it on for the en/ur locales (semantic mode by default there) and show the Arabic passage with its recording; the work is the UI and the copy that says what happened.
• Machine-translated transcripts (ar → en, ur). A second transcript per material, translated cue by cue so timestamps hold: a language toggle in the transcript panel, the same toggle in the text/SRT export (English subtitles for the video), and an English index in Meilisearch so English readers can search in English. Marked as machine translation, with a “report a mistake” that lands in the admin panel. Runs on the same worker fleet in its idle time with a local model, or through a cheap API; ~12,870 × ~9,000 words.
• Bilingual reading view. Arabic line, translation beneath; for learners of Arabic and for checking the translation.
• Translated titles and one-line summaries as localized fields on the material, so the en/ur archive pages have something to browse by; generated with the better-titles pass above.
• Urdu and English transcription. The 68 non-Arabic recordings have no transcript because the pipeline’s ASR is Arabic-only; a per-language ASR step (Whisper for ur/en) on the same claim flow.
• Term glossary. Tap a transliterated term (zakat, sunnah) in a translated transcript for a short definition; generated once, reviewed in the admin.
• Captions to YouTube. The exported SRT, Arabic or translated, uploaded to the channel’s videos as caption tracks.
• Dubbed audio (later). A generated English voice from the translated transcript; heavy, and only after translations exist and are reviewed.

What the verify gate does not check yet

Content of the materials index, not just its count; sentences and passages of unpublished material; cue shape; null-language transcripts; enum versus migration drift; sort honoured; an admin-created material reaching the index; the pure functions (search grouping, cue parsing, claim reclaim) without live services.