From 5ac8e330ca1f7a74aee0d49ea1f2a2bd069e110d Mon Sep 17 00:00:00 2001 From: Hananoshika Yomaru Date: Fri, 22 May 2026 11:42:00 -0700 Subject: [PATCH] Update build scripts and enhance development workflow: Revise package.json to streamline build commands, replacing deprecated scripts with new chapter-specific build commands. Update CONTRIBUTING.md to clarify local development instructions and improve chapter sync processes. Enhance README.md with updated serving and build commands for better user guidance. Add new types for Bun in devDependencies and update the feed and sitemap to reflect recent content changes. --- .agents/skills/ofb/SKILL.md | 18 ++- .agents/skills/zola/SKILL.md | 21 ++- CONTRIBUTING.md | 29 ++-- README.md | 30 ++-- bun.lock | 9 ++ data/github-sponsors.json | 2 +- frontend/src/main.tsx | 10 +- package.json | 23 +-- scripts/build-chapter-widgets.ts | 137 +++++++++++++++++ scripts/build-chapter.ts | 186 ++++++++++++++++++++++++ scripts/build-site.ts | 25 ++++ scripts/dev.sh | 21 --- scripts/sync-chapters.ts | 35 +++-- scripts/validate-curriculum.test.ts | 15 ++ scripts/validate-curriculum.ts | 34 +++-- spec.md | 12 +- static/feed.xml | 16 +- static/sitemap.xml | 2 +- themes/openfreebooks/sass/_catalog.scss | 1 + 19 files changed, 501 insertions(+), 125 deletions(-) create mode 100644 scripts/build-chapter-widgets.ts create mode 100644 scripts/build-chapter.ts create mode 100644 scripts/build-site.ts delete mode 100644 scripts/dev.sh create mode 100644 scripts/validate-curriculum.test.ts diff --git a/.agents/skills/ofb/SKILL.md b/.agents/skills/ofb/SKILL.md index e5eb874..c77aae9 100644 --- a/.agents/skills/ofb/SKILL.md +++ b/.agents/skills/ofb/SKILL.md @@ -32,11 +32,13 @@ frontend/ → catalog-app, Mermaid map view, site chrome ```bash bun install -bun run build:js # after frontend/ changes -bun run generate:site-metadata # regenerate static/sitemap.xml + static/feed.xml from content + git log -bun run index:search # once per dev session (copies public/pagefind → static/pagefind) -zola serve # http://127.0.0.1:1111 -bun run build # production: JS + Zola + pagefind → public/ +bun run dev # instant local serve (zola only) +bun run build:chapter math/measures-dispersion math/loci +bun run build:chapter math/* +bun run build:chapter # auto-detect changed chapters (interactive confirm) +bun run build:site # full rebuild (sync external data + all chapters + site) +bun run build # alias of build:site +bun test ``` ## Non-negotiables @@ -65,7 +67,7 @@ bun run build # production: JS + Zola + pagefind → public/ | Sitemap/feed coverage | `scripts/generate-site-metadata.ts`, `static/sitemap.xml`, `static/feed.xml`, `package.json` build script | | Homepage subject card | `themes/.../templates/index.html` (not yet data-driven) | | Browse / nav URL | `zola.toml` `[extra] browse_url` + `base.html` site-config + `main.tsx` fallback | -| Search UI / index | `base.html` (Pagefind assets + modal), `content/search.md`, `_search.scss`; `bun run index:search` for dev | +| Search UI / index | `base.html` (Pagefind assets + modal), `content/search.md`, `_search.scss`; chapter/site builds refresh Pagefind | ## Sitemap / feed auto-generation @@ -73,7 +75,7 @@ bun run build # production: JS + Zola + pagefind → public/ - `static/sitemap.xml` (all user-facing built pages + live chapters) - `static/feed.xml` (RSS 2.0 for pages + live chapters) - URL timestamps come from `git log -1` on content-source paths. If a page/chapter has no commit yet, generator falls back to filesystem mtime. -- Build pipeline runs this automatically before `zola build` via `package.json`. +- Build pipeline runs this automatically before `zola build`. - Include only resolvable pages; planned chapters without live content are excluded. ## Search (Pagefind) @@ -87,7 +89,7 @@ bun run build # production: JS + Zola + pagefind → public/ | Theming | `themes/.../sass/_search.scss` (`--pf-*` → site tokens) | - **Component UI** only (`pagefind-component-ui.js`), not legacy `pagefind-ui.js`. -- Index runs **after** `zola build`. Production: `bun run build`. Dev: `bun run index:search` copies bundle to `static/pagefind/` (gitignored) for `zola serve`. +- Index runs **after** `zola build` in build commands (`build:chapter`/`build:site`). - v1 indexes **live chapter pages only** (`data-pagefind-body` on `templates/chapter.html`); home, about, catalog, credits, and other site pages are excluded. Detailed checklists: [curriculum-data.md](curriculum-data.md). Human-facing guide: [CONTRIBUTING.md](../../../CONTRIBUTING.md). diff --git a/.agents/skills/zola/SKILL.md b/.agents/skills/zola/SKILL.md index 479d7ce..d1bd716 100644 --- a/.agents/skills/zola/SKILL.md +++ b/.agents/skills/zola/SKILL.md @@ -28,23 +28,22 @@ See [reference.md](reference.md) for Tera pitfalls and build checks. ## Commands ```bash -bun run build:js # sync chapters + Vite build (JS/CSS bundle outputs) -bun run build # smart local build (incremental; uses build cache) -bun run build:full # full deterministic rebuild (use in CI / clean validation) -bun run dev # smart local dev startup (incremental chapter sync + watch + zola serve) -bun run dev:full # force full chapter sync before starting dev server -zola build # required before indexing -bun run index:search # pagefind on public/ → copies to static/pagefind/ for zola serve -zola serve # dev at http://127.0.0.1:1111 +bun run dev # instant serve (zola only) +bun run build:chapter math/loci math/measures-dispersion +bun run build:chapter math/* # wildcard chapter builds +bun run build:chapter # auto-detect changed chapters (interactive confirm) +bun run build:site # full site rebuild (includes sponsor sync when token exists) +bun run build # alias of build:site +bun test ``` After Sass or template changes, run `zola build` (or `zola serve`). Do **not** copy `public/` into `themes/openfreebooks/static/`. ### Build mode behavior -- **First clone / cold start:** `bun run build` behaves like a full pipeline because caches are absent. -- **Continuous local development:** `bun run build` reuses caches and skips unchanged steps where possible. -- **CI:** prefer `bun run build:full` for deterministic, correctness-first builds. +- **Chapter workflow:** prefer `build:chapter` for normal content work; it rebuilds chapter assets and only runs Vite when frontend/site-shell code changes. +- **Site workflow:** use `build:site`/`build` for full rebuilds and release verification. +- **CI:** use `bun run build:site` and `bun test`. **Search (Pagefind):** read [zola-pagefind.md](zola-pagefind.md) before changing search UI, indexing, or `data-pagefind-*` attributes. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c219962..90afe29 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -19,11 +19,11 @@ Thank you for helping grow free, open textbooks. | Add a **catalog subject banner** | [§ Catalog subject banner](#catalog-subject-banner) | | Add a **curriculum** filter (e.g. GCSE) | [§ New curriculum label](#add-a-curriculum-label) | -**Browse the catalog locally:** `bun run build:js && zola serve` → [http://127.0.0.1:1111/catalog/](http://127.0.0.1:1111/catalog/) +**Browse the catalog locally:** `bun run dev` → [http://127.0.0.1:1111/catalog/](http://127.0.0.1:1111/catalog/) -**Search locally:** After `bun run build:js`, run `bun run index:search` once (or again after changing chapter HTML), then `zola serve`. Try [http://127.0.0.1:1111/search/](http://127.0.0.1:1111/search/) or press ⌘K / Ctrl+K. +**Search locally:** Run `bun run build:chapter ...` or `bun run build:site` after content changes to refresh the index, then use `bun run dev`. Try [http://127.0.0.1:1111/search/](http://127.0.0.1:1111/search/) or press ⌘K / Ctrl+K. -**Build modes:** `bun run build` is smart/incremental for local iteration; use `bun run build:full` for a deterministic full rebuild (CI/release checks). +**Build modes:** use `bun run build:chapter` for targeted chapter builds and `bun run build:site` for full rebuilds. --- @@ -37,7 +37,7 @@ content/{subject}/{slug}/ → everything for the live page (one folder) - **Catalog** (`/catalog/`) — list, map (`?view=tree`), and for Mathematics a Compare doc (`?view=compare`, HTML partial `themes/openfreebooks/templates/partials/math-curricula-compare.html`). Curriculum names (DSE, IB, …) appear only in the catalog — not inside chapter prose ([spec.md](spec.md)). - Every chapter needs a **`description`** in curriculum JSON (map clamps to three lines). -- **`bun run sync:chapters`** (also run automatically by `bun run build:js`) merges `core.html` + `supplement.html` into `_index.md` for Zola. Edit the HTML files, not the merged body in `_index.md`. +- Chapter sync (merging `core.html` + `supplement.html` into `_index.md`) runs automatically in build commands. Edit HTML sources, not merged `_index.md`. ### Map graph rules (summary) @@ -130,15 +130,15 @@ strand = "Number & Algebra" ### 4. Build and check ```bash -bun run build:js # sync:chapters + Vite bundle -zola serve +bun run build:chapter math/your-chapter-slug +bun run dev ``` - `/catalog/?subject=math` — chapter appears and links correctly. - `/math/your-chapter-slug/` — page renders. - `bun run build` — full production build. -New subjects with shared CSS/JS: add `content/{subject}/subject.scss` and/or `subject.ts`; `sync:chapters` wires them automatically. No per-chapter template branches. +New subjects with shared CSS/JS: add `content/{subject}/subject.scss` and/or `subject.ts`; chapter sync during build wires them automatically. No per-chapter template branches. --- @@ -163,7 +163,7 @@ Add edges in `data/{subject}-curriculum.json`: 1. `data/catalog.json` — add the subject to `subjects`. 2. Create `data/{id}-curriculum.json` (same schema as `math-curriculum.json`; `strands` + optional `graph.edges`). 3. Add `content/{subject}/` when you have live chapters. -4. Run `bun run build:js` (this syncs chapter content and auto-registers subject CSS/JS modules). +4. Run `bun run build:chapter {subject}/*` (this syncs chapters, rebuilds relevant widgets, and updates global outputs). ### Subject scaffold (copyable) @@ -210,7 +210,7 @@ cwebp -resize 1400 0 -q 82 your-banner.jpg -o static/catalog/banners/math.webp { "id": "math", "name": "Mathematics", "banner": "/catalog/banners/math.webp" } ``` -4. Check locally: `bun run build:js && zola serve` → `/catalog/?subject=math` +4. Check locally: `bun run build:chapter math/* && bun run dev` → `/catalog/?subject=math` Omit `banner` for subjects that should stay unchanged (e.g. roadmap-only placeholders). @@ -228,14 +228,13 @@ Omit `banner` for subjects that should stay unchanged (e.g. roadmap-only placeho ```bash bun install -bun run sync:chapters # after content/ chapter changes -bun run build:js # sync + Vite -bun run dev # incremental sync + watch + zola serve -bun run dev:full # force full sync before dev -zola serve +bun run dev +bun run build:chapter math/measures-dispersion math/loci +bun run build:chapter math/* +bun run build:chapter # auto-detect changed chapters, list, confirm +bun run build:site bun test bun run build -bun run build:full ``` - Do **not** commit `themes/openfreebooks/static/` (Vite output). diff --git a/README.md b/README.md index 6ac3df5..ee9de77 100644 --- a/README.md +++ b/README.md @@ -55,29 +55,37 @@ Install JS dependencies: bun install ``` -For day-to-day local development, run JS watch + Zola serve together: +Serve locally: ```bash bun run dev ``` -Open [http://127.0.0.1:1111](http://127.0.0.1:1111). Re-run `bun run index:search` when you change chapter HTML or add pages so search stays up to date. +Open [http://127.0.0.1:1111](http://127.0.0.1:1111). -Use `bun run dev:full` when you want to force a full chapter sync before starting dev servers. +Search UI is available in local serve; run a build command to refresh the search index (`static/pagefind/` is gitignored and copied from build output). -Search UI is available immediately; results require the index step above (`static/pagefind/` is gitignored and copied from the build output). - -## Production build +## Build commands ```bash -bun run build +bun run build:chapter math/measures-dispersion math/loci +bun run build:chapter math/* +bun run build:chapter # auto-detect changed chapters, list them, ask for confirmation +bun run build:site # full site build (includes sponsor sync when GITHUB_TOKEN is set) +bun run build # alias of build:site ``` -`bun run build` is smart/incremental for local development: -- first run (or missing cache) behaves like a full build, -- repeat runs skip unchanged steps. +`build:chapter` updates chapter sync, chapter widget JS (targeted), metadata, site output, and search index. -Use `bun run build:full` for a deterministic full rebuild (recommended for CI/release validation). +In non-interactive environments, `build:chapter` requires explicit chapter arguments. + +## Testing + +```bash +bun test +``` + +`bun test` includes curriculum validation. ## Deploy to Cloudflare diff --git a/bun.lock b/bun.lock index 6da812f..58c49ff 100644 --- a/bun.lock +++ b/bun.lock @@ -13,6 +13,7 @@ "three": "^0.184.0", }, "devDependencies": { + "@types/bun": "^1.3.14", "pagefind": "^1.5.0", "sass": "^1.89.2", "typescript": "^5.8.3", @@ -234,6 +235,8 @@ "@types/babel__traverse": ["@types/babel__traverse@7.28.0", "", { "dependencies": { "@babel/types": "^7.28.2" } }, "sha512-8PvcXf70gTDZBgt9ptxJ8elBeBjcLOAcOtoO/mPJjtji1+CdGbHgm77om1GrsPxsiE+uXIpNSK64UYaIwQXd4Q=="], + "@types/bun": ["@types/bun@1.3.14", "", { "dependencies": { "bun-types": "1.3.14" } }, "sha512-h1hFqFVcvAvD9j9K7ZW7vd82aSA+rTdznZa+5bwvCwqSB1jmmfLcbIWhOLx1/+boy/xmjgCs/OMUL8hRJSmnPw=="], + "@types/d3": ["@types/d3@7.4.3", "", { "dependencies": { "@types/d3-array": "*", "@types/d3-axis": "*", "@types/d3-brush": "*", "@types/d3-chord": "*", "@types/d3-color": "*", "@types/d3-contour": "*", "@types/d3-delaunay": "*", "@types/d3-dispatch": "*", "@types/d3-drag": "*", "@types/d3-dsv": "*", "@types/d3-ease": "*", "@types/d3-fetch": "*", "@types/d3-force": "*", "@types/d3-format": "*", "@types/d3-geo": "*", "@types/d3-hierarchy": "*", "@types/d3-interpolate": "*", "@types/d3-path": "*", "@types/d3-polygon": "*", "@types/d3-quadtree": "*", "@types/d3-random": "*", "@types/d3-scale": "*", "@types/d3-scale-chromatic": "*", "@types/d3-selection": "*", "@types/d3-shape": "*", "@types/d3-time": "*", "@types/d3-time-format": "*", "@types/d3-timer": "*", "@types/d3-transition": "*", "@types/d3-zoom": "*" } }, "sha512-lZXZ9ckh5R8uiFVt8ogUNf+pIrK4EsWrx2Np75WvF/eTpJ0FMHNhjXk8CKEx/+gpHbNQyJWehbFaTvqmHWB3ww=="], "@types/d3-array": ["@types/d3-array@3.2.2", "", {}, "sha512-hOLWVbm7uRza0BYXpIIW5pxfrKe0W+D5lrFiAEYR+pb6w3N2SwSMaJbXdUfSEv+dT4MfHBLtn5js0LAWaO6otw=="], @@ -300,6 +303,8 @@ "@types/geojson": ["@types/geojson@7946.0.16", "", {}, "sha512-6C8nqWur3j98U6+lXDfTUWIfgvZU+EumvpHKcYjujKH7woYyLj2sUmff0tRhrqM7BohUw7Pz3ZB1jj2gW9Fvmg=="], + "@types/node": ["@types/node@25.9.1", "", { "dependencies": { "undici-types": ">=7.24.0 <7.24.7" } }, "sha512-xfrlY7UD5rMJk3ZVJP8BNzS28J36YJg+xp+LPXV1TdWxr8uMH5A860QNxYDGQe/ylDSgjxE52Q9VnO7p75tJxg=="], + "@types/trusted-types": ["@types/trusted-types@2.0.7", "", {}, "sha512-ScaPdn1dQczgbl0QFTeTOmVHFULt394XJgOQNoyVhZ6r2vLnMLJfBPd53SB52T/3G36VI1/g2MZaX0cwDuXsfw=="], "@upsetjs/venn.js": ["@upsetjs/venn.js@2.0.0", "", { "optionalDependencies": { "d3-selection": "^3.0.0", "d3-transition": "^3.0.1" } }, "sha512-WbBhLrooyePuQ1VZxrJjtLvTc4NVfpOyKx0sKqioq9bX1C1m7Jgykkn8gLrtwumBioXIqam8DLxp88Adbue6Hw=="], @@ -316,6 +321,8 @@ "browserslist": ["browserslist@4.28.2", "", { "dependencies": { "baseline-browser-mapping": "^2.10.12", "caniuse-lite": "^1.0.30001782", "electron-to-chromium": "^1.5.328", "node-releases": "^2.0.36", "update-browserslist-db": "^1.2.3" }, "bin": { "browserslist": "cli.js" } }, "sha512-48xSriZYYg+8qXna9kwqjIVzuQxi+KYWp2+5nCYnYKPTr0LvD89Jqk2Or5ogxz0NUMfIjhh2lIUX/LyX9B4oIg=="], + "bun-types": ["bun-types@1.3.14", "", { "dependencies": { "@types/node": "*" } }, "sha512-4N0ig0fEomHt5R0KCFWjovxow98rIoRwKolrYdCcknNwMekCXRnWEUvgu5soYV8QXtVsrUD8B95MBOZGPvr6KQ=="], + "caniuse-lite": ["caniuse-lite@1.0.30001793", "", {}, "sha512-iwSsYWaCOoh26cV8NwNRViHlrfUvYsHDfRVcbtmw0Kg6PJIZZXwMkj1442FYLBGkeUf1juAsU3DTfxW579mrPA=="], "chokidar": ["chokidar@4.0.3", "", { "dependencies": { "readdirp": "^4.0.1" } }, "sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA=="], @@ -564,6 +571,8 @@ "uhyphen": ["uhyphen@0.2.0", "", {}, "sha512-qz3o9CHXmJJPGBdqzab7qAYuW8kQGKNEuoHFYrBwV6hWIMcpAmxDLXojcHfFr9US1Pe6zUswEIJIbLI610fuqA=="], + "undici-types": ["undici-types@7.24.6", "", {}, "sha512-WRNW+sJgj5OBN4/0JpHFqtqzhpbnV0GuB+OozA9gCL7a993SmU+1JBZCzLNxYsbMfIeDL+lTsphD5jN5N+n0zg=="], + "update-browserslist-db": ["update-browserslist-db@1.2.3", "", { "dependencies": { "escalade": "^3.2.0", "picocolors": "^1.1.1" }, "peerDependencies": { "browserslist": ">= 4.21.0" }, "bin": { "update-browserslist-db": "cli.js" } }, "sha512-Js0m9cx+qOgDxo0eMiFGEueWztz+d4+M3rGlmKPT+T4IS/jP4ylw3Nwpu6cpTTP8R1MAC1kF4VbdLt3ARf209w=="], "uuid": ["uuid@14.0.0", "", { "bin": { "uuid": "dist-node/bin/uuid" } }, "sha512-Qo+uWgilfSmAhXCMav1uYFynlQO7fMFiMVZsQqZRMIXp0O7rR7qjkj+cPvBHLgBqi960QCoo/PH2/6ZtVqKvrg=="], diff --git a/data/github-sponsors.json b/data/github-sponsors.json index bdbe449..441e3cf 100644 --- a/data/github-sponsors.json +++ b/data/github-sponsors.json @@ -2,7 +2,7 @@ "title": "GitHub Sponsors", "intro": "Supporters who help keep Open Free Books free and open for learners everywhere.", "profile_url": "https://github.com/sponsors/HananoshikaYomaru", - "fetched_at": "2026-05-21T18:06:17.197Z", + "fetched_at": "2026-05-22T18:41:21.577Z", "sponsors": [ { "login": "cybersader", diff --git a/frontend/src/main.tsx b/frontend/src/main.tsx index 6798395..c057ad4 100644 --- a/frontend/src/main.tsx +++ b/frontend/src/main.tsx @@ -82,20 +82,20 @@ function mountChapterWidgets() { if (mounts.length === 0) return; - void import("./generated/chapter-widgets").then(({ chapterWidgetLoaders }) => { + void import("./generated/chapter-widgets").then(({ chapterWidgetUrls }) => { const loadPromises: Promise[] = []; for (const el of mounts) { const widget = el.dataset.widget; if (!widget) continue; const loaderKey = `${chapterKey}/${widget}`; - const loader = chapterWidgetLoaders[loaderKey]; - if (!loader) { - console.warn(`No widget loader registered for ${loaderKey}`); + const moduleUrl = chapterWidgetUrls[loaderKey]; + if (!moduleUrl) { + console.warn(`No widget module URL registered for ${loaderKey}`); continue; } loadPromises.push( - loader().then((mod) => { + import(/* @vite-ignore */ moduleUrl).then((mod) => { render(() => mod.default({}), el); }) ); diff --git a/package.json b/package.json index 5ea30af..057f61a 100644 --- a/package.json +++ b/package.json @@ -4,21 +4,11 @@ "private": true, "type": "module", "scripts": { - "sync:chapters": "bun scripts/sync-chapters.ts", - "sync:sponsors": "bun scripts/fetch-github-sponsors.ts", - "generate:site-metadata": "bun scripts/generate-site-metadata.ts", - "build:js": "bun run sync:chapters && vite build", - "build:site": "bun run generate:site-metadata && zola build && bun run index:search", - "build:full": "bun run check:static-html && bun run validate:curriculum && rm -f themes/openfreebooks/static/main.css public/main.css && bun run sync:chapters -- --full && vite build && bun run build:site", - "build": "bun scripts/build-smart.ts", - "index:search": "pagefind --site public --glob \"**/index.html\" && rm -rf static/pagefind && cp -R public/pagefind static/pagefind", - "dev:js": "vite build --watch", - "dev:full": "bash scripts/dev.sh --full", - "dev": "bash scripts/dev.sh", - "check:static-html": "bash scripts/check-static-html.sh", - "scan:size": "bun scripts/scan-output-size.ts", - "test": "bun test", - "validate:curriculum": "bun scripts/validate-curriculum.ts" + "build:chapter": "bun scripts/build-chapter.ts", + "build:site": "bun scripts/build-site.ts", + "build": "bun run build:site", + "dev": "zola serve", + "test": "bun test" }, "dependencies": { "defuddle": "^0.18.1", @@ -33,6 +23,7 @@ "pagefind": "^1.5.0", "typescript": "^5.8.3", "vite": "^6.3.5", - "vite-plugin-solid": "^2.11.6" + "vite-plugin-solid": "^2.11.6", + "@types/bun": "^1.3.14" } } diff --git a/scripts/build-chapter-widgets.ts b/scripts/build-chapter-widgets.ts new file mode 100644 index 0000000..faf71e3 --- /dev/null +++ b/scripts/build-chapter-widgets.ts @@ -0,0 +1,137 @@ +import { existsSync } from "node:fs"; +import { mkdir, readdir, readFile, rm, writeFile } from "node:fs/promises"; +import { basename, dirname, join } from "node:path"; +import { build } from "vite"; +import solid from "vite-plugin-solid"; +import { fingerprintPaths } from "./build-change-detector"; + +const ROOT = join(import.meta.dir, ".."); +const CONTENT = join(ROOT, "content"); +const OUTPUT_ROOT = join(ROOT, "themes/openfreebooks/static/js/chapter-widgets"); +const CACHE_PATH = join(ROOT, "data/_generated/chapter-widgets-cache.json"); + +type Chapter = { subject: string; slug: string; dir: string; key: string }; +type CacheFile = { version: 1; chapters: Record }; + +async function discoverChapters(): Promise { + const output: Chapter[] = []; + const subjects = await readdir(CONTENT, { withFileTypes: true }); + for (const subjectEntry of subjects) { + if (!subjectEntry.isDirectory() || subjectEntry.name.startsWith(".")) continue; + const subject = subjectEntry.name; + const subjectDir = join(CONTENT, subject); + const chapterEntries = await readdir(subjectDir, { withFileTypes: true }); + for (const chapterEntry of chapterEntries) { + if (!chapterEntry.isDirectory() || chapterEntry.name.startsWith(".")) continue; + const slug = chapterEntry.name; + const dir = join(subjectDir, slug); + const indexPath = join(dir, "_index.md"); + if (!existsSync(indexPath)) continue; + const indexRaw = await readFile(indexPath, "utf8"); + if (!/template\s*=\s*["']chapter\.html["']/.test(indexRaw)) continue; + output.push({ subject, slug, dir, key: `${subject}/${slug}` }); + } + } + return output; +} + +async function listWidgets(chapter: Chapter): Promise { + const widgetsDir = join(chapter.dir, "widgets"); + if (!existsSync(widgetsDir)) return []; + const entries = await readdir(widgetsDir, { withFileTypes: true }); + return entries + .filter((entry) => entry.isFile() && entry.name.endsWith(".tsx") && !entry.name.startsWith("_")) + .map((entry) => basename(entry.name, ".tsx")) + .sort(); +} + +async function chapterFingerprint(chapter: Chapter): Promise { + return fingerprintPaths(ROOT, [join(chapter.dir, "widgets")]); +} + +async function readCache(): Promise { + if (!existsSync(CACHE_PATH)) return { version: 1, chapters: {} }; + try { + const parsed = JSON.parse(await readFile(CACHE_PATH, "utf8")) as CacheFile; + if (parsed.version !== 1) return { version: 1, chapters: {} }; + return parsed; + } catch { + return { version: 1, chapters: {} }; + } +} + +async function buildWidget(chapter: Chapter, widget: string) { + const entry = join(chapter.dir, "widgets", `${widget}.tsx`); + const outDir = join(OUTPUT_ROOT, chapter.subject, chapter.slug); + await mkdir(outDir, { recursive: true }); + await build({ + configFile: false, + plugins: [solid()], + publicDir: false, + resolve: { + alias: { + "@ofb/katex": join(ROOT, "frontend/src/lib/katex.ts"), + }, + }, + build: { + outDir, + emptyOutDir: false, + lib: { + entry, + formats: ["es"], + fileName: () => `${widget}.js`, + }, + rollupOptions: { + treeshake: false, + output: { + chunkFileNames: "assets/[name]-[hash].js", + assetFileNames: "assets/[name]-[hash][extname]", + }, + }, + }, + }); +} + +function parseTargets(args: string[]): { full: boolean; chapterKeys: string[] } { + const full = args.includes("--full"); + const chapterKeys = args.filter((arg) => !arg.startsWith("--")); + return { full, chapterKeys }; +} + +async function main() { + const { full, chapterKeys } = parseTargets(process.argv.slice(2)); + const chapters = await discoverChapters(); + const chapterByKey = new Map(chapters.map((chapter) => [chapter.key, chapter])); + const targets = + chapterKeys.length === 0 + ? chapters + : chapterKeys + .map((key) => chapterByKey.get(key)) + .filter((chapter): chapter is Chapter => Boolean(chapter)); + + const prev = await readCache(); + const next: CacheFile = { version: 1, chapters: { ...prev.chapters } }; + + for (const chapter of targets) { + const fingerprint = await chapterFingerprint(chapter); + const unchanged = !full && prev.chapters[chapter.key] === fingerprint; + if (unchanged) { + console.log(`Unchanged widgets ${chapter.key} (skipped)`); + continue; + } + + const widgets = await listWidgets(chapter); + const chapterOutDir = join(OUTPUT_ROOT, chapter.subject, chapter.slug); + await rm(chapterOutDir, { recursive: true, force: true }); + for (const widget of widgets) { + await buildWidget(chapter, widget); + } + next.chapters[chapter.key] = fingerprint; + console.log(`Built widgets ${chapter.key} (${widgets.length})`); + } + + await mkdir(dirname(CACHE_PATH), { recursive: true }); + await writeFile(CACHE_PATH, `${JSON.stringify(next, null, 2)}\n`); +} + +main(); diff --git a/scripts/build-chapter.ts b/scripts/build-chapter.ts new file mode 100644 index 0000000..fe2a368 --- /dev/null +++ b/scripts/build-chapter.ts @@ -0,0 +1,186 @@ +import { spawn } from "node:child_process"; +import { existsSync } from "node:fs"; +import { mkdir, readdir, readFile, writeFile } from "node:fs/promises"; +import { dirname, join } from "node:path"; +import { createInterface } from "node:readline/promises"; +import { stdin as input, stdout as output } from "node:process"; +import { $ } from "bun"; +import { fingerprintPaths } from "./build-change-detector"; + +const ROOT = join(import.meta.dir, ".."); +const CONTENT = join(ROOT, "content"); +const BUNDLE_PATH = join(ROOT, "themes/openfreebooks/static/js/bundle.js"); +const BUILD_CACHE_PATH = join(ROOT, "data/_generated/build-chapter-cache.json"); + +type Chapter = { key: string; subject: string; slug: string }; +type BuildCache = { version: 1; shellFingerprint: string }; + +async function runStep(command: string) { + const result = await $`bash -lc ${command}`; + if (result.exitCode !== 0) process.exit(result.exitCode || 1); +} + +async function discoverChapters(): Promise { + const output: Chapter[] = []; + const subjects = await readdir(CONTENT, { withFileTypes: true }); + for (const subjectEntry of subjects) { + if (!subjectEntry.isDirectory() || subjectEntry.name.startsWith(".")) continue; + const subject = subjectEntry.name; + const subjectDir = join(CONTENT, subject); + const chapterEntries = await readdir(subjectDir, { withFileTypes: true }); + for (const chapterEntry of chapterEntries) { + if (!chapterEntry.isDirectory() || chapterEntry.name.startsWith(".")) continue; + const slug = chapterEntry.name; + const indexPath = join(subjectDir, slug, "_index.md"); + if (!existsSync(indexPath)) continue; + const raw = await readFile(indexPath, "utf8"); + if (!/template\s*=\s*["']chapter\.html["']/.test(raw)) continue; + output.push({ key: `${subject}/${slug}`, subject, slug }); + } + } + return output.sort((a, b) => a.key.localeCompare(b.key)); +} + +function patternToRegex(pattern: string): RegExp { + const escaped = pattern.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*"); + return new RegExp(`^${escaped}$`); +} + +function resolvePatterns(chapters: Chapter[], patterns: string[]): Chapter[] { + const results = new Map(); + for (const pattern of patterns) { + const regex = patternToRegex(pattern); + const matches = chapters.filter((chapter) => regex.test(chapter.key)); + for (const chapter of matches) { + results.set(chapter.key, chapter); + } + } + return [...results.values()].sort((a, b) => a.key.localeCompare(b.key)); +} + +async function changedChapterKeys(chapterKeys: Set): Promise { + const trackedDiff = await $`git --no-pager diff --name-only HEAD -- content`.text(); + const untracked = await $`git --no-pager ls-files --others --exclude-standard -- content`.text(); + const files = new Set( + [...trackedDiff.split("\n"), ...untracked.split("\n")] + .map((line) => line.trim()) + .filter((line) => line.length > 0) + ); + + const keys = new Set(); + for (const file of files) { + const match = file.match(/^content\/([^/]+)\/([^/]+)\//); + if (!match) continue; + const key = `${match[1]}/${match[2]}`; + if (chapterKeys.has(key)) keys.add(key); + } + return [...keys].sort(); +} + +async function confirmDetectedChanges(chapters: Chapter[], frontendChanged: boolean): Promise { + if (chapters.length > 0) { + console.log("[build:chapter] Auto-detected changed chapters:"); + for (const chapter of chapters) { + console.log(` - ${chapter.key}`); + } + } + if (frontendChanged) { + console.log("[build:chapter] Auto-detected frontend/site-shell changes."); + } + const rl = createInterface({ input, output }); + const answer = (await rl.question("Proceed? [y/N] ")).trim().toLowerCase(); + rl.close(); + return answer === "y" || answer === "yes"; +} + +async function readBuildCache(): Promise { + if (!existsSync(BUILD_CACHE_PATH)) return null; + try { + const parsed = JSON.parse(await readFile(BUILD_CACHE_PATH, "utf8")) as BuildCache; + if (parsed.version !== 1) return null; + return parsed; + } catch { + return null; + } +} + +async function writeBuildCache(cache: BuildCache) { + await mkdir(dirname(BUILD_CACHE_PATH), { recursive: true }); + await writeFile(BUILD_CACHE_PATH, `${JSON.stringify(cache, null, 2)}\n`); +} + +async function shellFingerprint(): Promise { + return fingerprintPaths(ROOT, [join(ROOT, "frontend"), join(ROOT, "vite.config.ts")]); +} + +function runPagefindInBackground() { + const command = + 'pagefind --site public --glob "**/index.html" && rm -rf static/pagefind && cp -R public/pagefind static/pagefind'; + const child = spawn("bash", ["-lc", command], { + cwd: ROOT, + detached: true, + stdio: "ignore", + }); + child.unref(); + console.log("[build:chapter] Pagefind indexing started in background."); +} + +async function main() { + const patterns = process.argv.slice(2).filter((arg) => !arg.startsWith("--")); + const chapters = await discoverChapters(); + const chapterByKey = new Map(chapters.map((chapter) => [chapter.key, chapter])); + const previousBuild = await readBuildCache(); + const currentShellFingerprint = await shellFingerprint(); + const frontendChanged = + !existsSync(BUNDLE_PATH) || previousBuild?.shellFingerprint !== currentShellFingerprint; + + let targets: Chapter[] = []; + + if (patterns.length > 0) { + targets = resolvePatterns(chapters, patterns); + if (targets.length === 0) { + console.error(`[build:chapter] No chapters matched patterns: ${patterns.join(", ")}`); + process.exit(1); + } + } else { + if (!input.isTTY || !output.isTTY) { + console.error("[build:chapter] Non-interactive mode requires explicit chapter arguments."); + process.exit(1); + } + const changedKeys = await changedChapterKeys(new Set(chapters.map((chapter) => chapter.key))); + targets = changedKeys.map((key) => chapterByKey.get(key)).filter((c): c is Chapter => Boolean(c)); + + if (targets.length === 0 && !frontendChanged) { + console.log("[build:chapter] No changed chapters or frontend changes detected."); + return; + } + + const confirmed = await confirmDetectedChanges(targets, frontendChanged); + if (!confirmed) { + console.log("[build:chapter] Cancelled."); + return; + } + } + + if (targets.length > 0) { + await runStep("bun scripts/sync-chapters.ts"); + await runStep(`bun scripts/build-chapter-widgets.ts ${targets.map((t) => t.key).join(" ")}`); + } + + const needsShellBuild = !existsSync(BUNDLE_PATH) || frontendChanged; + if (needsShellBuild) { + await runStep("rm -f themes/openfreebooks/static/main.css public/main.css"); + await runStep("vite build"); + } + + await runStep("bun scripts/generate-site-metadata.ts"); + await runStep("rm -rf public"); + await runStep("zola build"); + runPagefindInBackground(); + + await writeBuildCache({ version: 1, shellFingerprint: currentShellFingerprint }); + const summaryTargets = targets.length > 0 ? targets.map((target) => target.key).join(", ") : "(frontend only)"; + console.log(`[build:chapter] Core build done: ${summaryTargets}`); +} + +void main(); diff --git a/scripts/build-site.ts b/scripts/build-site.ts new file mode 100644 index 0000000..7714b37 --- /dev/null +++ b/scripts/build-site.ts @@ -0,0 +1,25 @@ +import { $ } from "bun"; + +async function runStep(command: string) { + const result = await $`bash -lc ${command}`; + if (result.exitCode !== 0) process.exit(result.exitCode || 1); +} + +async function main() { + await runStep("bash scripts/check-static-html.sh"); + await runStep("rm -f themes/openfreebooks/static/main.css public/main.css"); + if (process.env.GITHUB_TOKEN?.trim()) { + await runStep("bun scripts/fetch-github-sponsors.ts"); + } else { + console.log("[build:site] GITHUB_TOKEN not set; skipping sponsor sync."); + } + await runStep("bun scripts/sync-chapters.ts --full"); + await runStep("bun scripts/build-chapter-widgets.ts --full"); + await runStep("vite build"); + await runStep("bun scripts/generate-site-metadata.ts"); + await runStep("rm -rf public"); + await runStep("zola build"); + await runStep('pagefind --site public --glob "**/index.html" && rm -rf static/pagefind && cp -R public/pagefind static/pagefind'); +} + +void main(); diff --git a/scripts/dev.sh b/scripts/dev.sh deleted file mode 100644 index cb2e091..0000000 --- a/scripts/dev.sh +++ /dev/null @@ -1,21 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -cleanup() { - if [[ -n "${JS_PID:-}" ]] && kill -0 "${JS_PID}" 2>/dev/null; then - kill "${JS_PID}" - fi -} - -trap cleanup EXIT INT TERM - -if [[ "${1:-}" == "--full" ]]; then - bun run sync:chapters -- --full -else - bun run sync:chapters -fi - -bun run dev:js & -JS_PID=$! - -zola serve diff --git a/scripts/sync-chapters.ts b/scripts/sync-chapters.ts index d0c27be..c509a56 100644 --- a/scripts/sync-chapters.ts +++ b/scripts/sync-chapters.ts @@ -123,6 +123,13 @@ async function copyDir(srcDir: string, destDir: string) { } } +async function writeIfChanged(path: string, content: string) { + const current = existsSync(path) ? await readFile(path, "utf8") : null; + if (current === content) return; + await mkdir(dirname(path), { recursive: true }); + await writeFile(path, content); +} + async function compileChapterScss( subject: string, slug: string, @@ -361,18 +368,14 @@ function buildWidgetRegistry( const lines: string[] = [ "// Auto-generated by scripts/sync-chapters.ts — do not edit.", "", - "import type { Component } from \"solid-js\";", - "", - "export type ChapterWidgetModule = { default: Component };", - "", - "export const chapterWidgetLoaders: Record Promise> = {", + "export const chapterWidgetUrls: Record = {", ]; - for (const { key, widgets, dir } of entries) { - const relFromGenerated = relative(dirname(GENERATED_WIDGETS), dir).replace(/\\/g, "/"); + for (const { key, widgets } of entries) { + const [subject, slug] = key.split("/"); for (const widget of widgets) { - const importPath = `${relFromGenerated}/widgets/${widget}`; - lines.push(` "${key}/${widget}": () => import("${importPath}"),`); + const importPath = `/js/chapter-widgets/${subject}/${slug}/${widget}.js`; + lines.push(` "${key}/${widget}": "${importPath}",`); } } @@ -411,14 +414,11 @@ async function main() { widgetEntries.push({ key, widgets, dir: chapter.dir }); } - await mkdir(dirname(GENERATED_META), { recursive: true }); - await writeFile(GENERATED_META, `${JSON.stringify(metaFile, null, 2)}\n`); - - await mkdir(dirname(GENERATED_WIDGETS), { recursive: true }); - await writeFile(GENERATED_WIDGETS, buildWidgetRegistry(widgetEntries)); + await writeIfChanged(GENERATED_META, `${JSON.stringify(metaFile, null, 2)}\n`); + await writeIfChanged(GENERATED_WIDGETS, buildWidgetRegistry(widgetEntries)); const chapterNav = await buildChapterNav(chapters); - await writeFile(GENERATED_NAV, `${JSON.stringify(chapterNav, null, 2)}\n`); + await writeIfChanged(GENERATED_NAV, `${JSON.stringify(chapterNav, null, 2)}\n`); const subjectEntries: Array<{ id: string; dir: string; hasJs: boolean }> = []; const subjectsMeta: SubjectsMetaFile = { subjects: {} }; @@ -446,9 +446,8 @@ async function main() { subjectsMeta.subjects[subject.id] = meta; subjectEntries.push({ id: subject.id, dir: subject.dir, hasJs: meta.hasJs }); } - await writeFile(GENERATED_SUBJECTS_META, `${JSON.stringify(subjectsMeta, null, 2)}\n`); - await mkdir(dirname(GENERATED_SUBJECT_MODULES), { recursive: true }); - await writeFile(GENERATED_SUBJECT_MODULES, buildSubjectModuleRegistry(subjectEntries)); + await writeIfChanged(GENERATED_SUBJECTS_META, `${JSON.stringify(subjectsMeta, null, 2)}\n`); + await writeIfChanged(GENERATED_SUBJECT_MODULES, buildSubjectModuleRegistry(subjectEntries)); await mkdir(dirname(SYNC_CACHE_PATH), { recursive: true }); await writeFile(SYNC_CACHE_PATH, `${JSON.stringify(nextCache, null, 2)}\n`); diff --git a/scripts/validate-curriculum.test.ts b/scripts/validate-curriculum.test.ts new file mode 100644 index 0000000..4c4a22d --- /dev/null +++ b/scripts/validate-curriculum.test.ts @@ -0,0 +1,15 @@ +import { describe, expect, it } from "bun:test"; +import { validateCurriculum } from "./validate-curriculum"; + +describe("validate-curriculum", () => { + it("has no curriculum validation errors", async () => { + const result = await validateCurriculum(); + if (result.errors.length > 0) { + const details = result.errors + .map((error) => `[${error.file}] ${error.message}`) + .join("\n"); + throw new Error(`Curriculum validation failed:\n${details}`); + } + expect(result.errors).toHaveLength(0); + }); +}); diff --git a/scripts/validate-curriculum.ts b/scripts/validate-curriculum.ts index 89436de..4d4f795 100644 --- a/scripts/validate-curriculum.ts +++ b/scripts/validate-curriculum.ts @@ -97,14 +97,24 @@ async function readJsonFile(path: string): Promise { } } -async function main() { +export async function validateCurriculum(): Promise<{ + curriculumFileCount: number; + errors: ValidationError[]; +}> { const errors: ValidationError[] = []; const catalogPath = join(DATA_DIR, "catalog.json"); const catalog = await readJsonFile(catalogPath); if (!catalog || !Array.isArray(catalog.curriculums)) { - console.error("Cannot validate curriculums: data/catalog.json must contain a curriculums array."); - process.exit(1); + return { + curriculumFileCount: 0, + errors: [ + { + file: "catalog.json", + message: "Cannot validate curriculums: data/catalog.json must contain a curriculums array.", + }, + ], + }; } const globalCurriculums = new Set( @@ -263,15 +273,21 @@ async function main() { } } - if (errors.length > 0) { - console.error(`\n✖ Curriculum validation failed with ${errors.length} error(s):`); - for (const error of errors) { + return { curriculumFileCount: curriculumFiles.length, errors }; +} + +async function main() { + const result = await validateCurriculum(); + if (result.errors.length > 0) { + console.error(`\n✖ Curriculum validation failed with ${result.errors.length} error(s):`); + for (const error of result.errors) { console.error(`- [${error.file}] ${error.message}`); } process.exit(1); } - - console.log(`✓ Curriculum validation passed for ${curriculumFiles.length} curriculum file(s).`); + console.log(`✓ Curriculum validation passed for ${result.curriculumFileCount} curriculum file(s).`); } -main(); +if (import.meta.main) { + void main(); +} diff --git a/spec.md b/spec.md index 24984bf..2de0844 100644 --- a/spec.md +++ b/spec.md @@ -35,4 +35,14 @@ each chapter needs to have the following other than the core content: 5. questions bank - some exams questions from easy, intermediate, difficult, hardcore 6. reference (if any) -the order can be adjustable base on importance. \ No newline at end of file +the order can be adjustable base on importance. + +## Non-functional requirements + +### Dev experience and performance + +1. `bun run dev` must start fast and directly run `zola serve` only (no implicit sync/build/watch pipeline). +2. Chapter build workflow must prioritize fast feedback for content edits: + - avoid full frontend bundle rebuild unless frontend/site-shell code actually changed, + - keep chapter core rebuild path as lightweight as possible. +3. `build:chapter` is the default iterative workflow for chapter updates; it should minimize unnecessary I/O and avoid rebuilding unrelated assets. \ No newline at end of file diff --git a/static/feed.xml b/static/feed.xml index 3a6db22..3818856 100644 --- a/static/feed.xml +++ b/static/feed.xml @@ -4,7 +4,14 @@ Open Free Books https://openfreebooks.org/ Recent updates to Open Free Books pages and live chapters. - Fri, 22 May 2026 06:55:33 GMT + Fri, 22 May 2026 07:32:28 GMT + + Written calculation + https://openfreebooks.org/math/written-calculation/ + https://openfreebooks.org/math/written-calculation/ + Fri, 22 May 2026 07:32:28 GMT + Add, subtract, multiply, and divide whole numbers using vertical written form: align columns, regroup, and read the result. + Pythagoras' theorem https://openfreebooks.org/math/pythagoras-theorem/ @@ -250,13 +257,6 @@ Thu, 21 May 2026 06:54:57 GMT Solve, represent, and apply linear inequalities in one unknown using number lines and interval notation. - - Written calculation - https://openfreebooks.org/math/written-calculation/ - https://openfreebooks.org/math/written-calculation/ - Thu, 21 May 2026 05:05:38 GMT - Add, subtract, multiply, and divide whole numbers using vertical written form: align columns, regroup, and read the result. - FAQ https://openfreebooks.org/faq/ diff --git a/static/sitemap.xml b/static/sitemap.xml index 77e9c50..26907f0 100644 --- a/static/sitemap.xml +++ b/static/sitemap.xml @@ -174,7 +174,7 @@ https://openfreebooks.org/math/written-calculation/ - 2026-05-21T05:05:38.000Z + 2026-05-22T07:32:28.000Z https://openfreebooks.org/math/zero-and-one/ diff --git a/themes/openfreebooks/sass/_catalog.scss b/themes/openfreebooks/sass/_catalog.scss index 47d40db..464cc25 100644 --- a/themes/openfreebooks/sass/_catalog.scss +++ b/themes/openfreebooks/sass/_catalog.scss @@ -741,6 +741,7 @@ display: -webkit-box; -webkit-line-clamp: 3; -webkit-box-orient: vertical; + line-clamp: 3; overflow: hidden; }