feat: support .mdx files in page resolver and DocTree scanner #45

Merged

mik-tf merged 6 commits from development_mdx_features into development 2026-02-13 06:03:34 +00:00

Conversation 0 Commits 6 Files changed 14 +160 -55

mik-tf commented 2026-02-12 11:48:33 +00:00

Owner

Copy link

Summary

Adds support for .mdx files throughout the pipeline — page discovery, path resolution, export, search indexing, and web serving. .mdx content is preserved as-is (no JSX stripping or transformation).

Changes (11 files, +108/-44)

File	Change
src/doctree/utils.rs	Add MARKDOWN_EXTENSIONS, is_markdown_extension(), strip_markdown_extension() helpers + tests
src/doctree/collection.rs	Page scanner: ext == "md" → is_markdown_extension(&ext). Preserve source extension on export.
src/book/resolver.rs	find_page_file(): try .mdx after .md for all candidate patterns
src/doctree/client.rs	get_page_path() / list_pages(): support .mdx lookup and filtering
src/pdf/mod.rs	PDF page loading: add .mdx fallback path
src/book/exporter.rs	Preserve source file extension (.md or .mdx) through export pipeline
src/doctree/link.rs	Link classification: treat .mdx as page (not file)
src/web/server.rs	HTML link rewriting, page lookup, MIME type: handle .mdx
src/webservice/server/rpc.rs	URL-to-PDF title extraction: strip .mdx
src/vectorsdk/indexer.rs	Use page.relative_path instead of hardcoded .md for metadata lookup
src/doctree/mod.rs	Re-export markdown extension helpers

Design

• Centralized MARKDOWN_EXTENSIONS constant — adding future formats (e.g. .markdown) requires one change
• .mdx content preserved as-is through the entire pipeline
• .md is always tried first (backward compatible)
• Q&A extraction and search indexing work on .mdx text content
• JSX components pass through as raw HTML in the current web viewer

Future: full MDX rendering

Full JSX/React component rendering in .mdx files requires a WASM/React runtime, tracked in issue #44. This PR is a prerequisite — it ensures .mdx files are discovered and their content preserved faithfully for when that frontend is built.

Testing

• 136 unit tests pass (2 new: test_is_markdown_extension, test_strip_markdown_extension)
• Clippy clean
• Live test: znzcybercity main_intro.mdx and presentation.mdx warnings resolved (pages now found by resolver)

Closes #43

## Summary Adds support for `.mdx` files throughout the pipeline — page discovery, path resolution, export, search indexing, and web serving. `.mdx` content is preserved as-is (no JSX stripping or transformation). ## Changes (11 files, +108/-44) | File | Change | |---|---| | `src/doctree/utils.rs` | Add `MARKDOWN_EXTENSIONS`, `is_markdown_extension()`, `strip_markdown_extension()` helpers + tests | | `src/doctree/collection.rs` | Page scanner: `ext == "md"` → `is_markdown_extension(&ext)`. Preserve source extension on export. | | `src/book/resolver.rs` | `find_page_file()`: try `.mdx` after `.md` for all candidate patterns | | `src/doctree/client.rs` | `get_page_path()` / `list_pages()`: support `.mdx` lookup and filtering | | `src/pdf/mod.rs` | PDF page loading: add `.mdx` fallback path | | `src/book/exporter.rs` | Preserve source file extension (`.md` or `.mdx`) through export pipeline | | `src/doctree/link.rs` | Link classification: treat `.mdx` as page (not file) | | `src/web/server.rs` | HTML link rewriting, page lookup, MIME type: handle `.mdx` | | `src/webservice/server/rpc.rs` | URL-to-PDF title extraction: strip `.mdx` | | `src/vectorsdk/indexer.rs` | Use `page.relative_path` instead of hardcoded `.md` for metadata lookup | | `src/doctree/mod.rs` | Re-export markdown extension helpers | ## Design - Centralized `MARKDOWN_EXTENSIONS` constant — adding future formats (e.g. `.markdown`) requires one change - `.mdx` content preserved as-is through the entire pipeline - `.md` is always tried first (backward compatible) - Q&A extraction and search indexing work on `.mdx` text content - JSX components pass through as raw HTML in the current web viewer ## Future: full MDX rendering Full JSX/React component rendering in `.mdx` files requires a WASM/React runtime, tracked in **issue #44**. This PR is a prerequisite — it ensures `.mdx` files are discovered and their content preserved faithfully for when that frontend is built. ## Testing - 136 unit tests pass (2 new: `test_is_markdown_extension`, `test_strip_markdown_extension`) - Clippy clean - Live test: znzcybercity `main_intro.mdx` and `presentation.mdx` warnings resolved (pages now found by resolver) Closes #43

mik-tf added 1 commit 2026-02-12 11:48:33 +00:00

feat: support .mdx files in page resolver and DocTree scanner ... 402f1b6c7f

- Add is_markdown_extension() and strip_markdown_extension() helpers
  to doctree/utils.rs with MARKDOWN_EXTENSIONS constant
- Update collection scanner to discover .mdx pages (collection.rs)
- Update find_page_file() to try .mdx after .md (resolver.rs)
- Update DocTreeClient get_page_path/list_pages for .mdx (client.rs)
- Update PDF page loading to try .mdx fallback (pdf/mod.rs)
- Preserve source file extension through export pipeline (exporter.rs)
- Update link classification, HTML link rewriting, MIME types (server.rs)
- Use page.relative_path in indexer instead of hardcoded .md
- Add unit tests for new helpers

.mdx content is preserved as-is (no JSX stripping). Full MDX rendering
(JSX components) will be added when WASM frontend is enabled (see #44).

Closes #43

mik-tf changed title from feat: support .mdx files in page resolver and DocTree scanner to WIP: feat: support .mdx files in page resolver and DocTree scanner 2026-02-12 14:57:16 +00:00

mik-tf added 1 commit 2026-02-13 03:57:18 +00:00

Merge branch 'development' into development_mdx_features 607498f3ce

mik-tf added 1 commit 2026-02-13 04:16:05 +00:00

fix: create parent dirs in export_page for nested page paths ... 493d2e0cbf

Pages with path separators in their name (e.g. 'main/main_intro') would
fail with 'No such file or directory' because the content subdirectory
didn't exist. This fixes the cybercity book export which uses nested
page paths like 'main/main_intro', 'main/location/fumba', etc.

mik-tf added 1 commit 2026-02-13 05:21:58 +00:00

feat: add --skip-discover flag and make serve target ... 4c597b6efc

- Add skip_discover field to ServerConfig
- Add --skip-discover CLI flag to skip library git clone/pull on startup
- Add 'make serve' Makefile target (build + serve without library discovery)
- Fix frontmatter stripping in strip_mdx_content() for YAML frontmatter
- Fix page lookup for nested paths (match full path before stem fallback)
- Fix prev/next navigation for nested page paths

mik-tf added 1 commit 2026-02-13 05:46:33 +00:00

fix: sanitize nested page paths in PDF cache filenames ... dc283fa1cc

Pages with nested paths like main/planning/planning_intro caused
'No such file or directory' errors when generating per-page PDFs
because the slash created subdirectory references in the filename.

Replace / with _ in page names used for PDF cache paths and hash
file paths.

mik-tf force-pushed development_mdx_features from dc283fa1cc to 948b7c4f30 2026-02-13 05:48:46 +00:00 Compare

mik-tf changed title from WIP: feat: support .mdx files in page resolver and DocTree scanner to feat: support .mdx files in page resolver and DocTree scanner 2026-02-13 05:49:10 +00:00

mik-tf added 1 commit 2026-02-13 05:52:39 +00:00

fix: update deps.txt crate names to snake_case per naming convention 6cc345b079

mik-tf merged commit 7aabb9d0eb into development 2026-02-13 06:03:34 +00:00

mik-tf deleted branch development_mdx_features 2026-02-13 06:03:35 +00:00

mik-tf referenced this pull request from a commit 2026-02-13 06:03:36 +00:00

Merge pull request 'feat: support .mdx files in page resolver and DocTree scanner' (#45) from development_mdx_features into development

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

lhumina_code/hero_books!45

No description provided.