Unify books_dir into the existing ~/hero/var/books/ library tree #46

New issue

Closed

opened 2026-02-13 05:57:31 +00:00 by mik-tf · 0 comments

mik-tf commented

2026-02-13 05:57:31 +00:00

Owner

Problem

The system currently maintains two separate locations for book data:

books_dir (CLI --books-dir, defaults to .) — flat TOML files with __ namespace prefix (e.g. geomind__nitrograph.toml)
~/hero/var/books/ — persistent library tree where exports go (rendered HTML, PDFs, book.json, collections, .ai/ metadata)

This split means:

Book source definitions live in a different place than their output
The __ prefix convention is non-obvious
You cannot browse one directory to understand what is deployed
books_dir defaults to . which is error-prone

Solution: Complete the Existing Library Tree

The ~/hero/var/books/{library}/ structure (implemented in library.rs) already organizes exports, collections, and metadata per library. The fix is to move book TOMLs into this same tree, eliminating the separate books_dir.

Final unified structure

~/hero/var/books/
  geomind/
    library.toml                    # library config
    doctree.toml                    # doctree state
    books/
      nitrograph/
        book.toml                   # source definition (currently in flat books_dir)
        book.json                   # exported metadata
        content/                    # rendered markdown pages
        img/                        # images
        files/                      # attachments
      mycelium_overview/
        book.toml
        ...
    collections/
      docs_mycelium/
        .collection                 # collection marker
        .ai/                        # Q&A metadata, embeddings
        getting_started.md
        img/
  ourworld/
    library.toml
    books/
      collaboration/
        book.toml
        ...
    collections/
      ...

Why this structure

Books and collections are siblings under each library — collections can be shared across books
library.toml and doctree.toml sit at the library root — metadata about the library, not mixed with individual books
book.toml lives alongside its output (book.json, content/, img/) — source and output together per book
No __ prefix convention — namespace is the parent directory
One tree to browse — ls ~/hero/var/books/ shows everything deployed
Builds on existing code — library.rs already implements this directory layout, we just add book.toml placement

Implementation

Replace --books-dir CLI flag with --libraries-dir (default ~/hero/var/books/), env var HERO_BOOKS_LIBRARIES_DIR
Update BooksScanner to scan {library}/books/{book}/book.toml recursively instead of flat *.toml
Update export_books_for_serving() to write output alongside book.toml (already mostly does this)
Update import pipeline to write TOMLs into {library}/books/{book_name}/book.toml
Auto-create directory structure on startup
make demo / make dev copy example TOMLs from examples/ into the library tree
Update docs/manual/ to reflect the unified structure
Migration: auto-detect old flat books_dir layouts and log a warning with migration instructions

Notes

examples/ebooks_local and examples/ebooks_git remain as dev-time seed data, copied into the library tree on startup
library.rs helper functions (library_book_dir(), ensure_library_dirs(), etc.) already support this layout
The export_dir concept in ServerConfig becomes redundant — remove it and always use libraries_base_dir()
Git source repos stay in ~/hero/code/{server}/{account}/{repo}/ — never mixed with ~/hero/var/books/

## Problem The system currently maintains two separate locations for book data: 1. **`books_dir`** (CLI `--books-dir`, defaults to `.`) — flat TOML files with `__` namespace prefix (e.g. `geomind__nitrograph.toml`) 2. **`~/hero/var/books/`** — persistent library tree where exports go (rendered HTML, PDFs, `book.json`, collections, `.ai/` metadata) This split means: - Book source definitions live in a different place than their output - The `__` prefix convention is non-obvious - You cannot browse one directory to understand what is deployed - `books_dir` defaults to `.` which is error-prone ## Solution: Complete the Existing Library Tree The `~/hero/var/books/{library}/` structure (implemented in `library.rs`) already organizes exports, collections, and metadata per library. The fix is to **move book TOMLs into this same tree**, eliminating the separate `books_dir`. ### Final unified structure ``` ~/hero/var/books/ geomind/ library.toml # library config doctree.toml # doctree state books/ nitrograph/ book.toml # source definition (currently in flat books_dir) book.json # exported metadata content/ # rendered markdown pages img/ # images files/ # attachments mycelium_overview/ book.toml ... collections/ docs_mycelium/ .collection # collection marker .ai/ # Q&A metadata, embeddings getting_started.md img/ ourworld/ library.toml books/ collaboration/ book.toml ... collections/ ... ``` ### Why this structure - **Books and collections are siblings** under each library — collections can be shared across books - **`library.toml` and `doctree.toml`** sit at the library root — metadata about the library, not mixed with individual books - **`book.toml` lives alongside its output** (`book.json`, `content/`, `img/`) — source and output together per book - **No `__` prefix convention** — namespace is the parent directory - **One tree to browse** — `ls ~/hero/var/books/` shows everything deployed - **Builds on existing code** — `library.rs` already implements this directory layout, we just add `book.toml` placement ### Implementation 1. Replace `--books-dir` CLI flag with `--libraries-dir` (default `~/hero/var/books/`), env var `HERO_BOOKS_LIBRARIES_DIR` 2. Update `BooksScanner` to scan `{library}/books/{book}/book.toml` recursively instead of flat `*.toml` 3. Update `export_books_for_serving()` to write output alongside `book.toml` (already mostly does this) 4. Update import pipeline to write TOMLs into `{library}/books/{book_name}/book.toml` 5. Auto-create directory structure on startup 6. `make demo` / `make dev` copy example TOMLs from `examples/` into the library tree 7. Update `docs/manual/` to reflect the unified structure 8. Migration: auto-detect old flat `books_dir` layouts and log a warning with migration instructions ### Notes - `examples/ebooks_local` and `examples/ebooks_git` remain as dev-time seed data, copied into the library tree on startup - `library.rs` helper functions (`library_book_dir()`, `ensure_library_dirs()`, etc.) already support this layout - The `export_dir` concept in `ServerConfig` becomes redundant — remove it and always use `libraries_base_dir()` - Git source repos stay in `~/hero/code/{server}/{account}/{repo}/` — never mixed with `~/hero/var/books/`

mik-tf referenced this issue from a commit

2026-02-13 06:26:36 +00:00

feat: issues #40, #46, #47 — PDF viewer, books_dir default, import UX

mik-tf referenced this issue from a pull request that will close it,

2026-02-13 06:26:45 +00:00

WIP: feat: issues #40, #46, #47 — PDF viewer, books_dir default, import UX #48

mik-tf changed title from ~~Default books_dir to ~/hero/var/hero_books/books/ per hero ecosystem conventions~~ to Unify books_dir and export_dir into a single libraries/ directory tree

2026-02-13 13:46:42 +00:00

mik-tf changed title from ~~Unify books_dir and export_dir into a single libraries/ directory tree~~ to Unify books_dir into the existing ~/hero/var/books/ library tree

2026-02-13 14:14:17 +00:00

mik-tf referenced this issue

2026-02-13 14:45:47 +00:00

No way to remove a library or book from the system #52

mik-tf referenced this issue from a commit

2026-02-13 16:43:59 +00:00

feat: unify books_dir into library tree (#46)

mik-tf referenced this issue from a pull request that will close it,

2026-02-13 16:44:15 +00:00

feat: Unify books_dir into library tree (#46) #55