doctree/impl_plan.md

DocTree WebBuilder Implementation Plan

Overview

This document outlines the implementation plan for the WebBuilder component of the DocTree project. The WebBuilder is designed to process hjson configuration files (like those in examples/doctreenew/sites/demo1/) and generate a webmeta.json file that can be used by a browser-based website generator.

Current Status

What's Implemented:

1. DocTree Core Functionality:

   • The main DocTree library with functionality for scanning directories, managing collections, processing includes, and converting markdown to HTML
   • Redis storage backend for storing document metadata
   • Command-line interface (doctreecmd) for interacting with collections

2. Example Structure for the New Approach:

   • Example hjson configuration files in examples/doctreenew/sites/demo1/
   • This includes main.hjson, header.hjson, footer.hjson, collection.hjson, and pages/mypages1.hjson

3. Specification Document:

   • Detailed specification in webbuilder/src/builder/specs.md
   • Example output format in webbuilder/src/builder/webmeta.json

What's Not Yet Implemented:

1. WebBuilder Implementation:

   • The actual Rust code for the webbuilder component

2. Hjson Parsing:

   • Code to parse the hjson files in the doctreenew directory

3. Git Repository Integration:

   • Functionality to download referenced collections from Git repositories

4. IPFS Export:

   • Complete functionality to export assets to IPFS

5. Browser-Based Generator:

   • The browser-based website generator that would use the webmeta.json file

Implementation Plan

Phase 1: Core WebBuilder Implementation (2-3 weeks)

1. Setup Project Structure:

   • Create necessary modules and files in webbuilder/src/
   • Define main data structures and traits

2. Implement Hjson Parsing:

   • Add hjson crate dependency
   • Create parsers for each hjson file type (main, header, footer, collection, pages)
   • Implement validation for hjson files

3. Implement Site Structure Builder:

   • Create a module to combine parsed hjson data into a cohesive site structure
   • Implement navigation generation based on page definitions

4. Implement WebMeta Generator:

   • Create functionality to generate the webmeta.json file
   • Ensure all required metadata is included

Phase 2: Git Integration and Collection Processing (2 weeks)

1. Implement Git Repository Integration:

   • Add git2 crate dependency
   • Create functionality to clone/pull repositories based on collection.hjson
   • Implement caching to avoid unnecessary downloads

2. Integrate with DocTree Library:

   • Create an adapter to use DocTree functionality with hjson-defined collections
   • Implement processing of includes between documents

3. Implement Content Processing:

   • Create functionality to process markdown content
   • Handle special directives or custom syntax

Phase 3: IPFS Integration (2 weeks)

1. Enhance IPFS Integration:

   • Complete the IPFS export functionality in DocTree
   • Create a module to handle IPFS uploads

2. Implement Asset Management:

   • Create functionality to track and process assets (images, CSS, etc.)
   • Ensure proper IPFS linking

3. Implement Content Hashing:

   • Add Blake hash calculation for content integrity verification
   • Store hashes in webmeta.json

Phase 4: CLI and Testing (1-2 weeks)

1. Implement Command-Line Interface:

   • Create a CLI for the webbuilder
   • Add commands for building, validating, and deploying sites

2. Write Comprehensive Tests:

   • Unit tests for each component
   • Integration tests for the full workflow
   • Test with example sites

3. Documentation:

   • Update README with usage instructions
   • Create detailed API documentation
   • Add examples and tutorials

Phase 5: Browser-Based Generator (Optional, 3-4 weeks)

1. Design Browser Component:

   • Create a JavaScript/TypeScript library to consume webmeta.json
   • Design component architecture

2. Implement Content Rendering:

   • Create components to render markdown content
   • Implement navigation and site structure

3. Implement IPFS Integration:

   • Add functionality to fetch content from IPFS
   • Implement content verification using Blake hashes

4. Create Demo Site:

   • Build a demo site using the browser-based generator
   • Showcase features and capabilities

Technical Details

Key Dependencies

• hjson: For parsing hjson configuration files
• git2: For Git repository integration
• ipfs-api: For IPFS integration
• blake3: For content hashing
• clap: For command-line interface
• tokio: For async operations

Data Flow

1. Parse hjson files from input directory
2. Download referenced Git repositories
3. Process content with DocTree
4. Export assets to IPFS
5. Generate webmeta.json
6. (Optional) Upload webmeta.json to IPFS

Key Challenges

1. Git Integration: Handling authentication, rate limits, and large repositories
2. IPFS Performance: Optimizing IPFS uploads for large sites
3. Content Processing: Ensuring proper handling of includes and special syntax
4. Browser Compatibility: Ensuring the browser-based generator works across different browsers

Milestones and Timeline

1. Core WebBuilder Implementation: Weeks 1-3
2. Git Integration and Collection Processing: Weeks 4-5
3. IPFS Integration: Weeks 6-7
4. CLI and Testing: Weeks 8-9
5. Browser-Based Generator (Optional): Weeks 10-13

Resources Required

1. Development Resources:

   • 1-2 Rust developers
   • 1 Frontend developer (for browser-based generator)

2. Infrastructure:

   • IPFS node for testing
   • Git repositories for testing
   • CI/CD pipeline

Conclusion

This implementation plan provides a roadmap for developing the WebBuilder component of the DocTree project. By following this plan, we can transform the current specification and example files into a fully functional system for generating websites from hjson configuration files and markdown content.