This commit is contained in:
2025-12-02 03:27:17 +01:00
parent 690af291b5
commit 29ab30788e
25 changed files with 510 additions and 142 deletions

View File

@@ -1,53 +0,0 @@
> NOTE: DocTree is the default document collections/export pipeline used by the current Docusaurus integration (see `lib/data/doctree/client`). The Doctree module described here is an alternative/legacy export mechanism that still exists but is not the primary path. Use DocTree by default unless you explicitly need Doctree.
# Doctree Export Specification
## Overview
The `doctree` module in `lib/data/doctree` is responsible for processing and exporting documentation trees. This involves taking a structured representation of documentation (collections, pages, images, files) and writing it to a specified file system destination. Additionally, it leverages Redis to store metadata about the exported documentation, facilitating quick lookups and integration with other systems.
## Key Components
### `lib/data/doctree/export.v`
This file defines the main `export` function for the `Tree` object. It orchestrates the overall export process:
- Takes `TreeExportArgs` which includes parameters like `destination`, `reset` (to clear destination), `keep_structure`, `exclude_errors`, `toreplace` (for regex replacements), `concurrent` (for parallel processing), and `redis` (to control Redis metadata storage).
- Processes definitions, includes, actions, and macros within the `Tree`.
- Generates file paths for pages, images, and other files.
- Iterates through `Collection` objects within the `Tree` and calls their respective `export` methods, passing down the `redis` flag.
### `lib/data/doctree/collection/export.v`
This file defines the `export` function for the `Collection` object. This is where the actual file system writing and Redis interaction for individual collections occur:
- Takes `CollectionExportArgs` which includes `destination`, `file_paths`, `reset`, `keep_structure`, `exclude_errors`, `replacer`, and the `redis` flag.
- Creates a `.collection` file in the destination directory with basic collection information.
- **Redis Integration**:
- Obtains a Redis client using `base.context().redis()`.
- Stores the collection's destination path in Redis using `redis.hset('doctree:path', 'collection_name', 'destination_path')`.
- Calls `export_pages`, `export_files`, `export_images`, and `export_linked_pages` which all interact with Redis if the `redis` flag is true.
- **`export_pages`**:
- Processes page links and handles not-found errors.
- Writes markdown content to the destination file system.
- Stores page metadata in Redis: `redis.hset('doctree:collection_name', 'page_name', 'page_file_name.md')`.
- **`export_files` and `export_images`**:
- Copies files and images to the destination directory (e.g., `img/`).
- Stores file/image metadata in Redis: `redis.hset('doctree:collection_name', 'file_name', 'img/file_name.ext')`.
- **`export_linked_pages`**:
- Gathers linked pages within the collection.
- Writes a `.linkedpages` file.
- Stores linked pages file metadata in Redis: `redis.hset('doctree:collection_name', 'linkedpages', 'linkedpages_file_name.md')`.
## Link between Redis and Export
The `doctree` export process uses Redis as a metadata store. When the `redis` flag is set to `true` (which is the default), the export functions populate Redis with key-value pairs that map collection names, page names, file names, and image names to their respective paths and file names within the exported documentation structure.
This Redis integration serves as a quick lookup mechanism for other applications or services that might need to access or reference the exported documentation. Instead of traversing the file system, these services can query Redis to get the location of specific documentation elements.
## Is Export Needed?
Yes, the export functionality is crucial for making the processed `doctree` content available outside the internal `doctree` representation.
- **File System Export**: The core purpose of the export is to write the documentation content (markdown files, images, other assets) to a specified directory. This is essential for serving the documentation via a web server, integrating with static site generators (like Docusaurus, as suggested by other files in the project), or simply providing a browsable version of the documentation.
- **Redis Metadata**: While the file system export is fundamental, the Redis metadata storage is an important complementary feature. It provides an efficient way for other systems to programmatically discover and locate documentation assets. If there are downstream applications that rely on this Redis metadata for navigation, search, or content delivery, then the Redis part of the export is indeed needed. If no such applications exist or are planned, the `redis` flag can be set to `false` to skip this step, but the file system export itself remains necessary for external consumption of the documentation.

View File

@@ -2,6 +2,7 @@ module db
import x.json2
import json
import incubaid.herolib.data.ourtime
import strconv
pub fn decode_int(data string) !int {

View File

@@ -498,7 +498,6 @@ pub fn calendar_event_handle(mut f ModelsFactory, rpcid int, servercontext map[s
}
else {
println('Method not found on calendar_event: ${method}')
$dbg;
return new_error(rpcid,
code: 32601
message: 'Method ${method} not found on calendar_event'

View File

@@ -0,0 +1,219 @@
module ledger
import incubaid.herolib.hero.db
import incubaid.herolib.data.encoder
import incubaid.herolib.data.ourtime
import incubaid.herolib.schemas.jsonrpc { Response, new_error, new_response, new_response_false, new_response_int, new_response_true }
import json
pub struct DBAccount {
pub mut:
db &db.DB @[skip; str: skip]
}
pub fn (self Account) type_name() string {
return 'Account'
}
@[params]
pub struct AccountPolicyArg {
pub mut:
policy_id u32
admins []u32
min_signatures u8
limits []AccountLimit
whitelist_out []u32
whitelist_in []u32
lock_till u32
admin_lock_type LockType
admin_lock_till u32
admin_unlock []u32
admin_unlock_min_signature u8
clawback_accounts []u32
clawback_min_signatures u8
clawback_from u32
clawback_till u32
}
pub struct AccountArg {
pub mut:
name string
description string
owner_id u32
location_id u32
accountpolicies []AccountPolicyArg
assets []AccountAsset
assetid u32
last_activity u32 // timestamp
administrators []u32
}
pub fn (mut self DBAccount) new(args AccountArg) !Account {
mut accountpolicies := []AccountPolicy{}
for policy_arg in args.accountpolicies {
accountpolicies << AccountPolicy{
policy_id: policy_arg.policy_id
admins: policy_arg.admins
min_signatures: policy_arg.min_signatures
limits: policy_arg.limits
whitelist_out: policy_arg.whitelist_out
whitelist_in: policy_arg.whitelist_in
lock_till: policy_arg.lock_till
admin_lock_type: policy_arg.admin_lock_type
admin_lock_till: policy_arg.admin_lock_till
admin_unlock: policy_arg.admin_unlock
admin_unlock_min_signature: policy_arg.admin_unlock_min_signature
clawback_accounts: policy_arg.clawback_accounts
clawback_min_signatures: policy_arg.clawback_min_signatures
clawback_from: policy_arg.clawback_from
clawback_till: policy_arg.clawback_till
}
}
mut o := Account{
owner_id: args.owner_id
location_id: args.location_id
accountpolicies: accountpolicies
assets: args.assets
assetid: args.assetid
last_activity: args.last_activity
administrators: args.administrators
}
o.name = args.name
o.description = args.description
o.updated_at = u32(ourtime.now().unix())
return o
}
pub fn (mut self DBAccount) set(o Account) !Account {
return self.db.set[Account](o)!
}
pub fn (mut self DBAccount) delete(id u32) !bool {
if !self.db.exists[Account](id)! {
return false
}
self.db.delete[Account](id)!
return true
}
pub fn (mut self DBAccount) exist(id u32) !bool {
return self.db.exists[Account](id)!
}
pub fn (mut self DBAccount) get(id u32) !Account {
mut o, data := self.db.get_data[Account](id)!
mut e_decoder := encoder.decoder_new(data)
self.load(mut o, mut e_decoder)!
return o
}
@[params]
pub struct AccountListArg {
pub mut:
filter string
status int = -1
limit int = 20
offset int = 0
}
pub fn (mut self DBAccount) list(args AccountListArg) ![]Account {
mut all_accounts := self.db.list[Account]()!.map(self.get(it)!)
mut filtered_accounts := []Account{}
for account in all_accounts {
// Add filter logic based on Account properties
if args.filter != '' && !account.name.contains(args.filter)
&& !account.description.contains(args.filter) {
continue
}
// We could add more filters based on status if the Account struct has a status field
filtered_accounts << account
}
// Apply pagination
mut start := args.offset
if start >= filtered_accounts.len {
start = 0
}
mut limit := args.limit
if limit > 100 {
limit = 100
}
if start + limit > filtered_accounts.len {
limit = filtered_accounts.len - start
}
if limit <= 0 {
return []Account{}
}
return if filtered_accounts.len > 0 {
filtered_accounts[start..start + limit]
} else {
[]Account{}
}
}
pub fn (mut self DBAccount) list_all() ![]Account {
return self.db.list[Account]()!.map(self.get(it)!)
}
pub struct UserRef {
pub mut:
id u32
}
pub fn account_handle(mut f ModelsFactory, rpcid int, servercontext map[string]string, userref UserRef, method string, params string) !Response {
match method {
'get' {
id := db.decode_u32(params)!
res := f.account.get(id)!
return new_response(rpcid, json.encode_pretty(res))
}
'set' {
mut args := db.decode_generic[AccountArg](params)!
mut o := f.account.new(args)!
if args.id != 0 {
o.id = args.id
}
o = f.account.set(o)!
return new_response_int(rpcid, int(o.id))
}
'delete' {
id := db.decode_u32(params)!
success := f.account.delete(id)!
if success {
return new_response_true(rpcid)
} else {
return new_response_false(rpcid)
}
}
'exist' {
id := db.decode_u32(params)!
if f.account.exist(id)! {
return new_response_true(rpcid)
} else {
return new_response_false(rpcid)
}
}
'list' {
args := db.decode_generic_or_default[AccountListArg](params, AccountListArg{})!
result := f.account.list(args)!
return new_response(rpcid, json.encode_pretty(result))
}
else {
return new_error(
rpcid: rpcid
code: 32601
message: 'Method ${method} not found on Account'
)
}
}
}

View File

@@ -0,0 +1,72 @@
module ledger
import incubaid.herolib.hero.db
// Account represents an account in the financial system
@[heap]
pub struct Account {
db.Base
pub mut:
owner_id u32 // link to user, o is not defined
location_id u32 // link to location, 0 is none
accountpolicies []AccountPolicy
assets []AccountAsset
assetid u32
last_activity u32
administrators []u32
status AccountStatus
}
// AccountStatus represents the status of an account
pub enum AccountStatus {
active
inactive
suspended
archived
}
// AccountPolicy represents a set of rules for an account
pub struct AccountPolicy {
pub mut:
policy_id u32 @[index]
admins []u32 // people who can transfer money out
min_signatures u8 // nr of people who need to sign
limits []AccountLimit
whitelist_out []u32 // where money can go to
whitelist_in []u32 // where money can come from
lock_till u32 // date in epoch till no money can be transfered, only after
admin_lock_type LockType
admin_lock_till u32 // date in epoch when admin can unlock (0 means its free), this is unlock for changing this policy
admin_unlock []u32 // users who can unlock the admin policy
admin_unlock_min_signature u8 // nr of signatures from the adminunlock
clawback_accounts []u32 // account(s) which can clawback
clawback_min_signatures u8
clawback_from u32 // from epoch money can be clawed back, 0 is always
clawback_till u32 // till which date
}
pub enum LockType {
locked_till
locked
free
}
pub struct AccountLimit {
pub mut:
amount u64 // in smallest unit
asset_id u32
period AccountLimitPeriodLimit
}
pub enum AccountLimitPeriodLimit {
daily
weekly
monthly
}
pub struct AccountAsset {
pub mut:
assetid u32
balance u64
metadata map[string]string
}

View File

@@ -0,0 +1,122 @@
module ledger
import incubaid.herolib.data.encoder
pub fn (self AccountPolicy) dump(mut e encoder.Encoder) ! {
e.add_u16(1) // version
e.add_u32(self.policy_id)
e.add_list_u32(self.admins)
e.add_u8(self.min_signatures)
e.add_u32(u32(self.limits.len))
for limit in self.limits {
limit.dump(mut e)!
}
e.add_list_u32(self.whitelist_out)
e.add_list_u32(self.whitelist_in)
e.add_u32(self.lock_till)
e.add_u8(u8(self.admin_lock_type))
e.add_u32(self.admin_lock_till)
e.add_list_u32(self.admin_unlock)
e.add_u8(self.admin_unlock_min_signature)
e.add_list_u32(self.clawback_accounts)
e.add_u8(self.clawback_min_signatures)
e.add_u32(self.clawback_from)
e.add_u32(self.clawback_till)
}
fn (mut self AccountPolicy) load(mut e encoder.Decoder) ! {
version := e.get_u16()!
assert version == 1, 'Unsupported AccountPolicyP version: ${version}'
self.policy_id = e.get_u32()!
self.admins = e.get_list_u32()!
self.min_signatures = e.get_u8()!
limits_len := e.get_u32()!
self.limits = []AccountLimit{cap: int(limits_len)}
for _ in 0 .. limits_len {
mut limit := AccountLimit{}
limit.load(mut e)!
self.limits << limit
}
self.whitelist_out = e.get_list_u32()!
self.whitelist_in = e.get_list_u32()!
self.lock_till = e.get_u32()!
self.admin_lock_type = unsafe { LockType(e.get_u8()!) }
self.admin_lock_till = e.get_u32()!
self.admin_unlock = e.get_list_u32()!
self.admin_unlock_min_signature = e.get_u8()!
self.clawback_accounts = e.get_list_u32()!
self.clawback_min_signatures = e.get_u8()!
self.clawback_from = e.get_u32()!
self.clawback_till = e.get_u32()!
}
pub fn (self AccountLimit) dump(mut e encoder.Encoder) ! {
e.add_u16(1) // version
e.add_u64(self.amount)
e.add_u32(self.asset_id)
e.add_u8(u8(self.period))
}
fn (mut self AccountLimit) load(mut e encoder.Decoder) ! {
version := e.get_u16()!
assert version == 1, 'Unsupported AccountLimitP version: ${version}'
self.amount = e.get_u32()!
self.asset_id = e.get_u32()!
self.period = unsafe { AccountLimitPeriodLimit(e.get_u8()!) }
}
pub fn (self AccountAsset) dump(mut e encoder.Encoder) ! {
e.add_u16(1) // version
e.add_u32(self.assetid)
e.add_u64(self.balance)
e.add_map_string(self.metadata)
}
fn (mut self AccountAsset) load(mut e encoder.Decoder) ! {
version := e.get_u16()!
assert version == 1, 'Unsupported AccountAssetP version: ${version}'
self.assetid = e.get_u32()!
self.balance = e.get_u32()!
self.metadata = e.get_map_string()!
}
pub fn (self Account) dump(mut e encoder.Encoder) ! {
e.add_u16(1) // version
e.add_u32(self.owner_id)
e.add_u32(self.location_id)
e.add_u32(u32(self.accountpolicies.len))
for policy in self.accountpolicies {
policy.dump(mut e)!
}
e.add_u32(u32(self.assets.len))
for asset in self.assets {
asset.dump(mut e)!
}
e.add_u32(self.assetid)
e.add_u32(self.last_activity)
e.add_list_u32(self.administrators)
}
fn (mut self DBAccount) load(mut o Account, mut e encoder.Decoder) ! {
version := e.get_u16()!
assert version == 1, 'Unsupported Account version: ${version}'
o.owner_id = e.get_u32()!
o.location_id = e.get_u32()!
policies_len := e.get_u32()!
o.accountpolicies = []AccountPolicy{cap: int(policies_len)}
for _ in 0 .. policies_len {
mut policy := AccountPolicy{}
policy.load(mut e)!
o.accountpolicies << policy
}
assets_len := e.get_u32()!
o.assets = []AccountAsset{cap: int(assets_len)}
for _ in 0 .. assets_len {
mut asset := AccountAsset{}
asset.load(mut e)!
o.assets << asset
}
o.assetid = e.get_u32()!
o.last_activity = e.get_u32()!
o.administrators = e.get_list_u32()!
}

View File

@@ -0,0 +1,61 @@
module ledger
import incubaid.herolib.hero.db
import json
pub struct ModelsFactory {
pub mut:
db &db.DB
account &DBAccount
// asset &DBAsset
// dnszone &DBDNSZone
// group &DBGroup
// member &DBMember
// notary &DBNotary
// signature &DBSignature
// transaction &DBTransaction
// user &DBUser
// userkvs &DBUserKVS
// userkvsitem &DBUserKVSItem
}
pub fn new_models_factory(mut database db.DB) !&ModelsFactory {
mut factory := &ModelsFactory{
db: database
account: &DBAccount{
db: database
}
}
// factory.asset = &DBAsset{
// db: database
// }
// factory.dnszone = &DBDNSZone{
// db: database
// }
// factory.group = &DBGroup{
// db: database
// }
// factory.member = &DBMember{
// db: database
// }
// factory.notary = &DBNotary{
// db: database
// }
// factory.signature = &DBSignature{
// db: database
// }
// factory.transaction = &DBTransaction{
// db: database
// }
// factory.user = &DBUser{
// db: database
// }
// factory.userkvs = &DBUserKVS{
// db: database
// }
// factory.userkvsitem = &DBUserKVSItem{
// db: database
// }
return factory
}

View File

@@ -1,4 +1,4 @@
module models_ledger
module ledger
pub fn (self AccountPolicy) dump(mut e encoder.Encoder) ! {
e.add_u32(self.policy_id)
@@ -10,15 +10,15 @@ pub fn (self AccountPolicy) dump(mut e encoder.Encoder) ! {
}
e.add_list_u32(self.whitelist_out)
e.add_list_u32(self.whitelist_in)
e.add_u64(self.lock_till)
e.add_u32(self.lock_till)
e.add_u8(u8(self.admin_lock_type))
e.add_u64(self.admin_lock_till)
e.add_u32(self.admin_lock_till)
e.add_list_u32(self.admin_unlock)
e.add_u8(self.admin_unlock_min_signature)
e.add_list_u32(self.clawback_accounts)
e.add_u8(self.clawback_min_signatures)
e.add_u64(self.clawback_from)
e.add_u64(self.clawback_till)
e.add_u32(self.clawback_from)
e.add_u32(self.clawback_till)
}
fn (mut self AccountPolicy) load(mut e encoder.Decoder) ! {
@@ -34,38 +34,38 @@ fn (mut self AccountPolicy) load(mut e encoder.Decoder) ! {
}
self.whitelist_out = e.get_list_u32()!
self.whitelist_in = e.get_list_u32()!
self.lock_till = e.get_u64()!
self.lock_till = e.get_u32()!
self.admin_lock_type = unsafe { LockType(e.get_u8()!) }
self.admin_lock_till = e.get_u64()!
self.admin_lock_till = e.get_u32()!
self.admin_unlock = e.get_list_u32()!
self.admin_unlock_min_signature = e.get_u8()!
self.clawback_accounts = e.get_list_u32()!
self.clawback_min_signatures = e.get_u8()!
self.clawback_from = e.get_u64()!
self.clawback_till = e.get_u64()!
self.clawback_from = e.get_u32()!
self.clawback_till = e.get_u32()!
}
pub fn (self AccountLimit) dump(mut e encoder.Encoder) ! {
e.add_u64(self.amount)
e.add_u32(self.amount)
e.add_u32(self.asset_id)
e.add_u8(u8(self.period))
}
fn (mut self AccountLimit) load(mut e encoder.Decoder) ! {
self.amount = e.get_u64()!
self.amount = e.get_u32()!
self.asset_id = e.get_u32()!
self.period = unsafe { AccountLimitPeriodLimit(e.get_u8()!) }
}
pub fn (self AccountAsset) dump(mut e encoder.Encoder) ! {
e.add_u32(self.assetid)
e.add_u64(self.balance)
e.add_u32(self.balance)
e.add_map_string(self.metadata)
}
fn (mut self AccountAsset) load(mut e encoder.Decoder) ! {
self.assetid = e.get_u32()!
self.balance = e.get_u64()!
self.balance = e.get_u32()!
self.metadata = e.get_map_string()!
}
@@ -136,7 +136,7 @@ pub fn (self Account) dump(mut e encoder.Encoder) ! {
asset.dump(mut e)!
}
e.add_u32(self.assetid)
e.add_u64(self.last_activity)
e.add_u32(self.last_activity)
e.add_list_u32(self.administrators)
}
@@ -158,7 +158,7 @@ fn (mut self DBAccount) load(mut o Account, mut e encoder.Decoder) ! {
o.assets << asset
}
o.assetid = e.get_u32()!
o.last_activity = e.get_u64()!
o.last_activity = e.get_u32()!
o.administrators = e.get_list_u32()!
}
@@ -312,59 +312,6 @@ pub fn (mut self DBAccount) list_all() ![]Account {
return self.db.list[Account]()!.map(self.get(it)!)
}
// Response struct for API
pub struct Response {
pub mut:
id int
jsonrpc string = '2.0'
result string
error ?ResponseError
}
pub struct ResponseError {
pub mut:
code int
message string
}
pub fn new_response(rpcid int, result string) Response {
return Response{
id: rpcid
result: result
}
}
pub fn new_response_true(rpcid int) Response {
return Response{
id: rpcid
result: 'true'
}
}
pub fn new_response_false(rpcid int) Response {
return Response{
id: rpcid
result: 'false'
}
}
pub fn new_response_int(rpcid int, result int) Response {
return Response{
id: rpcid
result: result.str()
}
}
pub fn new_error(rpcid int, code int, message string) Response {
return Response{
id: rpcid
error: ResponseError{
code: code
message: message
}
}
}
pub struct UserRef {
pub mut:
id u32

View File

@@ -1,4 +1,4 @@
module models_ledger
module ledger
import incubaid.herolib.data.ourtime
import incubaid.herolib.hero.db
@@ -37,15 +37,15 @@ pub mut:
limits []AccountLimit
whitelist_out []u32 // where money can go to
whitelist_in []u32 // where money can come from
lock_till u64 // date in epoch till no money can be transfered, only after
lock_till u32 // date in epoch till no money can be transfered, only after
admin_lock_type LockType
admin_lock_till u64 // date in epoch when admin can unlock (0 means its free), this is unlock for changing this policy
admin_lock_till u32 // date in epoch when admin can unlock (0 means its free), this is unlock for changing this policy
admin_unlock []u32 // users who can unlock the admin policy
admin_unlock_min_signature u8 // nr of signatures from the adminunlock
clawback_accounts []u32 // account(s) which can clawback
clawback_min_signatures u8
clawback_from ourtime.OurTime // from epoch money can be clawed back, 0 is always
clawback_till u64 // till which date
clawback_from u32 // from epoch money can be clawed back, 0 is always
clawback_till u32 // till which date
}

View File

@@ -1,4 +1,4 @@
module models_ledger
module ledger
import json

View File

@@ -1,5 +1,5 @@
// lib/threefold/models_ledger/asset.v
module models_ledger
module ledger
import incubaid.herolib.data.encoder
import incubaid.herolib.data.ourtime

View File

@@ -1,4 +1,4 @@
module models_ledger
module ledger
import json

View File

@@ -1,5 +1,5 @@
// lib/threefold/models_ledger/dnszone.v
module models_ledger
module ledger
import incubaid.herolib.data.encoder
import incubaid.herolib.data.ourtime

View File

@@ -1,4 +1,4 @@
module models_ledger
module ledger
fn test_setup_db_only() ! {
mut store := setup_test_db()!

View File

@@ -1,5 +1,5 @@
// lib/threefold/models_ledger/group.v
module models_ledger
module ledger
import incubaid.herolib.data.encoder
import incubaid.herolib.data.ourtime

View File

@@ -1,4 +1,4 @@
module models_ledger
module ledger
import incubaid.herolib.data.encoder
import incubaid.herolib.data.ourtime

View File

@@ -1,4 +1,4 @@
module models_ledger
module ledger
import incubaid.herolib.hero.db
import json

View File

@@ -1,4 +1,4 @@
module models_ledger
module ledger
import incubaid.herolib.data.encoder
import incubaid.herolib.data.ourtime

View File

@@ -1,4 +1,4 @@
module models_ledger
module ledger
import incubaid.herolib.data.encoder
import incubaid.herolib.data.ourtime

View File

@@ -1,4 +1,4 @@
module models_ledger
module ledger
import incubaid.herolib.hero.db

View File

@@ -1,5 +1,5 @@
// lib/threefold/models_ledger/transaction.v
module models_ledger
module ledger
import incubaid.herolib.data.encoder
import incubaid.herolib.data.ourtime

View File

@@ -1,5 +1,5 @@
// lib/threefold/models_ledger/user.v
module models_ledger
module ledger
import incubaid.herolib.data.encoder
import incubaid.herolib.data.ourtime

View File

@@ -1,4 +1,4 @@
module models_ledger
module ledger
import incubaid.herolib.data.encoder
import incubaid.herolib.data.ourtime

View File

@@ -1,4 +1,4 @@
module models_ledger
module ledger
import incubaid.herolib.data.encoder
import incubaid.herolib.hero.db