refactor

docs/main/key_innovations_overview/_category_.json

@@ -0,0 +1,7 @@
+{
+    "label": "Key Innovations",
+    "position": 7,
+    "link": {
+      "type": "generated-index",
+    }
+  }

docs/main/key_innovations_overview/compute/_category_.json

@@ -0,0 +1,7 @@
+{
+    "label": "Compute",
+    "position": 5,
+    "link": {
+      "type": "generated-index",
+    }
+  }

docs/main/key_innovations_overview/compute/cloud.md

@@ -0,0 +1,38 @@
+---
+title: Geo-Aware Cloud 
+sidebar_position: 3
+---
+
+
+## Zero OS as a generator for Compute, Storage, Network capacity
+
+### Compute (uses CU)
+
+- ZKube
+    - kubernetes deployment
+- Zero VM
+    - the container or virtual machine running inside ZOS
+- CoreX
+    - process manager (optional), can be used to get remote access to your zero_vm
+
+A 3Node is a Zero-OS enabled computer which is hosted with any of the Cloud Providers.
+
+### There are 4 storage mechanisms which can be used to store your data:
+
+- ZOS FS
+    - is our dedupe unique filesystem, replaces docker images. 
+- ZOS Mount
+    - is a mounted disk location on SSD, this can be used as faster storage location. 
+- Quantum Safe Filesystem
+    - this is a super unique storage system, data can never be lost or corrupted. Please be reminded that this storage layer is only meant to be used for secondary storage applications.
+- ZOS Disk
+    - a virtual disk technology, only for TFTech OEM partners.
+
+### There are 4 ways how networks can be connected to a Z-Machine.
+
+- Mycelium = Planetary network
+    - is a planetary scalable network, we have clients for windows, osx, android and iphone.
+- ZOS NIC
+    - connection to a public ipaddress
+- WEB GW
+    - web gateway, a secure way to allow internet traffic reach your secure Z-Machine.

docs/main/key_innovations_overview/compute/compute_compare.md

@@ -0,0 +1,22 @@
+---
+title: 'Compare'
+sidebar_position: 30
+description: The computer layer compared.
+---
+
+
+|                | Zos Compute Layer  Benefits                                                      | Default                                                          |
+|----------------|--------------------------------------------------------------------------------|------------------------------------------------------------------|
+| Management     | Full P2P, done by 3bot Agents, blockchain IT contract                          | Centralized e.g. Kubernetes, ...                                 |
+| OS Deploy      | Stateless, there are no files copied on local HDD/SSD.                         | Deploy image or execute installer on a physical server           |
+| OS Upgrade     | Seamless, rolling upgrades, 100% modular and pre-deterministic, decentralized  | Difficult and error prone + vulnerable from security perspective |
+| Tamperproof    | If file gets modified Zero-OS will not boot the file.                          | No, man in middle is possible.                                   |
+| Scalability    | To the world                                                                   | Expensive and depending on lots of capital                       |
+| Security       | A lot of effort went into the capability to deploy for high security usecases. | Very hard to deploy securely, and expensive                      |
+| Green          | For certain workloads we can safe upto 10x on power usage                      | Super power hungry.                                              |
+| Liquid Cooling | Easy to do because of autonomous behavior no need to replace HW.               | Hard to do, how to do maintenance.                               |
+| Sovereign      | Yes                                                                            | Mostly not.                                                               |
+| Complexity     | Anyone can do it, we made it to allow everyone to be a provider.               | Real experts needed.                                             |
+
+> We do not compare our system with those that claim to be full cloud solutions but merely deploy containers using other management systems and optionally connect to a blockchain for billing purposes. Nor do we compare with marketplace systems that simply act as frontends for other systems. We believe these systems, while visually impressive, lack substantial technological foundations and cannot serve as a fundamental base layer for others.
+

docs/main/key_innovations_overview/compute/for_everyone.md

@@ -0,0 +1,14 @@
+---
+title: 'For Everyone'
+sidebar_position: 2
+description: 'Everyone can build on top of the ThreeFold new internet'
+# hide_title: true
+---
+
+## Zero-OS is easy to dploy
+
+![](img/zos_simple.png)
+
+## Everyone Can Build
+
+![](img/for_everyone.png)

docs/main/key_innovations_overview/compute/super_hub.md

@@ -0,0 +1,16 @@
+---
+title: 'Super HUB'
+sidebar_position: 20
+description: 'Ultra scalable architecture.'
+hide_title: true
+---
+
+
+## Superhub Architecture
+
+
+![](img/superhub.png)
+
+
+
+

docs/main/key_innovations_overview/compute/zero_deploy_inno.md

@@ -0,0 +1,50 @@
+---
+title: Deterministic Deploy
+sidebar_position: 5
+hide_title: true
+---
+
+# Deterministic Deployment
+
+
+![](img/deterministic.png)
+
+
+
+
+The concept of Zero-Deploy is a key component of the **Smart Contract for IT** framework, which can be applied to any type of workload—whether it's containers, virtual machines (VMs), network gateways, volumes, Kubernetes resources, or other network elements. This framework serves as a formal agreement between a farmer (provider) and a user regarding the deployment of an IT workload.
+
+### Process
+
+1. **Build Your Code**  
+2. 
+   Develop and prepare your application (AI agents, web 2, web 3) code.
+
+3. **Convert to Zero-Image**  
+
+   Use a CI/CD solution to convert your Docker build (or other format) into a Zero-Image format. The 3Bot will do this on behalf of the user in the future.
+
+4. **Define the Workload**  
+
+   Specify all the details of your workload, including network bridges, web gateways, required machines, and more.
+
+5. **Register and Sign**  
+
+   Register the workload and sign it with your private key.
+
+6. **Automatic Detection**  
+   All necessary Zero-OS nodes (our infrastructure) will detect that a new workload needs to be deployed.
+
+7. **Deployment Process**  
+   The nodes will pull down the formal workload descriptions and initiate the deployment process.
+
+8. **Validation**  
+   Every step of the deployment is verified by Zero-OS (ZOS) to ensure that the intended result is accurately replicated. If any discrepancies are detected, ZOS will halt the deployment and provide an error message.
+
+### Benefits
+
+- **Deterministic Deployment**: There is no dynamic behavior during deployment at runtime, ensuring a consistent and predictable outcome.
+- **Strict Compliance**: No process can start unless all files and configurations are fully described at the flist level.
+
+
+

docs/main/key_innovations_overview/compute/zero_image_inno.md

@@ -0,0 +1,40 @@
+---
+title: Zero-Images
+sidebar_position: 3
+---
+
+![](../../img/zos_images.jpg)
+
+
+### The Problem
+
+The current method of deploying workloads in the cloud using Docker containers and virtual machine images has inherent issues. These images consume significant storage space, result in slow and bandwidth-intensive transfers to the internet's edge, drive up costs, introduce complexity, and pose security risks due to difficulties in tracking their contents over time.
+
+For instance, a complete Ubuntu image can easily be 2 GB in size, comprising millions of files. In contrast, the Flist (metadata for Zero-Image) for a full Ubuntu image is less than 2 MB (1000 times smaller). Based on this flist only the required files will be dowbloaded which can easily be 10x less compared to the original image size. These downloaded files (or subparts of files) are identified by a fingerprint (hash) and will only boot once authenticity can be verified.
+
+
+### Process
+
+- Zero-OS or the Zero-Image Command Line (works on linux) gets informed to provision a virtual filesystem based on a Zero-Image URL.
+- The Zero-Image Metadata is stored on e.g. an S3 Server or our Zero-Hub.
+
+
+### Introducing Flist
+
+A new image format that separates the image data (comprising files and subfile parts) from the metadata describing the image structure. 
+
+An Flist's format uniquely encompasses comprehensive file descriptions along with all relevant metadata such as size, modification and creation timestamps, and POSIX attributes. Additionally, it incorporates a fingerprint for each component, ensuring deterministic behavior—a crucial feature for security focused use cases.
+
+Flists provide the flexibility to manage metadata and data as separate entities, offering a versatile approach to handling various build and delivery scenarios.
+
+### The Benefits
+
+- **Rapid deployment:** Zero-OS enables containers and virtual machines to launch up to 100 times faster, especially in decentralized scenarios.
+- **Enhanced security:** Zero-OS prevents tampering with images, ensuring higher security levels.
+- **Reduced storage and bandwidth:** Zero-OS significantly reduces storage and bandwidth requirements, potentially achieving up to a 100-fold improvement.
+- **Deterministic deployments:** engineers can precisely define deployments beforehand, ensuring predictable outcomes without changes during deployment.
+- **100% compatible:** with existing standards, docker and virtual machines. The same format is useful for VM's as well as any container technology.
+
+### Status
+
+Usable for years, see Zero-OS.

docs/main/key_innovations_overview/compute/zero_install_inno.md

@@ -0,0 +1,32 @@
+---
+title: Zero-Install
+sidebar_position: 4
+---
+
+![](../../img/boot.png)  
+
+The Zero-OS is delivered to the 3Nodes over the internet network (network boot) and does not need to be installed.
+
+
+### 3Node Install
+
+1. Deploy a computer
+2. Configure a farm on the TFGrid explorer
+3. Download the bootloader and put on a USB stick or configure a network boot device
+4. Power on the computer and connect to the internet
+5. Boot! The computer will automatically download the components of the operating system (Zero-OS)
+
+The actual bootloader is very small, it brings up the network interface of your computer and queries TFGrid for the remainder of the boot files needed.
+
+The operating system is not installed on any local storage medium (hard disk, ssd), Zero-OS is stateless.
+
+The mechanism to allow this to work in a safe and efficient manner is an innovation called our container virtual filesystem. 
+
+### Process
+
+- optionally: configure booting from secure BIOS
+- optionally: install signing certificate in the BIOS, to make sure that only the right bootloader can be started
+- the bootloader (ISO, PXE, USB, ...) get's downloaded from Internet (TFGrid CDN or private deployment)
+- core-0 (the first boot process) starts, self verification happens
+- the metadata for the the required software modules is downloaded and checked against signature and hashes 
+- the core-0 zero_image service

docs/main/key_innovations_overview/compute/zos_compute.md

@@ -0,0 +1,67 @@
+---
+title: 'ZOS - geo-aware OS'
+sidebar_position: 1
+description: The computer layer
+hide_title: true
+---
+
+
+![](img/zos_intro.png)
+
+# ZOS - geo-aware OS
+
+ThreeFold has developed its own operating system, Zero-OS, which is based on the Linux Kernel. The purpose of Zero-OS is to strip away the unnecessary complexities commonly found in contemporary operating systems.
+
+
+### Imagine An Operating System With The Following Benefits
+
+- Up to 10x more efficient for certain workloads (e.g. storage)
+- No install required
+- All files are deduped for the VM's, containers and the ZOS itself, no more data duplicated filesystems
+- The hacking footprint is very small which leads to much safer systems
+    - Every file is fingerprinted and gets checked at launch time of an application
+    - There is no shell or server interface on the operating system
+    - The networks are end2end encrypted between all Nodes
+- It is possible to completely disconnect the compute/storage from the network service part which means hackers have a lot less chance to access the data
+- A smart contract for the IT layer allows groups of people to deploy IT workloads with consensus and full control
+- All workloads which can run on linux can run on Zero-OS but in a much more controlled, private and safe way
+
+> We have created an operating system from scratch. We used the Linux kernel and its components and then built further on it. We have been able to achieve all of the above benefits.
+
+## Requirements:
+
+- **Autonomy**: TF Grid needs to create compute, storage and networking capacity everywhere. We could not rely on a remote (or a local) maintenance of the operating system by owners or operating system administrators.
+- **Simplicity**: An operating system should be simple, able to exist anywhere for anyone, and be good for the planet.
+- **Stateless**: In a grid (peer-to-peer) set up, the sum of the components provides a stable basis for single elements to fail and not bring the whole system down. Therefore, it is necessary for single elements to be stateless, and the state needs to be stored within the grid.
+
+
+
+## Key Features of Zero-OS:
+
+Zero-OS is designed with minimalism in mind, supporting only a few fundamental primitives that handle essential low-level functions:
+
+1. **Storage Capacity**
+2. **Compute Capacity**
+3. **Network Capacity**
+
+Default features:
+
+- Compatible with Docker
+- Compatible with any VM (Virtual Machine)
+- Compatible with any Linux workload
+- Integrated unique storage & network primitives
+- Integrated smart contract for IT layer
+
+## benefits
+
+- No need to work with images, we work with our unique ZOS FS
+- Every container runs in a dedicated virtual machine providing more security
+- The containers talk to each other over a private network (Mycelium)
+- The containers can use a web gateway to allow internet users to connect to the applications which are running in their secure containers 
+- Can use core-x to manage the workload
+
+**Security and Simplicity:**
+
+Zero-OS provides a Autonomous Decentralized Cloud. 
+
+This not only blocks hacker access but also eliminates human error, enhancing both security and reliability.

docs/main/key_innovations_overview/compute/zos_contract_for_it.md

@@ -0,0 +1,35 @@
+---
+title: 'Smart Contract for IT'
+sidebar_position: 3
+description: 'How smart contract tech can be used to deploy IT workloads.'
+hide_title: true
+---
+
+
+## Deployment of Workloads Using Secure Smart IT Contracts
+
+
+![](img/smart_contract_it.png)
+
+
+
+Workloads can be deployed through a secure and decentralized system enabled by **smart IT contracts**. 
+
+
+These contracts ensure the following:
+
+1. **Multi-Signature Authorization**  
+   Before a workload is deployed, multiple authorized individuals must sign off on the deployment contract. This ensures a consensus-driven process, adding a layer of security and accountability. No single individual has unilateral control over the process.
+
+2. **Immutable and Autonomous Deployment**  
+   Once the deployment is signed and approved, the workload is executed as defined in the smart IT contract. The system ensures that:  
+   - No party, including the signers, can alter the deployed workload or access the stored data.  
+   - The deployment process is verified, authenticated, and recorded immutably on the blockchain, guaranteeing transparency and trust.
+
+3. **Managed by Virtual Administrators (3BOTs)**  
+   The workloads can optionally be autonomously managed by virtual administrators, known as **3BOTs**. These bots operate as trustworthy system administrators and ensure the deployed solution adheres strictly to the agreed-upon parameters.
+
+4. **Registered on a Decentralized Geo-Aware Ledger**  
+   Once the contract is finalized and deployment occurs, the details are permanently registered in the **TFChain blockchain**, providing an immutable record of the transaction. This further enhances security and transparency.
+
+By leveraging these mechanisms, the system ensures that IT workloads are deployed securely, remain tamper-proof, and operate in a decentralized, autonomous manner. This approach eliminates risks of unauthorized changes and protects the integrity of deployed solutions.

docs/main/key_innovations_overview/energy_efficient copy.md

@@ -0,0 +1,17 @@
+---
+sidebar_position: 1
+---
+
+# Geo Aware Sovereign AI Agent
+
+![](../ai_agent/img/ai_agent.png)
+
+> ask us for more information, the result of +10 years of research.
+
+unique
+
+- integrated personal blockchain with offline support (can even run on phone)
+- revolutionary backend database called OSIS with structured record (+10,000 records per second on a phone)
+- unbreakable network & data storage layer integration (in the phone and cloud)
+- easy to use very powerful development language with +30,000 github stars.
+- development 10x less time required compared to alternative web3 mechanisms.

docs/main/key_innovations_overview/energy_efficient.md

@@ -0,0 +1,14 @@
+---
+sidebar_position: 20
+---
+
+# Energy Efficient
+
+Below are some of the ways in which we achieve energy efficiency as compared to traditional models.
+
+![](img/energy_efficiency.png)
+
+In addition, a decentralized peer-to-peer infrastructure which finds the shortest path between end points is by nature energy-efficient. Data needs to travel a much shorter distance.
+
+> Depending on the use case the our approach can lead to 10x energy savings.
+

docs/main/key_innovations_overview/network/_category_.json

@@ -0,0 +1,7 @@
+{
+    "label": "Network",
+    "position": 6,
+    "link": {
+      "type": "generated-index",
+    }
+  }

docs/main/key_innovations_overview/network/compare.md

@@ -0,0 +1,20 @@
+---
+title: Compare
+sidebar_position: 20
+---
+
+
+|                             | ThreeFold Network Layer                                                           | Other Overlay Network Technologies (like VPN)                                                                               |
+|-----------------------------|-----------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
+| Management                  | Full P2P, done by 3bot Agents, blockchain IT contract                             | Centralized leading to security issues                                                                                      |
+| Locality                    | Find shortest path on latency and quality, this allows traffic to stay sovereign. | NO, based on centralized control mechanisms or inefficient algorithms that route traffic indiscriminately across the globe. |
+| Encryption                  | End2End ecryption, unique for every relation, linked to private key               | Normally based on key exchange, or pre-shared keys.                                                                         |
+| Post Quantum                | Possible (ask us)                                                                 | No                                                                                                                          |
+| Scalability                 | Our aim is to be planetary scalable, but we need more exposure.                   | Bad                                                                                                                         |
+| Compatibility               | We aim to support mobile, desktop, IOT, ...                                       | Depends, often not                                                                                                          |
+| Backdoors                   | NO, all is based on opensource                                                    | Often, yes, unfortunately.                                                                                                  |
+| Performance                 | Quite good, 1 gbit / sec can be achieved on std node (which is high for overlay)  | Often slow.                                                                                                                 |
+| Security Model              | Whitelist model                                                                   | Blacklist model, list who is bad e.g. firewalls                                                                             |
+| Fully integrated in compute | Yes                                                                               | Lots of different solutions                                                                                                 |
+
+

docs/main/key_innovations_overview/network/mycelium.md

@@ -0,0 +1,42 @@
+---
+title: Mycelium
+sidebar_position: 2
+---
+
+
+# Mycelium: Our Planetary Network
+
+![](img/mycelium_overview.png)
+
+
+The planetary network called Mycelium is an overlay network which lives on top of the existing Internet or other peer-to-peer networks created. 
+
+In the Mycelium network, everyone is connected to everyone. End-to-end encryption between users of an app and the app runs behind the network wall.
+
+## Mycelium: A New Network Layer for the Internet
+
+### The Problem
+
+The current centralized state of the internet poses significant security risks, with compromised routers and growing cyber threats (trillions of USD per year now), making everyone vulnerable to hacking. Industry responses involve disabling original features, hindering true peer-to-peer connectivity and personal server capabilities. Workarounds and system hacks have become the norm.
+
+**Our Internet is seriously broken. We need new ways to communicate**
+
+### Introducing Mycelium
+
+Mycelium is an overlay network layer designed to enhance the existing internet infrastructure while remaining compatible with all current applications. It empowers true peer-to-peer communication. By installing a Network Agent on your device, you gain the ability to securely connect with any other participant on this network. Mycelium intelligently reroutes traffic to maintain connectivity taking location of you and your peer into consideration.
+
+### The Benefits
+
+
+- **Continuous connectivity:** Mycelium ensures uninterrupted connectivity by dynamically rerouting traffic through available connections (friends, satellites, 4/5G, fiber).
+- **End-to-end encryption:** robust encryption stops man-in-the-middle attacks, guaranteeing secure communication.
+- **Proof of authenticity (POA): ensures that we know who we are communicating with 
+- **Optimized routing:** Mycelium finds the shortest path between network participants, reducing latency and keeping traffic localized.
+- **Universal server capability:** empowers individuals to act as servers, a foundational element for any peer-to-peer system.
+- **Full Compatibility:** Mycelium seamlessly integrates with the current internet, supporting any application.
+- **Impressive speed:** achieves 1 Gbps per Network Agent, ensuring rapid data transfer.
+
+### Status
+
+In beta and usable from TFGrid 3.13, its our 3e generation approach to networking and took us years to do. We are looking forward to your feedback.
+

docs/main/key_innovations_overview/network/mycelium_shortest_path_routing_inno.md

@@ -0,0 +1,63 @@
+---
+title: Shortest Path Routing
+sidebar_position: 5
+---
+
+
+# Shortest Path Routing
+
+## Empowering Connectivity with an End-to-End Encrypted Overlay Network
+
+### The Concept of End-to-End Encryption
+
+End-to-end encryption (E2EE) ensures that data is encrypted on the sender's device and only decrypted on the recipient's device. This means that no intermediaries, including service providers, can access or alter the data while it is in transit. 
+
+### Shortest Path Routing in Overlay Networks
+
+An overlay network is a virtual network built on top of an existing physical network. 
+
+Each enduser mycelium agent will execute custom routing logic and protocols to improve connectivity. 
+
+- In the context of a Mycelium peer-to-peer (P2P) overlay network, nodes (participants) can dynamically discover and connect to each other, forming a mesh-like structure.
+- Shortest Path Routing: The network can use algorithms to find the shortest or most efficient path between nodes. This ensures that data packets travel the minimum distance required to reach their destination, reducing latency and improving performance.
+
+### Multi-Hop Communication
+
+In a P2P overlay network, data can hop through multiple nodes to reach its destination. This means that if a direct connection is not available, the data can be relayed through intermediary nodes. For example:
+
+1. **Node A** wants to send data to **Node D**.
+2. There is no direct connection, but **Node A** can reach **Node B**, which can reach **Node C**, which finally reaches **Node D**.
+3. The data is encrypted end-to-end, so it remains secure throughout its journey.
+
+Network usage tracking and billing can be used to make sure all participants are rewarded.
+
+### Leveraging Existing Networks
+
+This overlay network operates on top of existing internet infrastructure. 
+
+This leads to:
+
+1. **Cost Efficiency**: By leveraging existing infrastructure, there is no need for extensive new investments in physical hardware.
+2. **Flexibility**: The network can dynamically adapt to changing conditions, such as network congestion or outages.
+
+### Improving Connectivity for Underserved Populations
+
+Currently, around 4 billion people lack decent internet access. 
+
+Mycelium can significantly improve their connectivity:
+
+1. **Decentralized Access**: People in remote or underserved areas can connect to the network through nearby nodes, which may belong to friends, community members, or even commercial providers offering bandwidth.
+2. **Community-Driven Networks**: Local communities can set up nodes that connect to the broader overlay network, creating a resilient and scalable web of connectivity.
+3. **Increased Bandwidth**: By aggregating available bandwidth from multiple sources, the overlay network can provide higher data rates and more reliable connections.
+
+### Example Scenario
+
+Imagine a remote village with limited internet access. The villagers set up several nodes that connect to each other and to nearby towns with better connectivity, also some of the nodes can be connected to Internet over satelite, mobile 4g or other mechanisms.
+
+Here’s how it works:
+
+1. **Local Node Setup**: Villagers install nodes on their devices, which form a local mesh network.
+2. **Connecting to Broader Network**: Some nodes have access to satellite internet or long-range Wi-Fi that connects to nearby towns.
+3. **Dynamic Routing**: When a villager wants to access online resources, their data is encrypted end-to-end and routed through the shortest path available, which may include local nodes, satellite links, and commercial internet providers.
+4. **Enhanced Access**: This setup leverages all available bandwidth sources, providing more reliable and faster internet access to the village.
+

docs/main/key_innovations_overview/network/mycelium_whiltelist.md

@@ -0,0 +1,57 @@
+---
+title: Whitelists
+sidebar_position: 4
+---
+
+# Mycelium Whitelists
+
+> Rethinking Network Security: Beyond Traditional Firewalls
+
+### The Limitations of Traditional Firewalls
+
+Firewalls have long been the cornerstone of network security, operating as gatekeepers to keep malicious actors out. 
+
+They work by monitoring incoming and outgoing network traffic and applying security rules to block or allow data packets based on predefined criteria. However, while firewalls are effective at creating a barrier, they have inherent limitations:
+
+1. **Perimeter Focus**: Firewalls are designed to protect the perimeter of the network. This approach assumes that threats come from outside the network, but it does not adequately address threats from within.
+2. **Static Rules**: Firewalls rely on static rules that can be bypassed by sophisticated attacks. They do not adapt dynamically to changing threat landscapes.
+3. **Single Point of Failure**: As a centralized barrier, firewalls represent a single point of failure. If a firewall is compromised, the entire network can be exposed.
+
+### The Need for Strong Authentication and Peer-to-Peer Communication
+
+To address these limitations, a more modern approach to network security involves strong authentication and decentralized communication. By ensuring that all participants on the network are strongly authenticated, we can establish trust at the individual level rather than relying solely on perimeter defenses.
+
+#### Strong Authentication
+
+Strong authentication involves verifying the identity of network participants using robust methods such as:
+
+- **Multi-Factor Authentication (MFA)**: Requires multiple forms of verification, such as passwords, biometrics, and hardware tokens.
+- **Public Key Infrastructure (PKI)**: Uses cryptographic keys to authenticate users and devices.
+
+By implementing strong authentication, we can ensure that only legitimate users and devices can access the network, significantly reducing the risk of unauthorized access.
+
+#### Peer-to-Peer Communication Over an Overlay Network
+
+Instead of routing all traffic through a central firewall, participants can communicate directly with each other and applications using a peer-to-peer (P2P) overlay network. An overlay network, called Mycelium, can facilitate this decentralized communication.
+
+- **Mycelium Overlay Network**: This overlay network functions like a mesh, allowing nodes (participants) to connect directly with each other and applications. It provides a resilient and scalable architecture where each node can dynamically find the best path for communication.
+
+### Whitelists and Group-Based Access Control
+
+To further enhance security, applications can use whitelists and group-based access control. This approach involves:
+
+1. **Whitelisting Users**: Only allowing access to users who are explicitly permitted. This can be based on strong authentication credentials.
+2. **Group-Based Access Control**: Organizing users into groups with specific permissions. Each application can define which groups have access based on their source IP addresses and other criteria.
+
+#### Example Scenario
+
+Consider an application hosted on the network. Instead of relying on a firewall to block unauthorized access, the application uses Mycelium to communicate with authenticated peers. It employs a whitelist to specify which users or groups can access the application. For instance:
+
+- **Group A**: Developers with access to development resources.
+- **Group B**: Administrators with access to administrative tools.
+- **Group C**: End-users with access to specific application features.
+
+Each group’s access is controlled by specifying the allowed source IP addresses and other authentication factors. This ensures that only authorized users can access the application, regardless of their location.
+
+> only available in the enterprise edition.
+

docs/main/key_innovations_overview/network/networking.md

@@ -0,0 +1,31 @@
+---
+sidebar_position: 1
+title: 'Overview'
+---
+
+
+# Network Technology Overview
+
+Our decentralized networking platform allows any compute and storage workload to be connected together on a private (overlay) network and exposed to the existing Internet network. The peer-to-peer network platform allows any workload to be connected over secure encrypted networks, which will look for the shortest path between nodes.
+
+### Secure Mesh Overlay Network (Peer-to-Peer)
+
+ZNet is the foundation of any architecture running on the TF Grid. It can be seen as a virtual private data center and the network allows all of the *N* containers to connect to all of the *(N-1)* other containers. Any network connection is a secure network connection between your containers, it creates a peer-to-peer network between containers. 
+
+No connection is made with the Internet. The ZNet is a single tenant network and by default not connected to the public Internet. Everything stays private. For connecting to the public Internet, a Web Gateway is included in the product to allow for public access, if and when required.
+
+### Redundancy
+
+As integrated with Web Gateway
+
+- Any app can get (securely) connected to the Internet by any chosen IP address made available by ThreeFold network farmers through WebW
+- An app can be connected to multiple web gateways at once, the DNS round robin principle will provide load balancing and redundancy
+- An easy clustering mechanism where web gateways and nodes can be lost and the public service will still be up and running
+- Easy maintenance. When containers are moved or re-created, the same end user connection can be reused as that connection is terminated on the Web Gateway. The moved or newly created Web Gateway will recreate the socket to the Web Gateway and receive inbound traffic.
+
+### Network Wall
+
+
+For OEM projects we can implement a cloud deployment without using TCP-IP or Ethernet this can lead to super secure environments, ideal to battle the Cuber Pandemic.
+
+

docs/main/key_innovations_overview/network/webgw.md

@@ -0,0 +1,41 @@
+---
+sidebar_position: 3
+title: Web Gateway
+---
+
+# Web Gateway
+
+The Web Gateway is a mechanism to connect private networks to the open Internet in such a way that there is no direct connection between the Internet and the secure workloads running in the Zero VMs.
+
+### Key Benefits
+
+- Separation between where compute workloads are and where services are exposed
+- Redundancy: Each app can be exposed on multiple web gateways at once
+- Support for many interfaces
+- Helps resolve shortage of IPv4 addresses
+
+### Implementation
+
+Some 3Nodes support gateway functionality (this is configured by the farmers). A 3Node with gateway configuration can then accept gateway workloads and forward traffic to Zero VMs that only have Planetary Network or IPv6 addresses.
+
+The gateway workloads consist of a name (prefix) that first needs to be reserved on the blockchain. Then, the list of backend IPs. There are other flags that can be set to control automatic TLS (please check Terraform documentation for the exact details of a reservation).
+
+Once the 3Node receives this workload, the network configures proxy for this name and the Planetary Network IPs.
+
+### Security
+
+Zero VMs have to have a Planetary Network IP or any other IPv6 (IPv4 is also accepted). This means that any person connected to the Planetary Network can also reach the Zero VM without the need for a proxy.
+
+So it's up to the Zero VM owner/maintainer to make sure it is secured and that only the required ports are open.
+
+### Redundant Network Connection
+
+![](../../img/redundant_net.jpg)
+
+### Unlimited Scale
+
+![](../../img/webgw_scaling.jpg)
+
+The network architecture is a pure scale-out network system. It can scale to unlimited size, there is simply no bottleneck. Network "supply" is created by network farmers, and network "demand" is done by TF Grid users.
+
+Supply and demand scale independently. For supply, there can be unlimited network farmers providing web gateways on their own 3Nodes, and unlimited compute farmers providing 3Nodes for compute and storage. The demand side is driven by developers creating software that runs on the grid, system integrators creating solutions for enterprises, and so on. Globally, there is exponentially-growing demand for data processing and storage use cases.

docs/main/key_innovations_overview/storage/1_storage.md

@@ -0,0 +1,28 @@
+---
+sidebar_position: 1
+---
+
+# Unbreakable Storage
+
+Our storage architecture follows the true peer2peer design of the cecentralized cloud system. 
+
+![](img/unbreakable_storage.png)
+
+Any participating node only stores small incomplete parts of objects (files, photos, movies, databases etc.) by offering a slice of the present (local) storage devices. Managing the storage and retrieval of all of these distributed fragments is done by a software that creates development or end-user interfaces for this storage algorithm. We call this '**dispersed storage**'.
+
+## Benefits
+
+- Not even a quantum computer can hack
+- Zetabytes can be stored as easily as Petabytes
+- The system is truly autonomous & self healing
+- Datarot is detected and fixed.
+- There is 100% contorl over where data is (GDPR)
+
+## Peer2Peer Advantages
+
+Peer2peer provides the unique proposition of selecting storage providers that match your application and service of business criteria. For example, you might be looking to store data for your application in a certain geographic area (for governance and compliance) reasons. You might also want to use different "storage policies" for different types of data. Examples are live versus archived data. All of these uses cases are possible with this storage architecture, and could be built by using the same building slice produced by farmers and consumed by developers or end-users.
+
+> There is 100% control over where the data is positioned and the security is incredible.
+
+
+

docs/main/key_innovations_overview/storage/20_compare.md

@@ -0,0 +1,20 @@
+---
+title: Compare
+sidebar_position: 20
+description: 'compare how its dont in other overlay network systems.'
+---
+
+
+
+|                             | ThreeFold Storage Layer                                                                  | Overlay Storage Systems                                                                                      |
+|-----------------------------|------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|
+| Unbreakable                | Yes, novel encoding system (not encryption based) makes impossible to hack data.         | No                                                                                                                      |
+| Post Quantum                | Possible (ask us)                                                                        | No                                                                                                                      |
+| Scalability                 | Yes, this system exists +10 years, is being used by large orgs for zetabytes.            | Some systems, most not, but centralized.                                                                                |
+| Compatibility               | Yes thanks to filesystem approach                                                        | Depends, often not                                                                                                      |
+| Backdoors                   | NO, all is based on opensource                                                           | ?                                                                                                                       |
+| Performance                 | Is not a super fast system but good for most cases, +- 100 MB / sec per content creator. | Variable, hard to say, some are                                                                                         |
+| Efficiency for redundancy   | Ultra efficient, only 20% overhead to allow 4 locations to go down                       | NO, sometimes +5 copies = 500%                                                                                          |
+| Fully integrated in compute | Yes                                                                                      | Lots of different solutions                                                                                             |
+| Management                  | Full P2P, done by 3bot Agents, blockchain IT contract                                    | Centralized leading to security issues                                                                                  |
+| Locality                    | Data can be local and sovereign, full control by the the user (data creator)             | Based on centralized control mechanisms or inefficient algorithms that route traffic indiscriminately across the globe. |

docs/main/key_innovations_overview/storage/2_qsfs_innovation.md

@@ -0,0 +1,89 @@
+---
+title: File System
+sidebar_position: 2
+---
+
+### The Problem
+
+There is a growing need for more accessible and user-friendly solutions to store and manage large volumes of data efficiently.
+
+While Zero-Stor addresses numerous storage challenges effectively, it may not be accessible or user-friendly for typical developers or system administrators. QSFS has been developed to bridge this gap and provide a more approachable storage solution.
+
+### Introducing QSFS
+
+A filesystem utilizing Zero-Stor as its backend. Metadata is safeguarded to prevent loss, inheriting Zero-Stor's benefits and simplifying usage for developers and system administrators.
+
+The filesystem is always deployed in one location, data is distributed (using zero-stor) across multiple sites for unparalleled reliability. 
+
+Metadata redundancy is included. While not consistently synchronized in real-time, the system allows configuration of consistency levels. Typically, the decentralized state may lag by up to 15 minutes.
+
+This filesystem can be mounted under various storage-aware applications, such as backup servers, file servers, or S3 servers, enhancing versatility.
+
+### Benefits
+
+- Inherits the advantages of Zero-Stor, including enhanced data security, efficiency, and scalability.
+- Provides a user-friendly interface for seamless integration with a wide range of applications.
+- Offers considerable scalability capabilities, although not unlimited in scale.
+- Achieves reasonable performance data transfer rates of up to 50 MB/sec, particularly for larger files.
+- Can scale to about 2 million files per filesystem.
+
+
+## Details
+
+Our Unbreakable filesystem technology has unique features.
+
+
+A redundant filesystem, can store PB's (millions of gigabytes) of information.
+
+Unique features:
+
+- Unlimited scalability (many petabytes)
+- Unbreakable:
+  - No farmer knows what the data is
+  - Even a quantum computer cannot decrypt the data
+- Data can't be lost
+  - Protection for datarot, data will autorepair
+- Data is kept forever (data does not get deleted)
+- Data is dispersed over multiple sites
+  - Even if the sites go down the data will not be lost
+- Up to 10x more efficient than storing on classic storage cloud systems
+- Can be mounted as filesystem on any OS or any deployment system (OSX, Linux, Windows, Docker, Kubernetes etc.)
+- Compatible with ± all data workloads (not high performance data driven workloads like a database)
+- Self-healing: when a node or disk is lost, the storage system can get back to the original redundancy level
+- Helps with compliance for regulations like GDPR (as the hosting facility has no view on what is stored: information is encrypted and incomplete)
+- Hybrid: can be installed onsite, public and private
+- Read-write caching on encoding node (the front end)
+
+
+### Mount Any Files In Your Storage Infrastructure
+
+The QSFS is a mechanism to mount any file system (in any format) on the grid, in a quantum secure way. 
+
+This storage layer relies on 3 primitives: 
+
+- [0-db](https://github.com/threefoldtech/0-db) is the storage engine.
+It is an always append database, which stores objects in an immutable format. It allows history to be kept out-of-the-box, good performance on disk, low overhead, easy data structure and easy backup (linear copy and immutable files).
+
+- [0-stor-v2](https://github.com/threefoldtech/0-stor_v2) is used to disperse the data into chunks by performing 'forward-looking error-correcting code' (FLECC) on it and send the fragments to safe locations.
+It takes files in any format as input, encrypts the file with AES based on a user-defined key, then FLECC-encodes the file and spreads out the result
+to multiple 0-DBs. The number of generated chunks is configurable to make it more or less robust against data loss through unavailable fragments. Even if some 0-DBs are unreachable, you can still retrieve the original data, and missing 0-DBs can even be rebuilt to have full consistency. It is an essential element of the operational backup. 
+
+- [0-db-fs](https://github.com/threefoldtech/0-db-fs) is the filesystem driver which uses 0-DB as a primary storage engine. It manages the storage of directories and metadata in a dedicated namespace and file payloads in another dedicated namespace.
+
+Together they form a storage layer that is quantum secure: even the most powerful computer can't hack the system because no single node contains all of the information needed to reconstruct the data.
+
+
+
+This concept scales forever, and you can bring any file system on top of it: 
+- S3 storage 
+- any backup system
+- an ftp-server
+- IPFS and Hypercore distributed file sharing protocols 
+
+
+### Architecture
+
+By using our filesystem inside a Virtual Machine or Kubernetes, the cloud user can deploy any storage application on top e.g. Minio for S3 storage, OwnCloud as online fileserver.
+
+Any storage workload can be deployed on top of Unbreakable Storage System.
+

docs/main/key_innovations_overview/storage/3_qss_algorithm.md

@@ -0,0 +1,118 @@
+---
+sidebar_position: 3
+---
+
+# Algoritm
+
+The UnbreakableStorage Algorithm is the heart of the Storage engine. The storage engine takes the original data objects and creates data part descriptions that it stores over many virtual storage devices (ZDB/s).
+
+Data gets stored over multiple ZDB's in such a way that data can never be lost.
+
+Unique features
+
+- Data always append, can never be lost
+- Even a quantum computer cannot decrypt the data
+- Data is spread over multiple sites. If these sites are lost the data will still be available
+- Protects from datarot
+
+## The Problem
+
+Today we produce more data than ever before. We cannot continue to make full copies of data to make sure it is stored reliably. This will simply not scale. We need to move from securing the whole dataset to securing all the objects that make up a dataset.
+
+We are using technology which was originally used for communication in space.
+
+The algo stores data fragments over multiple devices (physical storage devices ). 
+
+The solution is not based on replication or sharding, the algo represents the data as equasions which are distributed over multiple locations.
+
+
+## How Data Is Stored Today
+
+![](img/storage_old.png)
+
+In most distributed systems, as used on the Internet or in blockchain today, the data will get replicated (sometimes after sharding, which means distributed based on the content of the file and spread out over the world).
+
+This leads to a lot of overhead and minimal control where the data is.
+
+In well optimized systems overhead will be 400% but in some it can be orders of magnitude higher to get to a reasonable redundancy level.
+
+## The UnbreakableStorage System Works Differently
+
+![](img/unbreakable2.png)
+
+We have developed a new storage algorithm which is more efficient, ultra reliable and gives you full control over where your data is stored.
+
+Our approach is different. Let's try to visualize this new approach with a simple analogy using equations.
+
+Let a,b,c,d.... be the parts of the original object. You could create endless unique equations using these parts. A simple example: let's assume we have 3 parts of original objects that have the following values:
+
+```
+a=1
+b=2
+c=3
+```
+
+(and for reference the part of the real-world objects is not a simple number like `1` but a unique digital number describing the part, like the binary code for it `110101011101011101010111101110111100001010101111011.....`). 
+
+
+With these numbers we could create endless amounts of equations:
+
+```
+1: a+b+c=6
+2: c-b-a=0
+3: b-c+a=0
+4: 2b+a-c=2
+5: 5c-b-a=12
+
+etc.
+
+```
+
+Mathematically we only need 3 to describe the content (value) of the fragments. But creating more adds reliability. Now store those equations distributed (one equation per physical storage device) and forget the original object. So we no longer have access to the values of a, b, c and we just remember the locations of all the equations created with the original data fragments. 
+
+Mathematically we need three equations (any 3 of the total) to recover the original values for a, b or c. So do a request to retrieve 3 of the many equations and the first 3 to arrive are good enough to recalculate the original values. Three randomly retrieved equations are:
+
+```
+5c-b-a=12
+b-c+a=0
+2b+a-c=2
+```
+And this is a mathematical system we could solve:
+
+- First: `b-c+a=0 -> b=c-a`
+- Second: `2b+a-c=2 -> c=2b+a-2 -> c=2(c-a)+a-2 -> c=2c-2a+a-2 -> c=a+2`
+- Third: `5c-b-a=12 -> 5(a+2)-(c-a)-a=12 -> 5a+10-(a+2)+a-a=12 -> 5a-a-2=2 -> 4a=4 -> a=1`
+
+Now that we know `a=1` we could solve the rest `c=a+2=3` and `b=c-a=2`. And we have from 3 random equations regenerated the original fragments and could now recreate the original object. 
+
+The redundancy and reliability in this system results from creating equations (more than needed) and storing them. As shown these equations in any random order can recreate the original fragments and therefore redundancy comes in at a much lower overhead.
+
+In our system we don't do this with 3 parts but with thousands.
+
+### Example of 16/4
+
+
+Each object is fragmented into 16 parts. So we have 16 original fragments for which we need 16 equations to mathematically describe them. Now let's make 20 equations and store them dispersedly on 20 devices. To recreate the original object we only need 16 equations. The first 16 that we find and collect allows us to recover the fragment and in the end the original object. We could lose any 4 of those original 20 equations.
+
+The likelihood of losing 4 independent, dispersed storage devices at the same time is very low. Since we have continuous monitoring of all of the stored equations, we could create additional equations immediately when one of them is missing, making it an auto-regeneration of lost data and a self-repairing storage system. 
+
+> The overhead in this example is 4 out of 20 which is a mere **20%** instead of **400%** .
+
+## Content Delivery
+
+This system can be used as backend for content delivery networks. 
+
+E.g. content distribution policy could be a 10/50 distribution which means, the content of a movie would be distributed over 60 locations from which we can lose 50 at the same time.
+
+If someone now wants to download the data, the first 10 locations to answer will provide enough of the data parts to rebuild the data.
+
+The overhead here is more, compared to previous example, but stil orders of magnitude lower compared to other CDN systems. 
+
+## The UnbreakableStorage System Can Avoid Datarot
+
+Datarot is the fact that data storage degrades over time and becomes unreadable e.g. on a harddisk.
+
+The storage system provided by ThreeFold intercepts this silent data corruption ensurinf that data does not rot. 
+
+> See also https://en.wikipedia.org/wiki/Data_degradation
+

docs/main/key_innovations_overview/storage/_category_.json

@@ -0,0 +1,7 @@
+{
+    "label": "Storage",
+    "position": 7,
+    "link": {
+      "type": "generated-index",
+    }
+  }

docs/main/key_innovations_overview/storage/fungistor_innovation.md

@@ -0,0 +1,32 @@
+---
+title: FungiStor
+sidebar_position: 19
+---
+
+## FungiStor (H2 2025)
+
+### The Problem
+
+Existing blockchain, internet, and P2P content delivery and storage systems suffer from sluggish performance and are too expensive. Content retrieval is often slow, and the overhead for ensuring redundancy is excessive. We require innovative approaches to facilitate efficient information sharing among users.
+
+Content delivery frequently represents the most significant expense for social networks. Running a basic social video network for 10 million users currently costs approximately $2 million per month using traditional cloud providers. We have the potential to reduce this cost by several orders of magnitude.
+
+
+### Introducing FungiStor
+
+FungiStor is a peer-to-peer (P2P) content delivery layer designed to store and distribute an extensive range of objects, including images, videos, files, and more. It has the capability to handle trillions of objects and files efficiently. FungiStor serves as an excellent solution for content delivery networks (CDNs), significantly reducing costs for organizations seeking to stream or deliver substantial data volumes to their user base.
+
+
+### The Benefits
+
+- **Global scalability, sub-50ms lookups:** FungiStor scales worldwide with ultra-fast data retrieval under 50 milliseconds.
+- **Localized content delivery:** prioritizes local data access for optimized speed and efficiency.
+- **Quantum-Safe security:** incorporates robust quantum security measures.
+- **Interoperability:** works seamlessly with IPFS, Torrent, and more.
+- **Cost efficiency:** offers significant cost savings, potentially 10 to 100 times less than conventional solutions.
+
+### Status
+
+Planned for the end of 2024
+
+Remark, FungiStor will act as the backend infrastructure for the Flists within our own system. It is versatile and can be utilized by anyone in need of a global-level content delivery system for files, objects, and images.

docs/main/key_innovations_overview/storage/nft_storage.md

@@ -0,0 +1,89 @@
+---
+sidebar_position: 18
+title: NFT Storage
+---
+
+# Unbreakable Storage System for NFT
+
+Our technology enables Unbreakable storage for NFT.
+
+The owner of the NFT can upload the data using one of our supported interfaces:
+
+- Http upload (everything possible on https://nft.storage/ is also possible on our system)
+- Filesystem
+
+Anyone in the world can retrieve the NFT (if allowed) and the data will be verified when doing so. The data is available anywhere in the world using multiple interfaces again (IPFS, HTTP(S) etc.). Caching happens on a global level. No special software or account on ThreeFold is needed to do this.
+
+The NFT system operates on top of a very reliable storage system which is sustainable for the planet and ultra secure and private. The NFT owner also owns the data.
+
+
+## The Benefits
+
+#### Persistence = Owned by the data user, as represented by their associated 3Bot
+
+
+The system is not based on a shared-all architecture.
+
+Whoever stores the data has full control over:
+
+- Where data is stored (specific locations)
+- The redundancy policy which is used
+- How long the data is kept
+- CDN policy (where the data is available and for how long)
+
+
+#### Reliability
+
+- Data cannot be corrupted
+- Data cannot be lost
+- Each time data is fetched back the hash (fingerprint) is checked. If there are any issues then autorecovery occurs
+- All data is encrypted and compressed (unique per storage owner)
+- Data owner chooses the level of redundancy
+
+#### Lookup
+
+- Multi URL & storage network support (see more in the interfaces section)
+- IPFS, HyperDrive URL schema
+- Unique DNS schema (with long key which is globally unique)
+
+#### CDN Support
+
+Each file (movie, image etc.) stored is available in many locations worldwide.
+
+Each file gets a unique url pointing to the data which can be retrieved from all these locations.
+
+Caching happens at each endpoint.
+
+#### Self Healing & Auto Correcting Storage Interface
+
+Any corruption e.g. bitrot gets automatically detected and corrected.
+
+In case of a HD crash or storage node crash the data will automatically be expanded again to fit the chosen redundancy policy.
+
+#### The Storage Algoritm Uses Unbreakable Storage System As Its Base
+
+Not even a quantum computer can hack data stored on our QSSS.
+
+The QSSS is a super innovative storage system which works on planetary scale and has many benefits compared to shared and/or replicated storage systems.
+
+It uses forward looking error correcting codes inside.
+
+#### Green
+
+Storage uses upto 10x less energy compared to classic replicated system.
+
+#### Multi Interface
+
+The stored data is available over multiple interfaces at once.
+
+- Interfaces
+  - IPFS
+  - HTTP(S) on top of 3Bot
+  - Syncthing
+  - Filesystem
+
+This allows ultimate flexibility from the end user perspective.
+
+The object (video, image etc.) can easily be embedded in any website or other representation which supports http.
+
+

docs/main/key_innovations_overview/storage/qss_zero_knowledge_proof.md

@@ -0,0 +1,13 @@
+---
+sidebar_position: 3
+title: Zero Knowledge Proof
+---
+
+# Zero Knowledge Proof Storage System
+
+The Unbreakable Storage System is zero knowledge proof compliant. The storage system is made up of / split into 2 components: the actual storage devices use to store the data (ZDB's) and the Unbreakable Storage engine. 
+
+
+The zero proof knowledge compliancy comes from the fact that all of the physical storage nodes (3Nodes) can prove that they store a valid part of the data that the Unbreakable storage engine (QSSE) has stored on multiple independent devices. The QSSE can validate that all of the QSSE storage devices have a valid part of the original information. The storage devices however have no idea what the original stored data is as they only have a part (description) of the original data and have no access to the original data part or the complete original data objects.
+
+