manual added knowledge_base

collections/manual/knowledge_base/technology/primitives/compute/beyond_containers.md

@@ -0,0 +1,17 @@
+## Beyond Containers
+
+![](img/container_native.jpg)
+
+
+Default features:
+
+- compatible with Docker
+- compatible with any Linux workload
+
+We have following unique advantages:
+
+- no need to work with images, we work with our unique zos_fs.
+- every container runs in a dedicated virtual machine providing more security.
+- the containers talk to each other over a private network: zos_net.
+- the containers can use web_gw to allow users on the internet connect to the applications as running in their secure containers. 
+- can use core-x to manage the workload.

collections/manual/knowledge_base/technology/primitives/compute/compute_toc.md

@@ -0,0 +1,7 @@
+# Compute
+
+<h2>Table of Contents</h2>
+
+- [ZKube](./zkube.md)
+- [ZMachine](./zmachine.md)
+- [CoreX](./corex.md)

collections/manual/knowledge_base/technology/primitives/compute/corex.md

@@ -0,0 +1,21 @@
+
+<h1> CoreX </h1>
+
+<h2>Table of Contents </h2>
+
+- [Introduction](#introduction)
+- [ZMachine Process Manager](#zmachine-process-manager)
+
+***
+
+## Introduction
+
+CoreX allows you to manage your ZMachine over web remotely.
+
+## ZMachine Process Manager
+
+- Provide a web interface and a REST API to control your processes.
+- Allow to watch the logs of your processes.
+- Or use it as a web terminal (access over https to your terminal)!
+
+![](img/corex.jpg)

collections/manual/knowledge_base/technology/primitives/compute/tfgrid_compute.md

@@ -0,0 +1,25 @@
+
+## TFGrid Compute Layer
+
+![](img/tfgrid_compute_.jpg)
+
+We are more than just Container or VM technology, see [our Beyond Container Document](beyond_containers).
+
+A 3Node is a Zero-OS enabled computer which is hosted with any of the TF_Farmers.
+
+There are 4 storage mechanisms which can be used to store your data:
+
+- ZOS_FS is our dedupe unique filesystem, replaces docker images. 
+- ZOS_Mount is a mounted disk location on SSD, this can be used as faster storage location. 
+- QSFS, this is a super unique storage system, data can never be lost or corrupted. Please be reminded that this storage layer is only meant to be used for secondary storage applications.
+- ZOS_Disk, a virtual disk technology, only for TFTech OEM partners.
+
+There are 4 ways how networks can be connected to a Z-Machine.
+
+- Planetary_network : is a planetary scalable network, we have clients for windows, osx, android and iphone.
+- zos_net : is a fast end2end encrypted network technology, keep your traffic between your z_machines 100% private.
+- zos_bridge: connection to a public ipaddress
+- web_gw: web gateway, a secure way to allow internet traffic reach your secure Z-Machine.
+
+
+

collections/manual/knowledge_base/technology/primitives/compute/zkube.md

@@ -0,0 +1,40 @@
+<h1> ZKube </h1>
+
+<h2>Table of Contents </h2>
+
+- [Introduction](#introduction)
+- [Unique for our Kubernetes implementation](#unique-for-our-kubernetes-implementation)
+- [Features](#features)
+- [ZMachine Benefits](#zmachine-benefits)
+- [Architecture](#architecture)
+
+***
+
+## Introduction
+
+TFGrid is compatible with Kubernetes Technology.
+
+Each eVDC as shown above is a full blown Kubernetes deployment.
+
+## Unique for our Kubernetes implementation
+
+- The Kubernetes networks are on top of our [ZNet](../network/znet.md) technology which means all traffic between containers and kubernetes hosts is end2end encrypted independent of where your Kubernetes nodes are deployed.
+- You can mount a QSFS underneath a Kubernetes Node (VM), which means that you can deploy containers on top of QSFS to host unlimited amounts of storage in a super safe way.
+- You Kubernetes environment is for sure 100% decentralized, you define where you want to deploy your Kubernetes nodes and only you have access to the deployed workloads on the TFGrid.
+
+## Features
+
+*   integration with znet (efficient, secure encrypted network between the zmachines)
+*   can be easily deployed at the edge
+*   single-tenant!
+
+## ZMachine Benefits
+
+*   [ZOS Protect](../../zos/benefits/zos_advantages.md#zero-os-protect): no hacking surface to the Zero-Nodes, integrate silicon route of trust
+
+
+![](img/kubernetes_0_.jpg)
+
+## Architecture
+
+![](img/zkube_architecture_.jpg)

collections/manual/knowledge_base/technology/primitives/compute/zmachine.md

@@ -0,0 +1,30 @@
+<h1> ZMachine </h1>
+
+<h2>Table of Contents </h2>
+
+- [Introduction](#introduction)
+- [Features](#features)
+- [Architecture](#architecture)
+
+***
+
+## Introduction
+
+ZMachine is a unified container/virtual machine type. This can be used to start a virtual machine on a zos node.
+
+## Features
+
+*   import from docker (market std for containers)
+*   can be easily deployed at the edge (edge cloud)
+*   single-tenant, fully decentralized!
+*   can deploy unlimited amounts of storage using our qsfs.
+*   [ZOS Protect](../../zos/benefits/zos_advantages.md#zero-os-protect): no hacking surface to the Zero-Nodes, integrate silicon route of trust
+*   [ZOS Filesystem](../storage/qsfs.md): dedupe, zero-install, hacker-proof
+*   [WebGateway](../network/webgw3.md:) intelligent connection between web (internet) and container services
+*   integration with [ZNet](../network/znet.md) (efficient, secure encrypted network between the zmachines)
+
+## Architecture
+
+![](img/zmachine_zos_.jpg)
+
+A ZMachine is running as a virtual machine on top of Zero-OS.

collections/manual/knowledge_base/technology/primitives/network/network_toc.md

@@ -0,0 +1,7 @@
+# Network
+
+<h2>Table of Contents</h2>
+
+- [ZNET](./znet.md)
+- [ZNIC](./znic.md)
+- [WebGateway](./webgw3.md)

collections/manual/knowledge_base/technology/primitives/network/p2pagent.md

@@ -0,0 +1,5 @@
+# Peer2Peer Agent
+
+>TODO
+
+!!!include:zos_toc

collections/manual/knowledge_base/technology/primitives/network/planetary_network.md

@@ -0,0 +1,54 @@
+![](img/planetary_lan.jpg)
+
+# Planetary Network
+
+![](img/planet_net_.jpg)
+
+
+The planetary network is an overlay network which lives on top of the existing internet or other peer2peer networks created. In this network, everyone is connected to everyone. End-to-end encryption between users of an app and the app running behind the network wall.
+
+Each user end network point is strongly authenticated and uniquely identified, independent of the network carrier used. There is no need for a centralized firewall or VPN solutions, as there is a circle based networking security in place.
+
+Benefits :
+- It finds shortest possible paths between peers
+- There's full security through end-to-end encrypted messaging
+- It allows for peer2peer links like meshed wireless
+- It can survive broken internet links and re-route when needed
+- It resolves the shortage of IPV4 addresses
+
+
+Whereas current computer networks depend heavily on very centralized design and configuration, this networking concept breaks this mould by making use of a global spanning tree to form a scalable IPv6 encrypted mesh network.  This is a peer2peer implementation of a networking protocol.
+
+The following table illustrates high-level differences between traditional networks like the internet, and the planetary threefold network:
+
+| Characteristic                                                  | Traditional | Planetary Network |
+| --------------------------------------------------------------- | ----------- | ----------------- |
+| End-to-end encryption for all traffic across the network        | No          | Yes               |
+| Decentralized routing information shared using a DHT            | No          | Yes               |
+| Cryptographically-bound IPv6 addresses                          | No          | Yes               |
+| Node is aware of its relative location to other nodes           | No          | Yes               |
+| IPv6 address remains with the device even if moved              | No          | Yes               |
+| Topology extends gracefully across different mediums, i.e. mesh | No          | Yes               |
+
+## What are the problems solved here?
+
+The internet as we know it today doesn’t conform to a well-defined topology. This has largely happened over time - as the internet has grown, more and more networks have been “bolted together”. The lack of defined topology gives us some unavoidable problems:
+
+- The routing tables that hold a “map” of the internet are huge and inefficient
+- There isn’t really any way for a computer to know where it is located on the internet relative to anything else
+- It’s difficult to examine where a packet will go on its journey from source to destination without actually sending it
+- It’s very difficult to install reliable networks into locations that change often or are non-static, i.e. wireless mesh networks
+
+These problems have been partially mitigated (but not really solved) through centralization - rather than your computers at home holding a copy of the global routing table, your ISP does it for you. Your computers and network devices are configured just to “send it upstream” and to let your ISP decide where it goes from there, but this does leave you entirely at the mercy of your ISP who can redirect your traffic anywhere they like and to inspect, manipulate or intercept it.
+
+In addition, wireless meshing requires you to know a lot about the network around you, which would not typically be the case when you have outsourced this knowledge to your ISP. Many existing wireless mesh routing schemes are not scalable or efficient, and do not bridge well with existing networks.
+
+![](img/planetary_net.jpg)
+
+The planetary network is a continuation & implementation of the [Yggdrasil](https://yggdrasil-network.github.io/about.html) network initiative. This technology is in beta but has been proven to work already quite well.
+
+!!!def alias:planet_net,planetary_net,planetary_network,pan
+
+!!!include:zos_toc
+
+> Click [here](manual:planetary_network_connector) to read more about Planetary Network Connector Installation. Click [here](manual:yggdrasil_client) to read more about Planetary Network Installation (advanced). 

collections/manual/knowledge_base/technology/primitives/network/tfgrid_network.md

@@ -0,0 +1,7 @@
+# TFGrid networking
+
+- znet : private network between zmachines
+- [Planetary Network](planetary_network) : peer2peer end2end encrypted global network
+- znic : interface to planetary network
+- [WebGateway](webgw) : interface between internet and znet
+

collections/manual/knowledge_base/technology/primitives/network/webgw.md

@@ -0,0 +1,60 @@
+
+
+# WebGW 2.0
+
+The Web Gateway is a mechanism to connect the private networks to the open Internet, in such a way that there is no direct connection between internet and the secure workloads running in the ZMachines.
+
+![](img/webgateway.jpg)
+
+
+- Separation between where compute workloads are and where services are exposed.
+- Better Security
+- Redundant
+    - Each app can be exposed on multiple webgateways at once.
+- Support for many interfaces...
+- Helps resolve shortage of IPv4 addresses
+
+If (parts of) this private overlay network need to be reachable from the Internet, the zmachines initiate a secure connection *to* the web Gateway. 
+
+### Implementation
+
+It is important to mention that this connection is not a standard network connection, it is a [network socket](https://en.wikipedia.org/wiki/Network_socket) initiated by the container or VM to the web gateway. The container calls out to one or more web gateways and sets up a secure & private socket connection to the web gateway. The type of connection required is defined on the smart contract for IT layer and as such is very secure. There is no IP (TCP/UDP) coming from the internet towards the containers providing more security. 
+
+Up to the Web Gateway Internet traffic follows the same route as for any other network end point: A DNS entry tells the consumers client to what IP address to send traffic to.  This endpoint is the public interface of the Web Gateway.  That interface accepts the HTTP(s) (or any other TCP) packets and forward the packet payload  over the secure socket connection (initiated by the container) to the container.  
+
+No open pipe (NAT plus port forwarding) from the public internet to specific containers in the private (overlay) network exists.
+
+Web Gateways are created by so called network farmers.  Network farmers are people and companies that have access to good connectivity and have a large number of public IP routable IP networks.  They provide the facilities (hardware) for Web Gateways to run and terminate a lot of the public inbound and output traffic for the TF Grid.  Examples of network farmers are ISP's and (regional, national and international Telcos, internet exchanges etc.
+
+### Security
+
+Buy not providing an open and direct path in to the private network a lot of malicious phishing and hacking attempts are stopped at the Web Gateway.  By design any private network is meant to have multiple webgateways and by design these Web Gateways exist on different infrastructure in a different location.  Sniffing around and finding  out what can be done with a Web Gateway might (and will happen) but it will not compromise the containers in your private network.
+
+### Redundant Network Connection
+
+![](img/redundant_net.jpg)
+
+
+### Unlimited Scale
+
+![](img/webgw_scaling.jpg)
+
+
+The network architecture is a pure scale-out network system, it can scale to unlimited size, there is simply no bottleneck. Network "supply" is created by network farmers, and network "demand" is done by TF Grid users.  Supply and demand scale independently, for supply there can be unlimited network farmers providing the web gateways on their own 3nodes and unlimited compute farmers providing 3nodes for compute and storage. The demand side is driven by developers creating software that runs on the grid, system integrators creating solutions for enterprise and this demand side is exponentially growing for data processing and storage use cases.
+
+### Network Wall (future)
+
+see [Network Wall](network_wall)
+    
+## Roadmap 
+
+Above described Web Gateway is for 2.0.
+
+For 3.0 we start with a HTTP(S) proxy over Planetary network connection. Not all features from WebGW 2.0 have been ported.
+
+Further future, we envisage support for many other protocols: sql, redis, udp,  ...
+
+!!!def alias:web_gw,zos_web_gateway
+
+!!!include:zos_toc
+

collections/manual/knowledge_base/technology/primitives/network/webgw3.md

@@ -0,0 +1,50 @@
+<h1> WebGW 2.0 </h1>
+
+<h2>Table of Contents</h2>
+
+- [Introduction](#introduction)
+- [Implementation](#implementation)
+- [Security](#security)
+- [Redundant Network Connection](#redundant-network-connection)
+- [Unlimited Scale](#unlimited-scale)
+
+***
+
+## Introduction
+
+The Web Gateway is a mechanism to connect the private networks to the open Internet, in such a way that there is no direct connection between internet and the secure workloads running in the ZMachines.
+
+![](img/webgateway.jpg)
+
+
+- Separation between where compute workloads are and where services are exposed.
+- Redundant
+    - Each app can be exposed on multiple webgateways at once.
+- Support for many interfaces...
+- Helps resolve shortage of IPv4 addresses
+
+## Implementation
+
+Some 3nodes supports gateway functionality (configured by the farmers). A 3node with gateway config can then accept gateway workloads and then forward traffic to ZMachines that only has yggdrasil (planetary network) or Ipv6 addresses.
+
+The gateway workloads consists of a name (prefix) that need to be reserved on the block chain first. Then the list of backend IPs. There are other flags that can be set to control automatic TLS (please check terraform documentations for the exact details of a reservation)
+
+Once the 3node receives this workloads, the network configure proxy for this name and the yggdrasil ips.
+
+## Security
+
+ZMachines has to have an yggdrasil IP or any other IPv6 (also IPv4 are accepted) but it means that any person who is connected to the yggdrasil network, can also reach the ZMachine without the need for a proxy.
+
+So ti's up to the ZMachine owner/maintainer to make sure it is secured and only have the required ports open.
+
+## Redundant Network Connection
+
+![](img/redundant_net.jpg)
+
+
+## Unlimited Scale
+
+![](img/webgw_scaling.jpg)
+
+
+The network architecture is a pure scale-out network system, it can scale to unlimited size, there is simply no bottleneck. Network "supply" is created by network farmers, and network "demand" is done by TF Grid users.  Supply and demand scale independently, for supply there can be unlimited network farmers providing the web gateways on their own 3nodes and unlimited compute farmers providing 3nodes for compute and storage. The demand side is driven by developers creating software that runs on the grid, system integrators creating solutions for enterprise and this demand side is exponentially growing for data processing and storage use cases.

collections/manual/knowledge_base/technology/primitives/network/znet.md

@@ -0,0 +1,39 @@
+<h1> ZNET </h1>
+
+<h2>Table of Contents</h2>
+
+- [Introduction](#introduction)
+- [Secure mesh overlay network (peer2peer)](#secure-mesh-overlay-network-peer2peer)
+- [Redundancy](#redundancy)
+- [Interfaces in Zero-OS](#interfaces-in-zero-os)
+
+***
+
+## Introduction
+
+Decentralized networking platform allowing any compute and storage workload  to be connected together on a private (overlay) network and exposed to the existing internet network. The Peer2Peer network platform allows any workload  to be connected over secure encrypted networks which will look for the shortest path between the nodes.
+
+![](img/zos_network_overlay.jpg)
+
+## Secure mesh overlay network (peer2peer)
+
+Z_NET is the foundation of any architecture running on the TF Grid.  It can be seen as a virtual private datacenter and the network allows all of the *N* containers to connected to all of the *(N-1)* other containers. Any network connection is a secure network connection between your containers and creates  peer 2 peer network between containers. 
+
+![](img/overlay_net1.jpg)
+
+No connection is made with the internet.The ZNet is a single tenant network and by default not connected to the public internet.  Everything stays private. For connecting to the public internet a Web Gateway is included in the product to allows for public access if and when required.
+
+## Redundancy
+
+As integrated with [WebGW](./webgw3.md):
+
+![](img/znet_redundancy.jpg)
+
+- Any app can get (securely) connected to the internet by any chosen IP address made available by ThreeFold network farmers through WebGW.
+- An app can be connected to multiple web gateways at once, the DNS round robin principle will provide load balancing and redundancy.
+- An easy clustering mechanism where web gateways and nodes can be lost and the public service will still be up and running.
+- Easy maintenance. When containers are moved or re-created the same end user connection can be reused as that connection is terminated on the Web Gateway. The moved or newly created Web Gateway will recreate the socket to the Web Gateway and receive inbound traffic.
+
+## Interfaces in Zero-OS
+
+![](img/znet_znic1.jpg)

collections/manual/knowledge_base/technology/primitives/network/znic.md

@@ -0,0 +1,24 @@
+<h1> ZNIC </h1>
+
+<h2>Table of Contents</h2>
+
+- [Introduction](#introduction)
+- [Use Cases](#use-cases)
+- [Overview](#overview)
+
+***
+
+## Introduction
+
+ZNIC is the network interface which is connected to Z_Machine.
+
+## Use Cases
+
+Can be implemented as interface to
+
+- planetary_network.
+- public ip address on a Zero-OS.
+
+## Overview
+
+![](img/znet_znic.jpg)

collections/manual/knowledge_base/technology/primitives/primitives_toc.md

@@ -0,0 +1,19 @@
+# Primitives
+
+<h2>Table of Contents</h2>
+
+- [Compute](./compute/compute_toc.md)
+  - [ZKube](./compute/zkube.md)
+  - [ZMachine](./compute/zmachine.md)
+  - [CoreX](./compute/corex.md)
+- [Storage](./storage/storage_toc.md)
+  - [ZOS Filesystem](./storage/zos_fs.md)
+  - [ZOS Mount](./storage/zmount.md)
+  - [Quantum Safe File System](./storage/qsfs.md)
+  - [Zero-DB](./storage/zdb.md)
+  - [Zero-Disk](./storage/zdisk.md)
+- [Network](./network/network_toc.md)
+  - [ZNET](./network/znet.md)
+  - [ZNIC](./network/znic.md)
+  - [WebGateway](./network/webgw3.md)
+- [Zero-OS Advantages](../zos/benefits/zos_advantages.md)

collections/manual/knowledge_base/technology/primitives/storage/qsfs.md

@@ -0,0 +1,34 @@
+<h1> Quantum Safe Filesystem </h1>
+
+<h2>Table of Contents</h2>
+
+- [Introduction](#introduction)
+  - [Benefits](#benefits)
+  - [Use Cases](#use-cases)
+  - [Implementation](#implementation)
+ 
+***
+
+## Introduction
+
+Quantum safe filesystem presents itself as a filesystem to the ZMachine.
+
+### Benefits
+
+- Safe
+- Hacker Proof
+- Ultra Reliable
+- Low Overhead
+- Ultra Scalable
+- Self Healing = recovers service automatically in the event of outage with no human 
+
+![](img/zos_zstor.jpg)
+
+### Use Cases
+
+- Backup and archive system
+- Blockchain Storage Backend (OEM ONLY)
+
+### Implementation
+
+> QSFS is using QSSS inside.

collections/manual/knowledge_base/technology/primitives/storage/storage_toc.md

@@ -0,0 +1,9 @@
+# Storage
+
+<h2>Table of Contents</h2>
+
+- [ZOS Filesystem](./zos_fs.md)
+- [ZOS Mount](./zmount.md)
+- [Quantum Safe File System](./qsfs.md)
+- [Zero-DB](./zdb.md)
+- [Zero-Disk](./zdisk.md)

collections/manual/knowledge_base/technology/primitives/storage/zdb.md

@@ -0,0 +1,21 @@
+<h1> ZOS-DB (ZDB) </h1>
+
+<h2>Table of Contents</h2>
+
+- [Introduction](#introduction)
+- [Use Cases](#use-cases)
+- [Overview](#overview)
+
+***
+
+## Introduction
+
+0-db is a fast and efficient key-value store redis-protocol compatible, which makes data persistent inside an always append datafile, with namespaces support.
+
+## Use Cases
+
+> ZDB is being used as backend storage for [Quantum Safe Filesystem](./qsfs.md).
+
+## Overview
+
+![](img/zdb_arch.jpg)

collections/manual/knowledge_base/technology/primitives/storage/zdisk.md

@@ -0,0 +1,18 @@
+<h1> ZOS Disk </h1>
+
+<h2>Table of Contents</h2>
+
+- [Introduction](#introduction)
+- [Roadmap](#roadmap)
+
+***
+
+## Introduction
+
+Virtual disk creates the possibility to create and use virtual disks which can be attached to containers (and virtual machines).  
+
+The technology is designed to be redundant without having to do anything.
+
+## Roadmap
+
+- The virtual disk technology is  available for OEM's only, contact TF_Tech.

collections/manual/knowledge_base/technology/primitives/storage/zmount.md

@@ -0,0 +1,18 @@
+<h1> ZOS Mount </h1>
+
+<h2>Table of Contents</h2>
+
+- [Introduction](#introduction)
+- [Overview](#overview)
+
+***
+
+## Introduction
+
+ZOS mount is an SSD storage location on which can be written upon inside a VMachine and VKube.
+
+## Overview
+
+The SSD storage location is mounted on a chosen path inside your Z-Machine.
+
+![](img/zmount.jpg)

collections/manual/knowledge_base/technology/primitives/storage/zos_fs.md

@@ -0,0 +1,39 @@
+
+<h1> ZOS FileSystem (ZOS-FS) </h1>
+
+<h2>Table of Contents</h2>
+
+- [Introduction](#introduction)
+- [Uses Flist Inside](#uses-flist-inside)
+  - [Why this ZFlist Concept](#why-this-zflist-concept)
+- [Benefits](#benefits)
+
+***
+
+## Introduction
+
+A deduped filesystem which is more efficient compared to images as used in other Virtual Machine technology.
+
+## Uses Flist Inside
+
+In Zero-OS, `flist` is the format used to store zmachine images. This format is made to provide
+a complete mountable remote filesystem but downloading only the files contents that you actually needs.
+
+In practice, Flist itself is a small database which contains metadata about files and directories and file payload are stored on a tfgrid hub. You only need to download payload when you need it, this dramatically reduce zmachine boot time, bandwidth and disk overhead.
+
+### Why this ZFlist Concept
+
+Have you ever been in the following situation: you need two small files but they are embedded in a large archive.  How to get to those 2 files in an efficient way?  What a disappointment when you see this archive is 4 GB large and you only need 4 files of 2 MB inside. You'll need to download the full archive, store it somewhere to extract only what you need. Time, effort and bandwidth wasted.
+
+You want to start a Docker container and the base image you want to use is 2 GB. What do you need to do before being able to use your container ? Waiting to get the 2 GB downloaded.  This problem exists everywhere but in Europe and the US the bandwidth speeds are such that this does not present a real problem anymore, hence none of the leading (current) tech companies are looking for solutions for this.
+
+We believe that there should a smarter way of dealing with this then simply throwing larger bandwidth at the problem:  What if you could only download the files you actually want and not the full blob (archive, image, whatever...).
+
+ZFList is splitting metadata and data. Metadata is referential information about everything you need to know about content of the archive, but without the payload. Payload is the content of the referred files.  The ZFList is exactly that:  it consists of metadata with references that point to where to get the payload itself. So if you don't need it you won't get it.
+
+As soon as you have the flist mounted, you can see the full directory tree, and walk around it. The files are only downloaded and presented at moment that you try to access them. In other words, every time you want to read a file, or modify it, Zero FS will download it, so that the data is available too. You only download on-the-fly what you need which reduces dramatically the bandwidth requirement.
+
+
+## Benefits
+
+- Efficient usage of bandwidth makes this service perform with and without (much) bandwidth