lots of fixes and add ons

This commit is contained in:
2024-04-23 14:20:22 +02:00
parent b6889eec02
commit 7a2bb56a3b
83 changed files with 378 additions and 205 deletions

View File

@@ -1 +1,2 @@
qsss_overview.png
qss_fs_arch.png

View File

@@ -1,2 +1,5 @@
qss_scaleout.png
qsss_intro.png
qsss_intro2.png
qsfs_arch.png
storage_arch.png

Binary file not shown.

Before

Width:  |  Height:  |  Size: 170 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 262 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 801 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 286 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 233 KiB

View File

@@ -15,7 +15,11 @@ Unique features
Today we produce more data than ever before. We cannot continue to make full copies of data to make sure it is stored reliably. This will simply not scale. We need to move from securing the whole dataset to securing all the objects that make up a dataset.
ThreeFold is using space technology to store data fragments over multiple devices (physical storage devices in TFNodes). The solution does not distribute and store parts of an object (file, photo, movie etc.) but describes the part of an object. This can be visualized by thinking of it as equations.
We are using technology which was originally used for communication in space.
The algo stores data fragments over multiple devices (physical storage devices ).
The solution is not based on replication or sharding, the algo represents the data as equasions which are distributed over multiple locations.
## How Data Is Stored Today
@@ -32,9 +36,9 @@ In well optimized systems overhead will be 400% but in some it can be orders of
![alt text](qsss_overview.png)
ThreeFold has developed a new storage algorithm which is more efficient, ultra reliable and gives you full control over where your data is stored.
We have developed a new storage algorithm which is more efficient, ultra reliable and gives you full control over where your data is stored.
ThreeFold's approach is different. Let's try to visualize this new approach with a simple analogy using equations.
Our approach is different. Let's try to visualize this new approach with a simple analogy using equations.
Let a,b,c,d.... be the parts of the original object. You could create endless unique equations using these parts. A simple example: let's assume we have 3 parts of original objects that have the following values:

View File

@@ -1,6 +1,6 @@
# Quantum Safe Filesystem
![](img/qsss_intro.png)
![](qss_fs_arch.png)
A redundant filesystem, can store PB's (millions of gigabytes) of information.
@@ -8,7 +8,7 @@ Unique features:
- Unlimited scalability (many petabytes)
- Quantum Safe:
- On the TFGrid no farmer knows what the data is
- No farmer knows what the data is
- Even a quantum computer cannot decrypt the data
- Data can't be lost
- Protection for datarot, data will autorepair
@@ -16,20 +16,19 @@ Unique features:
- Data is dispersed over multiple sites
- Even if the sites go down the data will not be lost
- Up to 10x more efficient than storing on classic storage cloud systems
- Can be mounted as filesystem on any OS or any deployment system (OSX, Linux, Windows, Docker, Kubernetes, TFGrid etc.)
- Can be mounted as filesystem on any OS or any deployment system (OSX, Linux, Windows, Docker, Kubernetes etc.)
- Compatible with ± all data workloads (not high performance data driven workloads like a database)
- Self-healing: when a node or disk is lost, the storage system can get back to the original redundancy level
- Helps with compliance for regulations like GDPR (as the hosting facility has no view on what is stored: information is encrypted and incomplete)
- Hybrid: can be installed onsite, public and private
- Read-write caching on encoding node (the front end)
![](img/planet_fs.jpg)
## Mount Any Files In Your Storage Infrastructure
The QSFS is a mechanism to mount any file system (in any format) on the grid, in a quantum secure way.
This storage layer relies on 3 primitives of the ThreeFold technology :
This storage layer relies on 3 primitives:
- [0-db](https://github.com/threefoldtech/0-db) is the storage engine.
It is an always append database, which stores objects in an immutable format. It allows history to be kept out-of-the-box, good performance on disk, low overhead, easy data structure and easy backup (linear copy and immutable files).
@@ -42,7 +41,7 @@ to multiple 0-DBs. The number of generated chunks is configurable to make it mor
Together they form a storage layer that is quantum secure: even the most powerful computer can't hack the system because no single node contains all of the information needed to reconstruct the data.
![](img/quantum_safe_storage.jpg)
This concept scales forever, and you can bring any file system on top of it:
- S3 storage
@@ -50,30 +49,13 @@ This concept scales forever, and you can bring any file system on top of it:
- an ftp-server
- IPFS and Hypercore distributed file sharing protocols
![](img/quantum_safe_storage_scale.jpg)
## Architecture
By using our filesystem inside a Virtual Machine or Kubernetes, the TFGrid user can deploy any storage application on top e.g. Minio for S3 storage, OwnCloud as online fileserver.
By using our filesystem inside a Virtual Machine or Kubernetes, the cloud user can deploy any storage application on top e.g. Minio for S3 storage, OwnCloud as online fileserver.
![](img/qsstorage_architecture.jpg)
Any storage workload can be deployed on top of the zstor.
```mermaid
graph TD
subgraph Data Ingress and Egress
qss[Quantum Safe Storage Engine]
end
subgraph Physical Data storage
st1[Virtual Storage Device 1]
st2[Virtual Storage Device 2]
st3[Virtual Storage Device 3]
st4[Virtual Storage Device 4]
st5[Virtual Storage Device 5]
st6[...]
qss -.-> st1 & st2 & st3 & st4 & st5 & st6
end
```

Binary file not shown.

After

Width:  |  Height:  |  Size: 244 KiB

View File

@@ -6,6 +6,6 @@ The Quantum Safe Storage System is zero knowledge proof compliant. The storage s
![](img/qss_system.jpg)
The zero proof knowledge compliancy comes from the fact that all of the physical storage nodes (TFnodes) can prove that they store a valid part of the data that the quantum safe storage engine (QSSE) has stored on multiple independent devices. The QSSE can validate that all of the QSSE storage devices have a valid part of the original information. The storage devices however have no idea what the original stored data is as they only have a part (description) of the original data and have no access to the original data part or the complete original data objects.
The zero proof knowledge compliancy comes from the fact that all of the physical storage nodes (3Nodes) can prove that they store a valid part of the data that the quantum safe storage engine (QSSE) has stored on multiple independent devices. The QSSE can validate that all of the QSSE storage devices have a valid part of the original information. The storage devices however have no idea what the original stored data is as they only have a part (description) of the original data and have no access to the original data part or the complete original data objects.

View File

@@ -2,13 +2,29 @@
# Quantum Safe Storage System
i![](img/qsss_intro.png)
![](img/qsss_intro2.png)
Our storage architecture follows the true peer2peer design of the TF grid. Any participating node only stores small incomplete parts of objects (files, photos, movies, databases etc.) by offering a slice of the present (local) storage devices. Managing the storage and retrieval of all of these distributed fragments is done by a software that creates development or end-user interfaces for this storage algorithm. We call this '**dispersed storage**'.
Our storage architecture follows the true peer2peer design of the Decentralized Cloud System. Any participating node only stores small incomplete parts of objects (files, photos, movies, databases etc.) by offering a slice of the present (local) storage devices. Managing the storage and retrieval of all of these distributed fragments is done by a software that creates development or end-user interfaces for this storage algorithm. We call this '**dispersed storage**'.
![](img/qsss_intro_0_.jpg)
## Benefits
- Not even a quantum computer can hack
- Zetabytes can be stored as easily as Petabytes
- The system is truly autonomous & self healing
- Datarot is detected and fixed.
- There is 100% contorl over where data is (GDPR)
## Architecture
![](img/storage_arch.png)
The cloud user can mix and match storage technologies as are required for their application.
## Peer2Peer Advantages
Peer2peer provides the unique proposition of selecting storage providers that match your application and service of business criteria. For example, you might be looking to store data for your application in a certain geographic area (for governance and compliance) reasons. You might also want to use different "storage policies" for different types of data. Examples are live versus archived data. All of these uses cases are possible with this storage architecture, and could be built by using the same building blocks produced by farmers and consumed by developers or end-users.
> There is 100% control over where the data is positioned and the security is incredible.