info_tfgrid/collections/tech/key_innovations.md

Key Innovations

Mycelium: a new network layer for the internet

Why?

The current centralized state of the internet poses significant security risks, with compromised routers and growing cyber threats (trillions of USD per year now), making everyone vulnerable to hacking. Industry responses involve disabling original features, hindering true peer-to-peer connectivity and personal server capabilities. Workarounds and system hacks have become the norm.

Our Internet is seriously broken, we need new ways how to communicate

What?

Mycelium is an overlay network layer designed to enhance the existing internet infrastructure while remaining compatible with all current applications. It empowers true peer-to-peer communication. By installing a Network Agent on your device, you gain the ability to securely connect with any other participant on this network. Mycelium intelligently reroutes traffic to maintain connectivity taking location of you and your peer into consideration.

Benefits?

• Continuous Connectivity: Mycelium ensures uninterrupted connectivity by dynamically rerouting traffic through available connections (friends, satellites, 4/5G, fiber).
• End-to-End Encryption: Robust encryption stops man-in-the-middle attacks, guaranteeing secure communication.
• Proof of authenticity (POA): make sure we know who we communicate with
• Optimized Routing: Mycelium finds the shortest path between network participants, reducing latency and keeping traffic localized.
• Universal Server Capability: Empowers individuals to act as servers, a foundational element for any peer-to-peer system.
• Full Compatibility: Mycelium seamlessly integrates with the current internet, supporting any application.
• Impressive Speed: Achieves 1 Gbps per Network Agent, ensuring rapid data transfer.

Status?

• In beta and usable from TFGrid 3.13, its our 3e generation approach to networking and took us years to do. We are looking forward to your feedback.

Zero-OS: a minimalistic more efficient server operating system

Why?

Current Linux-based operating systems are challenging to use safely and efficiently on the edges of the Internet. They require central management, involve excessive complexity, and prove difficult to update and maintain, resulting in numerous security vulnerabilities. To revolutionize the internet, we must rethink how we host our applications, essentially reinventing the concept of a cloud-based operating system.

What?

An innovative operating system, retaining the Linux kernel but fundamentally redesigned for heightened security, efficiency, and support for true peer-to-peer and decentralized workloads.

This unique operating system doesn't require installation on hard disks or SSDs; it remains stateless, consistently up to date, and autonomously managed without human intervention. Its design enables self-healing application delivery, opening new possibilities in system resilience and reliability.

Benefits?

• Compatibility with Existing Workloads: Our primary goal is to ensure Zero-OS compatibility with over 99% of the workloads commonly hosted in centralized cloud environments today. This includes support for Docker containers, virtual machines, Kubernetes, and more.
• Reduced Attack Surface: Zero-OS boasts a smaller hacking surface, enhancing security by minimizing potential vulnerabilities.
• Stateless Design: Its statelessness simplifies deployment and updates, making it easier to maintain while ensuring it's always up to date.
• Autonomous Operation: Whether you have one instance or a billion, Zero-OS operates autonomously, streamlining management and maintaining consistency across all instances.
• Rapid Deployment: With Zero-OS, you can deploy 1000 virtual machines in just 2 minutes, ensuring agility and efficiency in scaling up resources.
• Unique Security Features: Zero-OS offers support for distinctive security features to enhance protection and fortify your infrastructure.
• Lower Cost, Easier to Maintain: Zero-OS significantly reduces the operational expenses associated with cloud infrastructure by automating most processes. This results in minimal operational costs and eliminates the need for extensive engineering efforts.
• Ready for a Decentralized World: Zero-OS empowers individuals to become hosts for required Internet capacity (storage, network, compute, gpu), allowing them to be rewarded for providing computing resources and internet connectivity. This aligns with the vision of a truly decentralized and distributed computing ecosystem.

Status?

• 3e generation, active for +4 years, tested by thousands
• Play with it yourself see www.threefold.io

FList: a new way how to deal with OS Images

Why?

The current method of deploying workloads in the cloud using Docker containers and virtual machine images has inherent issues. These images consume significant storage space, result in slow and bandwidth-intensive transfers to the internet's edge, drive up costs, introduce complexity, and pose security risks due to difficulties in tracking their contents over time.

For instance, a complete Ubuntu image can easily be 2 GB in size, comprising millions of files. In contrast, the Flist for a full Ubuntu image is less than 2 MB (1000 times smaller), containing only the necessary files required to launch an application.

What?

A new image format that separates the image data (comprising files and subfile parts) from the metadata describing the image structure.

An Flists format uniquely encompasses comprehensive file descriptions along with all relevant metadata such as size, modification and creation timestamps, and POSIX attributes. Additionally, it incorporates a fingerprint for each component, ensuring deterministic behavior—a crucial feature for security-focused use cases.

Flists provide the flexibility to manage metadata and data as separate entities, offering a versatile approach to handling various build and delivery scenarios.

Benefits

• Rapid Deployment: Zero-OS enables containers and virtual machines to launch up to 100 times faster, especially in decentralized scenarios.
• Enhanced Security: Zero-OS prevents tampering with images, ensuring higher security levels.
• Reduced Storage and Bandwidth: Zero-OS significantly reduces storage and bandwidth requirements, potentially achieving up to a 100-fold improvement.
• Deterministic Deployments: Engineers can precisely define deployments beforehand, ensuring predictable outcomes without changes during deployment.
• 100% compatible: with existing standards, docker, virtual machines... The same format is useful for VM's as well as any container technology.

Status?

• Usable for years, see Zero-OS.

Zero-Stor : a quantum safe backend storage system.

Why?

Traditional backend storage systems have their roots in centralized environments, focusing on low-latency and closed security setups. However, these characteristics make them less suitable for use in decentralized cloud contexts.

Newer-generation storage systems like protocol-driven or blockchain-based solutions may face scalability and performance limitations and may not fulfill certain critical requirements that we consider essential.

What?

A redesigned storage system which can scale to planet level, is super secure private and fast enough for more usecases. Its designed to operate in a decentralized context. Data can never be lost of corrupted.

This storage system is a backend storage system, cannot be used by end users, its meant to be integrated with a front end storage system like e.g. S3 or a filesystem (see next section).

Benefits?

• Data Resilience: Ensures data is never lost or corrupted.
• Planetary Scalability: Capable of scaling to a global level.
• Cost-Efficient: Offers exceptional cost efficiency.
• Versatility: Suitable for various use cases, including archiving, backup, files, and CDNs.
• Low Overhead: Requires only a 20% overhead for building a storage network where any four nodes can be lost simultaneously, compared to a 400% overhead in traditional storage systems.
• Security and Privacy: Provides robust security, even impervious to quantum computers.
• Data Sovereignty: Users have complete control over data placement.
• Empowering Front-End Applications: Can be integrated into various front-end storage applications, such as blockchains, archives, or S3.
• CDN Support: Functions effectively as a backend for CDN applications, facilitating content delivery.
• Sustainability: Uses 10 times less energy compared to traditional storage systems, contributing to sustainability efforts.
• Locality Aware: Data can be delivered to where the users are ideal for sovereign usecases.

Status?

• Zero-OS has been in beta for over four years, with continuous development and improvement.
• A notable deployment in Switzerland, with over 50 petabytes of storage capacity, served as a substantial test environment, although it's no longer active.
• Within the current TFGrid network, there's an impressive capacity of over 20 petabytes available for use.
• Lacking some monitoring, documentation ... will be added in TFGrid 3.15
• Previous releases have been successfully utilized by major government organizations on a massive scale (hundreds of petabytes), providing strong evidence of the concept's viability and effectiveness.

Quantum Safe File System

Why?

There is a growing need for more accessible and user-friendly solutions to store and manage large volumes of data efficiently.

While Zero-Stor addresses numerous storage challenges effectively, it may not be accessible or user-friendly for typical developers or system administrators. QSFS has been developed to bridge this gap and provide a more approachable storage solution.

What?

A FUSE-based filesystem utilizing Zero-Stor as its backend. Metadata is safeguarded to prevent loss, inheriting Zero-Stor's benefits and simplifying usage for developers and system administrators.

The filesystem is always deployed in one location, data is distributed (using zero-stor) across multiple sites for unparalleled reliability.

Metadata redundancy is included. While not consistently synchronized in real-time, the system allows configuration of consistency levels. Typically, the decentralized state may lag by up to 15 minutes.

This filesystem can be mounted under various storage-aware applications, such as backup servers, file servers, or S3 servers, enhancing versatility.

Benefits

• Inherits the advantages of Zero-Stor, including enhanced data security, efficiency, and scalability.
• Provides a user-friendly interface for seamless integration with a wide range of applications.
• Offers considerable scalability capabilities, although not unlimited in scale.
• Achieves reasonable performance data transfer rates of up to 50 MB/sec, particularly for larger files.
• Can scale to about 2 million files per filesystem.

FungiStor (end 2024)

Why?

Existing blockchain, internet, and P2P content delivery and storage systems suffer from sluggish performance and are too expensive. Content retrieval is often slow, and the overhead for ensuring redundancy is excessive. We require innovative approaches to facilitate efficient information sharing among users.

Content delivery frequently represents the most significant expense for social networks. Running a basic social video network for 10 million users currently costs approximately $2 million per month using traditional cloud providers. We have the potential to reduce this cost by several orders of magnitude.

What?

FungiStor is a peer-to-peer (P2P) content delivery layer designed to store and distribute an extensive range of objects, including images, videos, files, and more. It has the capability to handle trillions of objects and files efficiently. FungiStor serves as an excellent solution for content delivery networks (CDNs), significantly reducing costs for organizations seeking to stream or deliver substantial data volumes to their user base.

Furthermore, FungiStor will act as the backend infrastructure for the Flists within our own system. However, it is versatile and can be utilized by anyone in need of a global-level content delivery system for files, objects, and images.

Benefits?

• Global Scalability, Sub-50ms Lookups: FungiStor scales worldwide with ultra-fast data retrieval under 50 milliseconds.
• Localized Content Delivery: Prioritizes local data access for optimized speed and efficiency.
• Quantum-Safe Security: Incorporates robust quantum security measures.
• Interoperability: Works seamlessly with IPFS, Torrent, and more.
• Cost Efficiency: Offers significant cost savings, potentially 10 to 100 times less than conventional solutions.

NetworkWall (OEM only)

Why?

Traditional firewalls are increasingly ineffective in addressing modern security challenges. They struggle to mitigate emerging threats, particularly against backdoors and man-in-the-middle attacks. Backdoors can render firewalls obsolete as attackers find ways to bypass them. New, innovative approaches to cybersecurity are necessary to address these evolving security problems.

What?

Imagine a scenario where you deploy applications within secure, liquid-cooled physical containers or smaller POD's that operate without relying on traditional TCP/IP or Ethernet protocols. By avoiding the use of standard low-level protocols, the existing backdoors are unable to communicate with the external world.

Incoming traffic is intercepted at the application level and securely transported into the protected environment through physical communication mechanisms based on RDMA (Remote Direct Memory Access). The integration of this solution into Zero-OS ensures a seamless and transparent experience for developers and system administrators.

It's important to note that this solution is primarily intended for commercial use cases, but its existence is valuable knowledge in the realm of cybersecurity and network security.

Benefits

• Enhanced Security and Privacy: The solution offers significantly improved security and privacy measures, mitigating potential risks and vulnerabilities.
• Ultra-Fast Connectivity: Within the POD/Container, the connectivity is exceptionally fast, ensuring rapid data transfer and application performance.
• Robust Data and Application-Aware Proxies: Secure proxies between the Internet and the protected backend application provide an additional layer of security, safeguarding data and ensuring application-level awareness.
• Seamless Integration: The solution is designed for ease of integration within existing environments, minimizing disruptions and complexities during implementation.