tfgrid/info_tfgrid
33
3
Fork 0
Code Issues Pull Requests 2 Packages Projects Releases Activity

new book manual

Browse Source
This commit is contained in:
mik-tf
2024-04-15 17:42:25 +00:00
parent b46a6df7e0
commit a567404ef3
1772 changed files with 450 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
collections/manual/documentation/developers/grid_deployment/grid_deployment.md Normal file
View File

@@ -0,0 +1,9 @@
# Grid Deployment
The TFGrid whole source code is open-source and instances of the grid can be deployed by anyone thanks to the distribution of daily grid snapshots of the complete ThreeFold Grid stacks.
## Table of Contents
- [TFGrid Stacks](./tfgrid_stacks.md)
- [Full VM Grid Deployment](./grid_deployment_full_vm.md)
- [Grid Snapshots](./snapshots.md)

152
collections/manual/documentation/developers/grid_deployment/grid_deployment_full_vm.md Normal file
View File

@@ -0,0 +1,152 @@
<h1> Grid Deployment on a Full VM </h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Prerequisites](#prerequisites)
- [DNS Settings](#dns-settings)
- [DNS Verification](#dns-verification)
- [Prepare the VM](#prepare-the-vm)
- [Set the Firewall](#set-the-firewall)
- [Launch the Script](#launch-the-script)
- [Access the Grid Services](#access-the-grid-services)
- [Manual Commands](#manual-commands)
- [Update the Deployment](#update-the-deployment)
***
## Introduction
We present the steps to deploy a network instance of the TFGrid on a full VM.
For this guide, we will be deploying a mainnet instance. While the steps are similar for testnet and devnet, you will have to adjust your deployment depending on which network you use.
## Prerequisites
For this guide, you will need to deploy a full VM on the ThreeFold Grid with at least the following minimum specs:
- IPv4
- IPv6
- 32GB of RAM
- 1000 GB of SSD
- 8 vcores
After deploying the full VM, take note of the IPv4 and IPv6 addresses to properly set the DNS records and then SSH into the VM.
## DNS Settings
You need to set an A record for the IPv4 address and an AAAA record for the IPv6 address with a wildcard subdomain.
The following table explicitly shows how to set the A and AAAA records for your domain.
| Type | Host | Value |
| ---- | ---- | -------------- |
| A | \* | <ipv4_address> |
| AAAA | \* | <ipv6_address> |
### DNS Verification
You can use tools such as [DNSChecker](https://dnschecker.org/) or [dig](https://linux.die.net/man/1/dig) on a terminal to check if the DNS propagadation is complete.
## Prepare the VM
- Download the ThreeFold Tech `grid_deployment` repository
```
git clone https://github.com/threefoldtech/grid_deployment
cd grid_deployment/docker-compose/mainnet
```
- Generate a TFChain node key with `subkey`
```
echo .subkey_mainnet >> .gitignore
../subkey generate-node-key > .nodekey_mainnet
cat .nodekey_mainnet
```
- Create and the set environment variables file
```
cp .secrets.env-example .secrets.env
```
- Adjust the environment file
```
nano .secrets.env
```
- To adjust the `.secrets.env` file, take into account the following:
- **DOMAIN**="example.com"
- Write your own domain
- **TFCHAIN_NODE_KEY**="abc123"
- Write the output of the command `cat .nodekey_mainnet`
- **ACTIVATION_SERVICE_MNEMONIC**="word1 word2 ... word24"
- Write the seed phrase of an account on mainnet with at least 10 TFT in the wallet
- **GRID_PROXY_MNEMONIC**="word1 word2 ... word24"
- Write the seed phrase of an account on mainnet with at least 10 TFT in the wallet and a registered twin ID\*
> \*Note: If you've created an account using the ThreeFold Dashboard on mainnet, the twin ID is automatically registered.
## Set the Firewall
You can use UFW to set the firewall:
```
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 30333/tcp
ufw allow 22/tcp
ufw enable
ufw status
```
## Launch the Script
Once you've prepared the VM, you can simply run the script to install the grid stack and deploy it online.
```
sh install_grid_bknd.sh
```
This will take some time since you are downloading the whole mainnet grid snapshots.
## Access the Grid Services
Once you've deployed the grid stack online, you can access the different grid services by usual the usual subdomains:
```
dashboard.your.domain
metrics.your.domain
tfchain.your.domain
graphql.your.domain
relay.your.domain
gridproxy.your.domain
activation.your.domain
stats.your.domain
```
## Manual Commands
Once you've run the install script, you can deploy manually the grid stack with the following command:
```
docker compose --env-file .secrets.env --env-file .env up -d
```
You can also check if the environment variables are properly set:
```
docker compose --env-file .secrets.env --env-file .env config
```
If you want to see the output during deployment, remove `-d` in the command above as follows:
```
docker compose --env-file .secrets.env --env-file .env up
```
This can be helpful to troubleshoot errors.
## Update the Deployment
Go into the folder of the proper network, e.g. mainnet, and run the following commands:
```
git pull -r
docker compose --env-file .secrets.env --env-file .env up -d
```

196
collections/manual/documentation/developers/grid_deployment/snapshots.md Normal file
View File

@@ -0,0 +1,196 @@
<h1>Snapshots for Grid Backend Services</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Services](#services)
- [ThreeFold Public Snapshots](#threefold-public-snapshots)
- [Deploy the Services with Scripts](#deploy-the-services-with-scripts)
- [Create the Snapshots](#create-the-snapshots)
- [Start All the Services](#start-all-the-services)
- [Stop All the Services](#stop-all-the-services)
- [Expose the Snapshots with Rsync](#expose-the-snapshots-with-rsync)
- [Create the Service Configuration File](#create-the-service-configuration-file)
- [Start the Service](#start-the-service)
***
## Introduction
To facilitate deploying grid backend services, we provide snapshots to significantly reduce sync time. This can be setup anywhere from scratch. Once all services are synced, one can use the scripts to create snapshots automatically.
To learn how to deploy your own grid stack, read [this section](./grid_deployment_full_vm.md).
## Services
There are 3 grid backend services that collect enough data to justify creating snapshots:
- ThreeFold blockchain - TFChain
- Graphql - Indexer
- Graphql - Processor
## ThreeFold Public Snapshots
ThreeFold hosts all available snapshots at: [https://bknd.snapshot.grid.tf/](https://bknd.snapshot.grid.tf/). Those snapshots can be downloaded with rsync:
- Mainnet:
```
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshots/tfchain-mainnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshots/indexer-mainnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshots/processor-mainnet-latest.tar.gz .
```
- Testnet:
```
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotstest/tfchain-testnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotstest/indexer-testnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotstest/processor-testnet-latest.tar.gz .
```
- Devnet:
```
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotsdev/tfchain-devnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotsdev/indexer-devnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotsdev/processor-devnet-latest.tar.gz .
```
## Deploy the Services with Scripts
You can deploy the 3 individual services using known methods such as [Docker](../../system_administrators/computer_it_basics/docker_basics.md). To facilitate the process, scripts are provided that run the necessary docker commands.
The first script creates the snapshots, while the second and third scripts serve to start and stop all services.
You can use the start script to start all services and then set a cron job to execute periodically the snapshot creation script. This will ensure that you always have the latest version available on your server.
### Create the Snapshots
You can set a cron job to execute a script running rsync to create the snapshots and generate logs at a given interval.
- First download the script.
- Main net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/mainnet/create_snapshot.sh
```
- Test net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/testnet/create_snapshot.sh
```
- Dev net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/devnet/create_snapshot.sh
```
- Set the permissions of the script
```
chmod +x create_snapshot.sh
```
- Make sure to a adjust the snapshot creation script for your specific deployment
- Set a cron job
```
crontab -e
```
- Here is an example of a cron job where we execute the script every day at 1 AM and send the logs to `/var/log/snapshots/snapshots-cron.log`.
```sh
0 1 * * * sh /opt/snapshots/create-snapshot.sh > /var/log/snapshots/snapshots-cron.log 2>&1
```
### Start All the Services
You can start all services by running the provided scripts.
- Download the script.
- Main net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/mainnet/startall.sh
```
- Test net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/testnet/startall.sh
```
- Dev net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/devnet/startall.sh
```
- Set the permissions of the script
```
chmod +x startall.sh
```
- Run the script to start all services via docker engine.
```
./startall.sh
```
### Stop All the Services
You can stop all services by running the provided scripts.
- Download the script.
- Main net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/mainnet/stopall.sh
```
- Test net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/testnet/stopall.sh
```
- Dev net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/devnet/stopall.sh
```
- Set the permissions of the script
```
chmod +x stopall.sh
```
- Run the script to stop all services via docker engine.
```
./stopall.sh
```
## Expose the Snapshots with Rsync
We use rsync with a systemd service to expose the snapshots to the community.
### Create the Service Configuration File
To setup a public rsync server, create and edit the following file:
`/etc/rsyncd.conf`
```sh
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsync.log
port = 34873
max connections = 20
exclude = lost+found/
transfer logging = yes
use chroot = yes
reverse lookup = no
[gridsnapshots]
path = /storage/rsync-public/mainnet
comment = THREEFOLD GRID MAINNET SNAPSHOTS
read only = true
timeout = 300
list = false
[gridsnapshotstest]
path = /storage/rsync-public/testnet
comment = THREEFOLD GRID TESTNET SNAPSHOTS
read only = true
timeout = 300
list = false
[gridsnapshotsdev]
path = /storage/rsync-public/devnet
comment = THREEFOLD GRID DEVNET SNAPSHOTS
read only = true
timeout = 300
list = false
```
### Start the Service
Start and enable via systemd:
```sh
systemctl start rsync
systemctl enable rsync
systemctl status rsync
```

206
collections/manual/documentation/developers/grid_deployment/snapshots_archive.md Normal file
View File

@@ -0,0 +1,206 @@
<h1>Snapshots for Grid Backend Services</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Prerequisites](#prerequisites)
- [Services](#services)
- [ThreeFold Public Snapshots](#threefold-public-snapshots)
- [Deploy the Services with Scripts](#deploy-the-services-with-scripts)
- [Start All the Services](#start-all-the-services)
- [Stop All the Services](#stop-all-the-services)
- [Create the Snapshots](#create-the-snapshots)
- [Expose the Snapshots with Rsync](#expose-the-snapshots-with-rsync)
- [Create the Service Configuration File](#create-the-service-configuration-file)
- [Start the Service](#start-the-service)
***
## Introduction
To facilitate deploying grid backend services, we provide snapshots to significantly reduce sync time. This can be setup anywhere from scratch. Once all services are synced, one can use the scripts to create snapshots automatically.
## Prerequisites
There are a few prerequisites to properly run the ThreeFold services.
- [Docker engine](../computer_it_basics/docker_basics.md#install-docker-desktop-and-docker-engine)
- [Rsync](../computer_it_basics/file_transfer.md#rsync)
## Services
There are 3 grid backend services that collect enough data to justify creating snapshots:
- ThreeFold blockchain - TFChain
- Graphql - Indexer
- Graphql - Processor
## ThreeFold Public Snapshots
ThreeFold hosts all available snapshots at: [https://bknd.snapshot.grid.tf/](https://bknd.snapshot.grid.tf/). Those snapshots can be downloaded with rsync:
- Mainnet:
```
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshots/tfchain-mainnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshots/indexer-mainnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshots/processor-mainnet-latest.tar.gz .
```
- Testnet:
```
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotstest/tfchain-testnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotstest/indexer-testnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotstest/processor-testnet-latest.tar.gz .
```
- Devnet:
```
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotsdev/tfchain-devnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotsdev/indexer-devnet-latest.tar.gz .
rsync -Lv --progress --partial rsync://bknd.snapshot.grid.tf:34873/gridsnapshotsdev/processor-devnet-latest.tar.gz .
```
Let's now see how to use those snapshots to run the services via scripts.
## Deploy the Services with Scripts
You can deploy the 3 individual services using known methods such as [Docker](https://manual.grid.tf/computer_it_basics/docker_basics.html). To facilitate the process, scripts are provided that run the necessary docker commands.
The first script creates the snapshots, while the second and third scripts serve to start and stop all services.
You can use the start script to start all services and then set a cron job to execute periodically the snapshot creation script. This will ensure that you always have the latest version available on your server.
### Start All the Services
You can start all services by running the provided scripts.
- Download the script.
- Main net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/mainnet/startall.sh
```
- Test net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/testnet/startall.sh
```
- Dev net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/devnet/startall.sh
```
- Set the permissions of the script
```
chmod +x startall.sh
```
- Run the script to start all services via docker engine.
```
./startall.sh
```
### Stop All the Services
You can stop all services by running the provided scripts.
- Download the script.
- Main net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/mainnet/stopall.sh
```
- Test net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/testnet/stopall.sh
```
- Dev net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/devnet/stopall.sh
```
- Set the permissions of the script
```
chmod +x stopall.sh
```
- Run the script to stop all services via docker engine.
```
./stopall.sh
```
### Create the Snapshots
You can set a cron job to execute a script running rsync to create the snapshots and generate logs at a given interval.
- First download the script.
- Main net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/mainnet/create_snapshot.sh
```
- Test net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/testnet/create_snapshot.sh
```
- Dev net
```
wget https://github.com/threefoldtech/grid_deployment/blob/development/grid-snapshots/devnet/create_snapshot.sh
```
- Set the permissions of the script
```
chmod +x create_snapshot.sh
```
- Make sure to a adjust the snapshot creation script for your specific deployment
- Set a cron job
```
crontab -e
```
- Here is an example of a cron job where we execute the script every day at 1 AM and send the logs to `/var/log/snapshots/snapshots-cron.log`.
```sh
0 1 * * * sh /opt/snapshots/create-snapshot.sh > /var/log/snapshots/snapshots-cron.log 2>&1
```
## Expose the Snapshots with Rsync
We use rsync with a systemd service to expose the snapshots to the community.
### Create the Service Configuration File
To setup a public rsync server, create and edit the following file:
`/etc/rsyncd.conf`
```sh
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsync.log
port = 34873
max connections = 20
exclude = lost+found/
transfer logging = yes
use chroot = yes
reverse lookup = no
[gridsnapshots]
path = /storage/rsync-public/mainnet
comment = THREEFOLD GRID MAINNET SNAPSHOTS
read only = true
timeout = 300
list = false
[gridsnapshotstest]
path = /storage/rsync-public/testnet
comment = THREEFOLD GRID TESTNET SNAPSHOTS
read only = true
timeout = 300
list = false
[gridsnapshotsdev]
path = /storage/rsync-public/devnet
comment = THREEFOLD GRID DEVNET SNAPSHOTS
read only = true
timeout = 300
list = false
```
### Start the Service
Start and enable via systemd:
```sh
systemctl start rsync
systemctl enable rsync
systemctl status rsync
```
If you're interested about hosting your own instance of the grid to strenghten the ThreeFold ecosystem, make sure to read the next section, [Guardians of the Grid](./tfgrid_guardians.md).

32
collections/manual/documentation/developers/grid_deployment/tfgrid_stacks.md Normal file
View File

@@ -0,0 +1,32 @@
<h1> TFGrid Stacks </h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Advantages](#advantages)
- [Run Your Own Stack](#run-your-own-stack)
***
## Introduction
ThreeFold is an open-source project and anyone can run the full stack of the TFGrid in a totally decentralized manner. In practice, this means that anyone can grab a docker compose file shared by ThreeFold of the TFGrid stacks and run an instance of the grid services on their own domain.
This means that you could host your own instance of the ThreeFold Dashboard at `dashboard.yourdomain.com` that would serve your own instance of the complete TFGrid stack! Users could then access the ThreeFold Dashboard via your own domain.
The process is actually very straightforward and we even provide a script to streamline the process.
## Advantages
Setting such instances of the TFGrid ensures resiliency and decentralization of the ThreeFold ecosystem.
As a very concrete example, image that one instance of the Dashboard goes offline, `dashboard.grid.tf`, then users could still access the Dashboard from another instance. The more users of the TFGrid deploy their own instance, the more resilient the grid becomes.
The overall ThreeFold ecosystem becomes more resilient to failures of individual nodes.
## Run Your Own Stack
To set your own instance of the TFGrid, you can download a snapshot of the grid and deploy the TFGrid services with Docker. We even provide scripts to quicken the whole process!
Read more about snapshots in the [next section](./grid_deployment_full_vm.md).