tfgrid/info_tfgrid
33
3
Fork 0
Code Issues Pull Requests 2 Packages Projects Releases Activity

new book manual

Browse Source
This commit is contained in:
mik-tf
2024-04-15 17:42:25 +00:00
parent b46a6df7e0
commit a567404ef3
1772 changed files with 450 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files

127
collections/manual/documentation/developers/proxy/commands.md Normal file
View File

@@ -0,0 +1,127 @@
<h1>Commands</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Work on Docs](#work-on-docs)
- [To start the GridProxy server](#to-start-the-gridproxy-server)
- [Run tests](#run-tests)
***
## Introduction
The Makefile makes it easier to do mostly all the frequently commands needed to work on the project.
## Work on Docs
we are using [swaggo/swag](https://github.com/swaggo/swag) to generate swagger docs based on the annotation inside the code.
- install swag executable binary
```bash
go install github.com/swaggo/swag/cmd/swag@latest
```
- now if you check the binary directory inside go directory you will find the executable file.
```bash
ls $(go env GOPATH)/bin
```
- to run swag you can either use the full path `$(go env GOPATH)/bin/swag` or export go binary to `$PATH`
```bash
export PATH=$PATH:$(go env GOPATH)/bin
```
- use swag to format code comments.
```bash
swag fmt
```
- update the docs
```bash
swag init
```
- to parse external types from vendor
```bash
swag init --parseVendor
```
- for a full generate docs command
```bash
make docs
```
## To start the GridProxy server
After preparing the postgres database you can `go run` the main file in `cmds/proxy_server/main.go` which responsible for starting all the needed server/clients.
The server options
| Option | Description |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------- |
| -address | Server ip address (default `":443"`) |
| -ca | certificate authority used to generate certificate (default `"https://acme-staging-v02.api.letsencrypt.org/directory"`) |
| -cert-cache-dir | path to store generated certs in (default `"/tmp/certs"`) |
| -domain | domain on which the server will be served |
| -email | email address to generate certificate with |
| -log-level | log level `[debug\|info\|warn\|error\|fatal\|panic]` (default `"info"`) |
| -no-cert | start the server without certificate |
| -postgres-db | postgres database |
| -postgres-host | postgres host |
| -postgres-password | postgres password |
| -postgres-port | postgres port (default 5432) |
| -postgres-user | postgres username |
| -tfchain-url | tF chain url (default `"wss://tfchain.dev.grid.tf/ws"`) |
| -relay-url | RMB relay url (default`"wss://relay.dev.grid.tf"`) |
| -mnemonics | Dummy user mnemonics for relay calls |
| -v | shows the package version |
For a full server setup:
```bash
make restart
```
## Run tests
There is two types of tests in the project
- Unit Tests
- Found in `pkg/client/*_test.go`
- Run with `go test -v ./pkg/client`
- Integration Tests
- Found in `tests/queries/`
- Run with:
```bash
go test -v \
--seed 13 \
--postgres-host <postgres-ip> \
--postgres-db tfgrid-graphql \
--postgres-password postgres \
--postgres-user postgres \
--endpoint <server-ip> \
--mnemonics <insert user mnemonics>
```
- Or to run a specific test you can append the previous command with
```bash
-run <TestName>
```
You can found the TestName in the `tests/queries/*_test.go` files.
To run all the tests use
```bash
make test-all
```

55
collections/manual/documentation/developers/proxy/contributions.md Normal file
View File

@@ -0,0 +1,55 @@
<h1>Contributions Guide</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Project structure](#project-structure)
- [Internal](#internal)
- [Pkg](#pkg)
- [Writing tests](#writing-tests)
***
## Introduction
We propose a quick guide to learn how to contribute.
## Project structure
The main structure of the code base is as follows:
- `charts`: helm chart
- `cmds`: includes the project Golang entrypoints
- `docs`: project documentation
- `internal`: contains the explorer API logic and the cert manager implementation, this where most of the feature work will be done
- `pkg`: contains client implementation and shared libs
- `tests`: integration tests
- `tools`: DB tools to prepare the Postgres DB for testing and development
- `rootfs`: ZOS root endpoint that will be mounted in the docker image
### Internal
- `explorer`: contains the explorer server logic:
- `db`: the db connection and operations
- `mw`: defines the generic action mount that will be be used as http handler
- `certmanager`: logic to ensure certificates are available and up to date
`server.go` includes the logic for all the API operations.
### Pkg
- `client`: client implementation
- `types`: defines all the API objects
## Writing tests
Adding a new endpoint should be accompanied with a corresponding test. Ideally every change or bug fix should include a test to ensure the new behavior/fix is working as intended.
Since these are integration tests, you need to first make sure that your local db is already seeded with the ncessary data. See tools [doc](./db_testing.md) for more information about how to prepare your db.
Testing tools offer two clients that are the basic of most tests:
- `local`: this client connects to the local db
- `proxy client`: this client connects to the running local instance
You need to start an instance of the server before running the tests. Check [here](./commands.md) for how to start.

21
collections/manual/documentation/developers/proxy/database.md Normal file
View File

@@ -0,0 +1,21 @@
<h1>Database</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Max Open Connections](#max-open-connections)
***
## Introduction
The grid proxy has access to a postgres database containing information about the tfgrid, specifically information about grid nodes, farms, twins, and contracts.\
The database is filled/updated by this [indexer](https://github.com/threefoldtech/tfchain_graphql).
The grid proxy mainly retrieves information from the db with a few modifications for efficient retrieval (e.g. adding indices, caching node gpus, etc..).
## Max Open Connections
The postgres database can handle 100 open connections concurrently (that is the default value set by postgres), this number can be increased, depending on the infrastructure, by modifying it in the postgres.conf file where the db is deployed, or by executing the following query `ALTER system SET max_connections=size-of-connection`, but this requires a db restart to take effect.\
The explorer creates a connection pool to the postgres db, with a max open pool connections set to a specific number (currently 80).\
It's important to distinguish between the database max connections, and the max pool open connections, because if the pool did not have any constraints, it would try to open as many connections as it wanted, without any notion of the maximum connections the database accepts. It's the database responsibility then to accept or deny the connection.\
This is why the max number of open pool connections is set to 80: It's below the max connections the database could handle (100), and it gives room for other actors outside of the explorer to open connections with the database.\

45
collections/manual/documentation/developers/proxy/db_testing.md Normal file
View File

@@ -0,0 +1,45 @@
<h1>DB for testing</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Run postgresql container](#run-postgresql-container)
- [Create the DB](#create-the-db)
- [Method 1: Generate a db with relevant schema using the db helper tool:](#method-1-generate-a-db-with-relevant-schema-using-the-db-helper-tool)
- [Method 2: Fill the DB from a Production db dump file, for example if you have `dump.sql` file, you can run:](#method-2-fill-the-db-from-a-production-db-dump-file-for-example-if-you-have-dumpsql-file-you-can-run)
***
## Introduction
We show how to use a database for testing.
## Run postgresql container
```bash
docker run --rm --name postgres \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_DB=tfgrid-graphql \
-p 5432:5432 -d postgres
```
## Create the DB
you can either Generate a db with relevant schema to test things locally quickly, or load a previously taken DB dump file:
### Method 1: Generate a db with relevant schema using the db helper tool:
```bash
cd tools/db/ && go run . \
--postgres-host 127.0.0.1 \
--postgres-db tfgrid-graphql \
--postgres-password postgres \
--postgres-user postgres \
--reset \
```
### Method 2: Fill the DB from a Production db dump file, for example if you have `dump.sql` file, you can run:
```bash
psql -h 127.0.0.1 -U postgres -d tfgrid-graphql < dump.sql
```

38
collections/manual/documentation/developers/proxy/explorer.md Normal file
View File

@@ -0,0 +1,38 @@
<h1>The Grid Explorer</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Explorer Overview](#explorer-overview)
- [Explorer Endpoints](#explorer-endpoints)
***
## Introduction
The Grid Explorer is a rest API used to index a various information from the TFChain.
## Explorer Overview
- Due to limitations on indexing information from the blockchain, Complex inter-tables queries and limitations can't be applied directly on the chain.
- Here comes the TFGridDB, a shadow database contains all the data on the chain which is being updated each 2 hours.
- Then the explorer can apply a raw SQL queries on the database with all limitations and filtration needed.
- The used technology to extract the info from the blockchain is Subsquid check the [repo](https://github.com/threefoldtech/tfchain_graphql).
## Explorer Endpoints
| HTTP Verb | Endpoint | Description |
| --------- | --------------------------- | ---------------------------------- |
| GET | `/contracts` | Show all contracts on the chain |
| GET | `/farms` | Show all farms on the chain |
| GET | `/gateways` | Show all gateway nodes on the grid |
| GET | `/gateways/:node_id` | Get a single gateway node details |
| GET | `/gateways/:node_id/status` | Get a single node status |
| GET | `/nodes` | Show all nodes on the grid |
| GET | `/nodes/:node_id` | Get a single node details |
| GET | `/nodes/:node_id/status` | Get a single node status |
| GET | `/stats` | Show the grid statistics |
| GET | `/twins` | Show all the twins on the chain |
| GET | `/nodes/:node_id/statistics`| Get a single node ZOS statistics |
For the available filters on each node. check `/swagger/index.html` endpoint on the running instance.

117
collections/manual/documentation/developers/proxy/production.md Normal file
View File

@@ -0,0 +1,117 @@
<h1>Running Proxy in Production</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Production Run](#production-run)
- [To upgrade the machine](#to-upgrade-the-machine)
- [Dockerfile](#dockerfile)
- [Update helm package](#update-helm-package)
- [Install the chart using helm package](#install-the-chart-using-helm-package)
***
## Introduction
We show how to run grid proxy in production.
## Production Run
- Download the latest binary [here](https://github.com/threefoldtech/tfgrid-sdk-go/tree/development/grid-client)
- add the execution permission to the binary and move it to the bin directory
```bash
chmod +x ./gridproxy-server
mv ./gridproxy-server /usr/local/bin/gridproxy-server
```
- Add a new systemd service
```bash
cat << EOF > /etc/systemd/system/gridproxy-server.service
[Unit]
Description=grid proxy server
After=network.target
[Service]
ExecStart=gridproxy-server --domain gridproxy.dev.grid.tf --email omar.elawady.alternative@gmail.com -ca https://acme-v02.api.letsencrypt.org/directory --postgres-host 127.0.0.1 --postgres-db db --postgres-password password --postgres-user postgres --mnemonics <insert user mnemonics>
Type=simple
Restart=always
User=root
Group=root
[Install]
WantedBy=multi-user.target
Alias=gridproxy.service
EOF
```
- enable the service
```bash
systemctl enable gridproxy.service
```
- start the service
```bash
systemctl start gridproxy.service
```
- check the status
```bash
systemctl status gridproxy.service
```
- The command options:
- domain: the host domain which will generate ssl certificate to.
- email: the mail used to run generate the ssl certificate.
- ca: certificate authority server url, e.g.
- let's encrypt staging: `https://acme-staging-v02.api.letsencrypt.org/directory`
- let's encrypt production: `https://acme-v02.api.letsencrypt.org/directory`
- postgres -\*: postgres connection info.
## To upgrade the machine
- just replace the binary with the new one and apply
```bash
systemctl restart gridproxy-server.service
```
- it you have changes in the `/etc/systemd/system/gridproxy-server.service` you have to run this command first
```bash
systemctl daemon-reload
```
## Dockerfile
To build & run dockerfile
```bash
docker build -t threefoldtech/gridproxy .
docker run --name gridproxy -e POSTGRES_HOST="127.0.0.1" -e POSTGRES_PORT="5432" -e POSTGRES_DB="db" -e POSTGRES_USER="postgres" -e POSTGRES_PASSWORD="password" -e MNEMONICS="<insert user mnemonics>" threefoldtech/gridproxy
```
## Update helm package
- Do `helm lint charts/gridproxy`
- Regenerate the packages `helm package -u charts/gridproxy`
- Regenerate index.yaml `helm repo index --url https://threefoldtech.github.io/tfgridclient_proxy/ .`
- Push your changes
## Install the chart using helm package
- Adding the repo to your helm
```bash
helm repo add gridproxy https://threefoldtech.github.io/tfgridclient_proxy/
```
- install a chart
```bash
helm install gridproxy/gridproxy
```

149
collections/manual/documentation/developers/proxy/proxy.md Normal file
View File

@@ -0,0 +1,149 @@
<h1> Introducing Grid Proxy </h1>
<h2> Table of Content</h2>
- [About](#about)
- [How to Use the Project](#how-to-use-the-project)
- [Used Technologies \& Prerequisites](#used-technologies--prerequisites)
- [Start for Development](#start-for-development)
- [Setup for Production](#setup-for-production)
- [Get and Install the Binary](#get-and-install-the-binary)
- [Add as a Systemd Service](#add-as-a-systemd-service)
***
<!-- About -->
## About
The TFGrid client Proxy acts as an interface to access information about the grid. It supports features such as filtering, limitation, and pagination to query the various entities on the grid like nodes, contracts and farms. Additionally the proxy can contact the required twin ID to retrieve stats about the relevant objects and performing ZOS calls.
The proxy is used as the backend of several threefold projects like:
- [Dashboard](../../dashboard/dashboard.md)
<!-- Usage -->
## How to Use the Project
If you don't want to care about setting up your instance you can use one of the live instances. each works against a different TFChain network.
- Dev network: <https://gridproxy.dev.grid.tf>
- Swagger: <https://gridproxy.dev.grid.tf/swagger/index.html>
- Qa network: <https://gridproxy.qa.grid.tf>
- Swagger: <https://gridproxy.qa.grid.tf/swagger/index.html>
- Test network: <https://gridproxy.test.grid.tf>
- Swagger: <https://gridproxy.test.grid.tf/swagger/index.html>
- Main network: <https://gridproxy.grid.tf>
- Swagger: <https://gridproxy.grid.tf/swagger/index.html>
Or follow the [development guide](#start-for-development) to run yours.
By default, the instance runs against devnet. to configure that you will need to config this while running the server.
> Note: You may face some differences between each instance and the others. that is normal because each network is in a different stage of development and works correctly with others parts of the Grid on the same network.
<!-- Prerequisites -->
## Used Technologies & Prerequisites
1. **GoLang**: Mainly the two parts of the project written in `Go 1.17`, otherwise you can just download the compiled binaries from github [releases](https://github.com/threefoldtech/tfgrid-sdk-go/releases)
2. **Postgresql**: Used to load the TFGrid DB
3. **Docker**: Containerize the running services such as Postgres and Redis.
4. **Mnemonics**: Secret seeds for adummy identity to use for the relay client.
For more about the prerequisites and how to set up and configure them. follow the [Setup guide](./setup.md)
<!-- Development -->
## Start for Development
To start the services for development or testing make sure first you have all the [Prerequisites](#used-technologies--prerequisites).
- Clone this repo
```bash
git clone https://github.com/threefoldtech/tfgrid-sdk-go.git
cd tfgrid-sdk-go/grid-proxy
```
- The `Makefile` has all that you need to deal with Db, Explorer, Tests, and Docs.
```bash
make help # list all the available subcommands.
```
- For a quick test explorer server.
```bash
make all-start e=<MNEMONICS>
```
Now you can access the server at `http://localhost:8080`
- Run the tests
```bash
make test-all
```
- Generate docs.
```bash
make docs
```
To run in development environment see [here](./db_testing.md) how to generate test db or load a db dump then use:
```sh
go run cmds/proxy_server/main.go --address :8080 --log-level debug -no-cert --postgres-host 127.0.0.1 --postgres-db tfgrid-graphql --postgres-password postgres --postgres-user postgres --mnemonics <insert user mnemonics>
```
Then visit `http://localhost:8080/<endpoint>`
For more illustrations about the commands needed to work on the project, see the section [Commands](./commands.md). For more info about the project structure and contributions guidelines check the section [Contributions](./contributions.md).
<!-- Production-->
## Setup for Production
## Get and Install the Binary
- You can either build the project:
```bash
make build
chmod +x cmd/proxy_server/server \
&& mv cmd/proxy_server/server /usr/local/bin/gridproxy-server
```
- Or download a release:
Check the [releases](https://github.com/threefoldtech/tfgrid-sdk-go/releases) page and edit the next command with the chosen version.
```bash
wget https://github.com/threefoldtech/tfgrid-sdk-go/releases/download/v1.6.7-rc2/tfgridclient_proxy_1.6.7-rc2_linux_amd64.tar.gz \
&& tar -xzf tfgridclient_proxy_1.6.7-rc2_linux_amd64.tar.gz \
&& chmod +x server \
&& mv server /usr/local/bin/gridproxy-server
```
## Add as a Systemd Service
- Create the service file
```bash
cat << EOF > /etc/systemd/system/gridproxy-server.service
[Unit]
Description=grid proxy server
After=network.target
[Service]
ExecStart=gridproxy-server --domain gridproxy.dev.grid.tf --email omar.elawady.alternative@gmail.com -ca https://acme-v02.api.letsencrypt.org/directory --substrate wss://tfchain.dev.grid.tf/ws --postgres-host 127.0.0.1 --postgres-db db --postgres-password password --postgres-user postgres --mnemonics <insert user mnemonics>
Type=simple
Restart=always
User=root
Group=root
[Install]
WantedBy=multi-user.target
Alias=gridproxy.service
EOF
```

25
collections/manual/documentation/developers/proxy/proxy_readme.md Normal file
View File

@@ -0,0 +1,25 @@
<h1>Grid Proxy</h1>
Welcome to the *Grid Proxy* section of the TFGrid Manual!
In this comprehensive guide, we delve into the intricacies of the ThreeFold Grid Proxy, a fundamental component that empowers the ThreeFold Grid ecosystem.
This section is designed to provide users, administrators, and developers with a detailed understanding of the TFGrid Proxy, offering step-by-step instructions for its setup, essential commands, and insights into its various functionalities.
The Grid Proxy plays a pivotal role in facilitating secure and efficient communication between nodes within the ThreeFold Grid, contributing to the decentralized and autonomous nature of the network.
Whether you are a seasoned ThreeFold enthusiast or a newcomer exploring the decentralized web, this manual aims to be your go-to resource for navigating the ThreeFold Grid Proxy landscape.
To assist you on your journey, we have organized the content into distinct chapters below, covering everything from initial setup procedures and database testing to practical commands, contributions, and insights into the ThreeFold Explorer and the Grid Proxy Database functionalities.
<h2>Table of Contents</h2>
- [Introducing Grid Proxy](./proxy.md)
- [Setup](./setup.md)
- [DB Testing](./db_testing.md)
- [Commands](./commands.md)
- [Contributions](./contributions.md)
- [Explorer](./explorer.md)
- [Database](./database.md)
- [Production](./production.md)
- [Release](./release.md)

32
collections/manual/documentation/developers/proxy/release.md Normal file
View File

@@ -0,0 +1,32 @@
<h1>Release Grid-Proxy</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Steps](#steps)
- [Debugging](#debugging)
***
## Introduction
We show the steps to release a new version of the Grid Proxy.
## Steps
To release a new version of the Grid-Proxy component, follow these steps:
Update the `appVersion` field in the `charts/Chart.yaml` file. This field should reflect the new version number of the release.
The release process includes generating and pushing a Docker image with the latest GitHub tag. This step is automated through the `gridproxy-release.yml` workflow.
Trigger the `gridproxy-release.yml` workflow by pushing the desired tag to the repository. This will initiate the workflow, which will generate the Docker image based on the tag and push it to the appropriate registry.
## Debugging
In the event that the workflow does not run automatically after pushing the tag and making the release, you can manually execute it using the GitHub Actions interface. Follow these steps:
Go to the [GitHub Actions page](https://github.com/threefoldtech/tfgrid-sdk-go/actions/workflows/gridproxy-release.yml) for the Grid-Proxy repository.
Locate the workflow named gridproxy-release.yml.
Trigger the workflow manually by selecting the "Run workflow" option.

50
collections/manual/documentation/developers/proxy/setup.md Normal file
View File

@@ -0,0 +1,50 @@
<h1>Setup</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Install Golang](#install-golang)
- [Docker](#docker)
- [Postgres](#postgres)
- [Get Mnemonics](#get-mnemonics)
***
## Introduction
We show how to set up grid proxy.
## Install Golang
To install Golang, you can follow the official [guide](https://go.dev/doc/install).
## Docker
Docker is useful for running the TFGridDb in container environment. Read this to [install Docker engine](../../system_administrators/computer_it_basics/docker_basics.md#install-docker-desktop-and-docker-engine).
Note: it will be necessary to follow step #2 in the previous article to run docker without sudo. if you want to avoid that. edit the docker commands in the `Makefile` and add sudo.
## Postgres
If you have docker installed you can run postgres on a container with:
```bash
make db-start
```
Then you can either load a dump of the database if you have one:
```bash
make db-dump p=~/dump.sql
```
or easier you can fill the database tables with randomly generated data with the script `tools/db/generate.go` to do that run:
```bash
make db-fill
```
## Get Mnemonics
1. Install [polkadot extension](https://github.com/polkadot-js/extension) on your browser.
2. Create a new account from the extension. It is important to save the seeds.