manual transfer done for documentation, still hero issues for parsing

collections/manual/documentation/system_administrators/advanced/advanced.md

@@ -0,0 +1,14 @@
+<h1> TFGrid Advanced </h1>
+
+In this section, we delve into sophisticated topics and powerful functionalities that empower you to harness the full potential of TFGrid 3.0. Whether you're an experienced user seeking to deepen your understanding or a trailblazer venturing into uncharted territories, this manual is your gateway to mastering advanced concepts on the ThreeFold Grid.
+
+<h2>Table of Contents</h2>
+
+- [Token Transfer Keygenerator](./token_transfer_keygenerator.md)
+- [Cancel Contracts](./cancel_contracts.md)
+- [Contract Bills Reports](./contract_bill_report.md)
+- [Listing Free Public IPs](./list_public_ips.md)
+- [Redis](./grid3_redis.md)
+- [IPFS](./ipfs/ipfs_toc.md)
+  - [IPFS on a Full VM](./ipfs/ipfs_fullvm.md)
+  - [IPFS on a Micro VM](./ipfs/ipfs_microvm.md)

collections/manual/documentation/system_administrators/advanced/cancel_contracts.md

@@ -0,0 +1,48 @@
+<h1> Cancel Contracts </h1>
+
+<h2>Table of Contents </h2>
+
+- [Introduction](#introduction)
+- [Using the Dashboard](#using-the-dashboard)
+- [Using GraphQL and Polkadot UI](#using-graphql-and-polkadot-ui)
+- [Using grid3\_client\_ts](#using-grid3_client_ts)
+
+***
+
+## Introduction
+
+We present different methods to delete contracts on the TFGrid.
+
+## Using the Dashboard
+
+To cancel contracts with the Dashboard, consult the [Contracts List](../../dashboard/deploy/your_contracts.md) documentation.
+
+## Using GraphQL and Polkadot UI
+
+From the QraphQL service execute the following query.
+
+```
+query MyQuery {
+
+  nodeContracts(where: {twinId_eq: TWIN_ID, state_eq: Created}) {
+    contractId
+  }
+}
+
+```
+
+replace `TWIN_ID` with your twin id. The information should be available on the [Dashboard](../../dashboard/dashboard.md).
+
+Then from [polkadot UI](https://polkadot.js.org/apps/), add the tfchain endpoint to development.
+
+![](img/polka_web_add_development_url.png)
+
+Go to `Extrinsics`, choose the `smartContract` module and `cancelContract` extrinsic and use the IDs from GraphQL to execute the cancelation.
+
+![](img/polka_web_cancel_contracts.jpg)
+
+## Using grid3_client_ts
+
+In order to use the `grid3_client_ts` module, it is essential to first clone our official mono-repo containing the module and then navigate to it. If you are looking for a quick and efficient way to cancel contracts, we offer a code-based solution that can be found [here](https://github.com/threefoldtech/tfgrid-sdk-ts/blob/development/packages/grid_client/scripts/delete_all_contracts.ts).
+
+To make the most of `grid_client`, we highly recommend following our [Grid-Client guide](https://github.com/threefoldtech/tfgrid-sdk-ts/blob/development/packages/grid_client/README.md) for a comprehensive overview of the many advanced capabilities offered by this powerful tool. With features like contract creation, modification, and retrieval, `grid_client` provides an intuitive and easy-to-use solution for managing your contracts effectively.

collections/manual/documentation/system_administrators/advanced/contract_bill_report.md

@@ -0,0 +1,63 @@
+<h1> Contract Bills Reports </h1>
+
+<h2>Table of Contents</h2>
+
+- [Introduction](#introduction)
+- [Contract Billing Report (GraphQL)](#contract-billing-report-graphql)
+- [Consumption](#consumption)
+
+***
+
+## Introduction
+
+Now you can check the billing rate of your contracts directly from the `Contracts` tab in the Dashboard.
+
+> It takes an hour for the contract to display the billing rate (Until it reaches the first billing cycle).
+
+The `Billing Rate` is displayed in `TFT/Hour`
+
+![image](img/billing_rate.png)
+
+## Contract Billing Report (GraphQL)
+
+- you need to find the contract ID
+- ask the graphql for the consumption
+
+> example query for all contracts
+
+```graphql
+query MyQuery {
+  contractBillReports {
+    contractId
+    amountBilled
+    discountReceived
+  }
+}
+```
+
+And for a specific contract
+
+```graphql
+query MyQuery {
+  contractBillReports(where: { contractId_eq: 10 }) {
+    amountBilled
+    discountReceived
+    contractId
+  }
+}
+```
+
+## Consumption
+
+```graphql
+query MyQuery {
+  consumptions(where: { contractId_eq: 10 }) {
+    contractId
+    cru
+    sru
+    mru
+    hru
+    nru
+  }
+}
+```

collections/manual/documentation/system_administrators/advanced/grid3_redis.md

@@ -0,0 +1,46 @@
+<h1> Redis </h1>
+
+<h2> Table of Contents </h2>
+
+- [Introduction](#introduction)
+- [Install Redis](#install-redis)
+  - [Linux](#linux)
+  - [MacOS](#macos)
+- [Run Redis](#run-redis)
+
+***
+
+## Introduction
+
+Redis is an open-source, in-memory data structure store that is widely used as a caching layer, message broker, and database. It is known for its speed, versatility, and support for a wide range of data structures. Redis is designed to deliver high-performance data access by storing data in memory, which allows for fast read and write operations. It supports various data types, including strings, lists, sets, hashes, and more, and provides a rich set of commands for manipulating and querying the data.
+
+Redis is widely used in various use cases, including caching, session management, real-time analytics, leaderboards, task queues, and more. Its simplicity, speed, and flexibility make it a popular choice for developers who need a fast and reliable data store for their applications. In Threefold's ecosystem context, Redis can be used as a backend mechanism to communicate with the nodes on the ThreeFold Grid using the Reliable Message Bus.
+
+
+
+## Install Redis
+
+### Linux
+
+If you don't find Redis in your Linux distro's package manager, check the [Redis downloads](https://redis.io/download) page for the source code and installation instructions.
+
+### MacOS
+
+On MacOS, [Homebrew](https://brew.sh/) can be used to install Redis. The steps are as follow:
+
+```
+brew update
+brew install redis
+```
+
+Alternatively, it can be built from source, using the same [download page](https://redis.io/download/) as shown above.
+
+
+
+## Run Redis
+
+You can launch the Redis server with the following command:
+
+```
+redis-server
+```

collections/manual/documentation/system_administrators/advanced/grid3_stellar_tfchain_bridge.md

@@ -0,0 +1,39 @@
+<h1> Transferring TFT Between Stellar and TFChain</h1>
+
+<h2>Table of Contents</h2>
+
+- [Usage](#usage)
+- [Prerequisites](#prerequisites)
+- [Stellar to TFChain](#stellar-to-tfchain)
+- [TFChain to Stellar](#tfchain-to-stellar)
+
+***
+
+## Usage
+
+This document will explain how you can transfer TFT from Tfchain to Stellar and back.
+
+For more information on TFT bridges, read [this documentation](../threefold_token/tft_bridges/tft_bridges.md).
+
+## Prerequisites
+
+- [Stellar wallet](../threefold_token/storing_tft/storing_tft.md)
+
+- [Account on TFChain (use TF Dashboard to create one)](../dashboard/wallet_connector.md)
+
+![](./img/bridge.png)
+
+## Stellar to TFChain
+
+You can deposit to Tfchain using the bridge page on the TF Dashboard, click deposit:
+
+![bridge](./img/bridge_deposit.png)
+
+## TFChain to Stellar
+
+You can bridge back to stellar using the bridge page on the dashboard, click withdraw:
+
+![withdraw](./img/bridge_withdraw.png)
+
+A withdrawfee of 1 TFT will be taken, so make sure you send a larger amount as 1 TFT.
+The amount withdrawn from TFChain will be sent to your Stellar wallet.

collections/manual/documentation/system_administrators/advanced/ipfs/ipfs_fullvm.md

@@ -0,0 +1,190 @@
+<h1> IPFS on a Full VM</h1>
+
+<h2>Table of Contents</h2>
+
+- [Introduction](#introduction)
+- [Deploy a Full VM](#deploy-a-full-vm)
+- [Create a Root-Access User](#create-a-root-access-user)
+- [Set a Firewall](#set-a-firewall)
+  - [Additional Ports](#additional-ports)
+- [Install IPFS](#install-ipfs)
+- [Set IPFS](#set-ipfs)
+- [Final Verification](#final-verification)
+- [Questions and Feedback](#questions-and-feedback)
+
+***
+
+## Introduction
+
+In this ThreeFold guide, we explore how to set an IPFS node on a Full VM using the ThreeFold Playground.
+
+## Deploy a Full VM
+
+We start by deploying a full VM on the ThreeFold Playground.
+
+* Go to the [Threefold Playground](https://playground.grid.tf/#/)
+* Deploy a full VM (Ubuntu 20.04) with an IPv4 address and at least the minimum specs
+  * IPv4 Address
+  * Minimum vcores: 1vcore
+  * Minimum MB of RAM: 1024GB
+  * Minimum storage: 50GB
+* After deployment, note the VM IPv4 address
+* Connect to the VM via SSH
+  * ``` 
+    ssh root@VM_IPv4_address
+    ```
+
+## Create a Root-Access User
+
+We create a root-access user. Note that this step is optional.
+
+* Once connected, create a new user with root access (for this guide we use "newuser")
+  * ``` 
+    adduser newuser
+    ```
+  * You should now see the new user directory
+    * ``` 
+      ls /home
+      ```
+  * Give sudo capacity to the new user
+    * ```
+      usermod -aG sudo newuser
+      ```
+  * Switch to the new user
+    * ```
+      su - newuser
+      ```
+  * Create a directory to store the public key
+    * ```
+      mkdir ~/.ssh
+      ```
+  * Give read, write and execute permissions for the directory to the new user
+    * ```
+      chmod 700 ~/.ssh
+      ```
+  * Add the SSH public key in the file **authorized_keys** and save it
+    * ```
+      nano ~/.ssh/authorized_keys
+      ```
+* Exit the VM 
+  * ```
+    exit
+    ```
+* Reconnect with the new user
+  * ``` 
+    ssh newuser@VM_IPv4_address
+    ```
+
+## Set a Firewall
+
+We set a firewall to monitor and control incoming and outgoing network traffic. To do so, we will define predetermined security rules. As a firewall, we will be using [Uncomplicated Firewall](https://wiki.ubuntu.com/UncomplicatedFirewall) (ufw).
+For our security rules, we want to allow SSH, HTTP and HTTPS (443 and 8443).
+We thus add the following rules:
+* Allow SSH (port 22)
+  * ```
+    sudo ufw allow ssh
+    ```
+* Allow port 4001
+  * ```
+    sudo ufw allow 4001
+    ```
+* To enable the firewall, write the following:
+  * ```
+    sudo ufw enable
+    ```
+* To see the current security rules, write the following:
+  * ```
+    sudo ufw status verbose
+    ```
+You now have enabled the firewall with proper security rules for your IPFS deployment.
+
+### Additional Ports
+
+We provided the basic firewall ports for your IPFS instance. There are other more advanced configurations possible.
+If you want to access your IPFS node remotely, you can allow **port 5001**. This will allow anyone to access your IPFS node. Make sure that you know what you are doing if you go this route. You should, for example, restrict which external IP address can access port 5001.
+If you want to run your deployment as a gateway node, you should allow **port 8080**. Read the IPFS documentation for more information on this.
+If you want to run pubsub capabilities, you need to allow **port 8081**. For more information, read the [IPFS documentation](https://blog.ipfs.tech/25-pubsub/).
+
+## Install IPFS
+
+We install the [IPFS Kubo binary](https://docs.ipfs.tech/install/command-line/#install-official-binary-distributions).
+* Download the binary
+  * ```
+    wget https://dist.ipfs.tech/kubo/v0.24.0/kubo_v0.24.0_linux-amd64.tar.gz
+    ```
+* Unzip the file
+  * ```
+    tar -xvzf kubo_v0.24.0_linux-amd64.tar.gz
+    ```
+* Change directory
+  * ```
+    cd kubo
+    ```
+* Run the install script
+  * ```
+    sudo bash install.sh
+    ```
+* Verify that IPFS Kubo is properly installed
+  * ```
+    ipfs --version
+    ```
+
+## Set IPFS
+
+We initialize IPFS and run the IPFS daemon.
+
+* Initialize IPFS
+  * ```
+    ipfs init --profile server
+    ```
+* Increase the storage capacity (optional)
+  * ```
+    ipfs config Datastore.StorageMax 30GB
+    ```
+* Run the IPFS daemon
+  * ```
+    ipfs daemon
+    ```
+* Set an Ubuntu systemd service to keep the IPFS daemon running after exiting the VM
+  * ```
+    sudo nano /etc/systemd/system/ipfs.service
+    ```
+* Enter the systemd info
+  * ```
+    [Unit]
+    Description=IPFS Daemon
+    [Service]
+    Type=simple
+    ExecStart=/usr/local/bin/ipfs daemon --enable-gc
+    Group=newuser
+    Restart=always
+    Environment="IPFS_PATH=/home/newuser/.ipfs"
+    [Install]
+    WantedBy=multi-user.target
+    ```
+* Enable the service
+  * ```
+    sudo systemctl daemon-reload
+    sudo systemctl enable ipfs
+    sudo systemctl start ipfs
+    ```
+* Verify that the IPFS daemon is properly running
+  * ```
+    sudo systemctl status ipfs
+    ```
+## Final Verification
+We reboot and reconnect to the VM and verify that IPFS is properly running as a final verification.
+* Reboot the VM
+  * ```
+    sudo reboot
+    ```
+* Reconnect to the VM
+  * ```
+    ssh newuser@VM_IPv4_address
+    ```
+* Check that the IPFS daemon is running
+  * ```
+    ipfs swarm peers
+    ```
+## Questions and Feedback
+If you have any questions or feedback, please let us know by either writing a post on the [ThreeFold Forum](https://forum.threefold.io/), or by chatting with us on the [TF Grid Tester Community](https://t.me/threefoldtesting) Telegram channel.

collections/manual/documentation/system_administrators/advanced/ipfs/ipfs_microvm.md

@@ -0,0 +1,167 @@
+<h1> IPFS on a Micro VM</h1>
+
+<h2>Table of Contents</h2>
+
+- [Introduction](#introduction)
+- [Deploy a Micro VM](#deploy-a-micro-vm)
+- [Install the Prerequisites](#install-the-prerequisites)
+- [Set a Firewall](#set-a-firewall)
+  - [Additional Ports](#additional-ports)
+- [Install IPFS](#install-ipfs)
+- [Set IPFS](#set-ipfs)
+- [Set IPFS with zinit](#set-ipfs-with-zinit)
+- [Final Verification](#final-verification)
+- [Questions and Feedback](#questions-and-feedback)
+
+***
+
+## Introduction
+
+In this ThreeFold guide, we explore how to set an IPFS node on a micro VM using the ThreeFold Playground.
+
+## Deploy a Micro VM
+
+We start by deploying a micro VM on the ThreeFold Playground.
+
+* Go to the [Threefold Playground](https://playground.grid.tf/#/)
+* Deploy a micro VM (Ubuntu 22.04) with an IPv4 address
+  * IPv4 Address
+  * Minimum vcores: 1vcore
+  * Minimum MB of RAM: 1024MB
+  * Minimum storage: 50GB
+* After deployment, note the VM IPv4 address
+* Connect to the VM via SSH
+  * ``` 
+    ssh root@VM_IPv4_address
+    ```
+
+## Install the Prerequisites
+
+We install the prerequisites before installing and setting IPFS.
+
+* Update Ubuntu
+  * ```
+    apt update
+    ```
+* Install nano and ufw
+  * ```
+    apt install nano && apt install ufw -y
+    ```
+
+## Set a Firewall
+
+We set a firewall to monitor and control incoming and outgoing network traffic. To do so, we will define predetermined security rules. As a firewall, we will be using [Uncomplicated Firewall](https://wiki.ubuntu.com/UncomplicatedFirewall) (ufw).
+
+For our security rules, we want to allow SSH, HTTP and HTTPS (443 and 8443).
+
+We thus add the following rules:
+
+* Allow SSH (port 22)
+  * ```
+    ufw allow ssh
+    ```
+* Allow port 4001
+  * ```
+    ufw allow 4001
+    ```
+* To enable the firewall, write the following:
+  * ```
+    ufw enable
+    ```
+
+* To see the current security rules, write the following:
+  * ```
+    ufw status verbose
+    ```
+
+You have enabled the firewall with proper security rules for your IPFS deployment.
+
+### Additional Ports
+
+We provided the basic firewall ports for your IPFS instance. There are other more advanced configurations possible.
+
+If you want to access your IPFS node remotely, you can allow **port 5001**. This will allow anyone to access your IPFS node. Make sure that you know what you are doing if you go this route. You should, for example, restrict which external IP address can access port 5001.
+
+If you want to run your deployment as a gateway node, you should allow **port 8080**. Read the IPFS documentation for more information on this.
+
+If you want to run pubsub capabilities, you need to allow **port 8081**. For more information, read the [IPFS documentation](https://blog.ipfs.tech/25-pubsub/).
+
+## Install IPFS
+
+We install the [IPFS Kubo binary](https://docs.ipfs.tech/install/command-line/#install-official-binary-distributions).
+
+* Download the binary
+  * ```
+    wget https://dist.ipfs.tech/kubo/v0.24.0/kubo_v0.24.0_linux-amd64.tar.gz
+    ```
+* Unzip the file
+  * ```
+    tar -xvzf kubo_v0.24.0_linux-amd64.tar.gz
+    ```
+* Change directory
+  * ```
+    cd kubo
+    ```
+* Run the install script
+  * ```
+    bash install.sh
+    ```
+* Verify that IPFS Kubo is properly installed
+  * ```
+    ipfs --version
+    ```
+
+## Set IPFS
+
+We initialize IPFS and run the IPFS daemon.
+
+* Initialize IPFS
+  * ```
+    ipfs init --profile server
+    ```
+* Increase the storage capacity (optional)
+  * ```
+    ipfs config Datastore.StorageMax 30GB
+    ```
+* Run the IPFS daemon
+  * ```
+    ipfs daemon
+    ```
+
+## Set IPFS with zinit
+
+We set the IPFS daemon with zinit. This will make sure that the IPFS daemon starts at each VM reboot or if it stops functioning momentarily.
+
+* Create the yaml file
+  * ```
+    nano /etc/zinit/ipfs.yaml
+    ```
+* Set the execution command
+  * ```
+    exec: /usr/local/bin/ipfs daemon
+    ```
+* Run the IPFS daemon with the zinit monitor command
+  * ```
+    zinit monitor ipfs
+    ```
+* Verify that the IPFS daemon is running
+  * ```
+    ipfs swarm peers
+    ```
+
+## Final Verification
+
+We reboot and reconnect to the VM and verify that IPFS is properly running as a final verification.
+
+* Reboot the VM
+  * ```
+    reboot -f
+    ```
+* Reconnect to the VM and verify that the IPFS daemon is running
+  * ```
+    ipfs swarm peers
+    ```
+
+## Questions and Feedback
+
+If you have any questions or feedback, please let us know by either writing a post on the [ThreeFold Forum](https://forum.threefold.io/), or by chatting with us on the [TF Grid Tester Community](https://t.me/threefoldtesting) Telegram channel.

collections/manual/documentation/system_administrators/advanced/ipfs/ipfs_toc.md

@@ -0,0 +1,6 @@
+<h1>IPFS and ThreeFold</h1>
+
+<h2>Table of Contents</h2>
+
+- [IPFS on a Full VM](./ipfs_fullvm.md)
+- [IPFS on a Micro VM](./ipfs_microvm.md) 

collections/manual/documentation/system_administrators/advanced/list_public_ips.md

@@ -0,0 +1,22 @@
+<h1> Listing Public IPs </h1>
+
+<h2>Table of Contents </h2>
+
+- [Introduction](#introduction)
+- [Example](#example)
+
+***
+
+## Introduction
+
+Listing public IPs can be done by asking graphQL for all IPs that has `contractId = 0`
+
+## Example
+
+```graphql
+query MyQuery {
+  publicIps(where: {contractId_eq: 0}) {
+    ip
+  }
+}
+```

collections/manual/documentation/system_administrators/advanced/token_transfer_keygenerator.md

@@ -0,0 +1,88 @@
+
+<h1> Transfer TFT Between Networks by Using the Keygenerator </h1>
+
+<h2>Table of Contents </h2>
+
+- [Introduction](#introduction)
+- [Prerequisites](#prerequisites)
+  - [Keypair](#keypair)
+  - [Stellar to TFChain](#stellar-to-tfchain)
+  - [Alternative Transfer to TF Chain](#alternative-transfer-to-tf-chain)
+- [TFChain to Stellar](#tfchain-to-stellar)
+
+***
+
+## Introduction
+
+Using this method, only transfer is possible between accounts that are generated in the same manner and that are yours. Please find the keygen tooling for it below.
+
+## Prerequisites
+
+### Keypair
+
+- ed25519 keypair
+- Go installed on your local computer
+
+Create a keypair with the following tool: <https://github.com/threefoldtech/tfchain_tft/tree/main/tfchain_bridge/tools/keygen>
+
+```sh
+go build .
+./keygen
+```
+
+### Stellar to TFChain
+
+Create a Stellar wallet from the key that you generated.
+Transfer the TFT from your wallet to the bridge address. A deposit fee of 1 TFT will be taken, so make sure you send a larger amount as 1 TFT.
+
+Bridge addresses :
+
+- On Mainnet: `GBNOTAYUMXVO5QDYWYO2SOCOYIJ3XFIP65GKOQN7H65ZZSO6BK4SLWSC` on [Stellar Mainnet](https://stellar.expert/explorer/public).
+- On testnet: `GA2CWNBUHX7NZ3B5GR4I23FMU7VY5RPA77IUJTIXTTTGKYSKDSV6LUA4` on [Stellar MAINnet](https://stellar.expert/explorer/public)
+
+The amount deposited on TF Chain minus 1 TFT will be transferred over the bridge to the TFChain account.
+
+Effect will be the following :
+
+- Transferred TFTs from Stellar will be sent to a Stellar vault account representing all tokens on TFChain
+- TFTs will be minted on the TFChain for the transferred amount
+
+### Alternative Transfer to TF Chain
+
+We also enabled deposits to TF Grid objects. Following objects can be deposited to:
+
+- Twin
+- Farm
+- Node
+- Entity
+
+To deposit to any of these objects, a memo text in format `object_objectID` must be passed on the deposit to the bridge wallet. Example: `twin_1`.
+
+To deposit to a TF Grid object, this object **must** exists. If the object is not found on chain, a refund is issued.
+
+## TFChain to Stellar
+
+Create a TFChain account from the key that you generated. (TF Chain raw seed).
+Browse to :
+
+- For mainnet: <https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Ftfchain.grid.tf#/accounts>
+- For testnet: <https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Ftfchain.test.grid.tf#/accounts>
+- For Devnet: https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Ftfchain.dev.grid.tf#/accounts
+
+-> Add Account -> Click on mnemonic and select `Raw Seed` -> Paste raw TF Chain seed.
+
+Select `Advanced creation options` -> Change `keypair crypto type` to `Edwards (ed25519)`. Click `I have saved my mnemonic seed safely` and proceed.
+
+Choose a name and password and proceed.
+
+Browse to the [extrinsics](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Ftfchain.test.grid.tf#/extrinsics) <!--- or [Devnet](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Ftfchain.dev.grid.tf#/extrinsics) -->, select tftBridgeModule and extrinsic: `swap_to_stellar`. Provide your Bridge substrate address and the amount to transfer. Sign using your password.
+Again, a withdrawfee of 1 TFT will be taken, so make sure you send an amount larger than 1 TFT.
+
+The amount withdrawn from TFChain will be sent to your Stellar wallet.
+
+Behind the scenes, following will happen:
+
+- Transferred TFTs from Stellar will be sent from the Stellar vault account to the user's Stellar account
+- TFTs will be burned on the TFChain for the transferred amount
+
+Example: ![swap_to_stellar](img/swap_to_stellar.png ':size=400')