FullNode-HOWTO

The aim of this document is to provide step-by-step instructions for setting up a full node for the TON Blockchain. We assume some familiarity with the TON Blockchain Lite Client, at least to the extent explained in the Lite Client HOWTO.

Note that you need a machine with a public IP address and a high-bandwidth network connection to run a TON Blockchain Full Node. Typically you'll need a sufficiently powerful server in a datacenter with good network connectivity, using at least a 1 Gbit/s connection to reliably accommodate peak loads (the average load is expected to be approximately 100 Mbit/s). We recommend a dual-processor server with at least eight cores in each processor, at least 256 MiB RAM, at least 8 TB of conventional HDD storage and at least 512 GB of faster SSD storage. It is a bad idea to run a Full Node on your home computer; instead, you could run a Full Node on a remote server, and use the TON Blockchain Lite Client to connect to it from home.

0. Downloading and compiling
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The complete TON Blockchain Library and Validator software is downloaded and compiled similarly to the Lite Client. This process is outlined in the corresponding README file. The most important difference is that you have to download the complete sources from public GitHub repository https://github.com/ton-blockchain/ton (e.g., by running `git clone https://github.com/ton-blockchain/ton` and `git submodule update` afterwards) instead of downloading the smaller Lite Client source archive. You should also build all goals defined in CMakeLists.txt (e.g. by running `cmake <path-to-source-directory>` and `make` in your build directory), not only those specifically related to the Lite Client (which is also included in the larger distribution; you don't have to download and build it separately). We strongly recommend building a "release" or a "release with debug information" version of the TON Blockchain Library and especially of the Validator/Full Node by passing `-DCMAKE_BUILD_TYPE=Release` or `-DCMAKE_BUILD_TYPE=RelWithDebInfo` as an extra argument to `cmake` during its first run (if you forgot to do this, you can later delete file `CMakeCache.txt` from your build directory and re-run `cmake` with the appropriate options).

1. Full Node binaries
~~~~~~~~~~~~~~~~~~~~~

After the sources have been compiled successfully, you should obtain executable files `validator-engine/validator-engine` and `validator-engine-console/validator-engine-console` in your build directory. These are the most important files you need to run a TON Blockchain Full Node (or even a Validator), and to control it. You might wish to install them into your /usr/bin or similar directory. You are also likely to need the `generate-random-id` utility during setup.

2. Working directory of the Full Node
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The Full Node (also known as "validator-engine") stores its data in subdirectories of its working directory, e.g., `/var/ton-work/db`. It requires write access to this directory. If you want to use another directory as the working directory of the Full Node, you can use the command line option --db <path-to-work-dir>:

$ validator-engine --db ${DB_ROOT}

where ${DB_ROOT} is /var/ton-work/db or any other directory where validator-engine has write permissions.

3. Working directory layout
~~~~~~~~~~~~~~~~~~~~~~~~~~~

An approximate layout of the working directory of the TON Blockchain Full Node software is as follows:

* ${DB_ROOT}/config.json -- Automatically generated configuration file. It is automatically regenerated by validator-engine on some occasions. When validator-engine is not running, you can edit this file in a text editor because it is a JSON file.
* ${DB_ROOT}/static -- A directory with files that cannot be downloaded from the network, such as the "zerostate" (corresponding to the Genesis block of other blockchain architectures) of the masterchain and active workchains. Normally you don't have to initialize this directory unless you want to run your own instance of the TON Blockchain (for example, for testing or development purposes). Full nodes of existing instances of the TON Blockchain (such as the "testnet" and the "mainnet") will be able to download all required files from already running full nodes.
* ${DB_ROOT}/keyring -- Stores public and private keys known to validator-engine. For example, if your full node runs as a validator for some of the TON Blockchain shardchains, the validator block signing key is kept here. You may wish to set more restrictive permissions for this directory, such as 0700 (in *nix systems), so that only the user under which validator-engine is running would have access to this directory.
* ${DB_ROOT}/error -- A directory where validator-engine copies files related to severe error conditions (e.g., invalid block candidates) for further study. It is normally empty, and validator-engine never deletes files from this directory.
* ${DB_ROOT}/archive -- A directory where old and rarely used blocks are kept until their storage period expires. You can mount a larger but slower disk partition at this directory, or make this directory a symlink to a directory in such a partition. We recommend locating the remainder of ${DB_ROOT} in a fast storage device such as an SSD.
* ${DB_ROOT}/etc -- (Non-automatic) configuration files may be kept here, or in any other directory read-accessible to validator-engine.
* Other subdirectories of ${DB_ROOT} are used to keep ADNL cache data, recent blocks and states, and so on. They are not relevant to the purposes of this document.

4. Global configuration of the TON Blockchain
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In order to set up your Full Node, you'll need a special JSON file called the "global configuration (file)". It is called this because it is the same for all full nodes, and even nodes participating in different instances of the TON Blockchain (e.g., "testnet" vs. "mainnet") share an almost identical global configuration.

The "testnet" global configuration can be downloaded at https://test.ton.org/ton-global.config.json as follows:

$ wget https://test.ton.org/ton-global.config.json

You may wish to put this file into /var/ton-work/etc/ton-global.config.json .

We'll discuss the structure of this file later in more detail. For now, let us remark that the bulk of this file consists of a list of known TON DHT nodes required for the bootstrapping of the TON Network. A smaller section near the end of this file describes the particular instance of the TON Blockchain that we wish to connect to.

All instances of the TON Blockchain use the same "global" TON Network (i.e., the TON Network is not fragmented into several instances for each blockchain instance). While the global network configuration is therefore independent of the particular TON Blockchain instance chosen, the Full Nodes belonging to different instances will later connect to different overlay subnetworks inside the TON Network.

It is important to distinguish this "global configuration file", used for setting up a TON Blockchain Full Node, and the "local" or "automatic configuration file", automatically updated by validator-engine and usually stored in ${DB_ROOT}/config.json. The global configuration is used to generate the initial automatic configuration file, which is thereafter updated by validator-engine itself (e.g., by adding new DHT nodes or storing hashes of newer masterchain blocks).

5. Initializing the local configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once the global configuration file is downloaded, it can be used to create the initial local configuration in ${DB_ROOT}/config.json. To do this, execute validator-engine once:

$ validator-engine -C /var/ton-work/etc/ton-global.config.json --db /var/ton-work/db/ --ip <IP>:<PORT> -l /var/ton-work/log

Here `/var/ton-work/log` is the log directory of `validator-engine`, where it will create its log files. The argument to the `-C` command-line option is the global configuration file downloaded from test.ton.org as explained above, and `/var/ton-work/db/` is the working directory ${DB_ROOT}. Finally, <IP>:<PORT> are the global IP address of this full node (you need to indicate a public IPv4 address here) and the UDP port used to run TON Network protocols such as ADNL and RLDP. Make sure that your firewall is configured to pass UDP packets with source or destination <IP>:<PORT> at least for the `validator-engine` binary.

When validator-engine is invoked as above, and ${DB_ROOT}/config.json does not exist, it creates a new local configuration file ${DB_ROOT}/config.json using the information from the global configuration file and from the command-line options such as --ip, and then exits. If ${DB_ROOT}/config.json already exists, it is not rewritten; instead validator-engine starts up as a daemon using both the local and the global configuration.

If you need to change the local configuration afterwards, you'll need to either delete this file and regenerate it from the global configuration (potentially forgetting other important information accumulated in the local configuration), or edit the local configuration in a text editor (when validator-engine is not running).

6. Setting up remote control CLI
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You will almost surely want to enable validator-engine-console in the local configuration, to be able to control your Full Node (i.e., validator-engine daemon) when it is running. For this, you'll need to generate two keypairs, one for the server (validator-engine) and one for the client (validator-engine-console). In the examples below we assume that validator-engine-console runs on the same machine and connects to validator-engine through the loopback network interface. (This is not necessarily so; you can use validator-engine-console for remote control as well.)

As a first step, use the `generate-random-id` executable to create two keypairs, one for the server (on the machine running `validator-engine`) and one for the client (on the machine running `validator-engine-console`):

$ ./generate-random-id -m keys -n server
6E9FD109F76E08B5710445C72D2C5FEDE04A96357DAA4EC0DDAEA025ED3AC3F7 bp/RCfduCLVxBEXHLSxf7eBKljV9qk7A3a6gJe06w/c=

This utility generates a new keypair and saves the private key into file `server` and the public key into `server.pub`. The hexadecimal (6E9F...F7) and the base64 ("bp/RC...6wc/=") representations of the public key are displayed in the standard output, and are used henceforth to identify this key.

We have to install the private key `server` into the keyring of the Full Node (validator-engine):

$ mv server /var/ton-work/db/keyring/6E9FD109F76E08B5710445C72D2C5FEDE04A96357DAA4EC0DDAEA025ED3AC3F7

Notice that the file name to store this private key inside the keyring equals the hexadecimal identifier (which essentially is a hash of the public key) of this key.

Next, we generate the client keypair:

$ ./generate-random-id -m keys -n client
8BBA4F8FCD7CC4EF573B9FF48DC63B212A8E9292B81FC0359B5DBB8719C44656 i7pPj818xO9XO5/0jcY7ISqOkpK4H8A1m127hxnERlY=

We obtain a client keypair, saved into files `client` and `client.pub`. This second operation should be run in the working directory of the client (validator-engine-console), possibly on another machine.

Now we have to list the client's public key in the server's local configuration file ${DB_ROOT}/config.json. To do this, open the local configuration file in a text editor (after terminating validator-engine if it was running) and find the empty "control" section:

"control": [
]

Replace it with the following:

"control" : [
  { "id" : "bp/RCfduCLVxBEXHLSxf7eBKljV9qk7A3a6gJe06w/c=",
    "port" : <CONSOLE-PORT>,
    "allowed" : [
      { "id" : "i7pPj818xO9XO5/0jcY7ISqOkpK4H8A1m127hxnERlY=",
        "permissions" : 15
      }
    ]
  }
],

`control.0.id` is set to the base64 identifier of the server's public key, and `control.0.allowed.0.id` is the base64 identifier of the client's public key. <CONSOLE-PORT> is the TCP port the server will listen to for console commands.

7. Running the Full Node
~~~~~~~~~~~~~~~~~~~~~~~~

To run the full node, simply run the validator-engine binary in a console:

$ validator-engine --db ${DB_ROOT} -C /var/ton-work/etc/ton-global.config.json  -l /var/ton-work/log

It will read the global configuration from /var/ton-work/etc/ton-global.config.json, the local configuration from ${DB_ROOT}/config.json, and continue running silently. You should write suitable scripts for invoking validator-engine as a daemon (so that it does not terminate when the console is closed), but we'll skip these considerations for simplicity. (The command-line option `-d` of validator-engine should be sufficient for this on most *nix systems.)

If the configuration is invalid, validator-engine will terminate immediately and, in most cases, output nothing. You'll have to study the log files under /var/ton-work/log to find out what went wrong. Otherwise, validator-engine will keep working silently. Again, you can understand what's going on by inspecting the log files, and by looking into subdirectories of the ${DB_ROOT} directory.

If everything works as expected, validator-engine will locate other full nodes participating in the same instance of the TON Blockchain, and download recent blocks of the masterchain and all shardchains. (You can actually control the number of recent blocks to be downloaded, or even download them all starting from the zerostate---but this topic is outside the scope of this document; try running validator-engine with command-line option `-h` to find out the list of all available options with brief explanations).

8. Using the Console CLI
~~~~~~~~~~~~~~~~~~~~~~~~

If the validator-engine-console has been set up as explained in Section 6., you can use it to connect to the running validator-engine (i.e., your Full Node) and run simple status and key management queries:

$ ./validator-engine-console -k client -p server.pub -a <IP>:<CLIENT-PORT>

connecting to [<IP>:<CLIENT-PORT>]
local key: 8BBA4F8FCD7CC4EF573B9FF48DC63B212A8E9292B81FC0359B5DBB8719C44656
remote key: 6E9FD109F76E08B5710445C72D2C5FEDE04A96357DAA4EC0DDAEA025ED3AC3F7
conn ready
> gettime
received validator time: time=1566568904

The `gettime` command obtains the current Unix time at the validator. If everything has been configured properly, you'll see an output similar to the one above. Notice that you need both the client's private key ("client") and the server's public key ("server.pub") for the console to work. You might wish to move them (especially the client's private key) into a separate directory with suitable permissions.

Other console commands are available in validator-engine-console. For instance, `help` displays a list of all console commands with short descriptions. In particular, they are used to set up the Full Node as a Validator for the TON Blockchain, as explained in the separate document Validator-HOWTO.

9. Setting up the Full Node as a Lite Server
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can set up your Full Node to function as a Lite Server, so that you can use the Lite Client to connect to it from the same or a remote host. For instance, sending the command `last` in a Lite Client connected to your Full Node will display the identifier of the most recent masterchain block known to your Full Node, so that you can inspect the progress of block downloading. You can also inspect the state of all smart contracts, send external messages (e.g., wallet queries), and so on as explained in the Lite Client HOWTO.

In order to set up your Full Node as a Lite Server, you have to generate another keypair and install the private key into the server's keyring, similarly to what we did to enable the remote console:

$ utils/generate-random-id -m keys -n liteserver
BDFEA84525ADB3B16D0192488837C04883C10FF1F4031BB6DEECDD17544F5347 vf6oRSWts7FtAZJIiDfASIPBD/H0Axu23uzdF1RPU0c=
mv liteserver ${DB_ROOT}/keyring/BDFEA84525ADB3B16D0192488837C04883C10FF1F4031BB6DEECDD17544F5347

After that, stop validator-engine if it is running and open the local configuration file ${TON_DB}/config.json in a text editor. Find the empty section

"liteservers" : [
]

and replace it with a record containing the TCP port for listening to inbound Lite Client connections and the lite server's public key:

"liteservers" : [
{
"id" : "vf6oRSWts7FtAZJIiDfASIPBD/H0Axu23uzdF1RPU0c=",
"port" : <TCP-PORT>
}
],

Now start `validator-engine` again. If it does not terminate immediately, it is likely that you have re-configured it properly. Now you can use the lite-client binary (usually located as "lite-client/lite-client" with respect to the build directory) to connect to the Lite Server running as a part of your Full Node:

$ lite-client -a <IP>:<TCP-PORT> -p liteserver.pub

Again, `help` lists all commands available in the Lite Client. The Lite Client HOWTO contains some examples of what can be done with the Lite Client.