Merge pull request eth-educators#23 from yorickdowne/grafana-devel

Clarified the README. Many thanks to OhraeMez!

Author: yorickdowne

.eth2/.gitignore

@@ -1,3 +1,4 @@
 *
 !.gitignore
 !README.md
+!validator_keys

.eth2/README.md

@@ -1,4 +1,4 @@
-Th directory `validator_keys` will contain the `deposit_data-TIMESTAMP.json` and `keystore-m_ID.json`
+The directory `validator_keys` will contain the `deposit_data-TIMESTAMP.json` and `keystore-m_ID.json`
 files created by eth2.0-deposit-cli.
 
 Use `deposit_data-TIMESTAMP.json` for your initial deposit. After that, it can be disposed of.
@@ -7,7 +7,7 @@ Use `keystore-m_ID.json` files to import your validator secret keys into the val
 instance of the client you are running. These files need to be secured when you are done
 with the initial import.
 
-#Key Security
+# Validator Key Security
 
 The `keystore-m_ID.json` files have to be stored securely outside of this server. Offline
 is best, on media that cannot be remotely compromised. Keep the password(s) for
@@ -22,6 +22,8 @@ to slash your validator(s).
 
 For more on validator key security, read this article: https://www.attestant.io/posts/protecting-validator-keys/
 
+# Withdrawal Key Security
+
 **Critical**
 When you ran eth2.0-deposit-cli, a 24-word mnemonic was created. This mnemonic
 will be used for eth2 withdrawals in the future. It must be securely kept offline.

.eth2/validator_keys/.gitignore

@@ -0,0 +1,3 @@
+*
+!.gitignore
+!README.md

.eth2/validator_keys/README.md

@@ -0,0 +1,40 @@
+This directory `validator_keys` will contain the `deposit_data-TIMESTAMP.json` and `keystore-m_ID.json`
+files created by eth2.0-deposit-cli.
+
+Use `deposit_data-TIMESTAMP.json` for your initial deposit. After that, it can be disposed of.
+
+Use `keystore-m_ID.json` files to import your validator secret keys into the validator
+instance of the client you are running. These files need to be secured when you are done
+with the initial import.
+
+# Validator Key Security
+
+The `keystore-m_ID.json` files have to be stored securely outside of this server. Offline
+is best, on media that cannot be remotely compromised. Keep the password(s) for
+these files secure as well, for example in a local (not cloud-connected) password vault
+on a PC that is not on the network, or at the very least not used for online access. 
+
+These files will be needed in case you need to restore your validator(s).
+
+**Caution**
+An attacker with access to these files could slash your validator(s) or threaten
+to slash your validator(s).
+
+For more on validator key security, read this article: https://www.attestant.io/posts/protecting-validator-keys/
+
+# Withdrawal Key Security
+
+**Critical**
+When you ran eth2.0-deposit-cli, a 24-word mnemonic was created. This mnemonic
+will be used for eth2 withdrawals in the future. It must be securely kept offline.
+
+Precise methods are beyond this README, but consider something as simple as
+a sheet of paper kept in a fireproof envelope in a safe, or one of the steel
+mnemonic safeguards that are available.
+
+Test your mnemonic **before** you deposit, so you know that you will be able
+to withdraw funds in future.
+
+An attacker with access to your mnemonic can drain your funds.
+
+For more on withdrawal key security, read this article: https://www.attestant.io/posts/protecting-withdrawal-keys/

KEY-SECURITY.md

@@ -7,7 +7,7 @@ Use `keystore-m_ID.json` files to import your validator secret keys into the val
 instance of the client you are running. These files need to be secured when you are done
 with the initial import.
 
-# Key Security
+# Validator Key Security
 
 The `keystore-m_ID.json` files have to be stored securely outside of this server. Offline
 is best, on media that cannot be remotely compromised. Keep the password(s) for
@@ -25,6 +25,8 @@ to slash your validator(s).
 
 For more on validator key security, read this article: https://www.attestant.io/posts/protecting-validator-keys/
 
+# Withdrawal Key Security
+
 **Critical**
 When you ran eth2.0-deposit-cli, a 24-word mnemonic was created. This mnemonic
 will be used for eth2 withdrawals in the future. It must be securely kept offline.

README.md

@@ -11,11 +11,30 @@ Currently included clients:
 
 # USAGE
 
+## Before you start
+
 A file 'default.env' is provided and needs to be copied to '.env'.
 If this is not done, running docker-compose will fail.
+
 After copying the file, you **must** adjust the contents to your environment.
 Specifically, set the first entry `DEPCLI_UID` to your UID if you are running
-this on Linux. You can find your UID with `echo $UID`.
+this on Linux. You can find your UID with `echo $UID`.<br />
+You likely also *want* to set `GRAFFITI` inside the same file, and you might
+adjust `NUM_VAL` if you are going to create keys for more than one validator.
+
+On Linux, `docker-compose` runs as root. The individual containers
+do not, they run as local users inside the container.
+
+**Do not run two validators**<br />
+You'd get yourself slashed, and no-one wants that. Protecting you from this
+is a work in progress. Choose one client, and one client only, and run that.
+
+**You need an eth1 source**<br />
+This project assumes you'll use geth. It doesn't have to be that, it can
+be a 3rd party. You need some source for eth1, so that your validator can
+successfully propose blocks.
+
+Some troubleshooting commands are at the very end of the file.
 
 ## Install prerequisites
 
@@ -26,6 +45,23 @@ have write access to - your $HOME is fine - and pull this repo
 via git: `git clone https://github.com/eth2-educators/eth2-docker.git`,
 then `cd eth2-docker` into the newly created directory.
 
+## Choose a client
+
+I am deathly afraid of causing you to run two validators in parallel, leading to you getting slashed.
+
+Therefore, I am asking you to choose one client, and one client only, and copy its corresponding file in.
+This is not ideal. If you'd like to collaborate on a shell wrapper that makes this easier, please get in touch via
+the ethstaker Discord.
+
+There is a default file which runs lighthouse with local geth. If you are good with the default, just run with that.
+
+You'll be copying from the directory `clients` to the file `docker-compose.yml` in this directory. Current options are:
+- lighthouse, `cp clients/lh.yml ./docker-compose.yml`
+- prysm, `cp clients/prysm.yml ./docker-compose.yml`
+
+If you are going to use a 3rd-party eth1chain provider, edit `.env` and set either `LH_ETH1_NODE` or `PRYSM_ETH1_NODE` to
+point to your provider, and use the `eth2-3rd` target once you have imported keys and are ready.
+
 ## Create an eth2 wallet and deposit files
 
 You will deposit eth to the deposit contract, and receive locked eth2 in turn.
@@ -42,7 +78,9 @@ is just one (1) validator.
 With that said, this command will get you ready to deposit eth:
 `sudo docker-compose run deposit-cli`
 
-The created files will be in the directory `.eth2` in this project.
+The created files will be in the directory `.eth2/validator_keys` in this project.
+This is also where you'd place your own keystore files if you already have some for import.
+
 Please see the file `KEY-SECURITY.md` in this project for some notes on
 key security.
 
@@ -64,8 +102,16 @@ While this project gives you the freedom to shoot yourself in the foot like that
 
 ## Create a wallet by importing validator keys
 
+### You brought your own keys
+
+They go into `.eth2/validator_keys` in this project directory, not directly under `$HOME`. 
+Make **damn sure** your old validator is down and will stay down, you will get slashed
+if both are up at the same time.
+
 ### Lighthouse
 
+**Warning** Import your validator key(s) to only *one* client.
+
 Once both your withdrawal key (mnemonic) and validator key(s) (`keystore-m_ID.json` file(s))
 are secured offline, and **not** before, import the validator key(s) to the Lighthouse
 validator client:
@@ -120,25 +166,48 @@ Then, and **only** then:
 
 ### Lighthouse
 
-To start the lighthouse client, both beacon and validator, with local geth:
+To start the Lighthouse client, both beacon and validator, with local geth:
+
+```
+sudo docker-compose up -d eth2
+```
+
+Instead, if you are using a 3rd-party eth1chain, make sure that `LH_ETH1_NODE` in the file `.env` is pointing to it.
+
+To start the Lighthouse client, both beacon and validator, with 3rd party eth1chain:
 
 ```
-sudo docker-compose up -d lighthouse
+sudo docker-compose up -d eth2-3rd
 ```
 
 If, however, you chose not to store the wallet password locally, bring the services
 up individually instead:
 
+With local geth:
+
 ```
 sudo docker-compose up -d geth lh-beacon
+```
+
+Or with 3rd party eth1chain:
+```
+sudo docker-compose up -d lh-beacon
+
+```
+
+Then "run" the validator so it can prompt you for input:
+```
 sudo docker-compose run lh-validator
 ```
 
 After providing the wallet password, use the key sequence Ctrl-p Ctrl-q to detach
 from the running container.
 
+
 ### Prysm
 
+The Prysm client requires copying in a file, see the start of this document.
+
 Note that the Prysm client will find its external IP, but this repo assumes
 that IP is static. You can restart the container, possibly via crontab, with
 `docker-compose restart prysm-beacon` if your IP is dynamic. 
@@ -147,14 +216,31 @@ Work to support dynamic DNS would also be welcome.
 To start the Prysm client, both beacon and validator, with local geth:
 
 ```
-sudo docker-compose up -d prysm
+sudo docker-compose up -d eth2
+```
+
+Instead, if you are using a 3rd-party eth1chain, make sure that `PRYSM_ETH1_NODE` in the file `.env` is pointing to it.
+To start the Prysm client, both beacon and validator, with 3rd party eth1chain:
+
+```
+sudo docker-compose up -d eth2-3rd
 ```
 
 If, however, you chose not to store the wallet password locally, bring the services
 up individually instead:
 
+With local geth:
 ```
 sudo docker-compose up -d geth prysm-beacon
+```
+
+Or with 3rd-party eth1chain:
+```
+sudo docker-compose up -d prysm-beacon
+```
+
+Then "run" the validator so it can prompt you for input:
+```
 sudo docker-compose run prysm-validator
 ```
 
@@ -193,13 +279,23 @@ cd eth2-docker
 cp default.env .env
 ```
 
+After copying the `default.env` file, you **must** adjust the contents
+of `.env` to your environment.
+Specifically, set the first entry `DEPCLI_UID` to your UID if you are running
+this on Linux. You can find your UID with `echo $UID`.
+
+There is no need to touch the other UIDs in this file.
+
 Other distributions are expected to work as long as they support
 git, docker, and docker-compose.
 
 ## Windows 10 Prerequisites
 
 Install [Docker Desktop](https://www.docker.com/products/docker-desktop), [git](https://git-scm.com/download/win), and [Python 3](https://www.python.org/downloads/windows/). Note you can also type `python3` into a Powershell window and it will bring you to the Microsoft Store for a recent Python 3 version.
 
+You have to copy the `default.env` file to `.env`, from Powershell: `cp default.env .env`.
+After copying this file, you *should* adjust the contents of `.env` to your environment.
+
 Docker Desktop can be used with the WSL2 backend if desired, or without it.
 
 You will run the docker-compose and docker commands from Powershell. You do not need `sudo` in front of those commands.
@@ -217,21 +313,21 @@ show you how to.
 
 ### Geth
 
-Run:
+Run:<br />
 `sudo docker-compose build --no-cache geth`
 
-Then stop, remove and start geth:
-`sudo docker-compose stop geth && sudo docker-compose rm geth`
+Then stop, remove and start geth:<br />
+`sudo docker-compose stop geth && sudo docker-compose rm geth`<br />
 `sudo docker-compose up -d geth`
 
 ### Lighthouse
 
 lh-beacon and lh-validator share the same image, we only need to rebuild one.
 
-Run:
+Run:<br />
 `sudo docker-compose build --no-cache lh-beacon`
 
-Then restart the client:
+Then restart the client:<br />
 `sudo docker-compose down && sudo docker-compose up -d lighthouse`
 
 If you did not provide the wallet password to the container, come up more manually instead.
@@ -240,14 +336,42 @@ If you did not provide the wallet password to the container, come up more manual
 
 prysm-beacon and prysm-validator share the same image, we only need to rebuild one.
 
-Run:
+Run:<br />
 `sudo docker-compose build --no-cache prysm-beacon`
 
-Then restart the stack:
+Then restart the client:<br />
 `sudo docker-compose down && sudo docker-compose up -d prysm`
 
 If you did not provide the wallet password to the container, come up more manually instead.
 
+# Troubleshooting
+
+A few useful commands if you run into issues.
+
+`docker-compose stop servicename` brings a service down, for example `docker-compose stop lh-validator`.<br />
+`docker-compose down` will stop the entire stack.<br />
+`docker-compose up -d servicename` starts a single service, for example `docker-compose up -d lh-validator`.
+The `-d` means "detached", not connected to your input session.<br />
+`docker-compose run servicename` starts a single service and connects your input session to it. Use the Ctrl-p Ctrl-q
+key sequence to detach from it again.
+
+`docker ps` lists all running services, with the container name to the right.<br />
+`docker logs containername` shows logs for a container, `docker logs -f containername` scrolls them.<br />
+`docker exec -it containername /bin/bash` will connect you to a running service in a bash shell. The geth service doesn't have a shell.<br />
+
+If a service is continually restarting and you want to bring up its container manually, so you can investigate, first bring everything down:<br />
+`docker-compose down`, tear down everything first.<br />
+`docker ps`, make sure everything is down.<br />
+
+**HERE BE DRAGONS** You can totally run N copies of an image manually and then successfully start a validator in each and get yourself slashed.
+Take extreme care.
+
+Once your stack is down, to run an image and get into a shell, without executing the client automatically:<br />
+`docker run -it --entrypoint=/bin/bash imagename`, for example `docker run -it --entrypoint=/bin/bash lighthouse`.<br />
+You'd then run Linux commands manually in there, you could start components of the client manually. There is one image per client,
+the client images currently supplied are `lighthouse` and `prysm`.<br />
+`docker images` will show you all images.
+
 # Guiding principles:
 
 - Reduce the attack surface of the client where this is feasible. Not

TODO.md

@@ -1,13 +1,3 @@
-- Dynamic DNS for prysm. ddclient and --p2p-host-dns
 - Grafana and prometheus dashboards
-- ~~Documentation of how to do staking / onboarding with either client~~
-- traefik or nginx for both grafana and UI access
-- infura option
-- eventually: openethereum option
-- more clients?
-- binary client distribution docker images instead of compiled?
-- ~~Documentation of how to update the stack~~
 - docker-compose as a service on system boot
-- sh script to help a user come on board? But UI is likely better and comes from client team
-- Eventually: more conservative client build targets
-- Better secrets management than plaintext in the container. S3? Ansible?
+- Document mnemonic test