[BREAKING] TOML config (smartcontractkit#802)

* WIP#1

* define common TOML config components

* use generic and not interface{}

* add TOML config for Private Ethereum Network

* various little fixes, toml string duration, removed unnecessary logstream fields (will now use logging config), removed test log level from toml config

* move loki to separate struct, fix pyroscope file name

* add function that allows overriding chainlink image and version from TOML config instead of env vars

* adjust field names in structs

* add automatic forwarding of base64 config override env var to k8s

* add forwarding for network config as well

* fix go.mod

* remove unused k8s env vars, make pyroscope auth key setting based on TOML config

* make Pyroscope override optional

* fix reorg release name

* fix logstream docker tests

* nancy ignore x/crypto vulnerability, less code duplication in logstream test

* Update and add some readme

* add example of how to dynamically create TOML config in bash/CI

* examples, more readme.md

* fix go.mod

* better logstream readme, do not crash if there's no grafana url, remove hardcoded dashboard

* fix recursive files walker

* make the stop file error better

* try looking for the file in current directory first

* now do look for it first in the current directory for real

* allow to use even lower seconds_per_slot and slots_per_epoch values

* push up min slots per epoch

* rename Loki.url to Loki.endpoint

* [TT-755] grafana url shortener (smartcontractkit#811)

* shorten grafana urls
* do not expect dashboard url to have any query params

* remove usages of pkg/errors

* fix lint

* remve unused interfaces or default, simplify network config

* add test for recursive file searching utility function

* update config/README.md

* even better config readme, docs for config public methods

* fix unit tests

* fix eth private network config validation for pow

* fix typo and example

* fix last failing unit test

* make sure WillUseRemoteRunner() is not trying to access nil field

* add bearer token field for Loki config

* fix knonw networks test

* bump wasp to 0.4.1

* [TT-769]  customisable docker images (smartcontractkit#812)

* support for custom docker images in private eth network builder
* allow to pass custom execution/consensus/validator docker images via cmd line when starting new private chain

* tmp remove wasp bump

* new prom replace, prom bump

* bump wasp

* fix mergo import

---------

Co-authored-by: skudasov <[email protected]>

Makefile

@@ -73,7 +73,7 @@ compile_contracts:
 	python3 ./utils/compile_contracts.py
 
 test_unit: install_gotestfmt
-	go test -json -cover -covermode=count -coverprofile=unit-test-coverage.out ./client ./gauntlet ./testreporters ./k8s/config 2>&1 | tee /tmp/gotest.log | gotestfmt
+	go test -json -cover -covermode=count -coverprofile=unit-test-coverage.out ./client ./gauntlet ./testreporters ./k8s/config ./utils/osutil 2 2>&1 | tee /tmp/gotest.log | gotestfmt
 
 test_docker: install_gotestfmt
 	go test -json -cover -covermode=count -coverprofile=unit-test-coverage.out ./docker/test_env ./logstream 2>&1 | tee /tmp/gotest.log | gotestfmt	

README.md

@@ -39,6 +39,7 @@ make install_deps
 To read how to run a test in k8s, read [here](./k8s/REMOTE_RUN.md)
 
 ### Usage
+#### With env vars (deprecated)
 Create an env in a separate file and run it
 ```
 export CHAINLINK_IMAGE="public.ecr.aws/chainlink/chainlink"
@@ -48,6 +49,16 @@ go run k8s/examples/simple/env.go
 ```
 For more features follow [tutorial](./k8s/TUTORIAL.md)
 
+#### With TOML config
+It should be noted that using env vars for configuring CL nodes in k8s is deprecated. TOML config should be used instead:
+```toml
+[ChainlinkImage]
+image="public.ecr.aws/chainlink/chainlink"
+version="v2.8.0"
+```
+
+Check the example here: [env.go](./k8s/examples/simple_toml/env_toml_config.go)
+
 ### Development
 #### Running standalone example environment
 ```shell
@@ -104,6 +115,13 @@ We have extended support for execution layer clients in simulated networks. Foll
 
 When it comes to consensus layer we currently support only `Prysm`.
 
+Every component has some default Docker image it uses, but builder has a method that allows to pass custom one:
+```go
+WithCustomDockerImages(map[ContainerType]string{
+    ContainerType_Geth2: "my-custom-geth2-image:my-version"}).
+Build()
+```
+
 ## Command line
 
 You can start a simulated network with a single command:
@@ -120,6 +138,9 @@ Following cmd line flags are available:
   -t, --consensus-type string    consensus type (pow or pos) (default "pos")
   -e, --execution-layer string   execution layer (geth, nethermind, besu or erigon) (default "geth")
   -w, --wait-for-finalization    wait for finalization of at least 1 epoch (might take up to 5 mintues)
+      --consensus-client-image string   custom Docker image for consensus layer client  
+      --execution-layer-image string    custom Docker image for execution layer client
+      --validator-image string          custom Docker image for validator  
 ```
 
 To connect to that environment in your tests use the following code:
@@ -141,46 +162,102 @@ To connect to that environment in your tests use the following code:
 Builder will read the location of chain configuration from env var named `PRIVATE_ETHEREUM_NETWORK_CONFIG_PATH` (it will be printed in the console once the chain starts).
 
 `net` is an instance of `blockchain.EVMNetwork`, which contains characteristics of the network and can be used to connect to it using an EVM client. `rpc` variable contains arrays of public and private RPC endpoints, where "private" means URL that's accessible from the same Docker network as the chain is running in.
+
 # Using LogStream
 
 LogStream is a package that allows to connect to a Docker container and then flush logs to configured targets. Currently 3 targets are supported:
 * `file` - saves logs to a file in `./logs` folder
 * `loki` - sends logs to Loki
 * `in-memory` - stores logs in memory
 
-It can be configured to use multiple targets at once. If no target is specified, it becomes a no-op.
-
-Targets can be set in two ways:
-* using `LOGSTREAM_LOG_TARGETS` environment variable, e.g. `Loki,in-MemOry` (case insensitive)
-* using programmatic functional option `WithLogTarget()`
+It can be configured to use multiple targets at once. If no target is specified, it becomes a no-op. 
 
-Functional option has higher priority than environment variable.
+LogStream has to be configured by passing an instance of `LoggingConfig` to the constructor.
 
 When you connect a contaier LogStream will create a new consumer and start a detached goroutine that listens to logs emitted by that container and which reconnects and re-requests logs if listening fails for whatever reason. Retry limit and timeout can both be configured using functional options. In most cases one container should have one consumer, but it's possible to have multiple consumers for one container.
 
 LogStream stores all logs in gob temporary file. To actually send/save them, you need to flush them. When you do it, LogStream will decode the file and send logs to configured targets. If log handling results in an error it won't be retried and processing of logs for given consumer will stop (if you think we should add a retry mechanism please let us know).
 
 *Important:* Flushing and accepting logs is blocking operation. That's because they both share the same cursor to temporary file and otherwise it's position would be racey and could result in mixed up logs.
 
-When using `in-memory` or `file` target no other environment variables are required. When using `loki` target, following environment variables are required:
-* `LOKI_TENTANT_ID` - tenant ID
-* `LOKI_URL` - Loki URL to which logs will be pushed
-* `LOKI_BASIC_AUTH` -- only needed when running in CI and using public endpoint
+## Configuration
+
+Basic `LogStream` TOML configuration is following:
+```toml
+[LogStream]
+log_targets=["file"]
+log_producer_timeout="10s"
+log_producer_retry_limit=10
+```
+You can find it here: [logging_default.toml](config/tomls/logging_default.toml)
+
+When using `in-memory` or `file` target no other configuration variables are required. When using `loki` target, following ones must be set:
+```toml
+[Logging.Loki]
+tenant_id="promtail"
+url="https://change.me"
+basic_auth="my-secret-auth"
+bearer_token="bearer-token"
+```
+
+Also, do remember that different URL should be used when running in CI and everywhere else. In CI it should be a public endpoint, while in local environment it should be a private one.
+
+If your test has a Grafana dashboard in order for the url to be correctly printed you should provide the following config:
+```toml
+[Logging.Grafana]
+url="http://grafana.somwhere.com/my_dashboard"
+```
+
+## Initialisation
 
-Also, do remember that different `LOKI_URL` should be used when running in CI and everywhere else. In CI it should be a public endpoint, while in local environment it should be a private one.
+First you need to create a new instance:
+```golang
+// t - instance of *testing.T (can be nil)
+// testConfig.Logging - pointer to logging part of TestConfig
+ls := logstream.NewLogStream(t, testConfig.Logging)
+```
+
+## Listening to logs
+
+If using `testcontainers-go` Docker containers it is recommended to use life cycle hooks for connecting and disconnecting LogStream from the container. You can do that when creating `ContainerRequest` in the following way:
+```golang
+
+containerRequest := &tc.ContainerRequest{
+		LifecycleHooks: []tc.ContainerLifecycleHooks{
+			{PostStarts: []tc.ContainerHook{
+				func(ctx context.Context, c tc.Container) error {
+					if ls != nil {
+						return n.ls.ConnectContainer(ctx, c, "custom-container-prefix-can-be-empty")
+					}
+					return nil
+				},
+			},
+				PostStops: []tc.ContainerHook{
+					func(ctx context.Context, c tc.Container) error {
+						if ls != nil {
+							return n.ls.DisconnectContainer(c)
+						}
+						return nil
+					},
+				}},
+		},
+	}
+```
 
 You can print log location for each target using this function: `(m *LogStream) PrintLogTargetsLocations()`. For `file` target it will print relative folder path, for `loki` it will print URL of a Grafana Dashboard scoped to current execution and container ids. For `in-memory` target it's no-op.
 
 It is recommended to shutdown LogStream at the end of your tests. Here's an example:
-```go
+```golang
 
 t.Cleanup(func() {
     l.Warn().Msg("Shutting down Log Stream")
 
     if t.Failed() || os.Getenv("TEST_LOG_COLLECT") == "true" {
         // we can't do much if this fails, so we just log the error
         _ = logStream.FlushLogsToTargets()
+        // this will log log locations for each target (for file it will be a folder, for Loki Grafana dashboard -- remember to provide it's url in config!)
         logStream.PrintLogTargetsLocations()
+        // this will save log locations in test summary, so that they can be easily accessed in GH's step summary
         logStream.SaveLogLocationInTestSummary()
     }
 
@@ -189,6 +266,22 @@ t.Cleanup(func() {
     })
 ```
 
+or in a bit shorter way:
+```golang
+t.Cleanup(func() {
+    l.Warn().Msg("Shutting down Log Stream")
+
+    if t.Failed() || os.Getenv("TEST_LOG_COLLECT") == "true" {
+        // this will log log locations for each target (for file it will be a folder, for Loki Grafana dashboard -- remember to provide it's url in config!)
+        logStream.PrintLogTargetsLocations()
+        // this will save log locations in test summary, so that they can be easily accessed in GH's step summary
+    }
+
+    // we can't do much if this fails
+    _ = logStream.FlushAndShutdown()
+    })
+```
+
 ## Grouping test execution
 
 When running tests in CI you're probably interested in grouping logs by test execution, so that you can easily find the logs in Loki. To do that your job should set `RUN_ID` environment variable. In GHA it's recommended to set it to workflow id. If that variable is not set, then a run id will be automatically generated and saved in `.run.id` file, so that it can be shared by tests that are part of the same execution, but are running in different processes.
@@ -216,3 +309,6 @@ Example:
 ```
 
 In GHA after tests have ended we can use tools like `jq` to extract the information we need and display it in step summary.
+
+# TOML Config
+Basic and universal building blocks for TOML-based config are provided by `config` package. For more information do read [this](./config/README.md).

blockchain/config.go

@@ -33,7 +33,7 @@ var (
 			"58845406a51d98fb2026887281b4e91b8843bbec5f16b89de06d5b9a62b231e8",
 		},
 		ChainlinkTransactionLimit: 500000,
-		Timeout:                   JSONStrDuration{2 * time.Minute},
+		Timeout:                   StrDuration{2 * time.Minute},
 		MinimumConfirmations:      1,
 		GasEstimationBuffer:       10000,
 	}
@@ -59,7 +59,7 @@ type EVMNetwork struct {
 	// nodes require to run the tests.
 	ChainlinkTransactionLimit uint64 `envconfig:"evm_chainlink_transaction_limit" default:"500000" toml:"evm_chainlink_transaction_limit" json:"evm_chainlink_transaction_limit"`
 	// How long to wait for on-chain operations before timing out an on-chain operation
-	Timeout JSONStrDuration `envconfig:"evm_transaction_timeout" default:"2m" toml:"evm_transaction_timeout" json:"evm_transaction_timeout"`
+	Timeout StrDuration `envconfig:"evm_transaction_timeout" default:"2m" toml:"evm_transaction_timeout" json:"evm_transaction_timeout"`
 	// How many block confirmations to wait to confirm on-chain events
 	MinimumConfirmations int `envconfig:"evm_minimum_confirmations" default:"1" toml:"evm_minimum_confirmations" json:"evm_minimum_confirmations"`
 	// How much WEI to add to gas estimations for sending transactions
@@ -78,7 +78,7 @@ type EVMNetwork struct {
 	FinalityDepth uint64 `envconfig:"evm_finality_depth" default:"50" toml:"evm_finality_depth" json:"evm_finality_depth"`
 
 	// TimeToReachFinality is the time it takes for a block to be considered final. This is used to determine how long to wait for a block to be considered final.
-	TimeToReachFinality JSONStrDuration `envconfig:"evm_time_to_reach_finality" default:"0s" toml:"evm_time_to_reach_finality" json:"evm_time_to_reach_finality"`
+	TimeToReachFinality StrDuration `envconfig:"evm_time_to_reach_finality" default:"0s" toml:"evm_time_to_reach_finality" json:"evm_time_to_reach_finality"`
 
 	// Only used internally, do not set
 	URL string `ignored:"true"`
@@ -147,16 +147,16 @@ func (e *EVMNetwork) MustChainlinkTOML(networkDetails string) string {
 	return netString
 }
 
-// JSONStrDuration is JSON friendly duration that can be parsed from "1h2m0s" Go format
-type JSONStrDuration struct {
+// StrDuration is JSON/TOML friendly duration that can be parsed from "1h2m0s" Go format
+type StrDuration struct {
 	time.Duration
 }
 
-func (d *JSONStrDuration) MarshalJSON() ([]byte, error) {
+func (d *StrDuration) MarshalJSON() ([]byte, error) {
 	return json.Marshal(d.String())
 }
 
-func (d *JSONStrDuration) UnmarshalJSON(b []byte) error {
+func (d *StrDuration) UnmarshalJSON(b []byte) error {
 	var v interface{}
 	if err := json.Unmarshal(b, &v); err != nil {
 		return err
@@ -173,3 +173,19 @@ func (d *JSONStrDuration) UnmarshalJSON(b []byte) error {
 		return errors.New("invalid duration")
 	}
 }
+
+// MarshalText implements the text.Marshaler interface (used by toml)
+func (d StrDuration) MarshalText() ([]byte, error) {
+	return []byte(d.Duration.String()), nil
+}
+
+// UnmarshalText implements the text.Unmarshaler interface (used by toml)
+func (d *StrDuration) UnmarshalText(b []byte) error {
+	var err error
+	d.Duration, err = time.ParseDuration(string(b))
+	if err != nil {
+		return err
+	}
+	return nil
+
+}