Improve server_test and instructions on homebrew and dynamic config d…

…ocs (cadence-workflow#4557)

README.md

@@ -42,6 +42,7 @@ Cadence CLI can be used to operate workflows, tasklist, domain and even the clus
 
 You can use the following ways to install Cadence CLI:
 * Use brew to install CLI: `brew install cadence-workflow`
+  * Follow the [instructions](https://github.com/uber/cadence/discussions/4457) if you need to install older versions of CLI via homebrew. Usually this is only needed when you are running a server of a too old version.
 * Use docker image for CLI: `docker run --rm ubercadence/cli:<releaseVersion>`  or `docker run --rm ubercadence/cli:master ` . Be sure to update your image when you want to try new features: `docker pull ubercadence/cli:master `
 * Build the CLI binary yourself, check out the repo and run `make cadence` to build all tools. See [CONTRIBUTING](CONTRIBUTING.md) for prerequisite of make command.
 * Build the CLI image yourself, see [instructions](docker/README.md#diy-building-an-image-for-any-tag-or-branch)
@@ -80,6 +81,11 @@ The easiest way to get the schema tool is via homebrew.
 
 `brew install cadence-workflow` also includes `cadence-sql-tool` and `cadence-cassandra-tool`. 
  * The schema files are located at `/usr/local/etc/cadence/schema/`.
+ * To upgrade, make sure you remove the old ElasticSearch schema first: `mv /usr/local/etc/cadence/schema/elasticsearch /usr/local/etc/cadence/schema/elasticsearch.old && brew upgrade cadence-workflow`. Otherwise ElasticSearch schemas may not be able to get updated.
+ * Follow the [instructions](https://github.com/uber/cadence/discussions/4457) if you need to install older versions of schema tools via homebrew. 
+ However, easier way is to use new versions of schema tools with old versions of schemas. 
+ All you need is to check out the older version of schemas from this repo. Run `git checkout v0.21.3` to get the v0.21.3 schemas in [the schema folder](/schema). 
+
 
 ## License
 

canary/README.md

@@ -54,18 +54,21 @@ By default, it will load [the configuration in `config/canary/development.yaml`]
 Run `./cadence-canary -h` for details to understand the start options of how to change the loading directory if needed. 
 This will only start the workers. 
 
+In production, it's recommended to monitor the result of this canary. You can use [the workflow success metric](https://github.com/uber/cadence/blob/9336ed963ca1b5e0df7206312aa5236433e04fd9/service/history/execution/context_util.go#L138) 
+emitted by cadence history service `workflow_success`.  
+
 Configurations
 ----------------------
 Canary workers configuration contains two parts:
 - **Canary**: this part controls which domains canary workers are responsible for what tests the sanity workflow will exclude.
 ```yaml 
 canary:
   domains: ["cadence-canary"] # it will start workers on all those domains(also try to register if not exists) 
-  excludes: ["workflow.searchAttributes", "workflow.batch", "workflow.archival.visibility"] # it will exclude the three test cases
+  excludes: ["workflow.searchAttributes", "workflow.batch", "workflow.archival.visibility", "workflow.archival.history"] # it will exclude the three test cases. If archival is not enabled, you should exclude "workflow.archival.visibility" and"workflow.archival.history". If advanced visibility is not enabled, you should exclude "workflow.searchAttributes" and "workflow.batch". Otherwise canary will fail on those test cases.  
   cron: 
-    cronSchedule: #the schedule of cron canary, default to "@every 30s"
-    cronExecutionTimeout: #the timeout of each run of the cron execution, default to 18 minutes
-    startJobTimeout: #the timeout of each run of the sanity test suite, default to 9 minutes
+    cronSchedule: "@every 30s" #the schedule of cron canary, default to "@every 30s"
+    cronExecutionTimeout: 18m #the timeout of each run of the cron execution, default to 18 minutes
+    startJobTimeout: 9m #the timeout of each run of the sanity test suite, default to 9 minutes
 ``` 
 An exception here is `HistoryArchival` and `VisibilityArchival` test cases will always use `canary-archival-domain` domain. 
 

cmd/server/cadence/server_test.go

@@ -25,6 +25,7 @@ package cadence
 
 import (
 	"log"
+	"os"
 	"testing"
 	"time"
 
@@ -54,11 +55,9 @@ func (s *ServerSuite) SetupTest() {
 
 /*
 TestServerStartup tests the startup logic for the binary. When this fails, you should be able to reproduce by running "cadence-server start"
+If you need to run locally, make sure Cassandra is up and schema is installed(run `make install-schema`)
 */
 func (s *ServerSuite) TestServerStartup() {
-	// If you want to test it locally, change it to false
-	runInBuildKite := true
-
 	env := "development"
 	zone := ""
 	rootDir := "../../../"
@@ -71,8 +70,10 @@ func (s *ServerSuite) TestServerStartup() {
 	if err != nil {
 		log.Fatal("Config file corrupted.", err)
 	}
-	// replace local host to docker network
-	if runInBuildKite {
+
+	if os.Getenv("CASSANDRA_SEEDS") == "cassandra" {
+		// replace local host to docker network
+		// this env variable value is set by buildkite's docker-compose
 		ds := cfg.Persistence.DataStores[cfg.Persistence.DefaultStore]
 		ds.NoSQL.Hosts = "cassandra"
 		cfg.Persistence.DataStores[cfg.Persistence.DefaultStore] = ds