Skip to content

countertenor/mozart

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

103 Commits
cmd
cmd
 
 
gh images
gh images
 
 
internal
internal
 
 
pkg
pkg
 
 
static
static
 
 
.air.toml
.air.toml
 
 
.gitignore
.gitignore
 
 
LICENSE
LICENSE
 
 
Makefile
Makefile
 
 
README.md
README.md
 
 
go.mod
go.mod
 
 
go.sum
go.sum
 
 
main.go
main.go
 
 

Repository files navigation

Mozart

If you struggle with maintaining a lot of scripts:

â•°$ tree scripts
scripts
└── my-service
    ├── install
    │   ├── install-component1.sh
    │   ├── install-component2.sh
    │   └── install-component3.sh
    └── test
        ├── test-component1.sh
        ├── test-component2.sh
        └── test-component3.sh

then think of Mozart! Mozart is a script orchestrator. It can help package all scripts to a single binary, along with a CLI and and UI to manage them, with ZERO lines of code!

CLI:

Available CLI commands:

mozart execute my-service
mozart execute my-service install
mozart execute my-service install install-component1
mozart execute my-service install install-component2
mozart execute my-service install install-component3
mozart execute my-service test
mozart execute my-service test test-component1
mozart execute my-service test test-component2
mozart execute my-service test test-component3

What is mozart?

Mozart is a simple drop-in (no go-coding required) utility to orchestrate and attach a CLI and a UI to your scripts, making your independent, messy bunch of scripts into a well defined, orchestrated program complete with a CLI and UI!

Within minutes, instead of having hundreds of different scripts, you can have a single binary, complete with a CLI and a UI, which includes all those scripts. All you need to do is to drop the scripts into a folder structure (explained below). That's it!

Note: No code changes required, it is a simple drop-in type utility for your scripts.

What is an orchestrator?

What Docker compose is to Docker containers, Mozart is to scripts.

  • Manages packaging and distribution of scripts
  • Manages modularization – breaking down of big scripts into smaller scripts
  • Manages templating across scripts – runtime variable substitutions
  • Manages DRY policy across scripts
  • Manages all logs generated by scripts
  • Manages state of execution of all scripts – success/failure/last-exec time
  • Manages execution of all scripts – rerun/dry-run
  • Provides a CLI, a UI, a REST interface to interact with the scripts

How mozart differs from Ansible

Mozart Ansible
Easy learning curve Tough learning curve
No new software needed (only clone this repo) Initial setup required
Very less changes to existing scripts (simple drag and drop capability) More effort in making ansible modules
Working orchestrator within minutes Takes time to create playbooks
Provides single file packaging for all scripts Doesn't provide packaging (AFAIK?)
Useful for quick POC, testing, etc. Suitable for heavy duty production deployments
Simpler to use for single host (multiple hosts requires some changes) Suitable for multiple hosts from start

Table of Contents

What exactly does an orchestrator do?

Simply speaking, an orchestrator manages execution of all of your scripts.

Life without an orchestrator

Suppose you had 2 bash scripts which you use to test 2 different components. Without an orchestrator:

  1. You would have to ship both files to anyone who wants to use them. (imagine if you had 10 scripts, you would have to tar them up and send all).
  2. If running on a shared system, you can never know if another person already ran the scripts (unless actually going through the effort of seeing if the tests ran).
  3. Tendency to use lesser number of scripts for easy script execution management, but that makes each script huge. Breaking them down into multiple smaller scripts makes it easy to manage the code but it becomes harder to manage execution of all scripts.
  4. No way of using common variables across the scripts (like version of the component being tested).
  5. No way for one developer to look at the logs of a script executed by a second developer (unless the second developer explicitly routes the logs to an external file).
  6. No way of accidentally preventing execution of a script more than once (like prevent first script from running again if it already ran once).
  7. Manually execute each script through bash or python (no CLI or UI)

Benefits of using Mozart

  1. Simple migration to Mozart - no Go code change necessary. Just create a directory and dump your scripts in that - it's that easy. (Discussed in detail below)
  2. Single binary file which contains all your scripts AND brings with it the CLI along with the UI - so no need to send in bunch of scripts to anyone anymore.
  3. Lets you modularize the scripts, which means you can have more number of smaller scripts which do smaller tasks. No need to maintain huge bash files anymore. The smaller the scripts - the easier it is for you to manage and maintain them.
  4. Ability to use templating capabilities - similar to helm. Values in a yaml file are accessible by all scripts.
  5. Public visibility of the state of execution of scripts, so everyone has a clear idea of whether scripts were executed or not.
  6. Central logs for everyone to see.
  7. Make sure scripts are executed just once - the orchestrator will not allow you to run the same script again without explicitly mentioning a Re-Run flag, thereby preventing accidental execution of the same script.
  8. Ready to use CLI + UI to manage the execution.

Using your scripts with Mozart

Let us walk through how to actually add your scripts. There is one term that is of prime importance to Mozart - Modules.

Modules

Mozart works with the concept of modules and not the scripts directly. Modules are nothing but directories, created with the intention of performing 1 simple task. Each module (aka directory) can consist of either scripts or more nested modules (nested directories).

So in short, all your scripts need to be in a module, and Mozart will help you control the execution of those modules (instead of the scripts themselves).

Sample module

There is already a sample module present called test-module under resources/templates, which you can use to reference.

Steps

1. Initial setup

  1. Install go - https://golang.org/doc/install
  2. Clone repo - https://github.com/countertenor/mozart
  3. Run the command export PATH=$PATH:$(go env GOPATH)/bin) (Please note - to make it persistent, you will have to add the command to your .bashrc)
  4. Create a new directory inside resources/templates. This will be the base module under which all your modules will exist.
  5. (Optional) Delete the existing test-module inside resources/templates if you want a clean slate. That folder is only for reference. Leaving that folder as it is will not do any harm.

2. Modularize

  1. Look through your scripts, and identify the most basic steps that the scripts are supposed to be performing. Suppose I want to install a component called Symphony. I have one huge bash script for that. Some basic steps inside that bash script could be:

    1. Pre-requisite check.
    2. Installation of component.
    3. Validation of install.
    4. Uninstallation of component.

  2. For each identified step, create a module(directory) within the base directory and add that part of the script within that directory. For example, continuing with the above example, the directory structure should look like this:

    resources/templates
├── symphony                    (this is the base module)
│   ├── 00-pre-req              (first sub-module)
│   │   └── pre-req.sh
│   ├── 10-install              (second sub-module)
│   │   ├── 00-install-step1.sh
│   │   └── 10-install-step2.sh
│   ├── 20-validate             (third sub-module)
│   │   └── validate.sh
│   └── 30-uninstall            (fourth sub-module)
│       └── uninstall.sh

    The huge bash script is now broken down into smaller scripts, each in its own module. This makes the script easy to manage, while giving the option to add more scripts in the future as needed.

    Note: the xx-prefix before a module or script name is an optional prefix, through this you can control the order of execution of scripts/modules within the module.

Note: In case you don't want to break down your script into smaller scripts, you can create only the base module and drop your script in that directory.

3. (Optional) Use templating

You can do something like this within any script:

echo "{{.values.value1}} {{.values.value2}}"

These values are going to be fetched from a yaml file that you supply while invoking the CLI or the UI. (discussed later)

The yaml file should have something like this for the example above:

values:
value1: hello
value2: world

When you execute the corresponding module that contains the above script, you will see

echo hello world

This is discussed in detail below.

4. Build the binary

Run make - builds binaries for linux, mac and centOS environments, inside bin directory.

Voila! You have a single orchestrator binary with all your scripts in it.

Mozart yaml file

Providing an optional yaml file at runtime lets you enable certain templating features and configuration changes. If you do not need any changes, you can skip this section.

A sample blank configuration file can be generated from the binary itself, using the init command.

./bin/mozart init

Generated sample file :  mozart-sample.yaml

Templating

You can use the above yaml file to help with templating. If you refer to the test-module under resources/templates, you can see some examples of templating.

This idea is similar to helm.

Example:

In the file step1.sh, you see this:

#!/bin/bash

echo "{{.values.value1}} {{.values.value2}}"

The values in the brackets are the values that will be fetched from the yaml at runtime. So if you want to substitute some values at runtime, you replace the values with the {{ }} notation as you see above, and in the yaml file, add:

values:
  value1: hello
  value2: world

Mozart will substitute these values at runtime.

Optional configuration parameters

There are certain configuration parameters also that you can change using the same yaml file as above.

Log sub-directory

By default, log files are stored in /var/log/mozart directory (For linux and centOS), but if for some reason you want to add a sub-directory, you can do so by adding one line to the yaml file:

Example:

log_path: my-log-dir

Then all the logs will go to:

var/log/mozart/my-log-dir

Exec source

This lets you choose the execution environment of any type of script that you include.

The format is file_ext: source

Example:

exec_source:
  py: /usr/bin/python
  sh: /bin/bash

This lets Mozart know that if you place any file with the extension of .sh, then run it using /bin/bash. If you place any file with the extension .py, then run it using /usr/bin/python.

Note: These are the only 2 extensions added by default in Mozart. If you add any other type of script apart from python or bash, you will need to add the execution source in the yaml.

Delims

This lets you change the default delimiters (default - {{, }})

Example 1:

delims: ["[[", "]]"]

Adding this line in the yaml file changes the delimiters to [[ ]]. So after this, you can use templating like:

echo "[[.values.value1]] [[.values.value2]]"

Example 2:

delims: ["<<", ">>"]

Adding this line in the yaml file changes the delimiters to << >> . So after this, you can use templating like:

echo "<<.values.value1>> <<.values.value2>>"

Arguments

This lets you pass arguments to the scripts themselves when they are being executed.

Example:

args:
  00-step1.sh: ["-s", "hello"]

This makes sure that whenever the 00-step1.sh file is being executed by Mozart, the -s and hello arguments are passed to the file at runtime.

Using common snippets across scripts

There might be a scenario in which some scripts have a lot of common code. It is never a good idea to duplicate logic across scripts (DRY principle).

To tackle this, you can make use of the common folder that is present under the resources folder. This folder has one purpose and one purpose only - to hold common snippets of information that will be needed by more than one script.

You can take a look at the static/resources/templates/example-module/10-python-module/00-common-example/python-1.py file for an example.

Example:

Suppose if you have a function that you want in more than one script, say

def my_func(str):
print('inside funct - ' + str)

Instead of having this function be duplicated across scripts, you add this function as its own file under the common folder:

â•°$ cat static/resources/common/python/my_func

def my_func(str):
print('inside funct - ' + str)

Once the templating engine parses the files, it creates a mapping:

"key" - the filename ("my_func" in this case)
"value" - the content of the file itself

You can then substitute this function in any script using the key:

{{.my_func}}

This will substitute the contents of the file.

More info

  1. The files added under the common folder are also passes through the templating engine, so you can use templating in the files added to the common folder as well, something like this:

    echo "{{.values.value1}} {{.values.value2}}"

    These values will be parsed through the yaml file provided at runtime.

  2. Sometimes you might want to add indentation to the above substituted lines of code (It is essential in python scripts). You can do so by using nindent (courtesy of sprig functions)

    {{.my_func | nindent 4}}

Gotchas

  1. If you add a file under the common folder with an extension, for example, my_func.py, the key still remains my_func. That is because the templating engine gets confused when you try to access a key with a . So in cases where the file has an extension, the engine strips off the extension for the "key" value.

  2. Names with - are not permitted to be used in go templating, therefore filenames with - in them are not permitted.

Adding custom resources

Sometimes you might want to add files under certain modules which you don't want mozart to execute along with the module, instead these files will be used by your scripts.

In such cases, you can prefix ! to the file names (or prefix ! to an entire directory, in which case all files inside that directory will not be executed)

Example:

ls static/resources/templates/example-module/00-bash-module/02-external-example

!external_file.sh
step1.sh

As you see here, there's a file present with the name !external_file.sh. Since this file is prefixed with a !, this won't be executed by mozart when you run the module3 module.

If your script wants to execute this file, you can do so by adding these lines (example code in static/resources/templates/example-module/00-bash-module/02-external-example/step1.sh):

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd -P )"
bash $DIR/!external_file.sh

OR

You can write this file to a custom location and execute it:

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd -P )"
cat $DIR/!external_file.sh > /your/location/file.sh
bash /your/location/file.sh

Note: The DIR command is needed since the execution directory for the script is not where the script resides. So unfortunately you cannot run a script that's present in the same folder using something like ./script_name, since . assumes current execution directory.

CLI

Once you build your binary, Mozart gives you a CLI:

Mozart commands

- init          Generate a blank sample config yaml file for the orchestrator
- execute       (executes all scripts in specified directory)
- state         (displays execution state of all modules, accepts optional args [module name])
- server        Starts the REST server
- version       (displays version info for the application)
--json                (gives output in JSON)

Global flags
-c                  (optional) (configuration file, defaults to 'mozart-sample.yaml')
-v                  (optional) (prints verbosely, useful for debugging)
-d, --dry-run       (optional) shows what scripts will run, but does not run the scripts
-n, --no-generate   (optional) do not generate bash scripts as part of execution, instead use the ones in generated folder. Useful for running local change to the scripts
-p, --parallel      (optional) Run all scripts in parallel
-r, --re-run        (optional) re-run script from initial state, ignoring previously saved state

Executing modules

Running the binary built in the earlier step, you will see something like this:

$ ./bin/mozart execute -h

Execute scripts inside any folder.
*****************************************
Available commands:

mozart execute symphony-module
mozart execute symphony-module pre-req
mozart execute symphony-module install
mozart execute symphony-module validate
mozart execute symphony-module uninstall
*****************************************

If you select a module that contains other modules, something like mozart execute symphony-module, that's where the ordering of the sub-modules comes into play, which you control by adding the prefix. Or you can choose to execute a sub-module directly.

Note: Mozart automatically removes any prefix of the form xx-before the module name.

Checking the state

Once you start the execution, the state command shows you the current state of execution of the various modules within Mozart, along with other information.

State of all modules

$ ./bin/mozart state

State: {
  "generated/symphony-module/00-pre-req": {
    "pre-req.sh": {
      "startTime": "2020-11-04T15:08:12.5981-08:00",
      "timeTaken": "8.218745ms",
      "lastSuccessTime": "2020-11-04 15:08:12.606306 -0800 PST m=+0.016747847",
      "lastErrorTime": "",
      "state": "success",
      "logFilePath": "logs/2020-11-04--15-08-12.597-pre-req.log"
    }
  },
  "generated/symphony-module/10-install": {
    "00-install-step1.sh": {
      "startTime": "2020-11-04T15:08:12.607053-08:00",
      "timeTaken": "9.723926ms",
      "lastSuccessTime": "2020-11-04 15:08:12.616754 -0800 PST m=+0.027195761",
      "lastErrorTime": "",
      "state": "success",
      "logFilePath": "logs/2020-11-04--15-08-12.606-00-install-step1.log"
    },
    "10-install-step2.sh": {
      "startTime": "2020-11-04T15:08:12.617443-08:00",
      "timeTaken": "7.333338ms",
      "lastSuccessTime": "2020-11-04 15:08:12.624767 -0800 PST m=+0.035208922",
      "lastErrorTime": "",
      "state": "success",
      "logFilePath": "logs/2020-11-04--15-08-12.617-10-install-step2.log"
    }
  },
  "generated/symphony-module/20-validate": {
    "validate.sh": {
      "startTime": "2020-11-04T15:08:12.625411-08:00",
      "timeTaken": "7.542653ms",
      "lastSuccessTime": "2020-11-04 15:08:12.632945 -0800 PST m=+0.043386468",
      "lastErrorTime": "",
      "state": "success",
      "logFilePath": "logs/2020-11-04--15-08-12.625-validate.log"
    }
  },
  "generated/symphony-module/30-uninstall": {
    "uninstall.sh": {
      "startTime": "2020-11-04T15:08:12.633649-08:00",
      "timeTaken": "8.040003ms",
      "lastSuccessTime": "2020-11-04 15:08:12.641679 -0800 PST m=+0.052120673",
      "lastErrorTime": "",
      "state": "success",
      "logFilePath": "logs/2020-11-04--15-08-12.633-uninstall.log"
    }
  }
}

State of specific module

To get the state of a particular module:

./mozart state validate

State: {
  "generated/symphony-module/20-validate": {
    "validate.sh": {
      "startTime": "2020-11-04T15:08:12.625411-08:00",
      "timeTaken": "7.542653ms",
      "lastSuccessTime": "2020-11-04 15:08:12.632945 -0800 PST m=+0.043386468",
      "lastErrorTime": "",
      "state": "success",
      "logFilePath": "logs/2020-11-04--15-08-12.625-validate.log"
    }
  }

UI

Developed by @toshakamath

Helpful resources

Good links for templating

Bash scripts through go

Versioning with go

Web sockets

Command execution

Error stack

Embedding static content

Go build tools

About

A language-agnostic helm-like script orchestrator

Topics

go golang template framework helm scripts scripts-collection orchestrator goembed

Resources

Readme

License

MIT license
Activity
Custom properties

Stars

11 stars

Watchers

3 watching

Forks

2 forks
Report repository

Releases 3

Version 2.0 released! Latest
May 4, 2021
+ 2 releases

Contributors 2

  •  
  •  

Languages