START.md

Getting started

• Getting started
  • Requirements
  • Installation
    • Installation options
    • Install via cloudformation
    • Install via terraform
      • Default install
      • Install module from local file
      • Install module from S3
  • Enable autospotting
    • For an AutoScaling group
    • For Elastic Beanstalk
  • Configuration of autospotting
    • Testing configuration
    • Running configuration
      • Minimum on-demand configuration
    • Debug autospotting
  • Updates and Downgrades
    • Compatibility notices
  • Uninstallation
    • Uninstall via cloudformation
    • Uninstall via terraform

Binary License Notice

All pre-build binaries mentioned in this page are distributed under a proprietary license, and can only be used legally for up to 14 days, for evaluation purposes.

Project patrons and code contributors can get a long term license. Here you can also become a patron and see what other benefits this can bring you.

If you don't agree with the terms of this license nor you want to become a patron, you can still build it from source code yourself but you'll get very limited support if you do so.

Requirements

• You will need credentials to an AWS account able to start CloudFormation stacks.
• Some of the following steps assume you have the AWS cli tool installed, but the setup can also be done manually using the AWS console or using other tools able to launch CloudFormation stacks and set tags on AutoScaling groups.

Installation

Installation options

Autospotting can be installed via Cloudformation or Terraform, both install methods take a number of parameters, which allows you to configure it for your own environment. The defaults should be safe enough for most use cases, but for testing or more advanced use cases you may want to tweak some of them.

Some parameters control the Lambda runtime, while others allow tweaking the AutoSpotting algorithm, for example to keep a certain amount of on-demand capacity in the group, or run only against some AWS regions.

The algorithm parameters are just global defaults that can often be overridden at the AutoScaling group level based on additional tags set on the group.

The full list of parameters, including relatively detailed explanations about them and their overriding group tags can be seen in the CloudFormation AWS console or in the variables.tf file.

In case you may want to change some of them later, you can do it at any time by updating the stack via Cloudformation or Terraform.

Note: even though the Cloudformation stack template is not changing so often and it may often support multiple software versions, due to possible compatibility issues, it is recommended to also update the stack template when updating the software version.

Install via cloudformation

To install it via cloudformation, you only need to launch a CloudFormation stack in your account. Click the button below and follow the launch wizard to completion, you can safely use the default stack parameters.

If you are using the AWS command-line tool, you can use this command instead:

aws cloudformation create-stack \
--stack-name AutoSpotting \
--template-url https://s3.amazonaws.com/cloudprowess/nightly/template.yaml \
--capabilities CAPABILITY_IAM

Notes:

• For technical reasons the stack launched from the official binaries needs to be launched in the US-East-1(Virginia) region, so make sure it's not created in another region. Custom builds can be deployed in any region you prefer, just make sure your S3 bucket is in that region.
• The AutoScaling groups it runs against can be in any region, since all regions are processed at runtime, unless configured otherwise.

Install via terraform

You need to have terraform installed on your machine.

Default install

Pre-built binaries easier to install are available when using the Terraform module available from the Terraform Registry.

Copy and paste the below snippet into your Terraform configuration:

module "autospotting" {
  source  = "AutoSpotting/AutoSpotting/aws"
  version = "0.0.9" # this version is subject to change
}

You can also insert any of the variables you want to override, then:

terraform init # fetches required plugin and modules for terraform
export AWS_DEFAULT_REGION=XXXX
export AWS_ACCESS_KEY_ID=XXXX
export AWS_SECRET_ACCESS_KEY=XXXX
terraform plan
terraform apply -auto-approve

Custom binary installation

The lambda.zip file can be generated by building it locally using make(look for it in the build directory) but you'll need to set up your local development environment.

Instead, you can also download the latest official nightly build available here.

If you are only using autospotting as such, you can install the stack by doing:

 cd terraform/
 terraform init # fetches required plugin and modules for terraform
 export AWS_DEFAULT_REGION=XXXX
 export AWS_ACCESS_KEY_ID=XXXX
 export AWS_SECRET_ACCESS_KEY=XXXX
 terraform apply -auto-approve

Note: Apart from AWS variable, no variables are required. The module can be run as such, and would function. But you might want to tweak at least the default on-demand values and/or the regions in which autospotting runs. The extra parameters can also be overridden to suit your needs.

To use custom parameters, please refer to the variables.tf in terraform/. Here is an example modifying both autospotting and lambda configuration:

 terraform apply \
   -var asg_regions_enabled="eu*,us*" \
   -var asg_min_on_demand_percentage="33.3" \
   -var lambda_memory_size=1024

Install module from local file

When you are using autospotting integrated to your infrastructure, then you can use the module directly:

module "autospotting" {
  source = "github.com/AutoSpotting/AutoSpotting//terraform/autospotting"


  autospotting_disallowed_instance_types    = "t2.*"
  autospotting_min_on_demand_number         = "0"
  autospotting_min_on_demand_percentage     = "50.0"
  autospotting_regions_enabled              = "eu*,us*"
  autospotting_tag_filters                  = "spot-enabled=true,environment=dev,team=interactive"
  autospotting_on_demand_price_multiplier   = "1.0"
  autospotting_spot_price_buffer_percentage = "10.0"
  autospotting_bidding_policy               = "normal"
  autospotting_tag_filtering_mode           = "opt-in"
  autospotting_spot_product_description     = "Linux/UNIX (Amazon VPC)"

  lambda_zipname       = "./lambda.zip"
  lambda_runtime       = "go1.x"
  lambda_memory_size   = "256"
  lambda_timeout       = "300"
  lambda_run_frequency = "rate(5 minutes)"
}

Note: this snippet may be outdated, please see the Terraform module for the current module parameters.

Install module from S3

Instead of using a local ZIP file you can refer to the Lambda code in a location in S3:

module "autospotting" {
  source = "github.com/AutoSpotting/AutoSpotting//terraform/autospotting"

  lambda_s3_bucket = "lambda-releases"
  lambda_s3_key    = "autospotting.zip"
}

Install as Kubernetes cronjob

We have an example configuration file that allows you to run AutoSpotting as a Kubernetes cron job, instead of running it in AWS Lambda.

curl https://raw.githubusercontent.com/AutoSpotting/AutoSpotting/master/kubernetes/autospotting-cron.yaml.example > autospotting-cron.yaml

You can then edit it locally, tweaking it to suit your needs. Once you're happy with it, you can launch it on your Kubernetes cluster:

kubectl create -f kubernetes/autospotting-cron.yaml

You can tweak the configuration later using kubectl edit cronjob autospotting.

Keep in mind that this job automatically updates to the latest official binaries, so you may want to host your own Docker images if you want to stick to a certain version or you don't want to comply with the terms of our binary license.

Enable autospotting

For an AutoScaling group

Since AutoSpotting by default uses an opt-in model, no resources will be changed in your AWS account if you just launch the stack. You will need to explicitly enable it for each AutoScaling group where you want it to be used.

Enabling it for an AutoScaling group is a matter of setting a tag on the group:

Key: spot-enabled
Value: true

This can be configured with the AWS console from this view,

If you use the AWS command-line tools, the same can be achieved using this command:

aws autoscaling
--region eu-west-1 \
create-or-update-tags \
--tags ResourceId=my-auto-scaling-group,ResourceType=auto-scaling-group,Key=spot-enabled,Value=true,PropagateAtLaunch=false

Note: the above instructions use the eu-west-1 AWS region as an example. Depending on where your groups are defined, you may need to use a different region, since as mentioned before, your environments may be located anywhere.

This needs to be done for every single AutoScaling group where you want it enabled, otherwise the group is ignored. If you have lots of groups you may want to script it in some way.

One good way to automate is using CloudFormation, using this example snippet:

"MyAutoScalingGroup": {
  "Properties": {
    "Tags":[
    {
      "Key": "spot-enabled",
      "Value": "true",
      "PropagateAtLaunch": false
    }
    ]
  }
}

Note: The spot-enabled=true tag for opt-in is configurable. See the stack parameters for the way to override it.

Note: AutoSpotting now also supports an opt-out mode, in which it will take over all your groups except of those tagged with the configured tag. The default (but also configurable) opt-out tag is spot-enabled=false. This may be risky, please handle with care.

For Elastic Beanstalk

• In order to add tags to existing Elastic Beanstalk environment, you will need to rebuild the environment with the spot-enabled tag. Follow this guide

Configuration of autospotting

Testing configuration

Normally AutoSPotting runs from AWS Lambda, but for testing purposes it can also be compiled and executed locally as a command-line tool, which can be very useful for implementing and testing new functionality.

The algorithm can use custom command-line flags. Much like many other command-line tools, you can use the -h command line flag to see all the available options:

$ ./autospotting -h
Usage of ./autospotting:
  -allowed_instance_types="":
        If specified, the spot instances will be of these types.
        If missing, the type is autodetected frome each ASG based on it's Launch Configuration.
        Accepts a list of comma or whitespace seperated instance types (supports globs).
        Example: ./autospotting -allowed_instance_types 'c5.*,c4.xlarge'

  -bidding_policy="normal":
        Policy choice for spot bid. If set to 'normal', we bid at the on-demand price.
        If set to 'aggressive', we bid at a percentage value above the spot price configurable using the spot_price_buffer_percentage.

  -disallowed_instance_types="":
        If specified, the spot instances will _never_ be of these types.
        Accepts a list of comma or whitespace seperated instance types (supports globs).
        Example: ./autospotting -disallowed_instance_types 't2.*,c4.xlarge'

  -min_on_demand_number=0:
        On-demand capacity (as absolute number) ensured to be running in each of your groups.
        Can be overridden on a per-group basis using the tag autospotting_min_on_demand_number.

  -min_on_demand_percentage=0:
        On-demand capacity (percentage of the total number of instances in the group) ensured to be running in each of your groups.
        Can be overridden on a per-group basis using the tag autospotting_min_on_demand_percentage
        It is ignored if min_on_demand_number is also set.

  -on_demand_price_multiplier=1:
        Multiplier for the on-demand price. This is useful for volume discounts or if you want to
        set your bid price to be higher than the on demand price to reduce the chances that your
        spot instances will be terminated.

  -regions="":
        Regions where it should be activated (comma or whitespace separated list, also supports globs), by default it runs on all regions.
        Example: ./autospotting -regions 'eu-*,us-east-1'

  -spot_price_buffer_percentage=10:
        Percentage Value of the bid above the current spot price. A spot bid would be placed at a value :
        current_spot_price * [1 + (spot_price_buffer_percentage/100.0)]. The main benefit is that
        it protects the group from running spot instances that got significantly more expensive than
        when they were initially launched, but still somewhat less than the on-demand price. Can be
        enforced using the tag: autospotting_spot_price_buffer_percentage. If the bid exceeds
        the on-demand price, we place a bid at on-demand price itself.

  -spot_product_description="Linux/UNIX (Amazon VPC)":
        The Spot Product or operating system to use when looking up spot price history in the market.
        Valid choices: Linux/UNIX | SUSE Linux | Windows | Linux/UNIX (Amazon VPC) | SUSE Linux (Amazon VPC) | Windows (Amazon VPC)

  -tag_filters=[{spot-enabled true}]: Set of tags to filter the ASGs on.  Default is -tag_filters 'spot-enabled=true'
        Example: ./autospotting -tag_filters 'spot-enabled=true,Environment=dev,Team=vision'

The value of -min_on_demand_number has a higher priority than -min_on_demand_percentage, so if you specify both options in the command line, percentage will NOT be taken into account. It would be taken into account, ONLY if the -min_on_demand_number is invalid (negative, above the max number, etc).

The value of -regions controls the scope within which autospotting run, this is particularly useful when used during testing, in order to limit the scope of action and reduce the risk when evaluating it or experimenting with new functionality.

All the flags are also exposed as environment variables, expected in ALL_CAPS. For example using the -region command-line flag is equivalent to using the REGION environment variable.

When tag_filters is not passed, the default operation is to look for ASG's that have the tag spot-enabled=true. If you wish to narrow the operation of autospotting to ASGs that match more specific criteria you can specify the matching tags as you see fit. i.e. -tag_filters 'spot-enabled=true,Environment=dev,Team=vision'

Note: These configurations are also implemented when running from Lambda, where they are actually passed as environment variables set by CloudFormation in the Lambda function's configuration.

Running configuration

Minimum on-demand configuration

On top of the CLI configuration for the on-demand instances, autospotting can read those values from the tags of the auto-scaling groups. There are two available tags: autospotting_min_on_demand_number and autospotting_min_on_demand_percentage.

Just like for the CLI configuration the defined number has a higher priority than the percentage value. So the percentage will be ignored if autospotting_min_on_demand_number is present and valid.

The order of priority from strongest to lowest for minimum on-demand configuration is as following:

1. Tag autospotting_min_on_demand_number in ASG
2. Tag autospotting_min_on_demand_percentage in ASG
3. Option -min_on_demand_number in CLI
4. Option -min_on_demand_percentage in CLI

Note: the percentage does round up values. Therefore if we have for example 3 instances running in an autoscaling-group, and you specify 10%, autospotting will understand that you want 0 instances. If you specify 16%, then it will still understand that you want 0 instances, because 0.16 * 3 is equal to 0.47999 so it is rounded down to 0; but if you specify 17% (or more than 16.66667%) then the algorithm understands that you want at least one instance (0.17 * 3 = 0.51). All in all it should work as you expect, but this was just to explain some more the functionning of the percentage's math.

Debug autospotting

In certain situations you might want to add verbosity to the project in order to understand a bit better what its doing. If you want to do so please run it with the following environment variable AUTOSPOTTING_DEBUG.

You can do it locally with some custom binary:

 AUTOSPOTTING_DEBUG=true ./autospotting

Or you can do it via the Lambda console under the Environment variables section. Please note those variables aren't exposed via Cloudformation nor via terraform.

Updates and Downgrades

The software doesn't auto-update anymore(it used to in the first few versions), so you will need to manually perform updates using CloudFormation, based on the Travis CI build number of the version you would like to use going forward.

This method can be used both for upgrades and downgrades, so assuming you would like to switch to the build with the number 45, you will need to perform a CloudFormation stack update in which you change the "LambdaZipPath" stack parameter to a value that looks like nightly/lambda_build_45.zip.

Git commit SHAs(truncated to 7 characters) are also accepted instead of the build numbers, so for example nightly/lambda_build_f7f395d.zip should also be a valid parameter, as long as that build is available in the author's S3 bucket.

The full list of the objects available in the bucket can be seen here.

The full list of TravisCI builds and their respective git commits can be seen on the Travis CI builds page

Compatibility notices

• As of build 79 the CloudFormation template is also versioned for every subsequent build, but unfortunately this build also breaks compatibility with older stacks. If you run an older build you will also need to update the stack when updating to a build later than 79. Although the template rarely changes, it's recommended that you always keep it at the same build as the binary. Make sure you use the following stack parameter on any newer builds:

LambdaHandlerFunction: handler.Handle

Uninstallation

If at some point you want to uninstall it, the AutoScaling groups where it used to be enabled will keep running until their spot instances eventually get outbid and terminated, then replaced by AutoScaling with on-demand ones. This is eventually bringing the group to the initial state. If you want, you can speed up the process by gradually terminating the spot instances yourself.

The tags set on the group can be deleted at any time you want it to be disabled for that group.

Uninstall via cloudformation

You just need to delete the CloudFormation stack:

 aws cloudformation delete-stack --stack-name AutoSpotting

Uninstall via terraform

You just need to delete the elements via terraform:

 cd terraform/
 echo "yes" | terraform destroy