Skip to content

Commit

Permalink
UserData wrappers for CloudFormation helper scripts when using Beanst…
Browse files Browse the repository at this point in the history
…alk (LeanerCloud#366)

* Support custom role for cfn-init in Beanstalk UserData

* Wrappers & refactoring

* Docs

* Docs fixes

* More docs fixes

* Docs fixes

* Yet more docs fixes

* Lambda & Kubernetes config

* AutoSpottingElasticBeanstalk managed policy + Rename beanstalk_cfn_init_role to beanstalk_cfn_wrappers

* Update param description

* Kubernetes config fix

* Rename Beanstalk variable + Move test data

* Add missing permission for AutoSpottingBeanstalk role
  • Loading branch information
jawadst authored and cristim committed Oct 11, 2019
1 parent 89a188c commit 4382e90
Show file tree
Hide file tree
Showing 13 changed files with 638 additions and 6 deletions.
58 changes: 54 additions & 4 deletions START.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,10 +189,53 @@ be risky, please handle with care.

### For Elastic Beanstalk ###

* In order to add tags to existing Elastic Beanstalk environment, you will need
to rebuild or update the environment with the `spot-enabled` tag. For more
details you can follow this
[guide](http://www.boringgeek.com/add-or-update-tags-on-existing-elastic-beanstalk-environments)
Elastic Beanstalk uses CloudFormation to create an Auto-Scaling Group. The ASG
is then in charge of automatically scaling your application up and down. As a
result, AutoSpotting works natively with Elastic Beanstalk.

Follow these steps to configure AutoSpotting with Elastic Beanstalk.

#### 1 - Add the `spot-enabled` tag ####

Similar to standalone auto-scaling groups, you need to tag your Elastic Beanstalk
environment with the `spot-enabled` tag to let AutoSpotting manage the instances
in the group.

To add tags to an existing Elastic Beanstalk environment, you will need to rebuild
or update the environment with the `spot-enabled` tag. For more details you can
follow this [guide](http://www.boringgeek.com/add-or-update-tags-on-existing-elastic-beanstalk-environments).

#### 2 - Enable `patch_beanstalk_userdata` in AutoSpotting (optional) ####

Elastic Beanstalk leverages CloudFormation for creating resources and initializing
instances. When a new instance is launched, Elastic Beanstalk configures it through
the auto-scaling configuration (`UserData` and tags).

AutoSpotting launches spot instances outside of the auto-scaling group and attaches
them to the group after a grace period. As a result, the Elastic Beanstalk
initialization process can randomly fail or be delayed by 10+ minutes.
When it is delayed, the spot instances take a long time (10+ minutes) before being
initialized, appearing as healthy in Elastic Beanstalk and being added
to the load balancer.

As a solution, you can configure AutoSpotting to alter the Elastic Beanstalk
user-data so that the Elastic Beanstalk initialization process can run even
if the spot instance is not a part of the auto-scaling group.

To enable that option, set the `patch_beanstalk_userdata` variable to `true`
in your configuration.

You will also need to update the permissions of the role used by your instances
to authorize requests to the CloudFormation API. Add the `AutoSpottingElasticBeanstalk`
policy to the role `aws-elasticbeanstalk-ec2-role` or the custom instance profile/role
used by your Beanstalk instances.

The permissions contained in `AutoSpottingElasticBeanstalk` are required if you set
`patch_beanstalk_userdata` variable to `true`. If they are not added, your spot
instances will not be able to run correctly.

You can get more information on the need for this configuration variable and
the permissions in the [bug report](https://github.com/AutoSpotting/AutoSpotting/issues/344).

## Configuration of AutoSpotting ##

Expand Down Expand Up @@ -258,6 +301,13 @@ Usage of ./AutoSpotting:
-tag_filters=[{spot-enabled true}]: Set of tags to filter the ASGs on. Default is -tag_filters 'spot-enabled=true'
Example: ./AutoSpotting -tag_filters 'spot-enabled=true,Environment=dev,Team=vision'
-patch_beanstalk_userdata="true":
Controls whether AutoSpotting patches Elastic Beanstalk UserData
scripts to use the instance role when calling CloudFormation
helpers instead of the standard CloudFormation authentication
method
Example: ./AutoSpotting --patch_beanstalk_userdata true
```

<!-- markdownlint-enable MD013 -->
Expand Down
6 changes: 5 additions & 1 deletion autospotting.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,8 @@ func run() {
"termination_notification_action=%s "+
"cron_schedule=%s "+
"cron_schedule_state=%s "+
"license=%s \n",
"license=%s "+
"patch_beanstalk_userdata=%s \n",
conf.Regions,
conf.MinOnDemandNumber,
conf.MinOnDemandPercentage,
Expand All @@ -69,6 +70,7 @@ func run() {
conf.CronSchedule,
conf.CronScheduleState,
conf.LicenseType,
conf.PatchBeanstalkUserdata,
)

autospotting.Run(conf.Config)
Expand Down Expand Up @@ -215,6 +217,8 @@ func (c *cfgData) parseCommandLineFlags() {
flag.StringVar(&c.LicenseType, "license", "evaluation", "\n\tControls the terms under which you use AutoSpotting"+
"Allowed values: evaluation|I_am_supporting_it_on_Patreon|I_contributed_to_development_within_the_last_year|I_built_it_from_source_code\n"+
"\tExample: ./AutoSpotting --license evaluation\n")
flag.StringVar(&c.PatchBeanstalkUserdata, "patch_beanstalk_userdata", "", "\n\tControls whether AutoSpotting patches Elastic Beanstalk UserData scripts to use the instance role when calling CloudFormation helpers instead of the standard CloudFormation authentication method\n"+
"\tExample: ./AutoSpotting --patch_beanstalk_userdata true\n")

v := flag.Bool("version", false, "Print version number and exit.\n")
flag.Parse()
Expand Down
33 changes: 33 additions & 0 deletions cloudformation/stacks/AutoSpotting/template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,19 @@
binaries are restricted to up to $1000 in monthly savings, the others
are not restricted"
Type: "String"
PatchBeanstalkUserdata:
Default: "false"
AllowedValues:
- "false"
- "true"
Description: >
"Controls whether AutoSpotting patches Elastic Beanstalk UserData
scripts to use the instance role when calling CloudFormation helpers
instead of the standard CloudFormation authentication method.
After creating this CloudFormation stack, you must add the
AutoSpottingElasticBeanstalk managed policy to your Beanstalk
instance profile/role if you turn this option to On"
Type: "String"
Conditions:
StackSetsFalse: !Equals
- !Ref DeployUsingStackSets
Expand Down Expand Up @@ -409,6 +422,8 @@
Ref: "FilterByTags"
TERMINATION_NOTIFICATION_ACTION:
Ref: "TerminationNotificationAction"
PATCH_BEANSTALK_USERDATA:
Ref: "PatchBeanstalkUserdata"
Handler:
Ref: "LambdaHandlerFunction"
MemorySize:
Expand Down Expand Up @@ -729,3 +744,21 @@
except:
print_exc()
print("Unexpected error:", exc_info()[0])
ElasticBeanstalkPolicy:
Type: AWS::IAM::ManagedPolicy
Condition: "StackIsMain"
Properties:
Description: "Allow instances to get initialization information from CloudFormation when they are a part of a Beanstalk cluster. Add this policy to your Elastic Beanstalk instance profile (role) when enabling patch_beanstalk_userdata in AutoSpotting. (Managed policy created by AutoSpotting)"
ManagedPolicyName: AutoSpottingElasticBeanstalk
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Action:
- "cloudformation:DescribeStackResource"
- "cloudformation:DescribeStackResources"
- "cloudformation:SignalResource"
- "cloudformation:RegisterListener"
- "cloudformation:GetListenerCredentials"
Effect: "Allow"
Resource: "*"
20 changes: 20 additions & 0 deletions core/autoscaling_configuration.go
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,10 @@ const (
// CronScheduleStateTag is the name of the tag set on the AutoScaling Group that
// can override the global value of the CronScheduleState parameter
CronScheduleStateTag = "autospotting_cron_schedule_state"

// PatchBeanstalkUserdataTag is the name of the tag set on the AutoScaling Group that
// can override the global value of the PatchBeanstalkUserdata parameter
PatchBeanstalkUserdataTag = "patch_beanstalk_userdata"
)

// AutoScalingConfig stores some group-specific configurations that can override
Expand Down Expand Up @@ -99,6 +103,8 @@ type AutoScalingConfig struct {

CronSchedule string
CronScheduleState string // "on" or "off", dictate whether to run inside the CronSchedule or not

PatchBeanstalkUserdata string
}

func (a *autoScalingGroup) loadPercentageOnDemand(tagValue *string) (int64, bool) {
Expand Down Expand Up @@ -178,6 +184,19 @@ func (a *autoScalingGroup) loadConfOnDemand() bool {
return false
}

func (a *autoScalingGroup) loadPatchBeanstalkUserdata() {
tagValue := a.getTagValue(PatchBeanstalkUserdataTag)

if tagValue != nil {
logger.Printf("Loaded PatchBeanstalkUserdata value %v from tag %v\n", *tagValue, PatchBeanstalkUserdataTag)
a.config.PatchBeanstalkUserdata = *tagValue
return
}

debug.Println("Couldn't find tag", PatchBeanstalkUserdataTag, "on the group", a.name, "using the default configuration")
a.config.PatchBeanstalkUserdata = a.region.conf.PatchBeanstalkUserdata
}

func (a *autoScalingGroup) loadBiddingPolicy(tagValue *string) (string, bool) {
biddingPolicy := *tagValue
if biddingPolicy != "aggressive" {
Expand Down Expand Up @@ -255,6 +274,7 @@ func (a *autoScalingGroup) loadConfigFromTags() bool {

a.LoadCronSchedule()
a.LoadCronScheduleState()
a.loadPatchBeanstalkUserdata()

if resOnDemandConf {
logger.Println("Found and applied configuration for OnDemand value")
Expand Down
64 changes: 64 additions & 0 deletions core/autoscaling_configuration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1037,3 +1037,67 @@ func Test_autoScalingGroup_LoadCronScheduleState(t *testing.T) {
})
}
}

func Test_autoScalingGroup_LoadPatchBeanstalkUserdata(t *testing.T) {
tests := []struct {
name string
Group *autoscaling.Group
asgName string
config AutoScalingConfig
region *region
want string
}{
{
name: "No tag set on the group, use region config (no value)",
Group: &autoscaling.Group{},
region: &region{
conf: &Config{
PatchBeanstalkUserdata: "",
},
},
want: "",
},
{
name: "No tag set on the group, use region config (true)",
Group: &autoscaling.Group{},
region: &region{
conf: &Config{
PatchBeanstalkUserdata: "true",
},
},
want: "true",
},
{
name: "Tag set on the group",
Group: &autoscaling.Group{
Tags: []*autoscaling.TagDescription{
{
Key: aws.String(PatchBeanstalkUserdataTag),
Value: aws.String("false"),
},
},
},
region: &region{
conf: &Config{
PatchBeanstalkUserdata: "true",
},
},
want: "false",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
a := &autoScalingGroup{
Group: tt.Group,
name: tt.asgName,
config: tt.config,
region: tt.region,
}
a.loadPatchBeanstalkUserdata()
got := a.config.PatchBeanstalkUserdata
if got != tt.want {
t.Errorf("LoadPatchBeanstalkUserdata got %v, expected %v", got, tt.want)
}
})
}
}
80 changes: 80 additions & 0 deletions core/beanstalk.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
package autospotting

import (
"encoding/base64"
"strings"
)

// Beanstalk UserData wrappers for CloudFormation Helper scripts
// https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-helper-scripts-reference.html
//
// `cfn-init`, `cfn-get-metadata` and `cfn-signal` are wrapped by adding the
// instance role to the original code as `--role instance-role`
// `cfn-hup` does not accept a `--role` param so we write the role into the config file
// /etc/cfn/cfn-hup.conf
var beanstalkUserDataCFNWrappers = `---- modify CloudFormation helpers ----
# Modify cfn-init to use --role by default
echo -e '#!/bin/bash\nINROLE=$(curl -s 169.254.169.254/latest/meta-data/iam/security-credentials/)\n/opt/aws/bin/cfn-init-2 --role $INROLE "$@" \nexit $?' > /opt/aws/bin/cfn-init.tmp
mv /opt/aws/bin/cfn-init /opt/aws/bin/cfn-init-2
mv /opt/aws/bin/cfn-init.tmp /opt/aws/bin/cfn-init
chmod +x /opt/aws/bin/cfn-init
# Modify cfn-get-metadata to use --role by default
echo -e '#!/bin/bash\nINROLE=$(curl -s 169.254.169.254/latest/meta-data/iam/security-credentials/)\n/opt/aws/bin/cfn-get-metadata-2 --role $INROLE "$@" \nexit $?' > /opt/aws/bin/cfn-get-metadata.tmp
mv /opt/aws/bin/cfn-get-metadata /opt/aws/bin/cfn-get-metadata-2
mv /opt/aws/bin/cfn-get-metadata.tmp /opt/aws/bin/cfn-get-metadata
chmod +x /opt/aws/bin/cfn-get-metadata
# Modify cfn-signal to use --role by default
echo -e '#!/bin/bash\nINROLE=$(curl -s 169.254.169.254/latest/meta-data/iam/security-credentials/)\n/opt/aws/bin/cfn-signal-2 --role $INROLE "$@" \nexit $?' > /opt/aws/bin/cfn-signal.tmp
mv /opt/aws/bin/cfn-signal /opt/aws/bin/cfn-signal-2
mv /opt/aws/bin/cfn-signal.tmp /opt/aws/bin/cfn-signal
chmod +x /opt/aws/bin/cfn-signal
# Modify cfn-hup to use --role by default
echo -e '#!/bin/bash\nprintf "role=$(curl -s 169.254.169.254/latest/meta-data/iam/security-credentials/)" >> /etc/cfn/cfn-hup.conf\n/opt/aws/bin/cfn-hup-2 "$@" \nexit $?' > /opt/aws/bin/cfn-hup.tmp
mv /opt/aws/bin/cfn-hup /opt/aws/bin/cfn-hup-2
mv /opt/aws/bin/cfn-hup.tmp /opt/aws/bin/cfn-hup
chmod +x /opt/aws/bin/cfn-hup
---- modify CloudFormation helpers ----
`

func decodeUserData(userData *string) *string {
// UserData is sometimes encoded as base64 ; decoded it if needed
decodedUserData, err := base64.StdEncoding.DecodeString(*userData)

if err != nil {
// This is not Base64-encoded, return the original string
return userData
}

// This was Base64-encoded, return the decoded string
decodedUserDataString := string(decodedUserData)
return &decodedUserDataString
}

func encodeUserData(userData *string) *string {
// Encode UserData string to base64
encodedUserData := base64.StdEncoding.EncodeToString([]byte(*userData))

return &encodedUserData
}

func getPatchedUserDataForBeanstalk(userData *string) *string {
// Decode the UserData
decodedUserData := decodeUserData(userData)

// Patch the UserData if possible
if strings.Contains(*decodedUserData, "ebbootstrap") {
// Force set the role for calling CloudFormation helpers to be the instance role
// The UserData created by Beanstalk is encoded as a Mime Multi Part Archive
// with Cloud Init User-Data format (https://cloudinit.readthedocs.io/en/latest/topics/format.html)
// We can't simply append our extra code to it, we need to add it to the correct mime part
// Hence, we replace the first `#!/bin/bash` with our wrapper
patchedUserData := strings.Replace(*decodedUserData, "#!/bin/bash\n", "#!/bin/bash\n"+beanstalkUserDataCFNWrappers, 1)
return encodeUserData(&patchedUserData)
}

return userData
}
Loading

0 comments on commit 4382e90

Please sign in to comment.