diff --git a/docs/design/expansion.md b/docs/design/expansion.md new file mode 100644 index 0000000000000..00c32797b4f34 --- /dev/null +++ b/docs/design/expansion.md @@ -0,0 +1,407 @@ +# Variable expansion in pod command, args, and env + +## Abstract + +A proposal for the expansion of environment variables using a simple `$(var)` syntax. + +## Motivation + +It is extremely common for users to need to compose environment variables or pass arguments to +their commands using the values of environment variables. Kubernetes should provide a facility for +the 80% cases in order to decrease coupling and the use of workarounds. + +## Goals + +1. Define the syntax format +2. Define the scoping and ordering of substitutions +3. Define the behavior for unmatched variables +4. Define the behavior for unexpected/malformed input + +## Constraints and Assumptions + +* This design should describe the simplest possible syntax to accomplish the use-cases +* Expansion syntax will not support more complicated shell-like behaviors such as default values + (viz: `$(VARIABLE_NAME:"default")`), inline substitution, etc. + +## Use Cases + +1. As a user, I want to compose new environment variables for a container using a substitution + syntax to reference other variables in the container's environment and service environment + variables +1. As a user, I want to substitute environment variables into a container's command +1. As a user, I want to do the above without requiring the container's image to have a shell +1. As a user, I want to be able to specify a default value for a service variable which may + not exist +1. As a user, I want to see an event associated with the pod if an expansion fails (ie, references + variable names that cannot be expanded) + +### Use Case: Composition of environment variables + +Currently, containers are injected with docker-style environment variables for the services in +their pod's namespace. There are several variables for each service, but users routinely need +to compose URLs based on these variables because there is not a variable for the exact format +they need. Users should be able to build new environment variables with the exact format they need. +Eventually, it should also be possible to turn off the automatic injection of the docker-style +variables into pods and let the users consume the exact information they need via the downward API +and composition. + +#### Expanding expanded variables + +It should be possible to reference an variable which is itself the result of an expansion, if the +referenced variable is declared in the container's environment prior to the one referencing it. +Put another way -- a container's environment is expanded in order, and expanded variables are +available to subsequent expansions. + +### Use Case: Variable expansion in command + +Users frequently need to pass the values of environment variables to a container's command. +Currently, Kubernetes does not perform any expansion of varibles. The workaround is to invoke a +shell in the container's command and have the shell perform the substitution, or to write a wrapper +script that sets up the environment and runs the command. This has a number of drawbacks: + +1. Solutions that require a shell are unfriendly to images that do not contain a shell +2. Wrapper scripts make it harder to use images as base images +3. Wrapper scripts increase coupling to kubernetes + +Users should be able to do the 80% case of variable expansion in command without writing a wrapper +script or adding a shell invocation to their containers' commands. + +### Use Case: Images without shells + +The current workaround for variable expansion in a container's command requires the container's +image to have a shell. This is unfriendly to images that do not contain a shell (`scratch` images, +for example). Users should be able to perform the other use-cases in this design without regard to +the content of their images. + +### Use Case: See an event for incomplete expansions + +It is possible that a container with incorrect variable values or command line may continue to run +for a long period of time, and that the end-user would have no visual or obvious warning of the +incorrect configuration. If the kubelet creates an event when an expansion references a variable +that cannot be expanded, it will help users quickly detect problems with expansions. + +## Design Considerations + +### What features should be supported? + +In order to limit complexity, we want to provide the right amount of functionality so that the 80% +cases can be realized and nothing more. We felt that the essentials boiled down to: + +1. Ability to perform direct expansion of variables in a string +2. Ability to specify default values via a prioritized mapping function but without support for + defaults as a syntax-level feature + +### What should the syntax be? + +The exact syntax for variable expansion has a large impact on how users perceive and relate to the +feature. We considered implementing a very restrictive subset of the shell `${var}` syntax. This +syntax is an attractive option on some level, because many people are familiar with it. However, +this syntax also has a large number of lesser known features such as the ability to provide +default values for unset variables, perform inline substitution, etc. + +In the interest of preventing conflation of the expansion feature in Kubernetes with the shell +feature, we chose a different syntax similar to the one in Makefiles, `$(var)`. We also chose not +to support the bar `$var` format, since it is not required to implement the required use-cases. + +Nested references, ie, variable expansion within variable names, are not supported. + +#### How should unmatched references be treated? + +Ideally, it should be extremely clear when a variable reference couldn't be expanded. We decided +the best experience for unmatched variable references would be to have the entire reference, syntax +included, show up in the output. As an example, if the reference `$(VARIABLE_NAME)` cannot be +expanded, then `$(VARIABLE_NAME)` should be present in the output. + +#### Escaping the operator + +Although the `$(var)` syntax does overlap with the `$(command)` form of command substitution +supported by many shells, because unexpanded variables are present verbatim in the output, we +expect this will not present a problem to many users. If there is a collision between a varible +name and command substitution syntax, the syntax can be escaped with the form `$$(VARIABLE_NAME)`, +which will evaluate to `$(VARIABLE_NAME)` whether `VARIABLE_NAME` can be expanded or not. + +## Design + +This design encompasses the variable expansion syntax and specification and the changes needed to +incorporate the expansion feature into the container's environment and command. + +### Syntax and expansion mechanics + +This section describes the expansion syntax, evaluation of variable values, and how unexpected or +malformed inputs are handled. + +#### Syntax + +The inputs to the expansion feature are: + +1. A utf-8 string (the input string) which may contain variable references +2. A function (the mapping function) that maps the name of a variable to the variable's value, of + type `func(string) string` + +Variable references in the input string are indicated exclusively with the syntax +`$()`. The syntax tokens are: + +- `$`: the operator +- `(`: the reference opener +- `)`: the reference closer + +The operator has no meaning unless accompanied by the reference opener and closer tokens. The +operator can be escaped using `$$`. One literal `$` will be emitted for each `$$` in the input. + +The reference opener and closer characters have no meaning when not part of a variable reference. +If a variable reference is malformed, viz: `$(VARIABLE_NAME` without a closing expression, the +operator and expression opening characters are treated as ordinary characters without special +meanings. + +#### Scope and ordering of substitutions + +The scope in which variable references are expanded is defined by the mapping function. Within the +mapping function, any arbitrary strategy may be used to determine the value of a variable name. +The most basic implementation of a mapping function is to use a `map[string]string` to lookup the +value of a variable. + +In order to support default values for variables like service variables presented by the kubelet, +which may not be bound because the service that provides them does not yet exist, there should be a +mapping function that uses a list of `map[string]string` like: + +```go +func MakeMappingFunc(maps ...map[string]string) func(string) string { + return func(input string) string { + for _, context := range maps { + val, ok := context[input] + if ok { + return val + } + } + + return "" + } +} + +// elsewhere +containerEnv := map[string]string{ + "FOO": "BAR", + "ZOO": "ZAB", + "SERVICE2_HOST": "some-host", +} + +serviceEnv := map[string]string{ + "SERVICE_HOST": "another-host", + "SERVICE_PORT": "8083", +} + +// single-map variation +mapping := MakeMappingFunc(containerEnv) + +// default variables not found in serviceEnv +mappingWithDefaults := MakeMappingFunc(serviceEnv, containerEnv) +``` + +### Implementation changes + +The necessary changes to implement this functionality are: + +1. Add a new interface, `ObjectEventRecorder`, which is like the `EventRecorder` interface, but + scoped to a single object, and a function that returns an `ObjectEventRecorder` given an + `ObjectReference` and an `EventRecorder` +2. Introduce `third_party/golang/expansion` package that provides: + 1. An `Expand(string, func(string) string) string` function + 2. A `MappingFuncFor(ObjectEventRecorder, ...map[string]string) string` function +3. Add a new EnvVarSource for expansions and associated tests +4. Make the kubelet expand environment correctly +5. Make the kubelet expand command correctly + +#### Event Recording + +In order to provide an event when an expansion references undefined variables, the mapping function +must be able to create an event. In order to facilitate this, we should create a new interface in +the `api/client/record` package which is similar to `EventRecorder`, but scoped to a single object: + +```go +// ObjectEventRecorder knows how to record events about a single object. +type ObjectEventRecorder interface { + // Event constructs an event from the given information and puts it in the queue for sending. + // 'reason' is the reason this event is generated. 'reason' should be short and unique; it will + // be used to automate handling of events, so imagine people writing switch statements to + // handle them. You want to make that easy. + // 'message' is intended to be human readable. + // + // The resulting event will be created in the same namespace as the reference object. + Event(reason, message string) + + // Eventf is just like Event, but with Sprintf for the message field. + Eventf(reason, messageFmt string, args ...interface{}) + + // PastEventf is just like Eventf, but with an option to specify the event's 'timestamp' field. + PastEventf(timestamp util.Time, reason, messageFmt string, args ...interface{}) +} +``` + +There should also be a function that can construct an `ObjectEventRecorder` from a `runtime.Object` +and an `EventRecorder`: + +```go +type objectRecorderImpl struct { + object runtime.Object + recorder EventRecorder +} + +func (r *objectRecorderImpl) Event(reason, message string) { + r.recorder.Event(r.object, reason, message) +} + +func ObjectEventRecorderFor(object runtime.Object, recorder EventRecorder) ObjectEventRecorder { + return &objectRecorderImpl{object, recorder} +} +``` + +#### Expansion package + +The expansion package should provide two methods: + +```go +// MappingFuncFor returns a mapping function for use with Expand that +// implements the expansion semantics defined in the expansion spec; it +// returns the input string wrapped in the expansion syntax if no mapping +// for the input is found. If no expansion is found for a key, an event +// is raised on the given recorder. +func MappingFuncFor(recorder record.ObjectEventRecorder, context ...map[string]string) func(string) string { + // ... +} + +// Expand replaces variable references in the input string according to +// the expansion spec using the given mapping function to resolve the +// values of variables. +func Expand(input string, mapping func(string) string) string { + // ... +} +``` + +#### Expansion `EnvVarSource` + +In order to avoid changing the existing behavior of the `EnvVar.Value` field, there should be a new +`EnvVarSource` that represents a variable expansion that an env var's value should come from: + +```go +// EnvVarSource represents a source for the value of an EnvVar. +type EnvVarSource struct { + // Other fields omitted + + Expansion *EnvVarExpansion +} + +type EnvVarExpansion struct { + // The input string to be expanded + Expand string +} +``` + +#### Kubelet changes + +The Kubelet should change to: + +1. Correctly expand environment variables with `Expansion` sources +2. Correctly expand references in the Command and Args + +### Examples + +#### Inputs and outputs + +These examples are in the context of the mapping: + +| Name | Value | +|-------------|------------| +| `VAR_A` | `"A"` | +| `VAR_B` | `"B"` | +| `VAR_C` | `"C"` | +| `VAR_REF` | `$(VAR_A)` | +| `VAR_EMPTY` | `""` | + +No other variables are defined. + +| Input | Result | +|--------------------------------|----------------------------| +| `"$(VAR_A)"` | `"A"` | +| `"___$(VAR_B)___"` | `"___B___"` | +| `"___$(VAR_C)"` | `"___C"` | +| `"$(VAR_A)-$(VAR_A)"` | `"A-A"` | +| `"$(VAR_A)-1"` | `"A-1"` | +| `"$(VAR_A)_$(VAR_B)_$(VAR_C)"` | `"A_B_C"` | +| `"$$(VAR_B)_$(VAR_A)"` | `"$(VAR_B)_A"` | +| `"$$(VAR_A)_$$(VAR_B)"` | `"$(VAR_A)_$(VAR_B)"` | +| `"f000-$$VAR_A"` | `"f000-$VAR_A"` | +| `"foo\\$(VAR_C)bar"` | `"foo\Cbar"` | +| `"foo\\\\$(VAR_C)bar"` | `"foo\\Cbar"` | +| `"foo\\\\\\\\$(VAR_A)bar"` | `"foo\\\\Abar"` | +| `"$(VAR_A$(VAR_B))"` | `"$(VAR_A$(VAR_B))"` | +| `"$(VAR_A$(VAR_B)"` | `"$(VAR_A$(VAR_B)"` | +| `"$(VAR_REF)"` | `"$(VAR_A)"` | +| `"%%$(VAR_REF)--$(VAR_REF)%%"` | `"%%$(VAR_A)--$(VAR_A)%%"` | +| `"foo$(VAR_EMPTY)bar"` | `"foobar"` | +| `"foo$(VAR_Awhoops!"` | `"foo$(VAR_Awhoops!"` | +| `"f00__(VAR_A)__"` | `"f00__(VAR_A)__"` | +| `"$?_boo_$!"` | `"$?_boo_$!"` | +| `"$VAR_A"` | `"$VAR_A"` | +| `"$(VAR_DNE)"` | `"$(VAR_DNE)"` | +| `"$$$$$$(BIG_MONEY)"` | `"$$$(BIG_MONEY)"` | +| `"$$$$$$(VAR_A)"` | `"$$$(VAR_A)"` | +| `"$$$$$$$(GOOD_ODDS)"` | `"$$$$(GOOD_ODDS)"` | +| `"$$$$$$$(VAR_A)"` | `"$$$A"` | +| `"$VAR_A)"` | `"$VAR_A)"` | +| `"${VAR_A}"` | `"${VAR_A}"` | +| `"$(VAR_B)_______$(A"` | `"B_______$(A"` | +| `"$(VAR_C)_______$("` | `"C_______$("` | +| `"$(VAR_A)foobarzab$"` | `"Afoobarzab$"` | +| `"foo-\\$(VAR_A"` | `"foo-\$(VAR_A"` | +| `"--$($($($($--"` | `"--$($($($($--"` | +| `"$($($($($--foo$("` | `"$($($($($--foo$("` | +| `"foo0--$($($($("` | `"foo0--$($($($("` | +| `"$(foo$$var)` | `$(foo$$var)` | + +#### In a pod: building a URL + +Notice the `$(var)` syntax. + +```yaml +apiVersion: v1beta3 +kind: Pod +metadata: + name: expansion-pod +spec: + containers: + - name: test-container + image: gcr.io/google_containers/busybox + command: [ "/bin/sh", "-c", "env" ] + env: + - name: PUBLIC_URL + valueFrom: + expansion: + expand: "http://$(GITSERVER_SERVICE_HOST):$(GITSERVER_SERVICE_PORT)" + restartPolicy: Never +``` + +#### In a pod: building a URL using downward API + +```yaml +apiVersion: v1beta3 +kind: Pod +metadata: + name: expansion-pod +spec: + containers: + - name: test-container + image: gcr.io/google_containers/busybox + command: [ "/bin/sh", "-c", "env" ] + env: + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: "metadata.namespace" + - name: PUBLIC_URL + valueFrom: + expansion: + expand: "http://gitserver.$(POD_NAMESPACE):$(SERVICE_PORT)" + restartPolicy: Never +``` + +[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/expansion.md?pixel)]() diff --git a/third_party/golang/expansion/expand.go b/third_party/golang/expansion/expand.go new file mode 100644 index 0000000000000..468727dac9032 --- /dev/null +++ b/third_party/golang/expansion/expand.go @@ -0,0 +1,98 @@ +package expansion + +const ( + operator = '$' + referenceOpener = '(' + referenceCloser = ')' +) + +// syntaxWrap returns the input string wrapped the expansion syntax. +func syntaxWrap(input string) string { + return string(operator) + string(referenceOpener) + input + string(referenceCloser) +} + +// MappingFuncFor returns a mapping function for use with Expand that +// implements the expansion semantics defined in the expansion spec; it +// returns the input string wrapped in the expansion syntax if no mapping +// for the input is found. +func MappingFuncFor(context ...map[string]string) func(string) string { + return func(input string) string { + for _, vars := range context { + val, ok := vars[input] + if ok { + return val + } + } + + return syntaxWrap(input) + } +} + +// Expand replaces variable references in the input string according to +// the expansion spec using the given mapping function to resolve the +// values of variables. +func Expand(input string, mapping func(string) string) string { + buf := make([]byte, 0, 2*len(input)) + checkpoint := 0 + for cursor := 0; cursor < len(input); cursor++ { + if input[cursor] == operator && cursor+1 < len(input) { + // Copy the portion of the input string since the last + // checkpoint into the buffer + buf = append(buf, input[checkpoint:cursor]...) + + // Attempt to read the variable name as defined by the + // syntax from the input string + read, isVar, advance := tryReadVariableName(input[cursor+1:]) + + if isVar { + // We were able to read a variable name correctly; + // apply the mapping to the variable name and copy the + // bytes into the buffer + buf = append(buf, mapping(read)...) + } else { + // Not a variable name; copy the read bytes into the buffer + buf = append(buf, read...) + } + + // Advance the cursor in the input string to account for + // bytes consumed to read the variable name expression + cursor += advance + + // Advance the checkpoint in the input string + checkpoint = cursor + 1 + } + } + + // Return the buffer and any remaining unwritten bytes in the + // input string. + return string(buf) + input[checkpoint:] +} + +// tryReadVariableName attempts to read a variable name from the input +// string and returns the content read from the input, whether that content +// represents a variable name to perform mapping on, and the number of bytes +// consumed in the input string. +// +// The input string is assumed not to contain the initial operator. +func tryReadVariableName(input string) (string, bool, int) { + switch input[0] { + case operator: + // Escaped operator; return it. + return input[0:1], false, 1 + case referenceOpener: + // Scan to expression closer + for i := 1; i < len(input); i++ { + if input[i] == referenceCloser { + return input[1:i], true, i + 1 + } + } + + // Incomplete reference; return it. + return string(operator) + string(referenceOpener), false, 1 + default: + // Not the beginning of an expression, ie, an operator + // that doesn't begin an expression. Return the operator + // and the first rune in the string. + return (string(operator) + string(input[0])), false, 1 + } +} diff --git a/third_party/golang/expansion/expand_test.go b/third_party/golang/expansion/expand_test.go new file mode 100644 index 0000000000000..c2060e706e154 --- /dev/null +++ b/third_party/golang/expansion/expand_test.go @@ -0,0 +1,285 @@ +package expansion + +import ( + "testing" + + "github.com/GoogleCloudPlatform/kubernetes/pkg/api" +) + +func TestMapReference(t *testing.T) { + envs := []api.EnvVar{ + { + Name: "FOO", + Value: "bar", + }, + { + Name: "ZOO", + Value: "$(FOO)-1", + }, + { + Name: "BLU", + Value: "$(ZOO)-2", + }, + } + + declaredEnv := map[string]string{ + "FOO": "bar", + "ZOO": "$(FOO)-1", + "BLU": "$(ZOO)-2", + } + + serviceEnv := map[string]string{} + + mapping := MappingFuncFor(declaredEnv, serviceEnv) + + for _, env := range envs { + declaredEnv[env.Name] = Expand(env.Value, mapping) + } + + expectedEnv := map[string]string{ + "FOO": "bar", + "ZOO": "bar-1", + "BLU": "bar-1-2", + } + + for k, v := range expectedEnv { + if e, a := v, declaredEnv[k]; e != a { + t.Errorf("Expected %v, got %v", e, a) + } else { + delete(declaredEnv, k) + } + } + + if len(declaredEnv) != 0 { + t.Errorf("Unexpected keys in declared env: %v", declaredEnv) + } +} + +func TestMapping(t *testing.T) { + context := map[string]string{ + "VAR_A": "A", + "VAR_B": "B", + "VAR_C": "C", + "VAR_REF": "$(VAR_A)", + "VAR_EMPTY": "", + } + mapping := MappingFuncFor(context) + + doExpansionTest(t, mapping) +} + +func TestMappingDual(t *testing.T) { + context := map[string]string{ + "VAR_A": "A", + "VAR_EMPTY": "", + } + context2 := map[string]string{ + "VAR_B": "B", + "VAR_C": "C", + "VAR_REF": "$(VAR_A)", + } + mapping := MappingFuncFor(context, context2) + + doExpansionTest(t, mapping) +} + +func doExpansionTest(t *testing.T, mapping func(string) string) { + cases := []struct { + name string + input string + expected string + }{ + { + name: "whole string", + input: "$(VAR_A)", + expected: "A", + }, + { + name: "repeat", + input: "$(VAR_A)-$(VAR_A)", + expected: "A-A", + }, + { + name: "beginning", + input: "$(VAR_A)-1", + expected: "A-1", + }, + { + name: "middle", + input: "___$(VAR_B)___", + expected: "___B___", + }, + { + name: "end", + input: "___$(VAR_C)", + expected: "___C", + }, + { + name: "compound", + input: "$(VAR_A)_$(VAR_B)_$(VAR_C)", + expected: "A_B_C", + }, + { + name: "escape & expand", + input: "$$(VAR_B)_$(VAR_A)", + expected: "$(VAR_B)_A", + }, + { + name: "compound escape", + input: "$$(VAR_A)_$$(VAR_B)", + expected: "$(VAR_A)_$(VAR_B)", + }, + { + name: "mixed in escapes", + input: "f000-$$VAR_A", + expected: "f000-$VAR_A", + }, + { + name: "backslash escape ignored", + input: "foo\\$(VAR_C)bar", + expected: "foo\\Cbar", + }, + { + name: "backslash escape ignored", + input: "foo\\\\$(VAR_C)bar", + expected: "foo\\\\Cbar", + }, + { + name: "lots of backslashes", + input: "foo\\\\\\\\$(VAR_A)bar", + expected: "foo\\\\\\\\Abar", + }, + { + name: "nested var references", + input: "$(VAR_A$(VAR_B))", + expected: "$(VAR_A$(VAR_B))", + }, + { + name: "nested var references second type", + input: "$(VAR_A$(VAR_B)", + expected: "$(VAR_A$(VAR_B)", + }, + { + name: "value is a reference", + input: "$(VAR_REF)", + expected: "$(VAR_A)", + }, + { + name: "value is a reference x 2", + input: "%%$(VAR_REF)--$(VAR_REF)%%", + expected: "%%$(VAR_A)--$(VAR_A)%%", + }, + { + name: "empty var", + input: "foo$(VAR_EMPTY)bar", + expected: "foobar", + }, + { + name: "unterminated expression", + input: "foo$(VAR_Awhoops!", + expected: "foo$(VAR_Awhoops!", + }, + { + name: "expression without operator", + input: "f00__(VAR_A)__", + expected: "f00__(VAR_A)__", + }, + { + name: "shell special vars pass through", + input: "$?_boo_$!", + expected: "$?_boo_$!", + }, + { + name: "bare operators are ignored", + input: "$VAR_A", + expected: "$VAR_A", + }, + { + name: "undefined vars are passed through", + input: "$(VAR_DNE)", + expected: "$(VAR_DNE)", + }, + { + name: "multiple (even) operators, var undefined", + input: "$$$$$$(BIG_MONEY)", + expected: "$$$(BIG_MONEY)", + }, + { + name: "multiple (even) operators, var defined", + input: "$$$$$$(VAR_A)", + expected: "$$$(VAR_A)", + }, + { + name: "multiple (odd) operators, var undefined", + input: "$$$$$$$(GOOD_ODDS)", + expected: "$$$$(GOOD_ODDS)", + }, + { + name: "multiple (odd) operators, var defined", + input: "$$$$$$$(VAR_A)", + expected: "$$$A", + }, + { + name: "missing open expression", + input: "$VAR_A)", + expected: "$VAR_A)", + }, + { + name: "shell syntax ignored", + input: "${VAR_A}", + expected: "${VAR_A}", + }, + { + name: "trailing incomplete expression not consumed", + input: "$(VAR_B)_______$(A", + expected: "B_______$(A", + }, + { + name: "trailing incomplete expression, no content, is not consumed", + input: "$(VAR_C)_______$(", + expected: "C_______$(", + }, + { + name: "operator at end of input string is preserved", + input: "$(VAR_A)foobarzab$", + expected: "Afoobarzab$", + }, + { + name: "shell escaped incomplete expr", + input: "foo-\\$(VAR_A", + expected: "foo-\\$(VAR_A", + }, + { + name: "lots of $( in middle", + input: "--$($($($($--", + expected: "--$($($($($--", + }, + { + name: "lots of $( in beginning", + input: "$($($($($--foo$(", + expected: "$($($($($--foo$(", + }, + { + name: "lots of $( at end", + input: "foo0--$($($($(", + expected: "foo0--$($($($(", + }, + { + name: "escaped operators in variable names are not escaped", + input: "$(foo$$var)", + expected: "$(foo$$var)", + }, + { + name: "newline not expanded", + input: "\n", + expected: "\n", + }, + } + + for _, tc := range cases { + expanded := Expand(tc.input, mapping) + if e, a := tc.expected, expanded; e != a { + t.Errorf("%v: expected %q, got %q", tc.name, e, a) + } + } +}