forked from kubernetes/community
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request kubernetes#524 from deads2k/tpr-to-beta
Proposal: TPR to beta proposal
- Loading branch information
Showing
1 changed file
with
253 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,253 @@ | ||
| # Moving ThirdPartyResources to beta | ||
|
|
||
| ## Background | ||
| There are a number of important issues with the alpha version of | ||
| ThirdPartyResources that we wish to address to move TPR to beta. The list is | ||
| tracked [here](https://github.com/kubernetes/features/issues/95), and also | ||
| includes feedback from existing Kubernetes ThirdPartyResource users. This | ||
| proposal covers the steps we believe are necessary to move TPR to beta and to | ||
| prevent future challenges in upgrading. | ||
|
|
||
|
|
||
| ## Goals | ||
| 1. Ensure ThirdPartyResource APIs operate consistently with first party | ||
| Kubernetes APIs. | ||
| 2. Enable ThirdPartyResources to specify how they will appear in API | ||
| discovery to be consistent with other resources and avoid naming confilcts | ||
| 3. Move TPR into their own API group to allow the extensions group to be | ||
| [removed](https://github.com/kubernetes/kubernetes/issues/43214) | ||
| 4. Support cluster scoped TPR resources | ||
| 5. Identify other features required for TPR to become beta | ||
| 6. Minimize the impact to alpha ThirdPartyResources consumers and define a | ||
| process for how TPR migrations / breaking changes can be accomplished (for | ||
| both the cluster and for end users) | ||
|
|
||
| Non-goals | ||
| 1. Solve automatic conversion of TPR between versions or automatic migration of | ||
| existing TPR | ||
|
|
||
| ### Desired API Semantics | ||
| TPRs are intended to look like normal kube-like resources to external clients. | ||
| In order to do that effectively, they should respect the normal get, list, | ||
| watch, create, patch, update, and delete semantics. | ||
|
|
||
| In "normal" Kubernetes APIs, if I have a persisted resource in the same group | ||
| with the same name in v1 and v2, they are backed by the same underlying object. | ||
| A change made to one is reflected in the other. API clients, garbage collection, | ||
| namespace cleanup, version negotiation, and controllers all build on this. | ||
|
|
||
| The convertibility of Kubernetes APIs provides a seamless interaction between | ||
| versions. A TPR does not have the ability to convert between versions, which | ||
| focuses on the primary role of TPR as an easily extensible and simple mechanism | ||
| for adding new APIs. Conversion primarily allows structural, but not backwards | ||
| incompatible, changes. By not supporting conversion, all TPR use cases are | ||
| preserved, but a large amount of complexity is avoided for consumers of TPR. | ||
|
|
||
| Allowing a single, user specified version for a given TPR will provide this | ||
| semantic by preventing server-side versioning altogether. All instances of a | ||
| single TPR must have the same version or the Kubernetes API semantic of always | ||
| returning a resource encoded to the matching version will not be maintained. | ||
| Since conversions (even native Kubernetes conversions) cannot be used to handle | ||
| behavioral changes, the same effect can be achieved for TPRs client-side with | ||
| overlapping serialization changes. | ||
|
|
||
|
|
||
| ### Avoiding Naming Problems | ||
| There are several identifiers that a Kubernetes API resource has which share | ||
| value-spaces within an API group and must not conflict. They are: | ||
| 1. Resource-type value space | ||
| 1. plural resource-type name - like "configmaps" | ||
| 2. singular resource-type name - like "configmap" | ||
| 3. short names - like "cm" | ||
| 2. Kind-type value space - for group "example.com" | ||
| 1. Kind name - like "ConfigMap" | ||
| 2. ListKind name - like "ConfigMapList" | ||
| If these values conflict within their value-spaces then no client will be able | ||
| to properly distinguish intent. | ||
|
|
||
| The actual name of the TPR-registration (resource that describes the TPR to | ||
| create) resource can only protect one of these values from conflict. Since | ||
| Kubernetes API types are accessed via a URL that looks like `/apis/<group>/<version>/namespaces/<namespace-name>/<plural-resource-type>`, | ||
| the name of the TPR-registration object will be `<plural-resource-type>.<group>`. | ||
|
|
||
| Conflicts with other parts of the value-space can not be detected with static | ||
| validation, so there will be a spec/status split with `status.conditions` that | ||
| reflect the acceptance status of a TPR-registration. For instance, you cannot | ||
| determine whether two TPRs in the same group have the same short name without | ||
| inspecting the current state of existing TPRs. | ||
|
|
||
| Parts of the value-space will be "claimed" by making an entry in TPR.status to | ||
| include the accepted names which will be served. This prevents a new TPR from | ||
| disabling an existing TPR's name. | ||
|
|
||
|
|
||
| ## New API | ||
| In order to: | ||
| 1. eliminate opaquely derived information - deriving camel-cased kind names | ||
| from lower-case dash-delimited values as for instance. | ||
| 1. allow the expression of complex transformations - not all plurals are easily | ||
| determined (ox and oxen) and not all are English. Fields for complete | ||
| specification eliminates ambiguity. | ||
| 1. handle TPR-registration value-space conflicts | ||
| 1. [stop using the extensions API group](https://github.com/kubernetes/kubernetes/issues/43214) | ||
|
|
||
| We can create a type `ThirdPartyResource.apiextension.k8s.io`. | ||
| ```go | ||
| // ThirdPartyResourceSpec describe how a user wants their resource to appear | ||
| type ThirdPartyResourceSpec struct { | ||
| // Group is the group this resource belongs in | ||
| Group string `json:"group" protobuf:"bytes,1,opt,name=group"` | ||
| // Version is the version this resource belongs in | ||
| Version string `json:"version" protobuf:"bytes,2,opt,name=version"` | ||
| // Names holds the information about the resource and kind you have chosen which is | ||
| // surfaced through discovery. | ||
| Names ThirdPartyResourceNames | ||
|
|
||
| // Scope indicates whether this resource is cluster or namespace scoped. Default is namespaced | ||
| Scope ResourceScope `json:"scope" protobuf:"bytes,8,opt,name=scope,casttype=ResourceScope"` | ||
| } | ||
|
|
||
| type ThirdPartyResourceNames struct { | ||
| // Plural is the plural name of the resource to serve. It must match the name of the TPR-registration | ||
| // too: plural.group | ||
| Plural string `json:"plural" protobuf:"bytes,3,opt,name=plural"` | ||
| // Singular is the singular name of the resource. Defaults to lowercased <kind> | ||
| Singular string `json:"singular,omitempty" protobuf:"bytes,4,opt,name=singular"` | ||
| // ShortNames are short names for the resource. | ||
| ShortNames []string `json:"shortNames,omitempty" protobuf:"bytes,5,opt,name=shortNames"` | ||
| // Kind is the serialized kind of the resource | ||
| Kind string `json:"kind" protobuf:"bytes,6,opt,name=kind"` | ||
| // ListKind is the serialized kind of the list for this resource. Defaults to <kind>List | ||
| ListKind string `json:"listKind,omitempty" protobuf:"bytes,7,opt,name=listKind"` | ||
| } | ||
|
|
||
| type ResourceScope string | ||
|
|
||
| const ( | ||
| ClusterScoped ResourceScope = "Cluster" | ||
| NamespaceScoped ResourceScope = "Namespaced" | ||
| ) | ||
|
|
||
| type ConditionStatus string | ||
|
|
||
| // These are valid condition statuses. "ConditionTrue" means a resource is in the condition. | ||
| // "ConditionFalse" means a resource is not in the condition. "ConditionUnknown" means kubernetes | ||
| // can't decide if a resource is in the condition or not. In the future, we could add other | ||
| // intermediate conditions, e.g. ConditionDegraded. | ||
| const ( | ||
| ConditionTrue ConditionStatus = "True" | ||
| ConditionFalse ConditionStatus = "False" | ||
| ConditionUnknown ConditionStatus = "Unknown" | ||
| ) | ||
|
|
||
| // ThirdPartyResourceConditionType is a valid value for ThirdPartyResourceCondition.Type | ||
| type ThirdPartyResourceConditionType string | ||
|
|
||
| const ( | ||
| // NameConflict means the resource or kind names chosen for this ThirdPartyResource conflict with others in the group. | ||
| // The first TPR in the group to have the name reflected in status "wins" the name. | ||
| NameConflict ThirdPartyResourceConditionType = "NameConflict" | ||
| // Terminating means that the ThirdPartyResource has been deleted and is cleaning up. | ||
| Terminating ThirdPartyResourceConditionType = "Terminating" | ||
| ) | ||
|
|
||
| // ThirdPartyResourceCondition contains details for the current condition of this ThirdPartyResource. | ||
| type ThirdPartyResourceCondition struct { | ||
| // Type is the type of the condition. | ||
| Type ThirdPartyResourceConditionType `json:"type" protobuf:"bytes,1,opt,name=type,casttype=ThirdPartyResourceConditionType"` | ||
| // Status is the status of the condition. | ||
| // Can be True, False, Unknown. | ||
| Status ConditionStatus `json:"status" protobuf:"bytes,2,opt,name=status,casttype=ConditionStatus"` | ||
| // Last time the condition transitioned from one status to another. | ||
| // +optional | ||
| LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty" protobuf:"bytes,4,opt,name=lastTransitionTime"` | ||
| // Unique, one-word, CamelCase reason for the condition's last transition. | ||
| // +optional | ||
| Reason string `json:"reason,omitempty" protobuf:"bytes,5,opt,name=reason"` | ||
| // Human-readable message indicating details about last transition. | ||
| // +optional | ||
| Message string `json:"message,omitempty" protobuf:"bytes,6,opt,name=message"` | ||
| } | ||
|
|
||
| // ThirdPartyResourceStatus indicates the state of the ThirdPartyResource | ||
| type ThirdPartyResourceStatus struct { | ||
| // Conditions indicate state for particular aspects of a ThirdPartyResource | ||
| Conditions []ThirdPartyResourceCondition `json:"conditions" protobuf:"bytes,1,opt,name=conditions"` | ||
|
|
||
| // AcceptedNames are the names that are actually being used to serve discovery | ||
| // They may not be the same as names in spec. | ||
| AcceptedNames ThirdPartyResourceNames | ||
| } | ||
|
|
||
| // +genclient=true | ||
|
|
||
| // ThirdPartyResource represents a resource that should be exposed on the API server. Its name MUST be in the format | ||
| // <.spec.plural>.<.spec.group>. | ||
| type ThirdPartyResource struct { | ||
| metav1.TypeMeta `json:",inline"` | ||
| metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"` | ||
|
|
||
| // Spec describes how the user wants the resources to appear | ||
| Spec ThirdPartyResourceSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"` | ||
| // Status indicates the actual state of the ThirdPartyResource | ||
| Status ThirdPartyResourceStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"` | ||
| } | ||
|
|
||
| // ThirdPartyResourceList is a list of ThirdPartyResource objects. | ||
| type ThirdPartyResourceList struct { | ||
| metav1.TypeMeta `json:",inline"` | ||
| metav1.ListMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"` | ||
|
|
||
| // Items individual ThirdParties | ||
| Items []ThirdPartyResource `json:"items" protobuf:"bytes,2,rep,name=items"` | ||
| } | ||
| ``` | ||
|
|
||
|
|
||
| ## Behavior | ||
| ### Create | ||
| When a new TPR is created, no synchronous action is taken. | ||
| A controller will run to confirm that value-space of the reserved names doesn't | ||
| collide and sets the "KindNameConflict" condition to `false`. | ||
|
|
||
| A custom `http.Handler` will look at request and use the parsed out | ||
| GroupVersionResource information to match it to a ThirdPartyResource. The ThirdPartyResource | ||
| will be checked to make sure its valid enough in .Status to serve and will | ||
| response appropriated. If there is no ThirdPartyResource defined, it will delegate | ||
| to the next handler in the chain. | ||
|
|
||
| ### Delete | ||
| When a TPR-registration is deleted, it will be handled as a finalizer like a | ||
| namespace is done today. The `Terminating` condition will be updated (like | ||
| namespaces) and that will cause mutating requests to be rejected by the REST | ||
| handler (see above). The finalizer will remove all the associated storage. | ||
| Once the finalizer is done, it will delete the TPR-registration itself. | ||
|
|
||
|
|
||
| ## Migration from existing TPR | ||
| Because of the changes required to meet the goals, there is not a silent | ||
| auto-migration from the existing TPR to the new TPR. It will be possible, but | ||
| it will be manual. At a high level, you simply: | ||
| 1. Stop all clients from writing to TPR (revoke edit rights for all users) and | ||
| stop controllers. | ||
| 2. Get all your TPR-data. | ||
| `$ kubectl get TPR --all-namespaces -o yaml > data.yaml` | ||
| 3. Delete the old TPR-data. Be sure you orphan! | ||
| `$ kubectl delete TPR --all --all-namespaces --cascade=false` | ||
| 4. Delete the old TPR-registration. | ||
| `$ kubectl delete TPR/name` | ||
| 5. Create a new TPR-registration with the same GroupVersionKind as before. | ||
| `$ kubectl create -f new_tpr.name` | ||
| 6. Recreate your new TPR-data. | ||
| `$ kubectl create -f data.yaml` | ||
| 7. Restart controllers. | ||
|
|
||
| There are a couple things that you'll need to consider: | ||
| 1. Garbage collection. You may have created links that weren't respected by | ||
| the GC collector in 1.6. Since you orphaned your dependents, you'll probably | ||
| want to re-adopt them like the Kubernetes controllers do with their resources. | ||
| 2. Controllers will observe deletes. Part of this migration actually deletes | ||
| the resource. Your controller will see the delete. You ought to shut down | ||
| your TPR controller while you migrate your data. If you do this, your | ||
| controller will never see a delete. | ||
|
|