Skip to content

Commit

Permalink
Re-format controllers doc
Browse files Browse the repository at this point in the history
  • Loading branch information
Jamie Hannaford committed Apr 11, 2017
1 parent 0f7cc84 commit a217f64
Showing 1 changed file with 51 additions and 84 deletions.
135 changes: 51 additions & 84 deletions contributors/devel/controllers.md
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
# Writing Controllers

A Kubernetes controller is an active reconciliation process. That is, it watches some object for the world's desired
state, and it watches the world's actual state, too. Then, it sends instructions to try and make the world's current
state be more like the desired state.
A Kubernetes controller is an active reconciliation process. That is, it watches some object for the world's desired state, and it watches the world's actual state, too. Then, it sends instructions to try and make the world's current state be more like the desired state.

The simplest implementation of this is a loop:

Expand All @@ -18,88 +16,47 @@ Watches, etc, are all merely optimizations of this logic.

## Guidelines

When you're writing controllers, there are few guidelines that will help make sure you get the results and performance
you're looking for.
When you're writing controllers, there are few guidelines that will help make sure you get the results and performance you're looking for.

1. Operate on one item at a time. If you use a `workqueue.Interface`, you'll be able to queue changes for a
particular resource and later pop them in multiple “worker” gofuncs with a guarantee that no two gofuncs will
work on the same item at the same time.
1. Operate on one item at a time. If you use a `workqueue.Interface`, you'll be able to queue changes for a particular resource and later pop them in multiple “worker” gofuncs with a guarantee that no two gofuncs will work on the same item at the same time.

Many controllers must trigger off multiple resources (I need to "check X if Y changes"), but nearly all controllers
can collapse those into a queue of “check this X” based on relationships. For instance, a ReplicaSetController needs
to react to a pod being deleted, but it does that by finding the related ReplicaSets and queuing those.
Many controllers must trigger off multiple resources (I need to "check X if Y changes"), but nearly all controllers can collapse those into a queue of “check this X” based on relationships. For instance, a ReplicaSetController needs to react to a pod being deleted, but it does that by finding the related ReplicaSets and queuing those.

1. Random ordering between resources. When controllers queue off multiple types of resources, there is no guarantee of ordering amongst those resources.

1. Random ordering between resources. When controllers queue off multiple types of resources, there is no guarantee
of ordering amongst those resources.
Distinct watches are updated independently. Even with an objective ordering of “created resourceA/X” and “created resourceB/Y”, your controller could observe “created resourceB/Y” and “created resourceA/X”.

Distinct watches are updated independently. Even with an objective ordering of “created resourceA/X” and “created
resourceB/Y”, your controller could observe “created resourceB/Y” and “created resourceA/X”.
1. Level driven, not edge driven. Just like having a shell script that isn't running all the time, your controller may be off for an indeterminate amount of time before running again.

If an API object appears with a marker value of `true`, you can't count on having seen it turn from `false` to `true`, only that you now observe it being `true`. Even an API watch suffers from this problem, so be sure that you're not counting on seeing a change unless your controller is also marking the information it last made the decision on in the object's status.

1. Level driven, not edge driven. Just like having a shell script that isn't running all the time, your controller
may be off for an indeterminate amount of time before running again.
1. Use `SharedInformers`. `SharedInformers` provide hooks to receive notifications of adds, updates, and deletes for a particular resource. They also provide convenience functions for accessing shared caches and determining when a cache is primed.

If an API object appears with a marker value of `true`, you can't count on having seen it turn from `false` to `true`,
only that you now observe it being `true`. Even an API watch suffers from this problem, so be sure that you're not
counting on seeing a change unless your controller is also marking the information it last made the decision on in
the object's status.
Use the factory methods down in https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/informers/factory.go to ensure that you are sharing the same instance of the cache as everyone else.

This saves us connections against the API server, duplicate serialization costs server-side, duplicate deserialization costs controller-side, and duplicate caching costs controller-side.

1. Use `SharedInformers`. `SharedInformers` provide hooks to receive notifications of adds, updates, and deletes for
a particular resource. They also provide convenience functions for accessing shared caches and determining when a
cache is primed.
You may see other mechanisms like reflectors and deltafifos driving controllers. Those were older mechanisms that we later used to build the `SharedInformers`. You should avoid using them in new controllers

Use the factory methods down in https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/informers/factory.go
to ensure that you are sharing the same instance of the cache as everyone else.
1. Never mutate original objects! Caches are shared across controllers, this means that if you mutate your "copy" (actually a reference or shallow copy) of an object, you'll mess up other controllers (not just your own).

This saves us connections against the API server, duplicate serialization costs server-side, duplicate deserialization
costs controller-side, and duplicate caching costs controller-side.
The most common point of failure is making a shallow copy, then mutating a map, like `Annotations`. Use `api.Scheme.Copy` to make a deep copy.

You may see other mechanisms like reflectors and deltafifos driving controllers. Those were older mechanisms that we
later used to build the `SharedInformers`. You should avoid using them in new controllers
1. Wait for your secondary caches. Many controllers have primary and secondary resources. Primary resources are the resources that you'll be updating `Status` for. Secondary resources are resources that you'll be managing (creating/deleting) or using for lookups.

Use the `framework.WaitForCacheSync` function to wait for your secondary caches before starting your primary sync functions. This will make sure that things like a Pod count for a ReplicaSet isn't working off of known out of date information that results in thrashing.

1. Never mutate original objects! Caches are shared across controllers, this means that if you mutate your "copy"
(actually a reference or shallow copy) of an object, you'll mess up other controllers (not just your own).
1. There are other actors in the system. Just because you haven't changed an object doesn't mean that somebody else hasn't.

The most common point of failure is making a shallow copy, then mutating a map, like `Annotations`. Use
`api.Scheme.Copy` to make a deep copy.
Don't forget that the current state may change at any moment--it's not sufficient to just watch the desired state. If you use the absence of objects in the desired state to indicate that things in the current state should be deleted, make sure you don't have a bug in your observation code (e.g., act before your cache has filled).

1. Percolate errors to the top level for consistent re-queuing. We have a `workqueue.RateLimitingInterface` to allow simple requeuing with reasonable backoffs.

1. Wait for your secondary caches. Many controllers have primary and secondary resources. Primary resources are the
resources that you'll be updating `Status` for. Secondary resources are resources that you'll be managing
(creating/deleting) or using for lookups.
Your main controller func should return an error when requeuing is necessary. When it isn't, it should use `utilruntime.HandleError` and return nil instead. This makes it very easy for reviewers to inspect error handling cases and to be confident that your controller doesn't accidentally lose things it should retry for.

Use the `framework.WaitForCacheSync` function to wait for your secondary caches before starting your primary sync
functions. This will make sure that things like a Pod count for a ReplicaSet isn't working off of known out of date
information that results in thrashing.
1. Watches and Informers will “sync”. Periodically, they will deliver every matching object in the cluster to your `Update` method. This is good for cases where you may need to take additional action on the object, but sometimes you know there won't be more work to do.


1. There are other actors in the system. Just because you haven't changed an object doesn't mean that somebody else
hasn't.

Don't forget that the current state may change at any moment--it's not sufficient to just watch the desired state.
If you use the absence of objects in the desired state to indicate that things in the current state should be deleted,
make sure you don't have a bug in your observation code (e.g., act before your cache has filled).


1. Percolate errors to the top level for consistent re-queuing. We have a `workqueue.RateLimitingInterface` to allow
simple requeuing with reasonable backoffs.

Your main controller func should return an error when requeuing is necessary. When it isn't, it should use
`utilruntime.HandleError` and return nil instead. This makes it very easy for reviewers to inspect error handling
cases and to be confident that your controller doesn't accidentally lose things it should retry for.


1. Watches and Informers will “sync”. Periodically, they will deliver every matching object in the cluster to your
`Update` method. This is good for cases where you may need to take additional action on the object, but sometimes you
know there won't be more work to do.

In cases where you are *certain* that you don't need to requeue items when there are no new changes, you can compare the
resource version of the old and new objects. If they are the same, you skip requeuing the work. Be careful when you
do this. If you ever skip requeuing your item on failures, you could fail, not requeue, and then never retry that
item again.
In cases where you are *certain* that you don't need to requeue items when there are no new changes, you can compare the resource version of the old and new objects. If they are the same, you skip requeuing the work. Be careful when you do this. If you ever skip requeuing your item on failures, you could fail, not requeue, and then never retry that item again.


## Rough Structure
Expand All @@ -109,10 +66,11 @@ Overall, your controller should look something like this:
```go
type Controller struct{
// podLister is secondary cache of pods which is used for object lookups
podLister cache.StoreToPodLister
podLister cache.StoreToPodLister

// queue is where incoming work is placed to de-dup and to allow "easy" rate limited requeues on errors
queue workqueue.RateLimitingInterface
// queue is where incoming work is placed to de-dup and to allow "easy"
// rate limited requeues on errors
queue workqueue.RateLimitingInterface
}

func (c *Controller) Run(threadiness int, stopCh chan struct{}){
Expand All @@ -128,10 +86,11 @@ func (c *Controller) Run(threadiness int, stopCh chan struct{}){
return
}

// start up your worker threads based on threadiness. Some controllers have multiple kinds of workers
// start up your worker threads based on threadiness. Some controllers
// have multiple kinds of workers
for i := 0; i < threadiness; i++ {
// runWorker will loop until "something bad" happens. The .Until will then rekick the worker
// after one second
// runWorker will loop until "something bad" happens. The .Until will
// then rekick the worker after one second
go wait.Until(c.runWorker, time.Second, stopCh)
}

Expand All @@ -141,38 +100,46 @@ func (c *Controller) Run(threadiness int, stopCh chan struct{}){
}

func (c *Controller) runWorker() {
// hot loop until we're told to stop. processNextWorkItem will automatically wait until there's work
// available, so we don't worry about secondary waits
// hot loop until we're told to stop. processNextWorkItem will
// automatically wait until there's work available, so we don't worry
// about secondary waits
for c.processNextWorkItem() {
}
}

// processNextWorkItem deals with one key off the queue. It returns false when it's time to quit.
// processNextWorkItem deals with one key off the queue. It returns false
// when it's time to quit.
func (c *Controller) processNextWorkItem() bool {
// pull the next work item from queue. It should be a key we use to lookup something in a cache
// pull the next work item from queue. It should be a key we use to lookup
// something in a cache
key, quit := c.queue.Get()
if quit {
return false
}
// you always have to indicate to the queue that you've completed a piece of work
// you always have to indicate to the queue that you've completed a piece of
// work
defer c.queue.Done(key)

// do your work on the key. This method will contains your "do stuff" logic
err := c.syncHandler(key.(string))
if err == nil {
// if you had no error, tell the queue to stop tracking history for your key. This will
// reset things like failure counts for per-item rate limiting
// if you had no error, tell the queue to stop tracking history for your
// key. This will reset things like failure counts for per-item rate
// limiting
c.queue.Forget(key)
return true
}

// there was a failure so be sure to report it. This method allows for pluggable error handling
// which can be used for things like cluster-monitoring
// there was a failure so be sure to report it. This method allows for
// pluggable error handling which can be used for things like
// cluster-monitoring
utilruntime.HandleError(fmt.Errorf("%v failed with : %v", key, err))
// since we failed, we should requeue the item to work on later. This method will add a backoff
// to avoid hotlooping on particular items (they're probably still not going to work right away)
// and overall controller protection (everything I've done is broken, this controller needs to
// calm down or it can starve other useful work) cases.

// since we failed, we should requeue the item to work on later. This
// method will add a backoff to avoid hotlooping on particular items
// (they're probably still not going to work right away) and overall
// controller protection (everything I've done is broken, this controller
// needs to calm down or it can starve other useful work) cases.
c.queue.AddRateLimited(key)

return true
Expand Down

0 comments on commit a217f64

Please sign in to comment.