Task: Stateful Set force delete pods (kubernetes#1731)

* Adding task for force-deletion of pods in a statefulset.

* Fixed based on doc review.

* Reapply docs review fix

* Fixing nav and tasks from earlier.

_data/tasks.yml

@@ -27,12 +27,16 @@ toc:
   section:
   - title: Using an HTTP Proxy to Access the Kubernetes API
     path: /docs/tasks/access-kubernetes-api/http-proxy-access-api/
+
 - title: Administering a Cluster
   section:
   - title: Assigning Pods to Nodes
     path: /docs/tasks/administer-cluster/assign-pods-nodes/
   - title: Autoscaling the DNS Service in a Cluster
     path: /docs/tasks/administer-cluster/dns-horizontal-autoscaling/
+  - title: Safely Draining a Node while Respecting Application SLOs
+    path: /docs/tasks/administer-cluster/safely-drain-node/
+
 - title: Managing Stateful Applications
   section:
   - title: Upgrading from PetSets to StatefulSets
@@ -43,5 +47,5 @@ toc:
     path: /docs/tasks/manage-stateful-set/deleting-a-statefulset/
   - title: Debugging a StatefulSet
     path: /docs/tasks/manage-stateful-set/debugging-a-statefulset/
-  - title: Safely Draining a Node while Respecting Application SLOs
-    path: /docs/tasks/administer-cluster/safely-drain-node/
+  - title: Force Deleting StatefulSet Pods
+    path: /docs/tasks/manage-stateful-set/delete-pods/

docs/tasks/index.md

@@ -27,15 +27,15 @@ single thing, typically by giving a short sequence of steps.
 
 * [Assigning Pods to Nodes](/docs/tasks/administer-cluster/assign-pods-nodes/)
 * [Autoscaling the DNS Service in a Cluster](/docs/tasks/administer-cluster/dns-horizontal-autoscaling/)
+* [Safely Draining a Node while Respecting Application SLOs](/docs/tasks/administer-cluster/safely-drain-node/)
 
 #### Managing Stateful Applications
 
 * [Upgrading from PetSets to StatefulSets](/docs/tasks/manage-stateful-set/upgrade-pet-set-to-stateful-set/)
 * [Scaling a StatefulSet](/docs/tasks/manage-stateful-set/scale-stateful-set/)
 * [Deleting a StatefulSet](/docs/tasks/manage-stateful-set/deleting-a-statefulset/)
 * [Debugging a StatefulSet](/docs/tasks/manage-stateful-set/debugging-a-statefulset/)
-* [Safely Draining a Node while Respecting Application SLOs](/docs/tasks/administer-cluster/safely-drain-node/)
-
+* [Force Deleting StatefulSet Pods](/docs/tasks/manage-stateful-set/delete-pods/)
 
 ### What's next
 

docs/tasks/manage-stateful-set/delete-pods.md

@@ -0,0 +1,75 @@
+---
+assignees:
+- bprashanth
+- erictune
+- foxish
+- smarterclayton
+
+---
+
+{% capture overview %}
+This page shows how to delete Pods which are part of a stateful set, and explains the considerations to keep in mind when doing so.
+{% endcapture %}
+
+{% capture prerequisites %}
+
+* This is a fairly advanced task and has the potential to violate some of the properties inherent to StatefulSet.
+* Before proceeding, make yourself familiar with the considerations enumerated below.
+
+{% endcapture %}
+
+{% capture steps %}
+
+
+### StatefulSet considerations
+
+In normal operation of a StatefulSet, there is **never** a need to force delete a StatefulSet Pod. The StatefulSet controller is responsible for creating, scaling and deleting members of the StatefulSet. It tries to ensure that the specified number of Pods from ordinal 0 through N-1 are alive and ready. StatefulSet ensures that, at any time, there is at most one Pod with a given identity running in a cluster. This is referred to as *at most one* semantics provided by a StatefulSet.
+
+Manual force deletion should be undertaken with caution, as it has the potential to violate the at most one semantics inherent to StatefulSet. StatefulSets may be used to run distributed and clustered applications which have a need for a stable network identity and stable storage. These applications often have configuration which relies on an ensemble of a fixed number of members with fixed identities. Having multiple members with the same identity can be disastrous and may lead to data loss (e.g. split brain scenario in quorum-based systems). 
+
+### Deleting Pods
+
+You can perform a graceful pod deletion with the following command:
+
+```shell
+kubectl delete pods <pod>
+```
+
+For the above to lead to graceful termination, the Pod **must not** specify a `pod.Spec.TerminationGracePeriodSeconds` of 0. The practice of setting a `pod.Spec.TerminationGracePeriodSeconds` of 0 seconds is unsafe and strongly discouraged for StatefulSet Pods. Graceful deletion is safe and will ensure that the [Pod shuts down gracefully](/docs/user-guide/pods/#termination-of-pods) before the kubelet deletes the name from the apiserver. 
+
+Kubernetes (versions 1.5 or newer) will not delete Pods just because a Node is unreachable. The Pods running on an unreachable Node enter the 'Terminating' or 'Unknown' state after a [timeout](/docs/admin/node/#node-condition). Pods may also enter these states when the user attempts graceful deletion of a Pod on an unreachable Node. The only ways in which a Pod in such a state can be removed from the apiserver are as follows:
+   * The Node object is deleted (either by you, or by the [Node Controller](/docs/admin/node)).
+   * The kubelet on the unresponsive Node starts responding, kills the Pod and removes the entry from the apiserver. 
+   * Force deletion of the Pod by the user.
+
+The recommended best practice is to use the first or second approach. If a Node is confirmed to be dead (e.g. permanently disconnected from the network, powered down, etc), then delete the node object. If the node is suffering from a network partition, then try to resolve this or wait for it to resolve. When the partition heals, the kubelet will complete the deletion of the Pod and free up its name in the apiserver. 
+
+Normally, the system completes the deletion once the Pod is no longer running on a Node, or the Node is deleted by an administrator. You may override this by force deleting the Pod.
+
+#### Force Deletion
+
+Force deletions **do not** wait for confirmation from the kubelet that the Pod has been terminated. Irrespective of whether a force deletion is successful in killing a Pod, it will immediately free up the name from the apiserver. This would let the StatefulSet controller create a replacement Pod with that same identity; this can lead to the duplication of a still-running Pod, and if said Pod can still communicate with the other members of the StatefulSet, will violate the at most one semantics that StatefulSet is designed to guarantee.
+
+When you force delete a StatefulSet pod, you are asserting that the Pod in question will never again make contact with other Pods in the StatefulSet and its name can be safely freed up for a replacement to be created. 
+
+If you want to delete a Pod forcibly using kubectl version >= 1.5, do the following:
+
+```shell
+kubectl delete pods <pod> --grace-period=0 --force
+```
+
+If you're using any version of kubectl <= 1.4, you should omit the `--force` option and use:
+
+```shell
+kubectl delete pods <pod> --grace-period=0
+```
+
+Always perform force deletion of StatefulSet Pods carefully and with complete knowledge of the risks involved.
+
+{% endcapture %}
+
+{% capture whatsnext %}
+Learn more about debugging a StatefulSet. *TODO: Link to the task for debugging a StatefulSet*
+{% endcapture %}
+
+{% include templates/task.md %}