Document foreground/background deletion behavior more completely

[jbw976]

What's Missing?

@sttts was asking on Slack recently about the deletion of nested composite resources and understanding what is happening during that process, because it can be confusing.

@bobh66 explained this well in the below message about foreground and background deletion, which is copied below for convenience and posterity and with minor edits for readability:

The composite reconciler leaves deletion responsibilities to kubernetes - so when the default background deletion occurs and the k8s GC goes through the tree and marks everything deleted, the composite reconciler simply deletes the connection secret and removes it's finalizer which causes it to disappear. The MRs obviously have a lot more work to do on deletion and so they may stick around forever if they get stuck and cannot delete the remote resource, In the mean time the claim and all composites are long gone and the MRs are left orphaned.

We expose support for Foreground deletion with defaultCompositeDeletePolicy: Foreground, and implement it under the covers by setting the blockOwnerDeletion flag in the owner reference. We also updated the claim reconciler to delete the composite using the specified propagation policy so that we can now delete either a claim or a top-level composite using foreground deletion and kubernetes will do a "bottom up" deletion starting with the MRs. If an MR is stuck the entire chain of composites above it will stay in place until the MR is deleted. This makes finding stuck resources a lot easier. The helm and kubernetes providers act like the other providers in that the MR does not remove it's finalizers until the associated chart/object has been deleted.

We should add these details to the documentation in a discoverable place, so that users can make better decisions and be more informed about how deletion works in Crossplane and what their options are.

This is touched on in https://docs.crossplane.io/latest/concepts/composite-resource-definitions/#defaultcompositedeletepolicy, but it's not a complete explanation of the topic and it is hard to find that unless you already know what the field is called. Instead, there could be a general topic about XR deletion, perhaps added to the Usage documentation if not addressed as a topic on its own.