Update audit proposal for aggregated environment

contributors/design-proposals/auditing.md

@@ -155,7 +155,11 @@ const (
 
 The audit level is determined by the policy, which maps a combination of requesting user, namespace, verb, API group, and resource. The policy is described in detail [below](#policy).
 
-When the http request is processed, the request handler will close the audit event by filling in the http response. Then it will pass the event to the configured output backend.
+In an [aggregated](aggregated-api-servers.md) deployment, the `kube-aggregator` is able to fill in
+`Metadata` level audit events, but not above. For the higher audit levels, an audit event is
+generated _both_ in the `kube-aggregator` and in the end-user apiserver. The events can be
+de-duplicated in the audit backend based on the audit ID, which is generated from the `Audit-ID`
+header. The event generated by the end-user apiserver may not have the full authentication information.
 
 **Note:** for service creation and deletion there is special REST code in the apiserver which takes care of service/node port (de)allocation and removal of endpoints on service deletion. Hence, these operations are not visible on the API layer and cannot be audit logged therefore. **No other resources** (with the exception of componentstatus which is not of interest here) **implement this kind of custom CRUD operations.**
 
@@ -172,12 +176,13 @@ type Event struct {
 
     // below fields are filled at Metadata level and higher:
 
-    // Unique audit ID, generated for each request
+    // Unique ID of the request being audited, and able to de-dupe audit events.
+    // Set from the `Audit-ID` header.
     ID string
     // Time the event reached the apiserver
     Timestamp Timestamp
-    // Source IP, from where the request originates
-    SourceIP string
+    // Source IPs, from where the request originates, with intermediate proxy IPs.
+    SourceIPs []string
     // HTTP method sent by the client
     HttpMethod string
     // Verb is the kube verb associated with the request for API requests.
@@ -296,6 +301,14 @@ rules:
 The policy is checked immediately after authentication in the request handling, and determines how
 the `audit.Event` is formed.
 
+In an [aggregated](aggregated-api-servers.md) deployment, the `kube-aggregator` is responsible for
+checking the policy. The kube-aggregator writes the audit Level from the policy into a request
+header that is passed on to the end-user apiserver, e.g.
+
+```
+Audit-Level: Request
+```
+
 ### Filters
 
 In addition to the high-level policy rules, auditing can be controlled at a more fine-grained level
@@ -334,6 +347,31 @@ Following new flags should be introduced in the apiserver:
   which will log to `/var/log/apiserver-audit.log`, and additionally defines rotation arguments (analogically to the deprecated ones) and output format.
 * `--audit-policy` - which specifies a file with policy configuration, see [policy](#policy) for a sample file contents.
 
+### Audit Security
+
+Several parts of the audit system could be exposed to spoofing or tampering threats. This section
+lists the threats, and how we will mitigate them.
+
+**Audit ID.** The audit ID is set from the "front door" server, which could be a federation apiserver,
+a kube-aggregator, or end-user apiserver. Since the server can't currently know where in the serving
+chain it falls, it is possible for the client to set a `Audit-ID` header that is
+non-unique. For this reason, any aggregation that happens based on the audit ID must also sanity
+check the known fields (e.g. URL, source IP, time window, etc.). With this additional check, an
+attacker could generate a bit more noise in the logs, but no information would be lost.
+
+**Source IP.** Kubernetes requests may go through multiple hops before a response is generated
+(e.g. federation apiserver -> kube-aggregator -> end-user apiserver). Each hop must append the
+previous sender's IP address to the `X-Forwarded-For` header IP chain. If we simply audited the
+original sender's IP, an attacker could send there request with a bogus IP at the front of the
+`X-Forwarded-For` chain. To mitigate this, we will log the entire IP chain. This has the additional
+benefit of supporting external proxies.
+
+**Audit Policy.** The audit policy is intended to be set by the kube-aggregator and passed along in
+the `Audit-Level` header. However, we don't want an attacker to be able to simply set `Audit-Level:
+None` on their request. To prevent this, *if an apiserver has an audit policy configured, that
+policy overrides the header*. In practice, this means the kube-aggregator can be configured with an
+audit policy, and it will simply overwrite any audit level previously attached to the request.
+
 ## Sensible (not necessarily sequential) Milestones of Implementation
 
 1. Add `audit.Event` and `audit.OutputBackend` and implement [#27087](https://github.com/kubernetes/kubernetes/pull/27087)'s basic auditing using them, using a single global audit Level, up to `ResponseBody`.