Breakdown the Instantiation Throughput Costs (#41)

README.md

@@ -103,6 +103,7 @@ For further documentation, you may want to check these other documents:
 * [K8s](./docs/k8s.md) - documentation about configuring a single-node Kubernetes cluster.
 * [Kata](./docs/kata.md) - instructions to build our custom Kata fork and `initrd` images.
 * [Knative](./docs/knative.md) - documentation about Knative, our serverless runtime of choice.
+* [Local Registry](./docs/registry.md) - configuring a local registry to store OCI images.
 * [OVMF](./docs/ovmf.md) - notes on building OVMF and CoCo's OVMF boot process.
 * [SEV](./docs/sev.md) - speicifc documentation to get the project working with AMD SEV machines.
 * [Troubleshooting](./docs/troubleshooting.md) - tips to debug when things go sideways.

conf-files/knative_controller_custom_certs.yaml.j2

@@ -0,0 +1,17 @@
+spec:
+  template:
+    spec:
+      containers:
+        - name: controller
+          volumeMounts:
+            - name: custom-certs
+              mountPath: {{ path_to_certs }}
+          env:
+            - name: SSL_CERT_DIR
+              value: {{ path_to_certs }}
+      dnsPolicy: ClusterFirstWithHostNet
+      hostNetwork: true
+      volumes:
+        - name: custom-certs
+          secret:
+            secretName: {{ secret_name }}

docs/registry.md

@@ -0,0 +1,34 @@
+# Using a Local Registry
+
+In order to use a local image registry we need to configure `containerd`,
+`Kata`, and `containerd` to like our home-baked registry. In addition, Kata does
+not seem to be able to use HTTP registries inside the guest, so we need to go an extra
+step and configure HTTPS certificates for our registry too.
+
+To this extent, we first create a self-signed certificate, and give it the
+ALT name of our home-made registry. We must also include an entry in our DNS
+records to match our local (reachable from within the guest) IP, to this
+registry name.
+
+Second, we need to update the docker config to include our certificates for
+this registry, as well as containerd's.
+
+Third, we need to include both the updated `/etc/hosts` file with the DNS
+entries, as well as the certificate, inside the agent's `initrd`.
+
+Finally, we need to configure Knative to accept self-signed certificates. To
+do so, we need to update the `controller` deployment by applying a [patch](
+./conf-files/knative_controller_custom_certs.yaml.j2).
+
+All this process is automated when we start the local registry with the provided
+task:
+
+```bash
+inv registry.start
+```
+
+and is undone when we stop it:
+
+```bash
+inv registry.stop
+```

eval/README.md

@@ -33,12 +33,9 @@ Signing and encryption is an interactive process, hence why we do it once,
 in advance of the evaluation:
 
 ```bash
-# First encrypt (and sign) the image
-inv skopeo.encrypt-container-image "ghcr.io/csegarragonz/coco-helloworld-py:unencrypted" --sign
-
-# Then sign the unencrypted images used
-inv cosign.sign-container-image "ghcr.io/csegarragonz/coco-helloworld-py:unencrypted"
-inv cosign.sign-container-image "ghcr.io/csegarragonz/coco-knative-sidecar:unencrypted"
+# Enter an empty passphrase or click 'y' when prompted (it will happen
+# many times)
+inv eval.images.upload
 ```
 
 Now you are ready to run one of the experiments:
@@ -47,6 +44,7 @@ Now you are ready to run one of the experiments:
 * [Memory Size](#memory-size) - impact on initial VM memory size on start-up time.
 * [VM Start-Up](#vm-start-up) - breakdown of the cVM start-up costs
 * [Image Pull Costs](#image-pull) - breakdown of the costs associated to pulling an image on the guest.
+* [Throughput Detail](#throughput-detail) - breakdown of the costs associated to starting many services concurrently.
 
 ### Start-Up Costs
 
@@ -190,6 +188,12 @@ run:
 inv kata.replace-agent
 ```
 
+In addition, we want to configure the right debug logging settings:
+
+```bash
+inv kata.set-log-level debug containerd.set-log-level debug ovmf.set-log-level info
+```
+
 After that, you may run the experiment with:
 
 ```bash
@@ -215,6 +219,45 @@ You can see the plot below:
 
 ![plot](./plots/image-pull/image_pull.png)
 
+### Throughput Detail
+
+In this experiment, we pick one of the baselines in the [instantiation throughput](
+#instantiation-throughput) experiment, and try to analyze why the start-up
+latency increases linearly with the number of concurrent requests.
+
+To do so, we pick one of the data points in the aforementioned plot. In
+particular, we pick the most secure baseline (`coco-fw-sig-enc`), and the
+highest concurrency level (`16`), and record the timestamps of the basic VM
+creation events (as reported in the [start-up costs](#start-up-costs) plot).
+
+Given the amount of concurrent services, we want to use a more succinct
+logging configuration:
+
+```bash
+inv kata.set-log-level info containerd.set-log-level info
+```
+
+> Note that, given the volume of services we spin up, getting the logs from
+> `containerd` is unreliable, as `journalctl` will drop lines. Thus, for this
+> experiment we use the slightly less precise Kubernetes event's timestamps.
+
+To run the experiment you may run:
+
+```bash
+inv eval.xput-detail.run
+```
+
+and you may plot the results using:
+
+```bash
+inv eval.xput-detail.plot
+```
+
+which generates a plot in [`./plots/xput-detail/xput_detail.png`](
+./plots/xput-detail/xput_detail.png). You can also see the plot below:
+
+![plot](./plots/xput-detail/xput_detail.png)
+
 ## Benchmarks
 
 TODO

eval/apps/xput-detail/service.yaml.j2

@@ -0,0 +1,26 @@
+apiVersion: serving.knative.dev/v1
+kind: Service
+metadata:
+  name: helloworld-knative-{{ service_num }}
+  annotations:
+    "features.knative.dev/podspec-runtimeclassname": "enabled"
+spec:
+  template:
+    metadata:
+      labels:
+        apps.coco-serverless/name: helloworld-py
+        io.katacontainers.config.pre_attestation.enabled: "false"
+    spec:
+      {% if runtime_class is defined %}
+      runtimeClassName: {{ runtime_class }}
+      # coco-knative: need to run user container as root
+      securityContext:
+        runAsUser: 1000
+      {% endif %}
+      containers:
+        - image: {{ image_repo }}/{{ image_name }}:{{ image_tag }}
+          ports:
+            - containerPort: 8080
+          env:
+            - name: TARGET
+              value: "World"

eval/results/image-pull/coco-fw-sig-enc_cold.csv

@@ -1,25 +1,25 @@
 Run,ImageName,Event,TimeStampMs
-0,sidecar,StartGCImagePull,1698704327.368399
-0,sidecar,StartPullManifest,1698704327.376768
-0,sidecar,EndPullManifest,1698704327.990064
-0,sidecar,StartSignatureValidation,1698704327.990064
-0,sidecar,EndSignatureValidation,1698704329.296337
-0,sidecar,StartPullLayers,1698704329.296337
-0,sidecar,StartPullSingleLayer,1698704329.296337
-0,sidecar,EndPullSingleLayer,1698704330.0468361
-0,sidecar,StartHandleSingleLayer,1698704330.0468361
-0,sidecar,EndPullLayers,1698704330.819208
-0,sidecar,EndHandleSingleLayer,1698704330.819208
-0,sidecar,EndGCImagePull,1698704330.823149
-0,app,StartGCImagePull,1698704320.396366
-0,app,StartPullManifest,1698704320.411405
-0,app,EndPullManifest,1698704321.025102
-0,app,StartSignatureValidation,1698704321.025102
-0,app,EndSignatureValidation,1698704322.248573
-0,app,StartPullLayers,1698704322.248573
-0,app,StartPullSingleLayer,1698704322.248573
-0,app,EndPullSingleLayer,1698704322.8777323
-0,app,StartHandleSingleLayer,1698704322.8777323
-0,app,EndPullLayers,1698704327.303738
-0,app,EndHandleSingleLayer,1698704327.303738
-0,app,EndGCImagePull,1698704327.307644
+0,sidecar,StartGCImagePull,1698763808.3409
+0,sidecar,StartPullManifest,1698763808.354979
+0,sidecar,EndPullManifest,1698763809.100515
+0,sidecar,StartSignatureValidation,1698763809.100515
+0,sidecar,EndSignatureValidation,1698763810.754743
+0,sidecar,StartPullLayers,1698763810.754743
+0,sidecar,StartPullSingleLayer,1698763810.754743
+0,sidecar,EndPullSingleLayer,1698763812.3743262
+0,sidecar,StartHandleSingleLayer,1698763812.3743262
+0,sidecar,EndPullLayers,1698763812.793883
+0,sidecar,EndHandleSingleLayer,1698763812.793883
+0,sidecar,EndGCImagePull,1698763812.797953
+0,app,StartGCImagePull,1698763800.572303
+0,app,StartPullManifest,1698763800.590206
+0,app,EndPullManifest,1698763801.359332
+0,app,StartSignatureValidation,1698763801.359332
+0,app,EndSignatureValidation,1698763802.833511
+0,app,StartPullLayers,1698763802.833511
+0,app,StartPullSingleLayer,1698763802.833511
+0,app,EndPullSingleLayer,1698763804.1814477
+0,app,StartHandleSingleLayer,1698763804.1814477
+0,app,EndPullLayers,1698763808.241138
+0,app,EndHandleSingleLayer,1698763808.241138
+0,app,EndGCImagePull,1698763808.244155

eval/results/xput-detail/coco-fw-sig-enc_1.csv

@@ -0,0 +1,6 @@
+ServiceId,Event,TimeStampSecs
+0,Initialized,1698784625.0
+0,PodScheduled,1698784625.0
+0,SandboxReady,1698784631.003848
+0,Ready,1698784642.0
+0,ContainersReady,1698784642.0

eval/results/xput-detail/coco-fw-sig-enc_16.csv

@@ -0,0 +1,81 @@
+ServiceId,Event,TimeStampSecs
+0,Initialized,1698932745.0
+0,PodScheduled,1698932745.0
+0,SandboxReady,1698932754.265692
+0,Ready,1698932868.0
+0,ContainersReady,1698932868.0
+8,Initialized,1698932745.0
+8,PodScheduled,1698932745.0
+8,SandboxReady,1698932757.634508
+8,Ready,1698932872.0
+8,ContainersReady,1698932872.0
+9,Initialized,1698932746.0
+9,PodScheduled,1698932746.0
+9,SandboxReady,1698932757.652055
+9,Ready,1698932876.0
+9,ContainersReady,1698932876.0
+11,Initialized,1698932746.0
+11,PodScheduled,1698932746.0
+11,SandboxReady,1698932757.703852
+11,Ready,1698932880.0
+11,ContainersReady,1698932880.0
+10,Initialized,1698932746.0
+10,PodScheduled,1698932746.0
+10,SandboxReady,1698932757.744147
+10,Ready,1698932883.0
+10,ContainersReady,1698932883.0
+1,Initialized,1698932745.0
+1,PodScheduled,1698932745.0
+1,SandboxReady,1698932757.834407
+1,Ready,1698932888.0
+1,ContainersReady,1698932888.0
+2,Initialized,1698932748.0
+2,PodScheduled,1698932748.0
+2,SandboxReady,1698932760.166568
+2,Ready,1698932891.0
+2,ContainersReady,1698932891.0
+13,Initialized,1698932747.0
+13,PodScheduled,1698932747.0
+13,SandboxReady,1698932760.195484
+13,Ready,1698932894.0
+13,ContainersReady,1698932894.0
+12,Initialized,1698932747.0
+12,PodScheduled,1698932747.0
+12,SandboxReady,1698932760.214206
+12,Ready,1698932899.0
+12,ContainersReady,1698932899.0
+14,Initialized,1698932747.0
+14,PodScheduled,1698932747.0
+14,SandboxReady,1698932760.228735
+14,Ready,1698932902.0
+14,ContainersReady,1698932902.0
+15,Initialized,1698932748.0
+15,PodScheduled,1698932748.0
+15,SandboxReady,1698932760.229141
+15,Ready,1698932906.0
+15,ContainersReady,1698932906.0
+3,Initialized,1698932749.0
+3,PodScheduled,1698932749.0
+3,SandboxReady,1698932760.366882
+3,Ready,1698932909.0
+3,ContainersReady,1698932909.0
+5,Initialized,1698932750.0
+5,PodScheduled,1698932750.0
+5,SandboxReady,1698932760.57967
+5,Ready,1698932912.0
+5,ContainersReady,1698932912.0
+4,Initialized,1698932749.0
+4,PodScheduled,1698932749.0
+4,SandboxReady,1698932760.580918
+4,Ready,1698932915.0
+4,ContainersReady,1698932915.0
+6,Initialized,1698932750.0
+6,PodScheduled,1698932750.0
+6,SandboxReady,1698932760.605338
+6,Ready,1698932919.0
+6,ContainersReady,1698932919.0
+7,Initialized,1698932750.0
+7,PodScheduled,1698932750.0
+7,SandboxReady,1698932760.695114
+7,Ready,1698932923.0
+7,ContainersReady,1698932923.0

eval/results/xput-detail/coco-fw-sig-enc_2.csv

@@ -0,0 +1 @@
+Run,ServiceId,Event,TimeStampSecs

eval/results/xput-detail/coco-fw-sig-enc_4.csv

@@ -0,0 +1,21 @@
+ServiceId,Event,TimeStampSecs
+0,Initialized,1698785143.0
+0,PodScheduled,1698785143.0
+0,SandboxReady,1698785151.358817
+0,Ready,1698785182.0
+0,ContainersReady,1698785182.0
+2,Initialized,1698785144.0
+2,PodScheduled,1698785144.0
+2,SandboxReady,1698785151.607405
+2,Ready,1698785185.0
+2,ContainersReady,1698785185.0
+1,Initialized,1698785143.0
+1,PodScheduled,1698785143.0
+1,SandboxReady,1698785151.639349
+1,Ready,1698785189.0
+1,ContainersReady,1698785189.0
+3,Initialized,1698785144.0
+3,PodScheduled,1698785144.0
+3,SandboxReady,1698785151.931775
+3,Ready,1698785192.0
+3,ContainersReady,1698785192.0

eval/results/xput-detail/coco-fw-sig-enc_8.csv

@@ -0,0 +1,41 @@
+Run,ServiceId,Event,TimeStampSecs
+1,Initialized,1698774180.0
+1,PodScheduled,1698774180.0
+1,SandboxReady,1698774187.615061
+1,Ready,1698774247.0
+1,ContainersReady,1698774247.0
+0,Initialized,1698774180.0
+0,PodScheduled,1698774180.0
+0,SandboxReady,1698774187.648987
+0,Ready,1698774251.0
+0,ContainersReady,1698774251.0
+2,Initialized,1698774180.0
+2,PodScheduled,1698774180.0
+2,SandboxReady,1698774189.995768
+2,Ready,1698774255.0
+2,ContainersReady,1698774255.0
+4,Initialized,1698774181.0
+4,PodScheduled,1698774181.0
+4,SandboxReady,1698774190.651021
+4,Ready,1698774259.0
+4,ContainersReady,1698774259.0
+3,Initialized,1698774181.0
+3,PodScheduled,1698774181.0
+3,SandboxReady,1698774190.669562
+3,Ready,1698774263.0
+3,ContainersReady,1698774263.0
+5,Initialized,1698774182.0
+5,PodScheduled,1698774182.0
+5,SandboxReady,1698774190.725512
+5,Ready,1698774267.0
+5,ContainersReady,1698774267.0
+6,Initialized,1698774182.0
+6,PodScheduled,1698774182.0
+6,SandboxReady,1698774190.931352
+6,Ready,1698774271.0
+6,ContainersReady,1698774271.0
+7,Initialized,1698774182.0
+7,PodScheduled,1698774182.0
+7,SandboxReady,1698774191.112247
+7,Ready,1698774275.0
+7,ContainersReady,1698774275.0