Migration tests

This folder contains a set of deployments used to show the migration steps taken by the Helm chart from a SCK deployment.

You can see how this test executes as part of Github actions in .github/workflows/migration_tests.yaml

Local development

You can recreate the execution of the step on your machine with the following set up steps.

1. Check out the repository
2. Install kubectl, helm as per root README.
3. Set up a kind cluster with a few default utilities:

   export KUBECONFIG=/tmp/kube-config-splunk-otel-collector-chart-migration-testing
   export K8S_VERSION=v1.28.0
   kind create cluster --kubeconfig=$KUBECONFIG --config=.github/workflows/configs/kind-config.yaml --image=kindest/node:$K8S_VERSION --name=migration-tests
   kubectl get csr -o=jsonpath='{range.items[?(@.spec.signerName=="kubernetes.io/kubelet-serving")]}{.metadata.name}{" "}{end}' | xargs kubectl certificate approve
   make cert-manager
   

4. Deploy a collector as a service that can act as a log sink for HEC traffic, alongside with a deployment that outputs logs every second:

   kubectl apply -f migration_tests/collector_deployment.yaml
   

5. Deploy SCK

   helm repo add sck https://splunk.github.io/splunk-connect-for-kubernetes/
   helm install --wait sck -f migration_tests/sck_values.yaml sck/splunk-connect-for-kubernetes
   

6. Now that SCK is deployed, wait 10s to generate enough logs:

   sleep 10
   

7. Uninstall SCK:

   helm uninstall --wait sck
   

8. Collect logs collected so far:

   pod=$(kubectl get pods --no-headers -o custom-columns=":metadata.name" -lapp=collector-logsink)
   kubectl exec -it $pod -- cat /tmp/output.log | grep -Eo '"body":{"stringValue":"APP LOG LINE \d+' | awk '{print $4}' > sck_logs.log
   

9. Install SOCK:

   helm install --wait sock -f migration_tests/sock_values.yaml helm-charts/splunk-otel-collector
   

10. Wait additional 10s to produce more logs

    sleep 10
    

11. Check the collector deployment logs:

    pod=$(kubectl get pods --no-headers -o custom-columns=":metadata.name" -lapp=collector-logsink)
    kubectl exec $pod -- cat /tmp/output.log | grep -Eo '"body":{"stringValue":"APP LOG LINE [[:digit:]]+' | awk '{print $4}' > sock_logs.log
    

12. Check we have no duplicates:

    dupes=$(cat sock_logs.log | sort | uniq -d)
    if [[ -n $dupes ]]; then
      echo "Duplicates detected: $dupes"
    fi