[🚀 Feature]: Automatic creation of new jobs with browsers

[Doofus100500]

Feature and motivation

In my installation, I added another component for the automatic generation of jobs with browsers. And since you’ve already started building multiple browser images versions with the new Selenium, I’d like to share my implementation. It would be awesome if we could add similar logic to Selenium Grid.

#!/bin/bash
PVC_NAME=$(kubectl get pvc -n $SE_NAMESPACE -o jsonpath='{.items[0].metadata.name}')

function deployment_exists(){
  kubectl get scaledjob selenium-grid-selenium-${1,,}-node-v${2%.*} --namespace $SE_NAMESPACE > /dev/null 2>&1
}

while true
do
  readarray requests <<< "$(curl -X POST -H "Content-Type: application/json" --data '{"query":"{ sessionsInfo { sessionQueueRequests } }"}' -sfk $SE_NODE_GRID_GRAPHQL_URL | jq .data.sessionsInfo.sessionQueueRequests)"
  unset "requests[0]"
  if [[ ${#requests[*]} -gt 0 ]]; then
    unset "requests[-1]"
  fi
  for i in "${!requests[@]}"
    do
      name=$(echo -e ${requests[$i]} | grep browserName | awk -F'[:,]' '{for(i=1;i<=NF;i++){if($i~/browserName/){print $(i+1)}}}' | tr -d '[:space:]' | tr -d '\\"')
      version=$(echo -e ${requests[$i]} | grep browserVersion | awk -F'[:,]' '{for(i=1;i<=NF;i++){if($i~/browserVersion/){print $(i+1)}}}' | tr -d '[:space:]' | tr -d '\\"')
      platform=$(echo -e ${requests[$i]} | grep platformName | awk -F'[:,]' '{for(i=1;i<=NF;i++){if($i~/platformName/){print $(i+1)}}}' | tr -d '[:space:]' | tr -d '\\"')
      name=${name,,}
      platform=${platform,,}
      if ! [[ -z "$name" ]] && ! [[ -z "$version" ]] && [[ "$platform" == "linux" ]]; then
        pattern="\b[1-9][0-9][0-9]\.[0]\b"
        if [[ $version =~ $pattern ]]; then
          case $name in
            microsoftedge)
              if (( $(echo "$version < 120.0" |bc -l) )); then
                echo "This version of $name is no longer supported by our Grid. Version=$version"
              else
                deployments[$i]=edge:$version
              fi
              ;;
            chrome)
              if (( $(echo "$version < 120.0" |bc -l) )); then
                echo "This version of $name is no longer supported by our Grid. Version=$version"
              else
                deployments[$i]=chrome:$version
              fi
              ;;
            firefox)
              if (( $(echo "$version < 122.0" |bc -l) )); then
                echo "This version of $name is no longer supported by our Grid. Version=$version"
              else
                deployments[$i]=firefox:$version
              fi
              ;;
          esac
        else
          echo "Version of $name is incorrect, version=$version"
        fi
      fi
    done
  uniqs_deployments=($(for d in "${deployments[@]}"; do echo "${d}"; done | sort -u))
  for i in "${!uniqs_deployments[@]}"
    do
      name=$(echo ${uniqs_deployments[$i]} | cut -d ':' -f 1 )
      version=$(echo ${uniqs_deployments[$i]} | cut -d ':' -f 2 )
      if [ $name == "edge" ]; then kedabrowsername=MicrosoftEdge; sessionBrowserName=msedge; else kedabrowsername=$name; sessionBrowserName=$name; fi
      if [ $name == "firefox" ]; then disable_dshm=""; else disable_dshm="--disable-dev-shm-usage"; fi
      if [ $SE_NAMESPACE == "selenium-test" ]; then testTag="-test"; else testTag=""; fi
      if ! deployment_exists $name $version; then
        tags=$(curl -s -X GET https://registry.host/v2/$name/tags/list | jq -r '.tags[]')
        filtered_tags=$(echo "$tags" | grep "^$version")
        if [ "$filtered_tags" ]; then
          if [ -n "$testTag" ]; then
            filtered_tags=$(echo "$filtered_tags" | grep "\-test")
          else
            filtered_tags=$(echo "$filtered_tags" | grep -v "\-test")
          fi
          latest_tag=$(echo "$filtered_tags" | sort -V | tail -n 1)
          kubectl apply -f - <<EOF
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  annotations:
    helm.sh/hook: post-install,post-upgrade,post-rollback
    helm.sh/hook-weight: "1"
    autoscaling.keda.sh/paused: "false"
  finalizers:
  - finalizer.keda.sh
  generation: 2
  labels:
    app: selenium-grid-selenium-$name-node-v${version%.*}
    app.kubernetes.io/instance: selenium-grid
    app.kubernetes.io/name: selenium-grid-selenium-$name-node-v${version%.*}
  name: selenium-grid-selenium-$name-node-v${version%.*}
  namespace: $SE_NAMESPACE
spec:
  failedJobsHistoryLimit: 10
  jobTargetRef:
    activeDeadlineSeconds: 3600
    backoffLimit: 0
    completions: 1
    parallelism: 1
    template:
      metadata:
        labels:
          app: selenium-grid-selenium-$name-node-v${version%.*}
          app.kubernetes.io/name: selenium-grid-selenium-$name-node-v${version%.*}
          app.kubernetes.io/instance: selenium-grid
      spec:
        containers:
        - env:
          - name: KUBERNETES_NODE_HOST_IP
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
          - name: SE_NODE_MAX_SESSIONS
            value: "1"
          - name: SE_DRAIN_AFTER_SESSION_COUNT
            value: "1"
          - name: SE_NODE_BROWSER_VERSION
            value: "$version"
          - name: SE_NODE_PLATFORM_NAME
            value: linux
          - name: SE_NODE_STEREOTYPE_EXTRA
          - name: SE_NODE_CONTAINER_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: SE_BROWSER_ARGS_DISABLE_DSHM
            value: $disable_dshm
          - name: SE_OTEL_SERVICE_NAME
            value: selenium-node-$(echo "$SE_SUB_PATH" | cut -c 2-).$name:$version
          - name: SE_ENABLE_TRACING
            value: "true"
          - name: SE_NODE_HOST
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: SE_NODE_PORT
            value: "5555"
          - name: SE_NODE_REGISTER_PERIOD
            value: "120"
          - name: SE_NODE_REGISTER_CYCLE
            value: "5"
          - name: SE_SESSION_REQUEST_TIMEOUT
            value: "3600"
          - name: SE_VNC_NO_PASSWORD
            value: "1"
          - name: SE_VNC_VIEW_ONLY
            value: "1"
          - name: SE_OPTS
            value: "--enable-managed-downloads true"
          envFrom:
          - configMapRef:
              name: selenium-grid-selenium-event-bus-config
          - configMapRef:
              name: selenium-grid-selenium-node-config
          - configMapRef:
              name: selenium-grid-selenium-logging-config
          - configMapRef:
              name: selenium-grid-selenium-server-config
          - secretRef:
              name: selenium-grid-selenium-secrets
          - secretRef:
              name: selenium-grid-selenium-basic-auth-secrets
          image: registry.host/$name:$latest_tag
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                - bash
                - -c
                - /opt/bin/nodePreStop.sh >> /proc/1/fd/1
          name: selenium-grid-selenium-$name-node-v${version%.*}
          ports:
          - containerPort: 5555
            protocol: TCP
          resources:
            limits:
              cpu: "4"
              memory: 2Gi
              ephemeral-storage: 1Gi
            requests:
              cpu: "1"
              memory: 1Gi
              ephemeral-storage: 256Mi
          startupProbe:
            failureThreshold: 12
            httpGet:
              path: /status
              port: 5555
              scheme: ${SE_SERVER_PROTOCOL^^}
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 60
          volumeMounts:
          - name: dshm
            mountPath: /dev/shm
          - name: artifacts
            mountPath: /artifacts
          - mountPath: /opt/bin/nodeGridUrl.sh
            name: selenium-grid-selenium-node-config
            subPath: nodeGridUrl.sh
          - mountPath: /opt/bin/nodePreStop.sh
            name: selenium-grid-selenium-node-config
            subPath: nodePreStop.sh
          - mountPath: /opt/bin/nodeProbe.sh
            name: selenium-grid-selenium-node-config
            subPath: nodeProbe.sh
          - mountPath: /opt/selenium/secrets
            name: selenium-grid-selenium-tls-secret
            readOnly: true
        - env:
          - name: SE_NODE_MAX_SESSIONS
            value: "1"
          - name: SE_DRAIN_AFTER_SESSION_COUNT
            value: "1"
          - name: SE_NODE_PORT
            value: "5555"
          - name: DISPLAY_CONTAINER_NAME
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          envFrom:
          - configMapRef:
              name: selenium-grid-selenium-event-bus-config
          - configMapRef:
              name: selenium-grid-selenium-node-config
          - configMapRef:
              name: selenium-grid-selenium-recorder-config
          - configMapRef:
              name: selenium-grid-selenium-server-config
          - secretRef:
              name: selenium-grid-selenium-basic-auth-secrets
          - secretRef:
              name: selenium-grid-selenium-secrets
          image: docker-proxy.host/selenium/video:$SE_FF_MPEG_IMAGE
          imagePullPolicy: IfNotPresent
          name: video
          ports:
          - containerPort: 9000
            protocol: TCP
          resources:
            limits:
              cpu: "1"
              memory: 1Gi
              ephemeral-storage: 300Mi
            requests:
              cpu: 100m
              memory: 128Mi
              ephemeral-storage: 128Mi
          volumeMounts:
          - name: dshm
            mountPath: /dev/shm
          - mountPath: /opt/selenium/upload.conf
            name: selenium-grid-selenium-secrets
            subPath: upload.conf
          - mountPath: /videos
            name: videos
        initContainers:
        - command:
          - bash
          - -c
          - '''true'''
          image: registry.host/$name:$latest_tag
          imagePullPolicy: IfNotPresent
          name: pre-puller-selenium-grid-selenium-node
          resources:
            limits:
              cpu: "4"
              memory: 2Gi
              ephemeral-storage: 1Gi
            requests:
              cpu: "1"
              memory: 1Gi
              ephemeral-storage: 10Mi
        - command:
          - bash
          - -c
          - '''true'''
          image: docker-proxy.host/selenium/video:$SE_FF_MPEG_IMAGE
          imagePullPolicy: IfNotPresent
          name: pre-puller-video
          resources:
            limits:
              cpu: "1"
              memory: 1Gi
              ephemeral-storage: 300Mi
            requests:
              cpu: 100m
              memory: 128Mi
              ephemeral-storage: 128Mi
        restartPolicy: Never
        serviceAccount: $SE_NAMESPACE-sa
        serviceAccountName: $SE_NAMESPACE-sa
        shareProcessNamespace: false
        terminationGracePeriodSeconds: 3600
        volumes:
        - configMap:
            defaultMode: 493
            name: selenium-grid-selenium-node-config
          name: selenium-grid-selenium-node-config
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 1Gi
        - name: selenium-grid-selenium-tls-secret
          secret:
            secretName: selenium-grid-selenium-tls-secret
        - name: artifacts
          persistentVolumeClaim:
            claimName: $PVC_NAME
        - emptyDir: {}
          name: videos
        - configMap:
            defaultMode: 493
            name: selenium-grid-selenium-recorder-config
          name: selenium-grid-selenium-recorder-config
        - configMap:
            defaultMode: 493
            name: selenium-grid-selenium-uploader-config
          name: selenium-grid-selenium-uploader-config
        - name: selenium-grid-selenium-secrets
          secret:
            secretName: selenium-grid-selenium-secrets
  maxReplicaCount: 220
  minReplicaCount: 0
  pollingInterval: 20
  rollout:
    strategy: gradual
  scalingStrategy:
    strategy: default
  successfulJobsHistoryLimit: 0
  triggers:
  - authenticationRef:
      name: selenium-grid-selenium-scaler-trigger-auth
    metadata:
      browserName: $kedabrowsername
      browserVersion: "$version"
      capabilities: ""
      nodeMaxSessions: "1"
      platformName: linux
      sessionBrowserName: $sessionBrowserName
      unsafeSsl: "true"
    type: selenium-grid
EOF
        else
          echo "This $name version is not available on registry, version=$version"
        fi
      fi
  done
    unset name
    unset version
    unset uniqs_deployments
    unset deployments
    unset requests
    unset kedabrowsername
    unset tracesexporter
  sleep 5
done

Usage example

If we add this logic to Selenium Grid, then we’ll be able to support all existing browser versions, as well as those that will be released later.

[github-actions]

@Doofus100500, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[VietND96]

I see this looks like a scaler job listen the on GraphQL endpoint, then creates KEDA ScaledObject (if it is not present in the cluster yet), and the rest of works let KEDA scaler take care.
Let me think how to integrate this to the Helm chart.