Merge pull request apache#22096 from [Playground] Infrastructure for …

…sharing any code * [Playground] [Infrastrusture] add one more action to install a cloud datastore emulator * [Playground] [Infrastrusture] add variables * [Playground] [Infrastrusture] change default value for datastore emulator host * [Playground] [Infrastructure] add variables and their description * [Playground] [Infra] add tfvars file * [Playground] [Infra] add extra step to check changes * Update terraform.tfvars * Update variables.tf * [Playground] [Infra] removed extra variable from route dockerfile * [Playground] [Infra] changed backend workflow for production * [Playground] [Infra] removed extra files after testing * [Playground] [Infra] removed extra newline * [Playground] [Infra] removed extra env variables * [Playground] [Infra] updated step to set up cloud sdk * [Playground] [Infra] added viper lib for application props * [Playground] [Infra] added docker compose for local deployment * [Playground] [Infra] edited docker compose * [Playground] [Infrastructure] updated documentation * [Playgroumd] [Infrastructure] renamed db type to remote * [Playgorund] [Infrastructure] remove unused variable * [Playground] [Infrastructure] updated gradle tasks and dockerfile * [Playground] [Infrastucture] removed extra shell script * [Playground] [Infrastructure] updated docker compose file for local deployment * added newlines for git scanning Co-authored-by: MakarkinSAkvelon <[email protected]>
twosom · Jul 8, 2022 · 70585ad · 70585ad
1 parent d44c044
commit 70585ad
Show file tree

Hide file tree

Showing 18 changed files with 659 additions and 55 deletions.
diff --git a/.github/workflows/build_playground_backend.yml b/.github/workflows/build_playground_backend.yml
@@ -32,6 +32,7 @@ jobs:
       BEAM_VERSION: 2.33.0
       TERRAFORM_VERSION: 1.0.9
       STAND_SUFFIX: ''
+      DATASTORE_EMULATOR_HOST: 127.0.0.1:8888
     steps:
       - name: Check out the repo
         uses: actions/checkout@v2
@@ -45,6 +46,12 @@ jobs:
       - name: Remove default github maven configuration
         # This step is a workaround to avoid a decryption issue
         run: rm ~/.m2/settings.xml
+
+      - name: Set up Cloud SDK and its components
+        uses: google-github-actions/setup-gcloud@v0
+        with:
+            install_components: 'beta,cloud-datastore-emulator'
+
       - name: Run PreCommit
         run: ./gradlew playground:backend:precommit
       - name: install npm

diff --git a/playground/CI.md b/playground/CI.md
@@ -47,20 +47,21 @@ after merge even if they do not fail in isolation.
 
 #### Build And Deploy Playground Backend Application - [build_playground_backend.yml](.github/workflows/build_playground_backend.yml)
 
-| Job                                     | Description                                                         | Pull Request Run | Direct Push/Merge Run | Requires GCP Credentials |
-|-----------------------------------------|---------------------------------------------------------------------|------------------|-----------------------|--------------------------|
-| Check out the repo                      | GitHub Action used to check-out a repository.                       | Yes              | Yes                   | No                       |
-| setup-java                              | Install Java.                                                       | Yes              | Yes                   | No                       |
-| setup-go                                | Install Go.                                                         | Yes              | Yes                   | No                       |
-| maven config clean                      | Clean maven settings                                                | Yes              | Yes                   | No                       |
-| playground:backend:precommit            | Pre commit  playground.                                             | Yes              | Yes                   | No                       |
-| npm install                             | Install nmp package.                                                | Yes              | Yes                   | No                       |
-| lint dockerfile                         | Install and  lint docker file.                                      | Yes              | Yes                   | No                       |
-| setup-terraform                         | Install terraform.                                                  | Yes              | Yes                   | No                       |
-| Docker Tag                              | Add tag , if it not set.                                            | Yes              | Yes                   | No                       |
-| GCP account                             | Connect to Gcp.                                                     | Yes              | Yes                   | Yes                      |
-| Login to Docker                         | Login to docker repository.                                         | Yes              | Yes                   | Yes                      |
-| Deploy Backend                          | Build docker container,  push it  to repository,  deploy it to Gcp. | Yes              | Yes                   | Yes                      |
+| Job                                 | Description                                                         | Pull Request Run | Direct Push/Merge Run | Requires GCP Credentials |
+|-------------------------------------|---------------------------------------------------------------------|------------------|-----------------------|--------------------------|
+| Check out the repo                  | GitHub Action used to check-out a repository.                       | Yes              | Yes                   | No                       |
+| setup-java                          | Install Java.                                                       | Yes              | Yes                   | No                       |
+| setup-go                            | Install Go.                                                         | Yes              | Yes                   | No                       |
+| maven config clean                  | Clean maven settings                                                | Yes              | Yes                   | No                       |
+| set up cloud sdk and its components | Setting up cloud client and its components for tests                | Yes              | Yes                   | No                       |
+| playground:backend:precommit        | Pre commit  playground.                                             | Yes              | Yes                   | No                       |
+| npm install                         | Install nmp package.                                                | Yes              | Yes                   | No                       |
+| lint dockerfile                     | Install and  lint docker file.                                      | Yes              | Yes                   | No                       |
+| setup-terraform                     | Install terraform.                                                  | Yes              | Yes                   | No                       |
+| Docker Tag                          | Add tag , if it not set.                                            | Yes              | Yes                   | No                       |
+| GCP account                         | Connect to Gcp.                                                     | Yes              | Yes                   | Yes                      |
+| Login to Docker                     | Login to docker repository.                                         | Yes              | Yes                   | Yes                      |
+| Deploy Backend                      | Build docker container,  push it  to repository,  deploy it to Gcp. | Yes              | Yes                   | Yes                      |
 
 #### Build And Deploy Playground Frontend Application - [build_playground_frontend.yml](.github/workflows/build_playground_frontend.yml)
 

diff --git a/playground/README.md b/playground/README.md
@@ -31,6 +31,11 @@ The following requirements are needed for development, testing, and deploying.
 - Go protobuf dependencies (See [Go gRPC Quickstart](https://grpc.io/docs/languages/go/quickstart/))
 - Dart protobuf dependencies (See [Dart gRPC Quickstart](https://grpc.io/docs/languages/dart/))
 - [buf](https://docs.buf.build/installation)
+- [Docker](https://docs.docker.com/desktop/)
+- [Docker Compose](https://docs.docker.com/compose/install/)
+- [gcloud CLI](https://cloud.google.com/sdk/docs/install)
+- [gcloud Beta Commands](https://cloud.google.com/sdk/gcloud/reference/components/install)
+- [Cloud Datastore Emulator](https://cloud.google.com/sdk/gcloud/reference/components/install)
 
 # Available Gradle Tasks
 
@@ -55,7 +60,21 @@ cd beam
 ./gradlew playground:generateProto
 ```
 
+## Run local environment using docker compose
+
+```
+cd beam
+./gradlew playground:backend:containers:router:dockerComposeLocalUp
+```
+
+## Stop containers and removes containers created by dockerComposeLocalUp
+
+```
+cd beam
+./gradlew playground:backend:containers:router:dockerComposeLocalDown
+```
+
 # Deployment
 
 See [terraform](./terraform/README.md) for details on how to build and deploy
-the application and its dependent infrastructure.
+the application and its dependent infrastructure.
diff --git a/playground/backend/README.md b/playground/backend/README.md
@@ -91,6 +91,17 @@ default value and there is no need to set them up to launch locally:
   incoming requests to other instances while the instance will not ready.
 - `LAUNCH_SITE` - is the value to configure log (default value = local). If developers want to use log service on the
   App Engine then need to change this value to `app_engine`.
+- `SDK_CONFIG` - is the sdk configuration file path, e.g. default example for corresponding sdk. It will be saved to cloud datastore during application startup (default value = `../sdks.yaml`)
+- `DATASTORE_EMULATOR_HOST` - is the datastore emulator address. If it is given in the environment, the application will connect to the datastore emulator.
+- `PROPERTY_PATH` - is the application properties path (default value = `.`)
+
+### Application properties
+
+These properties are stored in `backend/properties.yaml` file:
+
+- `playground_salt` - is the salt to generate the hash to avoid whatever problems a collision may cause.
+- `max_snippet_size` - is the file content size limit. Since 1 character occupies 1 byte of memory, and 1 MB is approximately equal to 1000000 bytes, then maximum size of the snippet is 1000000.
+- `id_length` - is the length of the identifier that is used to store data in the cloud datastore. It's appropriate length to save storage size in the cloud datastore and provide good randomnicity.
 
 ### Running the server app via Docker
 
@@ -113,4 +124,4 @@ The following diagram represents the execution of beam code at the server:
 
 ### Validators/preparators representation
 To clarify which validators and preparators used with the code:
-![](ValidatorsPreparators.png)
+![](ValidatorsPreparators.png)
diff --git a/playground/backend/cmd/server/controller_test.go b/playground/backend/cmd/server/controller_test.go
@@ -24,6 +24,7 @@ import (
 	"github.com/google/uuid"
 	"go.uber.org/goleak"
 	"google.golang.org/grpc"
+	"google.golang.org/grpc/credentials/insecure"
 	"google.golang.org/grpc/test/bufconn"
 	"io/fs"
 	"log"
@@ -173,7 +174,7 @@ func TestPlaygroundController_RunCode(t *testing.T) {
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			conn, err := grpc.DialContext(tt.args.ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithInsecure())
+			conn, err := grpc.DialContext(tt.args.ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithTransportCredentials(insecure.NewCredentials()))
 			if err != nil {
 				t.Fatalf("Failed to dial bufnet: %v", err)
 			}
@@ -207,7 +208,7 @@ func TestPlaygroundController_CheckStatus(t *testing.T) {
 	ctx := context.Background()
 	pipelineId := uuid.New()
 	wantStatus := pb.Status_STATUS_FINISHED
-	conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithInsecure())
+	conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithTransportCredentials(insecure.NewCredentials()))
 	if err != nil {
 		t.Fatalf("Failed to dial bufnet: %v", err)
 	}
@@ -286,7 +287,7 @@ func TestPlaygroundController_GetCompileOutput(t *testing.T) {
 	ctx := context.Background()
 	pipelineId := uuid.New()
 	compileOutput := "MOCK_COMPILE_OUTPUT"
-	conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithInsecure())
+	conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithTransportCredentials(insecure.NewCredentials()))
 	if err != nil {
 		t.Fatalf("Failed to dial bufnet: %v", err)
 	}
@@ -365,7 +366,7 @@ func TestPlaygroundController_GetRunOutput(t *testing.T) {
 	ctx := context.Background()
 	pipelineId := uuid.New()
 	runOutput := "MOCK_RUN_OUTPUT"
-	conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithInsecure())
+	conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithTransportCredentials(insecure.NewCredentials()))
 	if err != nil {
 		t.Fatalf("Failed to dial bufnet: %v", err)
 	}
@@ -474,7 +475,7 @@ func TestPlaygroundController_GetLogs(t *testing.T) {
 	ctx := context.Background()
 	pipelineId := uuid.New()
 	logs := "MOCK_LOGS"
-	conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithInsecure())
+	conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithTransportCredentials(insecure.NewCredentials()))
 	if err != nil {
 		t.Fatalf("Failed to dial bufnet: %v", err)
 	}
@@ -583,7 +584,7 @@ func TestPlaygroundController_GetRunError(t *testing.T) {
 	ctx := context.Background()
 	pipelineId := uuid.New()
 	runError := "MOCK_RUN_ERROR"
-	conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithInsecure())
+	conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithTransportCredentials(insecure.NewCredentials()))
 	if err != nil {
 		t.Fatalf("Failed to dial bufnet: %v", err)
 	}
@@ -675,7 +676,7 @@ func TestPlaygroundController_Cancel(t *testing.T) {
 	defer goleak.VerifyNone(t, opt)
 	ctx := context.Background()
 	pipelineId := uuid.New()
-	conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithInsecure())
+	conn, err := grpc.DialContext(ctx, "bufnet", grpc.WithContextDialer(bufDialer), grpc.WithTransportCredentials(insecure.NewCredentials()))
 	if err != nil {
 		t.Fatalf("Failed to dial bufnet: %v", err)
 	}

diff --git a/playground/backend/containers/router/Dockerfile b/playground/backend/containers/router/Dockerfile
@@ -69,11 +69,15 @@ ENV SERVER_PORT=8080
 ENV APP_WORK_DIR=/opt/playground/backend/
 ENV BEAM_SDK="SDK_UNSPECIFIED"
 ENV BUCKET_NAME="playground-precompiled-objects"
+ENV SDK_CONFIG=/opt/playground/backend/sdks.yaml
+ENV PROPERTY_PATH=/opt/playground/backend/
 
 # Copy build result
 COPY --from=build /go/bin/server_go_backend /opt/playground/backend/
 COPY --from=build /go/src/playground/backend/configs /opt/playground/backend/configs/
 
+COPY sdks.yaml /opt/playground/backend/sdks.yaml
+COPY src/properties.yaml /opt/playground/backend/properties.yaml
 COPY entrypoint.sh /
 
 ENTRYPOINT ["/entrypoint.sh"]
diff --git a/playground/backend/containers/router/build.gradle b/playground/backend/containers/router/build.gradle
@@ -47,6 +47,10 @@ task copyDockerfileDependencies(type: Copy) {
       from '../../../playground'
       into 'build/playground'
    }
+   copy {
+      from '../../../sdks.yaml'
+      into 'build/'
+   }
 }
 
 docker {
@@ -62,5 +66,22 @@ docker {
                            "golang:1.16-bullseye" ])
 }
 
+task dockerComposeLocalUp() {
+    dependsOn ':playground:backend:containers:router:docker'
+    doLast {
+        exec {
+            commandLine "docker-compose", "-f", "docker-compose.local.yml", "up", "-d"
+        }
+    }
+}
+
+task dockerComposeLocalDown() {
+    doLast {
+        exec {
+            commandLine "docker-compose", "-f", "docker-compose.local.yml", "down"
+        }
+    }
+}
+
 // Ensure that we build the required resources and copy and file dependencies from related projects
-dockerPrepare.dependsOn copyDockerfileDependencies
+dockerPrepare.dependsOn copyDockerfileDependencies
diff --git a/playground/backend/containers/router/datastore/Dockerfile b/playground/backend/containers/router/datastore/Dockerfile
@@ -0,0 +1,30 @@
+###############################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+###############################################################################
+
+# Docker hub: https://hub.docker.com/r/google/cloud-sdk
+FROM google/cloud-sdk:alpine
+
+# Install required components
+RUN apk add --update --no-cache openjdk8-jre \
+    && gcloud components install cloud-datastore-emulator beta --quiet
+
+COPY entrypoint.sh /entrypoint.sh
+
+RUN chmod +x ./entrypoint.sh
+
+ENTRYPOINT ["./entrypoint.sh"]
diff --git a/playground/backend/containers/router/datastore/entrypoint.sh b/playground/backend/containers/router/datastore/entrypoint.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+gcloud config set project "$DATASTORE_PROJECT_ID"
+gcloud beta emulators datastore start --host-port="$DATASTORE_LISTEN_ADDRESS"
diff --git a/playground/backend/containers/router/docker-compose.local.yml b/playground/backend/containers/router/docker-compose.local.yml
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "2"
+
+services:
+  datastore:
+    build: "./datastore"
+    environment:
+      DATASTORE_PROJECT_ID: project-test
+      DATASTORE_LISTEN_ADDRESS: 0.0.0.0:8081
+    ports:
+      - "8081:8081"
+  router:
+    image: apache/beam_playground-backend-router
+    environment:
+      GOOGLE_CLOUD_PROJECT: project-test
+      DATASTORE_EMULATOR_HOST: datastore:8081
+    ports:
+      - "8080:8080"
+    depends_on:
+      - datastore
diff --git a/playground/backend/go.mod b/playground/backend/go.mod
@@ -19,14 +19,15 @@ go 1.16
 
 require (
 	cloud.google.com/go/logging v1.4.2
-	cloud.google.com/go/storage v1.18.2
+	cloud.google.com/go/storage v1.23.0
 	github.com/go-redis/redis/v8 v8.11.4
 	github.com/go-redis/redismock/v8 v8.0.6
 	github.com/google/uuid v1.3.0
 	github.com/improbable-eng/grpc-web v0.14.1
 	github.com/rs/cors v1.8.0
+	github.com/spf13/viper v1.12.0
 	go.uber.org/goleak v1.1.12
-	google.golang.org/api v0.58.0
-	google.golang.org/grpc v1.41.0
-	google.golang.org/protobuf v1.27.1
+	google.golang.org/api v0.85.0
+	google.golang.org/grpc v1.47.0
+	google.golang.org/protobuf v1.28.0
 )