Skip to content

Commit

Permalink
Merge pull request apache#16285 from [BEAM-13492][Playground] Update …
Browse files Browse the repository at this point in the history
…backend README file

* [BEAM-13492][Playground]
update README.md file

* Update README.md file for the backend

* [BEAM-13492][Playground]
add info about NUM_PARALLEL_JOBS environment variable.

* [BEAM-13492][Playground]
Add description about LAUNCH_SITE environment variable

* [BEAM-13492][Playground]
update README file

Co-authored-by: Artur Khanin <[email protected]>
  • Loading branch information
Aydar Zainutdinov and Artur Khanin authored Dec 26, 2021
1 parent 1fa1e3c commit 6d35bf9
Showing 1 changed file with 53 additions and 26 deletions.
79 changes: 53 additions & 26 deletions playground/backend/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,47 +21,82 @@

## About

Apache Beam is an open-source, unified model for defining parallel processing pipelines for batch and streaming data.
It provides a portable API layer for building sophisticated data-parallel processing pipelines that may be executed
across a diversity of execution engines, or runners.

Beam Playground helps facilitate trying out and adopting Apache Beam by providing a very quick way for prospective Beam
users to see and run examples of Apache Beam pipelines in a web interface that requires no setup.
Beam Playground helps facilitate trying out and adopting [Apache Beam](https://beam.apache.org/) by providing a very
quick way for prospective Beam users to see and run examples of Apache Beam pipelines in a web interface that requires
no setup.

## Getting Started

See [playground/README.md](../README.md) for details on requirements and setup.

This section describes what is needed to run the backend application.
- Generating models from proto file
- Go commands to run/test application locally

- Go commands to run/test the backend locally
- Set up environment variables to run the backend locally
- Running the backend via Docker

### Go commands to run/test application locally

The following command is used to build and serve the backend locally:
Go to the backend directory:

```shell
$ cd backend
```
go run ./cmd/server/server.go

The following command is used to build and serve the backend locally:

```shell
$ go run ./cmd/server/server.go
```

Run the following command to generate a release build file:

```
go build ./cmd/server/server.go
```shell
$ go build ./cmd/server/server.go
```

Playground tests may be run using this command:

```
go test ... -v
```shell
$ go test ... -v
```

The full list of commands can be found [here](https://pkg.go.dev/cmd/go).

## Running the server app

To run the server using Docker images there are `Docker` files in the `containers` folder for Go and Java languages.
Each of them processes the corresponding SDK, so the backend with Go SDK will work with Go examples/katas/tests only.
### Set up environment variables to run the backend locally

These environment variables should be set to run the backend locally:

- `BEAM_SDK` - is the SDK which backend could process (`SDK_GO` / `SDK_JAVA` / `SDK_PYTHON` / `SDK_SCIO`)
- `APP_WORK_DIR` - is the directory where all folders will be placed to process each code processing request
- `PREPARED_MOD_DIR` - is the directory where prepared go.mod and go.sum files are placed. It is used only for Go SDK

There are also environment variables which are needed for the deployment of Apache Beam Playground. These variables have
default value and there is no need to set them up to launch locally:

- `SERVER_IP` - is the IP address of the backend server (default value = `localhost`)
- `SERVER_PORT` - is the PORT of the backend server (default value = `8080`)
- `CACHE_TYPE` - is a type of the cache service which is used for the backend server. If it is set as a `remote`, then
the backend server will use Redis to keep all cache values (default value = `local`)
- `CACHE_ADDRESS` - is an address of the Redis server. It is used only when `CACHE_TYPE=remote` (default value
= `localhost:6379`)
- `BEAM_PATH` - it is the place where all required for the Java SDK libs are placed
(default value = `/opt/apache/beam/jars/*`)
- `KEY_EXPIRATION_TIME` - is the expiration time of the keys in the cache (default value = `15 min`)
- `PIPELINE_EXPIRATION_TIMEOUT` - is the expiration time of the code processing (default value = `15 min`)
- `PROTOCOL_TYPE` - is the type of the backend server protocol. It could be `TCP` or `HTTP` (default value = `HTTP`)
- `NUM_PARALLEL_JOBS` - is the max number of the code processing requests which could be processed on the backend server
at the same time (default value = `20`). This value is used to check the readiness of the backend server. If the
server reaches the max number of concurrent code-processing requests, then the load-balancer will route all other
incoming requests to other instances while the instance will not ready.
- `LAUNCH_SITE` - is the value to configure log (default value = local). If developers want to use log service on the
App Engine then need to change this value to `app_engine`.

### Running the server app via Docker

To run the server using Docker images there are `Docker` files in the `containers` folder for Java, Python and Go
languages. Each of them processes the corresponding SDK, so the backend with Go SDK will work with Go
examples/katas/tests only.

One more way to run the server is to run it locally how it is described above.

Expand All @@ -70,11 +105,3 @@ One more way to run the server is to run it locally how it is described above.
To call the server from another client – models and client code should be generated using the
`playground/api/v1/api.proto` file. More information about generating models and client's code using `.proto`
files for each language can be found [here](https://grpc.io/docs/languages/).

## Deployment

TBD

## How to Contribute

TBD

0 comments on commit 6d35bf9

Please sign in to comment.