Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
  • Loading branch information
cppwfs authored and markpollack committed May 17, 2019
1 parent bc2953e commit ea72403
Show file tree
Hide file tree
Showing 8 changed files with 26 additions and 26 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Assuming Data Flow is [installed](%currentPath%/installation/) and running on on

Applications in Data Flow are registered as named resources so that they may be referenced when using the Data Flow DSL to configure and compose streaming pipelines. Registration associates a logical application name and type with a physical resource, given by a URI.

The URI conforms to a [schema](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#spring-cloud-dataflow-register-stream-apps) and may represent a Maven artifact, a Docker image, or an actual `http(s)` or `file` URL. Data Flow defines some logical application types which indicate its role as a streaming component, a task, or a standalone application. For streaming applications, as you might expect, we will use `Source`,`Processor`, and `Sink` types.
The URI conforms to a [schema](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#spring-cloud-dataflow-register-stream-apps) and may represent a Maven artifact, a Docker image, or an actual `http(s)` or `file` URL. Data Flow defines some logical application types which indicate its role as a streaming component, a task, or a standalone application. For streaming applications, as you might expect, we will use `Source`,`Processor`, and `Sink` types.

The Data Flow Dashboard will land on the Application Registration view where we will register the source, processor, and sink apps.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,11 @@ graph TD;
Application log files can be inspected on a per application basis.
To aggregate logs from all applications into one, the deployer property `inheritLogging=true` can be set.
See
[Log Redirect](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_log_redirect)
for more information and [Deployment Logs](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#troubleshooting-deployment-logs) for enabling more log output.
[Log Redirect](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_log_redirect)
for more information and [Deployment Logs](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#troubleshooting-deployment-logs) for enabling more log output.

Debugging applications via JDWP can be accomplished by setting the deployer property `debugPort`.
See [Remote Debugging](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_remote_debugging) for more information.
See [Remote Debugging](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_remote_debugging) for more information.

### Docker Compose - Startup

Expand All @@ -49,7 +49,7 @@ graph TD;
```

The environment variables `DATAFLOW_VERSION` and `SKIPPER_VERSION` must be available in the current terminal environment via `export` or prefixing the `docker-compose` command.
See [Starting Docker Compose](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#getting-started-local-deploying-spring-cloud-dataflow-docker-starting) for more information.
See [Starting Docker Compose](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#getting-started-local-deploying-spring-cloud-dataflow-docker-starting) for more information.

### Docker Compose - Runtime

Expand All @@ -69,7 +69,7 @@ Consult the [Docker documentation](https://docs.docker.com/) for your platform.
As streams are deployed via Skipper, applications that are part of that stream will be launched as Java processes on the Skipper container.
For every part of a stream, an application is deployed.
The overall resource allocation (memory, CPU, etc) provided to Docker should account for the number of deployed applications.
See [Viewing Stream Logs](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#getting-started-local-deploying-spring-cloud-dataflow-docker-viewing-stream-logs) for more information on viewing log files.
See [Viewing Stream Logs](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#getting-started-local-deploying-spring-cloud-dataflow-docker-viewing-stream-logs) for more information on viewing log files.

## Cloud Foundry

Expand Down Expand Up @@ -97,7 +97,7 @@ graph TD;
```

When debugging deployment issues, raising deployer and Cloud Foundry related log levels may be useful.
See [Deployment Logs](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#troubleshooting-deployment-logs) for more information.
See [Deployment Logs](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#troubleshooting-deployment-logs) for more information.

## Kubernetes

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ Follow the instructions to code and build the task if you have not done so alrea
## Create Task Definition

We will register a task application, create a simple task definition, and launch the task using the Data Flow Server.
The Data Flow server provides a comprehensive [API](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide) to perform the necessary steps.
The Data Flow server includes a Data Flow Dashboard web UI client. In addition there is a [Data Flow Shell](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#shell) CLI, available as separate download.
The Data Flow server provides a comprehensive [API](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide) to perform the necessary steps.
The Data Flow server includes a Data Flow Dashboard web UI client. In addition there is a [Data Flow Shell](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#shell) CLI, available as separate download.
The CLI and the UI both expose the complete API functionality.
Which one to use is a matter of preference, but the UI is quite nice so we will feature it here.

Expand All @@ -45,7 +45,7 @@ The Data Flow Dashboard will land on the Application Registration view where we

Applications in Data Flow are registered as named resources so that they may be referenced when using the Data Flow DSL to configure and compose tasks.
Registration associates a logical application name and type with a physical resource, given by a URI.
The URI conforms to a [schema](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#spring-cloud-dataflow-register-stream-apps) and may represent a Maven artifact, a Docker image, or an actual `http(s)` or `file` URL.
The URI conforms to a [schema](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#spring-cloud-dataflow-register-stream-apps) and may represent a Maven artifact, a Docker image, or an actual `http(s)` or `file` URL.
Data Flow defines a some logical application types which indicate its role as a streaming component, a task, or a standalone application.
Spring Cloud Task applications, as you might guess, are always registered as a `task` type.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ Follow the instructions to code and build the Spring Batch application if you ha
## Create Task Definition

We will register the batch application, create a simple task definition for the batch application, and launch the task definition using the Data Flow Server.
The Data Flow server provides a comprehensive [API](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide) to perform the necessary steps.
The Data Flow server includes a Data Flow Dashboard web UI client. In addition there is a [Data Flow Shell](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#shell) CLI, available as separate download.
The Data Flow server provides a comprehensive [API](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide) to perform the necessary steps.
The Data Flow server includes a Data Flow Dashboard web UI client. In addition there is a [Data Flow Shell](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#shell) CLI, available as separate download.
The CLI and the UI both expose the complete API functionality.
Which one to use is a matter of preference, but the UI is quite nice so we will feature it here.

Expand All @@ -45,7 +45,7 @@ The Data Flow Dashboard will land on the Application Registration view where we

Applications in Data Flow are registered as named resources so that they may be referenced when using the Data Flow DSL to configure and compose tasks.
Registration associates a logical application name and type with a physical resource, given by a URI.
The URI conforms to a [schema](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#spring-cloud-dataflow-register-stream-apps) and may represent a Maven artifact, a Docker image, or an actual `http(s)` or `file` URL.
The URI conforms to a [schema](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#spring-cloud-dataflow-register-stream-apps) and may represent a Maven artifact, a Docker image, or an actual `http(s)` or `file` URL.
Data Flow defines a some logical application types which indicate its role as a streaming component, a task, or a standalone application.
In this case our Spring Batch application, as you might guess, are always registered as a `task` type.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,11 @@ graph TD;
Application log files can be inspected on a per application basis.
To aggregate logs from all applications into one, the deployer property `inheritLogging=true` can be set.
See
[Log Redirect](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_log_redirect)
for more information and [Deployment Logs](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#troubleshooting-deployment-logs) for enabling more log output.
[Log Redirect](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_log_redirect)
for more information and [Deployment Logs](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#troubleshooting-deployment-logs) for enabling more log output.

Debugging applications via JDWP can be accomplished by setting the deployer property `debugPort`.
See [Remote Debugging](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_remote_debugging) for more information.
See [Remote Debugging](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_remote_debugging) for more information.

#### Docker Compose - Startup

Expand All @@ -68,7 +68,7 @@ graph TD;
```

The environment variables `DATAFLOW_VERSION` and `SKIPPER_VERSION` must be available in the current terminal environment via `export` or prefixing the `docker-compose` command.
See [Starting Docker Compose](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#getting-started-local-deploying-spring-cloud-dataflow-docker-starting) for more information.
See [Starting Docker Compose](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#getting-started-local-deploying-spring-cloud-dataflow-docker-starting) for more information.

#### Docker Compose - Runtime

Expand Down Expand Up @@ -113,7 +113,7 @@ graph TD;
```

When debugging deployment issues, raising deployer and Cloud Foundry related log levels may be useful.
See [Deployment Logs](http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#troubleshooting-deployment-logs) for more information.
See [Deployment Logs](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#troubleshooting-deployment-logs) for more information.

### Kubernetes

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: 'Publish and subscribe to multiple destinations using the fan-in an

# Fan-in and Fan-out

By using [named destinations](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#spring-cloud-dataflow-stream-dsl-named-destinations), you can support fan-in and fan-out use cases.
By using [named destinations](https://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#spring-cloud-dataflow-stream-dsl-named-destinations), you can support fan-in and fan-out use cases.
Fan-in use cases are when multiple sources all send data to the same named destination, as shown in the following example:

```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: 'Create and Deploy a Python Task'

# Create and Deploy a Python Task

This recipe shows how to run a custom Python script as a [Data Flow Task](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#spring-cloud-dataflow-task) and how to orchestrate later as [Composed Tasks](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#spring-cloud-dataflow-composed-tasks).
This recipe shows how to run a custom Python script as a [Data Flow Task](https://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#spring-cloud-dataflow-task) and how to orchestrate later as [Composed Tasks](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#spring-cloud-dataflow-composed-tasks).

The approach requires the Python script to be bundled in a docker image, which can then be used in SCDF's `Local` and `Kubernetes` implementations.

Expand Down Expand Up @@ -75,7 +75,7 @@ To parse the input arguments and to manage its state in Data Flow, the custom sc

The `spring.cloud.task.executionid` property represents the Task id as known inside Data Flow and persisted in the `TASK_EXECUTION` table.

- The [task_args.py](https://github.com/spring-cloud/spring-cloud-dataflow-samples/blob/master/dataflow-website/recipes/polyglot/polyglot-python-task/util/task_args.py) utility helps extracting the task arguments for default (e.g. exec) [entry point style](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#_entry_point_style_2).
- The [task_args.py](https://github.com/spring-cloud/spring-cloud-dataflow-samples/blob/master/dataflow-website/recipes/polyglot/polyglot-python-task/util/task_args.py) utility helps extracting the task arguments for default (e.g. exec) [entry point style](https://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#_entry_point_style_2).
The utility also constructs [sqlalchemy](https://www.sqlalchemy.org/) urls for the different databases, that might be configured with SCDF (currently only mysql is tested). Check the [get_db_url()](https://github.com/spring-cloud/spring-cloud-dataflow-samples/blob/master/dataflow-website/recipes/polyglot/polyglot-python-task/util/task_args.py#L22) implementation.

For the `python_task.py` to act as a Data Flow task it needs to be bundled in a docker image and uploaded to `DockerHub`. Following [Dockerfile](https://github.com/spring-cloud/spring-cloud-dataflow-samples/blob/master/dataflow-website/recipes/polyglot/polyglot-python-task/Dockerfile) illustrates how to bundle a Python script into docker image:
Expand Down Expand Up @@ -171,7 +171,7 @@ then the second task execution (e. g. #2) fails as shown in the Data Flow Task U

## Use with Composed Tasks

With the provided task status management the Docker/Python tasks can be used inside [Composed Tasks](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#spring-cloud-dataflow-composed-tasks).
With the provided task status management the Docker/Python tasks can be used inside [Composed Tasks](https://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#spring-cloud-dataflow-composed-tasks).

For example for a parallel task execution:

Expand Down
8 changes: 4 additions & 4 deletions content/documentation/pages/7-resources/3-faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ Though batch/task applications are standalone Spring Boot applications, to recor

What is the relationship of [Composed Task Runner](https://github.com/spring-cloud-task-app-starters/composed-task-runner) and SCDF?

The [Composed Tasks](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#spring-cloud-dataflow-composed-tasks) delegates the execution of the collection of Tasks to a separate application, named the Composed Task Runner (CTR).
The [Composed Tasks](https://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#spring-cloud-dataflow-composed-tasks) delegates the execution of the collection of Tasks to a separate application, named the Composed Task Runner (CTR).
The CTR orchestrates the launching of Tasks defined in the composed task graph.
To use Composed Tasks, it is required to connect SCDF, CTR, and batch applications to a shared database. Only then, you will be able to track all of their execution history from SCDF’s dashboard.

Expand All @@ -118,7 +118,7 @@ SCDF delegates and relies on Skipper for the life cycle management of streaming

What tools are available to interact with SCDF?

[Shell](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#shell), [Dashboard](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#dashboard), [Java DSL](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#spring-cloud-dataflow-stream-java-dsl), and [REST-APIs](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#api-guide-resources).
[Shell](https://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#shell), [Dashboard](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#dashboard), [Java DSL](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#spring-cloud-dataflow-stream-java-dsl), and [REST-APIs](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#api-guide-resources).

<!--END_QUESTION-->

Expand All @@ -137,7 +137,7 @@ As such, we ship the binaries directly instead. And we expect the users either u

Can SCDF work with Oracle database?

Yes. Read more about the [supported databases here.](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#configuration-local-rdbms).
Yes. Read more about the [supported databases here.](https://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#configuration-local-rdbms).

<!--END_QUESTION-->

Expand Down Expand Up @@ -188,7 +188,7 @@ See the [compatibility matrix](https://github.com/spring-cloud/spring-cloud-stre

What is a Composed Task Runner (CTR)?

The [Composed Tasks](http://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#spring-cloud-dataflow-composed-tasks) feature in SCDF that delegates the execution of the composed to an separate application, named the Composed Task Runner (CTR).
The [Composed Tasks](https://docs.spring.io/spring-cloud-dataflow/docs/%dataflow-version%/reference/htmlsingle/#spring-cloud-dataflow-composed-tasks) feature in SCDF that delegates the execution of the composed to an separate application, named the Composed Task Runner (CTR).
The CTR to orchestrate the launching of Tasks defined in the composed task graph.
The Composed Task Runner (CTR) parses the graph DSL and for each node in the graph it will execute a RESTful call against a specified Spring Cloud Data Flow instance to launch the associated task definition.
For each task definition that is executed the Composed Task Runner will poll the database to verify that the task completed.
Expand Down

0 comments on commit ea72403

Please sign in to comment.