Skip to content

Commit

Permalink
Add settings for centralized pipeline mgmt
Browse files Browse the repository at this point in the history
  • Loading branch information
karenzone committed May 24, 2018
1 parent eae1cc0 commit 3f58461
Showing 1 changed file with 53 additions and 34 deletions.
87 changes: 53 additions & 34 deletions x-pack/docs/en/management/centralized-pipelines.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
=== Centralized Pipeline Management

The pipeline management feature centralizes the creation and
management of Logstash configuration pipelines.
management of Logstash configuration pipelines in {kib}.

NOTE: Centralized pipeline management is an {xpack} feature that is not included
with the basic license. If you want to try all of the features, you can start a
Expand All @@ -13,23 +13,13 @@ information, see https://www.elastic.co/subscriptions and
{stack-ov}/license-management.html[License
Management].

From within the pipeline
management UI in {kib}, you can control multiple Logstash instances. You can
add, edit, and delete pipeline configurations. On the Logstash side, you simply
need to enable configuration management and register Logstash to use the
centrally managed pipeline configurations.
You can control multiple Logstash instances from the pipeline management UI in
{kib}. You can add, edit, and delete pipeline configurations. On the Logstash
side, you simply need to enable configuration management and register Logstash
to use the centrally managed pipeline configurations.

The pipeline configurations, along with some metadata, are stored in
Elasticsearch. Any changes that you make to a pipeline definition in the UI are
picked up and loaded automatically by all Logstash instances registered to use
the pipeline. The changes are applied immediately; you do not have to restart
Logstash to pick up the changes, as long as Logstash is already registered to
use the pipeline.

NOTE: Centralized management is disabled until you configure and enable
{security}.

==== Managing Pipelines
==== Manage pipelines

Before using the pipeline management UI, you must:

Expand All @@ -38,16 +28,16 @@ Before using the pipeline management UI, you must:
the `logstash_admin` role as well as the `logstash_writer` role that you created
when you <<ls-security,configured Logstash to use basic authentication>>.

To centrally manage Logstash pipelines:
To manage Logstash pipelines in {kib}:

. Open {kib} in your browser and go to the Management tab. If you've set up
configuration management correctly, you'll see an area for managing Logstash.
Click the *Pipelines* link.
+
image::management/images/centralized_config.png[]

. To add a new pipeline, click the *Add* button and specify values for the
following fields:
. Click the *Pipelines* link.

. To add a new pipeline, click *Create pipeline* and specify values.
+
--
[horizontal]
Expand All @@ -65,26 +55,55 @@ Pipeline::
The pipeline configuration. You can treat the editor in the pipeline management
UI like any other editor. You don't have to worry about whitespace or indentation.

image::management/images/new_pipeline.png[]
Pipeline workers::
The number of parallel workers used to run the filter and output stages of the pipeline.

Pipeline batch size::
The maximum number of events an individual worker thread collects before
executing filters and outputs.

Pipeline batch delay::
Time in milliseconds to wait for each event before sending an undersized
batch to pipeline workers.

Queue type::
The internal queueing model for event buffering. Options are *memory* for
in-memory queueing, or *persisted* for disk-based acknowledged queueing.

Queue max bytes::
The total capacity of the queue.

Queue checkpoint writes::
The maximum number of events written before a checkpoint is forced when
persistent queues are enabled.

--

. Click *Save*.
===== Pipeline behavior

* The pipeline configurations and metadata are stored in Elasticsearch. Any
changes that you make to a pipeline definition are picked up and loaded
automatically by all Logstash instances registered to use the pipeline. The
changes are applied immediately. If Logstash is registered to use the pipeline,
you do not have to restart Logstash to pick up the changes.

The pipeline runs on all Logstash instances that are registered to use the
pipeline. There is no validation done at the UI level. The UI will save the new
configuration, and Logstash will attempt to load it. You need to check the local
Logstash logs for configuration errors. If you're using the Logstash monitoring
feature in {xpack}, you can also navigate to the Monitoring tab to check the
status of your Logstash nodes.
* The pipeline runs on all Logstash instances that are registered to use the
pipeline. {kib} saves the new configuration, and Logstash will attempt to load
it. There is no validation done at the UI level.

You can specify multiple pipeline configurations that run in parallel on the
* You need to check the local Logstash logs for configuration errors. If you're
using the Logstash monitoring feature in {xpack}, use the Monitoring tab to
check the status of your Logstash nodes.

* You can specify multiple pipeline configurations that run in parallel on the
same Logstash node.

If you edit a pipeline configuration and save the changes, Logstash reloads
* If you edit and save a pipeline configuration, Logstash reloads
the configuration in the background and continues processing events.

If you delete a pipeline (for example, `apache`) from the UI, Logstash will
attempt to stop the pipeline if it's running. Logstash will wait until all
events have been fully processed by the pipeline. Before deleting a pipeline,
make sure you understand your data sources because stopping a pipeline may
* If you try to delete a pipeline that is running (for example, `apache`) in {kib}, Logstash will
attempt to stop the pipeline. Logstash waits until all
events have been fully processed by the pipeline. Before you delete a pipeline,
make sure you understand your data sources. Stopping a pipeline may
lead to data loss.

0 comments on commit 3f58461

Please sign in to comment.