setting-up-logstash.asciidoc

Setting Up and Running Logstash

Before reading this section, see [installing-logstash] for basic installation instructions to get you started.

This section includes additional information on how to set up and run Logstash, including:

• Logstash Directory Layout

• Logstash Configuration Files

• [logstash-settings-file]

• [keystore]

• [running-logstash-command-line]

• [running-logstash]

• [docker]

• [docker-config]

• [logging]

• [shutdown]

Logstash Directory Layout

This section describes the default directory structure that is created when you unpack the Logstash installation packages.

Directory Layout of .zip and .tar.gz Archives

The .zip and .tar.gz packages are entirely self-contained. All files and directories are, by default, contained within the home directory — the directory created when unpacking the archive.

This is very convenient because you don’t have to create any directories to start using Logstash, and uninstalling Logstash is as easy as removing the home directory. However, it is advisable to change the default locations of the config and the logs directories so that you do not delete important data later on.

Type	Description	Default Location	Setting
home	Home directory of the Logstash installation.	{extract.path}- Directory created by unpacking the archive
bin	Binary scripts, including logstash to start Logstash and logstash-plugin to install plugins	{extract.path}/bin
settings	Configuration files, including logstash.yml and jvm.options	{extract.path}/config	path.settings
logs	Log files	{extract.path}/logs	path.logs
plugins	Local, non Ruby-Gem plugin files. Each plugin is contained in a subdirectory. Recommended for development only.	{extract.path}/plugins	path.plugins
data	Data files used by logstash and its plugins for any persistence needs.	{extract.path}/data	path.data

Directory Layout of Debian and RPM Packages

The Debian package and the RPM package each place config files, logs, and the settings files in the appropriate locations for the system:

Type	Description	Default Location	Setting
home	Home directory of the Logstash installation.	/usr/share/logstash
bin	Binary scripts including logstash to start Logstash and logstash-plugin to install plugins	/usr/share/logstash/bin
settings	Configuration files, including logstash.yml, jvm.options, and startup.options	/etc/logstash	path.settings
conf	Logstash pipeline configuration files	/etc/logstash/conf.d/*.conf	See /etc/logstash/pipelines.yml
logs	Log files	/var/log/logstash	path.logs
plugins	Local, non Ruby-Gem plugin files. Each plugin is contained in a subdirectory. Recommended for development only.	/usr/share/logstash/plugins	path.plugins
data	Data files used by logstash and its plugins for any persistence needs.	/var/lib/logstash	path.data

Directory Layout of Docker Images

The Docker images are created from the .tar.gz packages, and follow a similar directory layout.

Type	Description	Default Location	Setting
home	Home directory of the Logstash installation.	/usr/share/logstash
bin	Binary scripts, including logstash to start Logstash and logstash-plugin to install plugins	/usr/share/logstash/bin
settings	Configuration files, including logstash.yml and jvm.options	/usr/share/logstash/config	path.settings
conf	Logstash pipeline configuration files	/usr/share/logstash/pipeline	path.config
plugins	Local, non Ruby-Gem plugin files. Each plugin is contained in a subdirectory. Recommended for development only.	/usr/share/logstash/plugins	path.plugins
data	Data files used by logstash and its plugins for any persistence needs.	/usr/share/logstash/data	path.data

Note	Logstash Docker containers do not create log files by default. They log to standard output.

Logstash Configuration Files

Logstash has two types of configuration files: pipeline configuration files, which define the Logstash processing pipeline, and settings files, which specify options that control Logstash startup and execution.

Pipeline Configuration Files

You create pipeline configuration files when you define the stages of your Logstash processing pipeline. On deb and rpm, you place the pipeline configuration files in the /etc/logstash/conf.d directory. Logstash tries to load only files with .conf extension in the /etc/logstash/conf.d directory and ignores all other files.

See [configuration] for more info.

Settings Files

The settings files are already defined in the Logstash installation. Logstash includes the following settings files:

logstash.yml

Contains Logstash configuration flags. You can set flags in this file instead of passing the flags at the command line. Any flags that you set at the command line override the corresponding settings in the logstash.yml file. See [logstash-settings-file] for more info.

jvm.options

Contains JVM configuration flags. Specify each flag on a separate line. You can also use this file to set the locale for Logstash.

startup.options (Linux)

Contains options used by the system-install script in /usr/share/logstash/bin to build the appropriate startup script for your system. When you install the Logstash package, the system-install script executes at the end of the installation process and uses the settings specified in startup.options to set options such as the user, group, service name, and service description. By default, Logstash services are installed under the user logstash. The startup.options file makes it easier for you to install multiple instances of the Logstash service. You can copy the file and change the values for specific settings. Note that the startup.options file is not read at startup. If you want to change the Logstash startup script (for example, to change the Logstash user or read from a different configuration path), you must re-run the system-install script (as root) to pass in the new settings.