Skip to content

Latest commit

 

History

History
361 lines (258 loc) · 12.2 KB

getting-started-with-logstash.asciidoc

File metadata and controls

361 lines (258 loc) · 12.2 KB
Raw

Getting Started with Logstash

This section guides you through the process of installing Logstash and verifying that everything is running properly. After learning how to stash your first event, you go on to create a more advanced pipeline that takes Apache web logs as input, parses the logs, and writes the parsed data to an Elasticsearch cluster. Then you learn how to stitch together multiple input and output plugins to unify data from a variety of disparate sources.

This section includes the following topics:

Installing Logstash

Installing from a Downloaded Binary

The {ls} binaries are available from https://www.elastic.co/downloads. Download the Logstash installation file for your host environment—​TARG.GZ, DEB, ZIP, or RPM.

Unpack the file. Do not install Logstash into a directory path that contains colon (:) characters.

Note

These packages are free to use under the Elastic license. They contain open source and free commercial features and access to paid commercial features. {kibana-ref}/managing-licenses.html[Start a 30-day trial] to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

Alternatively, you can download an oss package, which contains only features that are available under the Apache 2.0 license.

On supported Linux operating systems, you can use a package manager to install Logstash.

Installing from Package Repositories

We also have repositories available for APT and YUM based distributions. Note that we only provide binary packages, but no source packages, as the packages are created as part of the Logstash build.

We have split the Logstash package repositories by version into separate urls to avoid accidental upgrades across major versions. For all {major-version}.y releases use {major-version} as version number.

We use the PGP key D88E42B4, Elastic’s Signing Key, with fingerprint

4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4

to sign all our packages. It is available from https://pgp.mit.edu.

Note

When installing from a package repository (or from the DEB or RPM installation file), you will need to run Logstash as a service. Please refer to {logstash-ref}/running-logstash.html[Running Logstash as a Service] for more information.

For testing purposes, you may still run Logstash from the command line, but you may need to define the default setting options (described in {logstash-ref}/dir-layout.html[Logstash Directory Layout]) manually. Please refer to {logstash-ref}/running-logstash-command-line.html[Running Logstash from the Command Line] for more information.

APT

Download and install the Public Signing Key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

You may need to install the apt-transport-https package on Debian before proceeding:

sudo apt-get install apt-transport-https
Warning

Use the echo method described above to add the Logstash repository. Do not use add-apt-repository as it will add a deb-src entry as well, but we do not provide a source package. If you have added the deb-src entry, you will see an error like the following:

Unable to find expected entry 'main/source/Sources' in Release file (Wrong sources.list entry or malformed file)

Just delete the deb-src entry from the /etc/apt/sources.list file and the installation should work as expected.

Run sudo apt-get update and the repository is ready for use. You can install it with:

sudo apt-get update && sudo apt-get install logstash

See {logstash-ref}/running-logstash.html[Running Logstash] for details about managing Logstash as a system service.

YUM

Download and install the public signing key:

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example logstash.repo

And your repository is ready for use. You can install it with:

sudo yum install logstash
Warning
 The repositories do not work with older rpm based distributions that still use RPM v3, like CentOS5.

See the {logstash-ref}/running-logstash.html[Running Logstash] document for managing Logstash as a system service.

Installing {ls} on macOS with Homebrew

Elastic publishes Homebrew formulae so you can install {ls} with the Homebrew package manager.

To install with Homebrew, you first need to tap the Elastic Homebrew repository:

brew tap elastic/tap

After you’ve tapped the Elastic Homebrew repo, you can use brew install to install the default distribution of {ls}:

brew install elastic/tap/logstash-full

This installs the most recently released default distribution of {ls}. To install the OSS distribution, specify elastic/tap/logstash-oss.

Starting {ls} with Homebrew

To have launchd start elastic/tap/logstash-full now and restart at login, run:

brew services start elastic/tap/logstash-full

To run {ls}, in the foreground, run:

logstash

Docker

Images are available for running Logstash as a Docker container. They are available from the Elastic Docker registry.

See Running Logstash on Docker for details on how to configure and run Logstash Docker containers.

Stashing Your First Event

First, let’s test your Logstash installation by running the most basic Logstash pipeline.

A Logstash pipeline has two required elements, input and output, and one optional element, filter. The input plugins consume data from a source, the filter plugins modify the data as you specify, and the output plugins write the data to a destination.

basic logstash pipeline

To test your Logstash installation, run the most basic Logstash pipeline.

MacOS, Linux

cd logstash-{logstash_version}
bin/logstash -e 'input { stdin { } } output { stdout {} }'

Windows

cd logstash-{logstash_version}
.\bin\logstash.bat -e "input { stdin { } } output { stdout {} }"

The command might vary slightly, depending on the terminal or shell you are using.

Note
 The location of the bin directory varies by platform. See {logstash-ref}/dir-layout.html[Directory layout] to find the location of bin\logstash on your system.
Important
macOS Gatekeeper warnings

Apple’s rollout of stricter notarization requirements affected the notarization of the {version} {ls} artifacts. If macOS Catalina displays a dialog when you first run {ls} that interrupts it, you will need to take an action to allow it to run. To prevent Gatekeeper checks on the {ls} files, run the following command on the downloaded .tar.gz archive or the directory to which was extracted:

xattr -d -r com.apple.quarantine <archive-or-directory>

For example, if the .tar.gz file was extracted to the default logstash-{version} directory, the command is:

xattr -d -r com.apple.quarantine logstash-{version}

Alternatively, you can add a security override if a Gatekeeper popup appears by following the instructions in the How to open an app that hasn’t been notarized or is from an unidentified developer section of Safely open apps on your Mac.

The -e flag enables you to specify a configuration directly from the command line. Specifying configurations at the command line lets you quickly test configurations without having to edit a file between iterations. The pipeline in the example takes input from the standard input, stdin, and moves that input to the standard output, stdout, in a structured format.

After starting Logstash, wait until you see "Pipeline main started" and then enter hello world at the command prompt:

hello world
2013-11-21T01:22:14.405+0000 0.0.0.0 hello world

Logstash adds timestamp and IP address information to the message. Exit Logstash by issuing a CTRL-D command in the shell where Logstash is running.

Congratulations! You’ve created and run a basic Logstash pipeline. Next, you learn how to create a more realistic pipeline.