Skip to content

Latest commit

 

History

History
122 lines (87 loc) · 5.68 KB

introduction.asciidoc

File metadata and controls

122 lines (87 loc) · 5.68 KB
Raw

The Power of Logstash

The ingestion workhorse for Elasticsearch and more

Horizontally scalable data processing pipeline with strong Elasticsearch and Kibana synergy

Pluggable pipeline architecture

Mix, match, and orchestrate different inputs, filters, and outputs to play in pipeline harmony

Community-extensible and developer-friendly plugin ecosystem

Over 200 plugins available, plus the flexibility of creating and contributing your own

logstash

Logstash Loves Data

Collect more, so you can know more. Logstash welcomes data of all shapes and sizes.

Logs and Metrics

Where it all started.

  • Handle all types of logging data

    • Easily ingest a multitude of web logs like {logstash-ref}/advanced-pipeline.html[Apache], and application logs like {logstash-ref}/plugins-inputs-log4j.html[log4j] for Java

    • Capture many other log formats like {logstash-ref}/plugins-inputs-syslog.html[syslog], networking and firewall logs, and more

  • Enjoy complementary secure log forwarding capabilities with Filebeat

  • Collect metrics from {logstash-ref}/plugins-inputs-ganglia.html[Ganglia], {logstash-ref}/plugins-codecs-collectd.html[collectd], {logstash-ref}/plugins-codecs-netflow.html[NetFlow], {logstash-ref}/plugins-inputs-jmx.html[JMX], and many other infrastructure and application platforms over {logstash-ref}/plugins-inputs-tcp.html[TCP] and {logstash-ref}/plugins-inputs-udp.html[UDP]

The Web

Unlock the World Wide Web.

  • Transform {logstash-ref}/plugins-inputs-http.html[HTTP requests] into events

    • Consume from web service firehoses like {logstash-ref}/plugins-inputs-twitter.html[Twitter] for social sentiment analysis

    • Webhook support for GitHub, HipChat, JIRA, and countless other applications

    • Enables many Watcher alerting use cases

  • Create events by polling {logstash-ref}/plugins-inputs-http_poller.html[HTTP endpoints] on demand

    • Universally capture health, performance, metrics, and other types of data from web application interfaces

    • Perfect for scenarios where the control of polling is preferred over receiving

Data Stores and Streams

Discover more value from the data you already own.

  • Better understand your data from any relational database or NoSQL store with a {logstash-ref}/plugins-inputs-jdbc.html[JDBC] interface

  • Unify diverse data streams from messaging queues like Apache {logstash-ref}/plugins-outputs-kafka.html[Kafka], {logstash-ref}/plugins-outputs-rabbitmq.html[RabbitMQ], and {logstash-ref}/plugins-outputs-sqs.html[Amazon SQS]

Sensors and IoT

Explore an expansive breadth of other data.

  • In this age of technological advancement, the massive IoT world unleashes endless use cases through capturing and harnessing data from connected sensors.

  • Logstash is the common event collection backbone for ingestion of data shipped from mobile devices to intelligent homes, connected vehicles, healthcare sensors, and many other industry specific applications.

Easily Enrich Everything

The better the data, the better the knowledge. Clean and transform your data during ingestion to gain near real-time insights immediately at index or output time. Logstash comes out-of-box with many aggregations and mutations along with pattern matching, geo mapping, and dynamic lookup capabilities.

  • {logstash-ref}/plugins-filters-grok.html[Grok] is the bread and butter of Logstash filters and is used ubiquitously to derive structure out of unstructured data. Enjoy a wealth of integrated patterns aimed to help quickly resolve web, systems, networking, and other types of event formats.

  • Expand your horizons by deciphering {logstash-ref}/plugins-filters-geoip.html[geo coordinates] from IP addresses, normalizing {logstash-ref}/plugins-filters-date.html[date] complexity, simplifying {logstash-ref}/plugins-filters-kv.html[key-value pairs] and {logstash-ref}/plugins-filters-csv.html[CSV] data, {logstash-ref}/plugins-filters-fingerprint.html[fingerprinting](anonymizing) sensitive information, and further enriching your data with {logstash-ref}/plugins-filters-translate.html[local lookups] or Elasticsearch {logstash-ref}/plugins-filters-elasticsearch.html[queries].

  • Codecs are often used to ease the processing of common event structures like {logstash-ref}/plugins-codecs-json.html[JSON] and {logstash-ref}/plugins-codecs-multiline.html[multiline] events.

See {logstash-ref}/transformation.html[Transforming Data] for an overview of some of the popular data processing plugins.

Choose Your Stash

Route your data where it matters most. Unlock various downstream analytical and operational use cases by storing, analyzing, and taking action on your data.

Analysis

  • {logstash-ref}/plugins-outputs-elasticsearch.html[Elasticsearch]

  • Data stores such as {logstash-ref}/plugins-outputs-mongodb.html[MongoDB] and {logstash-ref}/plugins-outputs-riak.html[Riak]

Archiving

  • {logstash-ref}/plugins-outputs-webhdfs.html[HDFS]

  • {logstash-ref}/plugins-outputs-s3.html[S3]

Monitoring

  • {logstash-ref}/plugins-outputs-nagios.html[Nagios]

  • {logstash-ref}/plugins-outputs-ganglia.html[Ganglia]

  • {logstash-ref}/plugins-outputs-zabbix.html[Zabbix]

  • {logstash-ref}/plugins-outputs-graphite.html[Graphite]

  • {logstash-ref}/plugins-outputs-datadog.html[Datadog]

  • {logstash-ref}/plugins-outputs-cloudwatch.html[CloudWatch]

Alerting

  • Watcher with Elasticsearch

  • {logstash-ref}/plugins-outputs-email.html[Email]

  • {logstash-ref}/plugins-outputs-pagerduty.html[Pagerduty]

  • {logstash-ref}/plugins-outputs-irc.html[IRC]

  • {logstash-ref}/plugins-outputs-sns.html[SNS]