This section summarizes the changes in the following releases:
-
Updates bundled JDK #15840
Http Filter - 1.5.1
-
Don’t process response when the body is empty. #50
Syslog_pri Filter - 3.2.1
-
Remove spurious leftover text from "use_labels" docs #15
Logstash Integration - 1.0.2
-
Fix: input plugin now correctly applies common event decorators
type,tags, andadd_fieldto events after receiving them #21
-
Add missing method of logger wrapper for puma #15640
-
Fix logstash-keystore multiple keys operations with command flags #15737
-
Separate scheduling of segments flushes from time #15697
-
Add system properties to configure Jackson’s stream read constraints #15763
-
Fix issue with Jackson 2.15: Can not write a field name, expecting a value #15564
-
Add info and link to {ls} running on a {k8s} cluster through {eck} (ECK) #15565
-
Add info for sending {ls} monitoring data to Elastic {serverless-short} #15636
-
Add docs for extending integrations with filter-elastic_integration #15674
-
Update Logstash intro and security overview for {serverless-short} #15663
-
Update the {ls}-to-{ls} communication docs to reflect the multiple hosts usage #15512
Elasticsearch Input - 4.19.1
Http Input - 3.8.0
-
Fixed SSL Java KeyStore support #171
-
Added
ssl_keystore_typeconfiguration -
Added SSL Java TrustStore configurations (
ssl_truststore_type,ssl_truststore_pathandssl_truststore_password)
Elastic_enterprise_search Integration - 3.0.0
-
[BREAKING] Swiftype endpoints are no longer supported for both plugins App Search and Workplace Search
-
Bumped Enterprise Search clients to version
>= 7.16,< 9#18 -
Added support to SSL configurations (
ssl_certificate_authorities,ssl_truststore_path,ssl_truststore_password,ssl_truststore_type,ssl_verification_mode,ssl_supported_protocolsandssl_cipher_suites) -
The App Search deprecated options
hostandpathwere removed
Kafka Integration - 11.3.3
-
Fixed: "Can’t modify frozen string" error when record value is
nil(tombstones) #155
Logstash Integration - 1.0.1
Elasticsearch Output - 11.22.2
-
Fixed: avoid to populate
versionandversion_typeattributes when processing integration metadata and datastream is enabled. #1161 -
Added support for propagating event processing metadata when this output is downstream of an Elastic Integration Filter and configured without explicit
version,version_type, orroutingdirectives #1158 -
Added support for propagating event processing metadata when this output is downstream of an Elastic Integration Filter and configured without explicit
index,document_id, orpipelinedirectives #1155 -
Changed the register to initiate pipeline shutdown upon bootstrap failure instead of simply logging the error #1151
-
Doc: Replace
document_already_exist_exceptionwithversion_conflict_engine_exceptionin thesilence_errors_in_logsetting example #1159 -
Doc: Add content for sending data to Elasticsearch on serverless #1164
No user-facing changes in Logstash core.
Netflow Codec - 4.3.2
Json Filter - 3.2.1
-
Fix tag on failure test #52
File Input - 4.4.6
-
Change read mode to immediately stop consuming buffered lines when shutdown is requested #322
Twitter Input - 4.1.1
-
Bumped
public_suffixgem version to> 4< 6#77
Csv Output - 3.0.10
-
Extend
spreadsheet_safeprefix guard to '-', '+', and '@' #27
-
Document how to further transform events processed by the filter-elastic_integration plugin #15675
-
Update JRuby to 9.4.5.0 #15670
Mutate Filter - 3.5.8
-
Fix "Can’t modify frozen string" error when converting boolean to
string#171
Beats Input - 6.7.2
-
Restore Lumberjack event parsing source code #486
Elastic_serverless_forwarder Input - 0.1.4
-
[DOC] Adds tips for using the logstash-input-elastic_serverless_forwarder plugin with the Elasticsearch output plugin #7
Validator_support Mixin - 1.1.1
-
Allow single-word host names such as "localhost". This addresses the inability to set "hosts" to "localhost" in the logstash-filter-elastic_integration plugin. #7
-
Downgrade jackson to avoid serialization issues when log.format is set to "json" #15549
-
Input imap plugin’s behavior with Logstash 8.10+ versions, is broken after upgrading its
maildependency #61
-
Update callsite syntax for i18n.t method to avoid deprecated and prohibited format #15500
Elasticsearch Filter - 3.16.1
Http Filter - 1.5.0
-
Standardize SSL settings and deprecate their non-standard counterparts. Deprecated settings will continue to work, and will provide pipeline maintainers with guidance toward using their standardized counterparts #49
-
Introduce new
ssl_truststore_path,ssl_truststore_password, andssl_truststore_typesettings for configuring SSL-trust using a PKCS-12 or JKS trust store, deprecate theirtruststore,truststore_password, andtruststore_typecounterparts. -
Introduce new
ssl_certificate_authoritiessetting for configuring SSL-trust using a PEM-formatted list certificate authorities, deprecate itscacertcounterpart. -
Introduce new
ssl_keystore_path,ssl_keystore_password, andssl_keystore_typesettings for configuring SSL-identity using a PKCS-12 or JKS key store, deprecate theirkeystore,keystore_password, andkeystore_typecounterparts. -
Introduce new
ssl_certificateandssl_keysettings for configuring SSL-identity using a PEM-formatted certificate/key pair, deprecate theirclient_certandclient_keycounterparts. -
Introduce the
ssl_cipher_suitesoption.
-
Beats Input - 6.7.0
-
Add explicit support for receiving a 0-length window to encapsulate an empty batch. Empty batches are acknowledged with the same 0-sequence ACK’s that are used as keep-alives during processing #479
Elasticsearch Input - 4.18.0
-
Add request header
Elastic-Api-Versionfor serverless #195
Http_poller Input - 5.5.0
-
Standardize SSL settings and deprecate their non-standard counterparts. Deprecated settings will continue to work, and will provide pipeline maintainers with guidance toward using their standardized counterparts #141
-
Introduce new
ssl_truststore_path,ssl_truststore_password, andssl_truststore_typesettings for configuring SSL-trust using a PKCS-12 or JKS trust store, deprecate theirtruststore,truststore_password, andtruststore_typecounterparts. -
Introduce new
ssl_certificate_authoritiessetting for configuring SSL-trust using a PEM-formatted list certificate authorities, deprecate itscacertcounterpart. -
Introduce new
ssl_keystore_path,ssl_keystore_password, andssl_keystore_typesettings for configuring SSL-identity using a PKCS-12 or JKS key store, deprecate theirkeystore,keystore_password, andkeystore_typecounterparts. -
Introduce new
ssl_certificateandssl_keysettings for configuring SSL-identity using a PEM-formatted certificate/key pair, deprecate theirclient_certandclient_keycounterparts. -
Introduce the
ssl_cipher_suitesoption.
-
Imap Input - 3.2.1
-
Upgrade
emaildependency so that supports Ruby 3.1. This also fixes thenet-smtpload regression #60
Jdbc Integration - 5.4.6
-
Update sequel version to >= 5.73.0. The ibmdb and jdbc/db2 adapters were fixed to properly handle disconnect errors, removing the related connection from the pool #144
Logstash Integration - 0.0.5
-
Logstash 8.11 version embeds the
logstash-integration-logstashplugin.
Http_client Mixin - 7.3.0
-
Standardize SSL settings and deprecate their non-standard counterparts. Deprecated settings will continue to work, and will provide pipeline maintainers with guidance toward using their standardized counterparts #42
-
Introduce new
ssl_truststore_path,ssl_truststore_password, andssl_truststore_typesettings for configuring SSL-trust using a PKCS-12 or JKS trust store, deprecate theirtruststore,truststore_password, andtruststore_typecounterparts. -
Introduce new
ssl_certificate_authoritiessetting for configuring SSL-trust using a PEM-formated list certificate authorities, deprecate itscacertcounterpart. -
Introduce new
ssl_keystore_path,ssl_keystore_password, andssl_keystore_typesettings for configuring SSL-identity using a PKCS-12 or JKS key store, deprecate theirkeystore,keystore_password, andkeystore_typecounterparts. -
Introduce new
ssl_certificateandssl_keysettings for configuring SSL-identity using a PEM-formatted certificate/key pair, deprecate theirclient_certandclient_keycounterparts. -
Introduce a way for plugin maintainers to include this mixin without supporting the now-deprecated SSL options.
-
Introduce the
ssl_cipher_suitesoption.
-
Validator_support Mixin - 1.1.0
-
Introduce
:required_host_optional_portvalidator #4
Elasticsearch Output - 11.18.0
Email Output - 4.1.3
-
Upgrade
emaildependency so that supports Ruby 3.1. This also fixes thenet-smtpload regression #69
Http Output - 5.6.0
-
Standardize SSL settings and deprecate their non-standard counterparts. Deprecated settings will continue to work, and will provide pipeline maintainers with guidance toward using their standardized counterparts #140
-
Introduce new
ssl_truststore_path,ssl_truststore_password, andssl_truststore_typesettings for configuring SSL-trust using a PKCS-12 or JKS trust store, deprecate theirtruststore,truststore_password, andtruststore_typecounterparts. -
Introduce new
ssl_certificate_authoritiessetting for configuring SSL-trust using a PEM-formatted list certificate authorities, deprecate itscacertcounterpart. -
Introduce new
ssl_keystore_path,ssl_keystore_password, andssl_keystore_typesettings for configuring SSL-identity using a PKCS-12 or JKS key store, deprecate theirkeystore,keystore_password, andkeystore_typecounterparts. -
Introduce new
ssl_certificateandssl_keysettings for configuring SSL-identity using a PEM-formatted certificate/key pair, deprecate theirclient_certandclient_keycounterparts. -
Introduce the
ssl_cipher_suitesoption.
-
Webhdfs Output - 3.1.0
-
Fix: remove snappy gem as a dependency in favor of directly vendoring the snappy jar. #46
These plugins may fail in Logstash 8.10.4:
-
Avro codec plugin. The avro codec plugin may fail with an
(Errno::ENOENT) No such file or directoryerror #42. This issue has been resolved in plugin v3.4.1 #43. -
Imap input plugin. Due to JRuby upgrade, the plugin is broken and will be unbundled. Details and updates are available in GitHub issue #61.
-
Email output plugin. Plugin raises
LoadError: no such file to load — net/smtpruntime error. See the issue details and work around in GitHub issue #68.
-
Update Guava dependency to 32.1.2 #15419
These plugins may fail in Logstash 8.10.3:
-
Avro codec plugin. The avro codec plugin may fail with an
(Errno::ENOENT) No such file or directoryerror #42. This issue has been resolved in plugin v3.4.1 #43. -
Imap input plugin. Due to JRuby upgrade, the plugin is broken and will be unbundled. Details and updates are available in GitHub issue #61.
-
Email output plugin. Plugin raises
LoadError: no such file to load — net/smtpruntime error. See the issue details and work around in GitHub issue #68.
Elasticsearch Filter - 3.15.3
-
Fixes a memory leak that occurs when a pipeline containing this filter terminates, which could become significant if the pipeline is cycled repeatedly #173
Useragent Filter - 3.3.5
-
Upgrade
snakeyamldependency #89
Beats Input - 6.6.4
-
[DOC] Fix misleading
enrich/source_datainput beats documentation about the Logstash host. #478
Elastic_serverless_forwarder Input - 0.1.3
-
Deprecates the
ssloption in favor ofssl_enabled#6 -
Bumps
logstash-input-httpgem version to>= 3.7.2(SSL-normalized)
Aws Integration - 7.1.6
-
Clean up plugin created temporary dirs at startup #39
Jdbc Integration - 5.4.5
-
Pin sequel to < 5.72.0 due to ruby/bigdecimal#169 #141
Kafka Integration - 11.3.1
-
Fix: update snappy dependency #148
These plugins may fail in Logstash 8.10.2:
-
Avro codec plugin. The avro codec plugin may fail with an
(Errno::ENOENT) No such file or directoryerror #42. This issue has been resolved in plugin v3.4.1 #43. -
Imap input plugin. Due to JRuby upgrade, the plugin is broken and will be unbundled. Details and updates are available in GitHub issue #61.
-
Email output plugin. Plugin raises
LoadError: no such file to load — net/smtpruntime error. See the issue details and work around in GitHub issue #68.
These plugins may fail in Logstash 8.10.1:
-
Avro codec plugin. The avro codec plugin may fail with an
(Errno::ENOENT) No such file or directoryerror #42. This issue has been resolved in plugin v3.4.1 #43. -
Imap input plugin. Due to JRuby upgrade, the plugin is broken and will be unbundled. Details and updates are available in GitHub issue #61.
-
Email output plugin. Plugin raises
LoadError: no such file to load — net/smtpruntime error. See the issue details and work around in GitHub issue #68.
These plugins may fail in Logstash 8.10.0:
-
Avro codec plugin. The avro codec plugin may fail with an
(Errno::ENOENT) No such file or directoryerror #42. This issue has been resolved in plugin v3.4.1 #43. -
Imap input plugin. Due to JRuby upgrade, the plugin is broken and will be unbundled. Details and updates are available in GitHub issue #61.
-
Email output plugin. Plugin raises
LoadError: no such file to load — net/smtpruntime error. See the issue details and work around in GitHub issue #68.
Elasticsearch Output - 11.16.0
-
Added support to Serverless Elasticsearch #114
Elastic_serverless_forwarder input 0.1.2
-
Introduces a dedicated input plugin for receiving events from Elastic Serverless Forwarder. This plugin is in late Technical Preview, which means that as we iterate toward a stable API both configuration options and implementation details may change in subsequent releases without the usual deprecation warnings. If you use this plugin, please keep your eye on the relevant changelogs when upgrading. #4
Kafka Integration - 11.3.0
-
Changed Kafka client to 3.4.1 #145
Tcp Input - 6.3.5
-
Standardized SSL settings #213
-
deprecated
ssl_enablein favor ofssl_enabled -
deprecated
ssl_certin favor ofssl_certificate -
deprecated
ssl_verifyin favor ofssl_client_authenticationwhen mode isserver -
deprecated
ssl_verifyin favor ofssl_verification_modewhen mode isclient
-
-
Update JDK to 17.0.8+7 #15237
-
Fixed an issue where installs and updates of certain {ls} plugins could fail when located behind a proxy #15131. This issue surfaced after
logstash-filter-translatewas updated to require that thejar-dependenciesgem be used to retrieve artifacts from maven when the plugin was installed. This requirement could prevent the plugin update when a proxy was in use. -
Improved logging when {ls} is stalled on shutdown #15056. We now provide additional information about the main thread if it is causing the shutdown to stall.
-
Improved SSL settings for connection to {es} for central management and monitoring #15045. This commit adds settings support for file-based certificates and cipher suites for management and monitoring settings, and removes the deprecation warnings from the logs that have been in since SSL configuration settings were revamped in the {es} output.
-
Update Bundler to version 2.4 #14995
Azure_event_hubs Input - 1.4.5
-
Update multiple dependencies such as
gson,log4j2,jackson#83
Beats Input - 6.6.3
Http Input - 3.7.2
-
Update netty to 4.1.94 #167
Snmp Input - 1.3.2
-
[DOC] Add troubleshooting help for "failed to locate MIB module" error when using smidump to convert MIBs #112
Tcp Input - 6.3.5
Tcp Output - 6.1.2
-
Changed the client mode to write using the non-blocking method. #52
No user-facing changes in Logstash core.
Translate Filter - 3.4.2
-
Fix JRuby 9.4 compatibility issue #98
Aws Integration - 7.1.4
Jdbc Integration - 5.4.4
-
Fix: adaptations for JRuby 9.4 #125
Rabbitmq Integration - 7.3.3
-
Fix the cancellation flow to avoid multiple invocations of basic.cancel #55
Csv Output - 3.0.9
-
Fix JRuby 9.4 compatibility issue #25
Elasticsearch Output - 11.15.8
-
Fix a regression introduced in 11.14.0 which could prevent Logstash 8.8 from establishing a connection to Elasticsearch for Central Management and Monitoring core features #1141
-
Remove obsolete notice when using plugins with version < 1.0.0 #15077
-
Docs: Add instructions to verify Docker install images #15064
-
Fixes a regression introduced in 8.8.0 which could prevent Monitoring or Central Management from establishing a connection to Elasticsearch in some SSL configurations #15068
Cef Codec - 6.2.7
-
Fix: when decoding in an ecs_compatibility mode, timestamp-normalized fields now handle provided-but-empty values #102
Anonymize Filter - 3.0.7
-
Pin murmurhash3 to 0.1.6 #16
Elasticsearch Filter - 3.15.1
-
Fixes a regression introduced in 3.15.0 which could prevent a connection from being established to Elasticsearch in some SSL configurations #169
Fingerprint Filter - 3.4.3
-
Pin murmurhash3 to 0.1.6 #74
Mutate Filter - 3.5.7
-
Docs: Clarify that
splitandjoinalso support strings #164
Translate Filter - 3.4.1
-
Fix the limitation of the size of yaml file that exceeds 3MB #97
Truncate Filter - 1.0.6
-
Make 0.0.8 the lower bound for flores dependency #9
Beats Input - 6.6.1
-
Update netty to 4.1.93 and jackson to 2.13.5 #472
Elasticsearch Input - 4.17.2
Http Input - 3.7.1
-
Update netty to 4.1.93 #166
Tcp Input - 6.3.3
-
Update netty to 4.1.93 #212
Jdbc Integration - 5.4.3
Rabbitmq Integration - 7.3.2
-
Change
tls_certificate_passwordtype topasswordto protect from leaks in the logs #54
Elasticsearch Output - 11.15.7
-
Fixes a regression introduced in 11.14.0 which could prevent a connection from being established to Elasticsearch in some SSL configurations #1138
-
Fixes possiblity of data loss when pipeline terminates very quickly after startup #1132.
-
Fixes
undefined 'shutdown_requested' methoderror when plugin checks if shutdown request is received #1134 -
Improves connection handling under several partial-failure scenarios #1130
-
Ensures an HTTP connection can be established before adding the connection to the pool
-
Ensures that the version of the connected Elasticsearch is retrieved successfully before the connection is added to the pool.
-
Fixes a crash that could occur when the plugin is configured to connect to a live HTTP resource that is not Elasticsearch
-
-
Removes the ECS v8 unreleased preview warning #1131
-
Restores DLQ logging behavior from 11.8.x to include the action-tuple as structured #1105
Email Output - 4.1.2
-
Change
passwordconfig type toPasswordto prevent leaks in debug logs #65
Logstash 8.8.0 may fail to start when SSL/TLS is enabled in monitoring and/or central management, due to a change introduced in version 11.14.0 of the logstash-output-elasticsearch plugin. When impacted by this issue, Logstash fails to start and logs an error similar to the following:
[logstash.licensechecker.licensereader] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}Resolution
A successful Elasticsearch output plugin update to version 11.15.8 or higher will
resolve this issue:
bin/logstash-plugin update logstash-output-elasticsearchOR
Specify the ca_trusted_fingerprint setting in the logstash.yml.
The certificate fingerprint can be extract with:
cat your_ca.cert | openssl x509 -outform der | sha256sum | awk '{print $1}'Then set the following on logstash.yml using the output from the previous command:
xpack.monitoring.elasticsearch.ssl.ca_trusted_fingerprint: "<value>"
xpack.management.elasticsearch.ssl.ca_trusted_fingerprint: "<value>"-
Fix a race condition that prevents Logstash from updating a pipeline’s configuration with in-flight events experiencing connection errors. #14739 This issue primarily manifests following the update of Elasticsearch credentials through Central Management, after credentials expired while events were in-flight. It causes the Elasticsearch Output to get stuck attempting to send events with the expired credentials instead of using the updated ones. To address this problem, Logstash has improved the pipeline shutdown phase functionality to allow an output plugin to request the termination of the in-flight batch of events; hence preventing the need for administrators to manually restart Logstash. Furthermore, when used in combination with a persistent queue to prevent data loss, the batch is eligible for reprocessing on pipeline restart. Plugin developers can now decide whether to make use of such functionality on output plugins. #14940
-
Updates Bundler to version 2.4 #14995
Elasticsearch Filter - 3.15.0
Memcached Filter - 1.2.0
-
Upgrade Dalli to 3.x #33
Beats Input - 6.6.0
-
Standardize SSL settings to comply with Logstash’s naming convention #470
Elasticsearch Input - 4.17.0
-
Standardize SSL settings to comply with Logstash’s naming convention #185
Http Input - 3.7.0
-
Standardize SSL settings to comply with Logstash’s naming convention #165
Kafka Integration - 11.2.1
-
Fix nil exception to empty headers of record during event metadata assignment #140
-
Added TLS truststore and keystore settings specifically to access the schema registry #137
-
Added config
group_instance_idto use the Kafka’s consumer static membership feature #135 -
Changed Kafka client to 3.3.1, requires Logstash >= 8.3.0.
-
Deprecated
defaultvalue for settingclient_dns_lookupforcing touse_all_dns_ipswhen explicitly used #130 -
Changed the consumer’s poll from using the one that blocks on metadata retrieval to the one that doesn’t #136
Normalize_config_support Mixin - 1.0.0
Elasticsearch Output - 11.15.1
-
Document meaning of infinite flow metric rates #14999
-
Update JDK to 17.0.7+7 #15015
Fluent Codec - 3.4.2
-
Fix: Convert LogStash::Timestamp values to iso-8601 to resolve crash issue with
msgpackserialization #30
Http Filter - 1.4.3
Useragent Filter - 3.3.4
-
Upgrade
snakeyamldependency to 1.33 #84
Aws Integration - 7.1.1
-
Fix failure to load Java dependencies making v7.1.0 unusable #24
-
Fix: DLQ writer isn’t properly created due to inversion of parameteres #14900
-
Logstash fails to start on OracleLinux7 #14890
-
Fix: DLQ age policy isn’t executed if the current head segment haven’t receives any write #14878
-
Fixes an issue during process shutdown in which the stalled shutdown watcher incorrectly reports
inflight_countas0even when there are events in-flight #14760
-
Allow
dead_letter_queue.retain.ageusage in pipeline settings #14954 -
Improved logging behavior in a docker container #14949
-
snakeyamlupgraded to1.33version #14881 -
Update bundeld JDK to
17.0.6+10#14849 -
jrjacksonto0.4.17andjacksonto2.14.1versions upgraded #14845 -
Pins
murmurhash3to0.1.6version #14832 -
Guard reserved tags field against incorrect use #14822
-
Remove unnecessary pipeline configuration logging #14779
-
Developers using the Ruby- or Java-based Plugin APIs will have access to a new API boundary for acquiring a timer object to track execution time #14748
Dns Filter - 3.2.0
-
Add tag(s) on DNS lookup times outs, defaults to
["_dnstimeout"]#67
Syslog_pri Filter - 3.2.0
-
Add tag on unrecognized
facility_labelcode #11
Beats Input - 6.5.0
-
Added
enrichenrichment option to control ECS passthrough.ssl_peer_metadataandinclude_codec_tagconfigurations are deprecated and can be managed through theenrich#464
Aws Integration - 7.1.0
-
Restore and upload corrupted GZIP files to AWS S3 after abnormal termination #20
Elasticsearch Output - 11.13.1
-
Avoid a crash by ensuring ILM settings are injected in the correct location depending on the default (or custom) template format,
template_apisetting and ES version #1102 -
Technology preview support for allowing events to individually encode a default pipeline with
[@metadata][target_ingest_pipeline](as part of a technology preview, this feature may change without notice) #1113
Fingerprint Filter - 3.4.2
-
Key config type changed to
passwordtype for better protection from leaks. #71
Aws Integration - 7.0.1
-
Resolved race conditions in the S3 Output plugin when handling temporary files #19
Elasticsearch Output - 11.12.4
-
Changed
manage_templatedefault value tofalsewhen data streams is enabled #1111
-
Updated snakeyaml to 1.33 #14848
-
Extends the flow rates introduced to the Node Stats API in 8.5.0 (which included windows for
currentandlifetime) to include a Technology Preview of several additional windows such aslast_15_minutes,last_24_hours, etc.. #14571 -
Logstash introduced instance and pipeline level flow metrics,
growth_bytesandgrowth_eventsfor persisted queue to provide a better visibility about how fast pipeline queue is growing. #14554
Netflow Codec - 4.3.0
-
Added Gigamon ipfix definitions #199
Elasticsearch Filter - 3.13.0
-
Added support for this plugin identifying itself to Elasticsearch with an SSL/TLS client certificate using a new
keystoreoption #162
Jdbc Integration - 5.4.1
-
Bugfix leak which happened in creating a new Database pool for every query. The pool is now crated on registration and closed on plugin’s
stop#119 -
Ambiguous Timestamp Support #92
-
FIX: when encountering an ambiguous timestamp, the JDBC Input no longer crashes
-
Added support for disambiguating timestamps in daylight saving time (DST) overlap periods
-
Elasticsearch Output - 11.12.1
-
Log bulk request response body on error, not just when debug logging is enabled #1096
-
Add legacy template API support for Elasticsearch 8 #1092
-
When using an
api_keyalong with eithercloud_idor httpshosts, you no longer need to also specifyssl ⇒ true#1065 -
Feature: expose
dlq_routeddocument metric to track the documents routed into DLQ #1090
No user-facing changes in Logstash core.
No user-facing changes in Logstash core.
-
Fixes the reporting of configuration errors when using multiple-pipelines to make them more actionable #14713
-
The bundled JDK has been updated to 17.0.5+8 #14728
Cef Codec - 6.2.6
-
Fix: when decoding, escaped newlines and carriage returns in extension values are now correctly decoded into literal newlines and carriage returns respectively #98
-
Fix: when decoding, non-CEF payloads are identified and intercepted to prevent data-loss and corruption. They now cause a descriptive log message to be emitted, and are emitted as their own
_cefparsefailure-tagged event containing the original bytes in itsmessagefield #99 -
Fix: when decoding while configured with a
delimiter, flushing this codec now correctly consumes the remainder of its internal buffer. This resolves an issue where bytes that are written without a trailing delimiter could be lost #100
Json Codec - 3.1.1
-
Fix: when decoded JSON includes an
[event][original]field, havingecs_compatibilityenabled will no longer overwrite the decoded field #43
Grok Filter - 4.4.3
-
Minor typos in docs examples #176
Tcp Input - 6.3.1
-
Fixes a regression in which the ssl_subject was missing for SSL-secured connections in server mode #199
Unix Input - 3.1.2
-
Fix: eliminate high CPU usage when data timeout is disabled and no data is available on the socket #30
Rabbitmq Integration - 7.3.1
-
DOCS: clarify the availability and cost of using the
metadata_enabledoption #52
Elasticsearch Output - 11.9.3
-
DOC: clarify that
http_compressionoption only affects requests; compressed responses have always been read independent of this setting #1030 -
Fix broken link to Logstash Reference #1085
-
Fixes a possible infinite-retry-loop that could occur when this plugin is configured with an
actionwhose value contains a sprintf-style placeholder that fails to be resolved for an individual event. Events in this state are routed to the pipeline’s dead letter queue (DLQ) if the DLQ is enabled. Otherwise, these events are logged-and-dropped so that the remaining events in the batch can be processed. #1080
Due to a recent change in the Red Hat scan verification process, this version of Logstash is not available in the Red Hat Ecosystem Catalog. This bug will be fixed in the next release. Please use the Elastic docker registry to download the 8.5.0 Logstash image.
-
It is often difficult to understand the health of a pipeline, including whether it is exerting or propagating back-pressure or otherwise staying reasonably “caught up” with its inputs. This release adds pipeline "flow" metrics to the node_stats API for each pipeline, which includes the current and lifetime rates for five key pipeline metrics: input_throughput, filter_throughput, output_throughput, queue_backpressure, and worker_concurrency. #14518
Translate Filter - 3.4.0
-
Refactor: leverage scheduler mixin #93
Elasticsearch Input - 4.16.0
Exec Input - 3.6.0
File Input - 4.4.4
-
Fixes gzip file handling in read mode when run on JDK12+, including JDK17 that is bundled with Logstash 8.4+ #312
Http_poller Input - 5.4.0
-
Refactor: start using scheduler mixin #134
Elasticsearch Output - 11.9.0
-
Feature: force unresolved dynamic index names to be sent into DLQ. This feature could be explicitly disabled using
dlq_on_failed_indexname_interpolationsetting #1084 -
Feature: Adds a new
dlq_custom_codesoption to customize DLQ codes #1067 -
Feature: deprecates the
failure_type_logging_whitelistconfiguration option, renaming itsilence_errors_in_log#1068
-
Fixed the inability to configure "monitoring.cluster_uuid" in docker #14496
-
Disabled DES-CBC3-SHA cipher in some plugins that still supported it #14501
-
Upgraded JRuby the CSV gem to fix a thread leak in Logstash 8.4.0 when using the CSV filter #14508 #14526
-
Fixed Windows .bat scripts that prevented the use of the Plugin Manager and Keystore in Logstash 8.3.3/8.4.0 #14516
-
Added documentation for using Winlogbeat with Logstash #14512
No user-facing changes in Logstash core.
This release brings significant improvements to help users manage their dead letter queues, including:
-
A new
clean_consumedoption on the Dead Letter Queue input plugin. It can automatically delete segments from a dead letter queue after all events in the segment have been consumed by a Logstash pipeline. -
A new age retention policy, enabling the automatic removal of segments from a dead letter queue based on the age of events within those segments.
-
Additional dead letter queue metrics available from the monitoring API #14324
Several AWS plugins are now bundled in a single {logstash-ref}/plugins-integrations-aws.html[AWS integration plugin], enabling easier maintenance and upgrades of AWS-based plugins. They all use version 3 of the AWS Ruby SDK.
Logstash now comes bundled with JDK17, while still providing compatibility with user-supplied JDK11. The new JDK includes an update pertaining to a potential security vulnerability. Please see our security statement for details.
-
Remove
/etc/systemd/system/logstash.serviceonly when file is installed by Logstash #14200 -
Fix Arcsight module compatibility with Elasticsearch
8.x#13874 -
Ensure that timestamp values are serialized with a minimum of 3 decimal places to guarantee that millisecond precision timestamps match those from Logstash
7.x#14299 -
Fix issue with native Java plugin thread-safety and concurrency #14360
-
Allow the ability to use Ruby codecs inside native Java plugins #13523
Dead Letter Queue Input - 2.0.0
Xml Filter - 4.2.0
-
Update Nokogiri dependency version #78
Aws Integration Plugin - 7.0.0
-
This new integration plugin incorporates and replaces the use of the these individual plugins: individual plugins:
-
logstash-input-s3
-
logstash-input-sqs
-
logstash-mixin-aws
-
logstash-output-cloudwatch
-
logstash-output-s3
-
logstash-output-sns
-
logstash-output-sqs
-
-
This replaces the use of the single combined aws
2.xsdk gem, with the modularized aws3.xgems.
-
We fixed an issue which occurred when users ran the plugin manager or the keystore with the bundled JVM. Some mandatory JVM options were not being picked up from the JvmOptionsParser, breaking compatibility with Windows on certain versions of the JDK. #14355
Date Filter - 3.1.15
-
Build: review build to be more reliable/portable #139
-
Cleaned up Java dependencies
Fingerprint Filter - 3.4.1
-
Added backward compatibility of timestamp format to provide consistent fingerprint #67
Http_poller Input - 5.3.1
-
Fix: Make sure plugin is closing the http client #130
Scheduler Mixin - 1.0.1
-
Refactor: Initialize time zone data eagerly #2
Core Patterns - 4.3.4
-
Fix: Correct typo in CISCOFW302013_302014_302015_302016 grok pattern #313
-
We fixed an event serializing incompatibility introduced in 8.3.0 #14314 If you’re using dead letter queues or persistent queues we recommend that you do not use Logstash 8.3.0 and upgrade to 8.3.1.
An event serialization bug was discovered, which causes an issue when trying to read dead letter or persistent queues created in previous versions of Logstash.
We recommend not upgrading to Logstash 8.3.0 if you are using dead letter or persistent queues.
-
{ls} is more efficient at fetching pipelines as of 8.3.0. When a {ls} instance sends its pipeline IDs to {es} or central pipeline management in {kib}, it gets back only the pipeline configs that belong to that instance. These enhancements required changes to both {ls} (#14076) and {es} (#85847).
These improvements dramatically decrease network load while also giving users the ability to control pipelines dynamically using wildcards.
-
Dead Letter Queues can now be configured to drop older events instead of new ones when they’re full. The setting "dead_letter_queue.storage_policy" has been introduced for this purpose, and new metrics - such as a counter for dropped events - are now exposed in the API to better monitor the DLQ behavior. #13923 #14058
-
To improve security of Logstash deployments, 8.3.0 brings a new setting "allow_superuser" that defaults to false. When enabled it prevents Logstash from starting as super user ("root"). This setting will be enabled by default in the future. Consider explicitly enabling it. Otherwise a deprecation log entry will be emitted. #14046 #14089
-
Continuing with the focus on security, we’ve introduced "api.auth.basic.password_policy.mode" to ensure the password used to guard Logstash’s HTTP API has a minimum set of strength requirements. By default a warning will be emitted if the defined password doesn’t meet the criteria, but in a future release the mode will be set to "ERROR". #14045 #14105 #14159
-
Elasticsearch introduced "security on by default" back in 8.0.0, with TLS enabled by default in its HTTP and transport layers. To facilitate connecting to 8.x clusters, Elasticsearch displays the fingerprint of the Certificate Authority it generates on startup. This release of Logstash introduces support for setting "ca_trusted_fingerprint" in Elasticsearch input, filter and outputs plugins. #14120
-
Technical Preview: Receiving events containing keys with characters that have special meaning to Logstash such as
[and](for field references) has always causes issues to data ingestion. A new setting in Technical Preview, disabled by default, called "config.field_reference.escape_style" was introduced to handle such special characters by escaping them. #14044
-
Don’t display values of password type settings in conditionals #13997
-
Introduce a retry mechanism in pipeline-to-pipeline instead of crashing #14027
-
Add thread safety around Puma startup/shutdown #14080
-
Add value converters for java.time classes #13972
-
Correct the class reference to the MetricNotFound exception #13970
-
Fix a possible corruption of Persistent Queue during a crash of the Logstash process #14165
Avro Codec - 3.4.0
-
Add
encodingoption to select the encoding of Avro payload, could bebinaryorbase64#39
Elasticsearch Filter - 3.12.0
-
Add support for
ca_trusted_fingerprintwhen run on Logstash 8.3+ #158
Fingerprint Filter - 3.4.0
-
Add support for 128bit murmur variant #66.
Azure_event_hubs Input - 1.4.4
-
Fix: Replace use of block with lambda to fix wrong number of arguments error on jruby-9.3.4.0 #75
Beats Input - 6.4.0
-
Feat: review and deprecate ssl protocol/cipher settings #450
Elasticsearch Input - 4.14.0
Http Input - 3.6.0
-
Feat: review and deprecate ssl protocol/cipher related settings #151
Jms Input - 3.2.2
-
Fix: Remove usage of
java_kind_of?to allow this plugin to be supported for versions of Logstash using jruby-9.3.x #54
S3 Input - 3.8.4
-
Refactoring, reuse code to manage
additional_settingsfrom mixin-aws #237
Sqs Input - 3.3.2
-
Fix an issue that prevented timely shutdown when subscribed to an inactive queue
-
Refactoring: used logstash-mixin-aws to leverage shared code to manage
additional_settings#64
Tcp Input - 6.3.0
-
Feat: ssl_supported_protocols (TLSv1.3) + ssl_cipher_suites #198
Jdbc Integration - 5.3.0
Kafka Integration - 10.12.0
Aws Mixin - 5.1.0
-
Add support for 'additional_settings' configuration options used by S3 and SQS input plugins #53.
-
Drop support for aws-sdk-v1
Elasticsearch Output - 11.6.0
S3 Output - 4.3.7
Tcp Output - 6.1.0
-
Feat: add support for TLS v1.3 #47
-
Fix: close server and client sockets on plugin close
-
Updated bundled JDK to 11.0.15+10 #14152
-
Avoid unnecessary thread synchronization when the Persistent Queue is full #14141
-
Added mandatory JVM option to avoid strict path checking introduced with recent JVM versions, starting from 11.0.15+10, 17.0.3+7. #14066
-
Fixed Dead Letter Queue bug happening in position retrieval and restore. This happened when the DLQ input plugin used
commit_offsetfeature. #14093 -
Fixes an issue where custom java plugins were unable to be installed and run correctly when retrieved from rubygems.org. #14060
-
Fixed no metrics update issue when PQ is draining. #13935
Cef Codec - 6.2.5
-
[DOC] Update link to CEF implementation guide #97
Dns Filter - 3.1.5
-
Fixed an issue where a non-string value existing in the resolve/reverse field could cause the plugin to crash #65
Grok Filter - 4.4.2
-
Clarify the definition of matches that depend on previous captures #169
Http Filter - 1.4.1
-
Fix: don’t process response body for HEAD requests #40
Beats Input - 6.3.1
-
Fix: Removed use of deprecated
importof java classes in ruby #449
File Input - 4.4.2
Http Input - 3.5.1
-
Fix: codecs provided with
additional_codecsnow correctly run in the pipeline’s context, which means that they respect thepipeline.ecs_compatibilitysetting #152
Jdbc Integration - 5.2.5
-
Fix: do not execute more queries with debug logging #109
Core Patterns - 4.3.3
-
Fix: parsing x-edge-location in CLOUDFRONT_ACCESS_LOG (ECS mode) #311
-
Improved resiliency of Central Management requests when an Elasticsearch node is down #13689 #13941
-
Ensure safe retrieval of queue stats that may not yet be populated #13942
-
Print bundled JDK’s version in launch scripts when
LS_JAVA_HOMEis provided #13880 -
Updated jackson-databind to 2.13.2 in ingest-converter tool #13900
-
Updated google-java-format dependency to 1.13.0 and guava to 31.0.1 in core #13700
-
Multiple documentation improvements related to: Logstash to Logstash communication #13999, docker variable injection #12198, LS-ES security configuration #14012, JDK 11 Bundling #14022, and other overall documentation restructuring #14015.
Http Filter - 1.4.0
-
Feat: added ssl_supported_protocols option #38
Kv Filter - 4.7.0
-
Allow attaching multiple tags on failure. The
tag_on_failureoption now also supports an array of strings #92
Beats Input - 6.3.0
-
Added support for TLSv1.3. #447
Elasticsearch Input - 4.12.3
-
Fix: update Elasticsearch Ruby client to correctly customize 'user-agent' header #171
Http Input - 3.5.0
-
Feat: TLSv1.3 support #146
Http_poller Input - 5.3.0
-
Feat: added ssl_supported_protocols option #133
Sqs Input - 3.3.0
-
Feature: Add
additional_settingsoption to fine-grain configuration of AWS client #61
Kafka Integration - 10.10.0
-
Added config setting to enable 'zstd' compression in the Kafka output #112
Http_client Mixin - 7.2.0
-
Feat: add
ssl_supported_protocolsoption #40
Http Output - 5.5.0
-
Fixed issue where Logstash crashed if Central Management couldn’t reach Elasticsearch #13689
Cef Codec - 6.2.4
-
[DOC] Emphasize importance of delimiter setting for byte stream inputs #95
Geoip Filter - 7.2.12
-
[DOC] Add
http_proxyenvironment variable for GeoIP service endpoint. The feature is included in 8.1.0, and was back-ported to 7.17.2 #207
Truncate Filter - 1.0.5
-
Switches behavior of add_tag and add_field, now tags and fields are added only when the truncation happens on any field or nested field #7.
Tcp Output - 6.0.2
-
Fix: unable to start with password protected key #45
-
The
bin/logstash-plugin uninstall <plugin>command works as expected, successfully uninstalling the specified plugin #13823 -
Logstash CLI tools are now able to use the selected JDK on Windows #13839
-
Logstash can successfully locate the Windows JVM, even if the path includes spaces #13881
-
The GeoIP database lookup will now respect a proxy defined with the http_proxy environment variable. #13840
-
The version of the bundled JDK has been updated to 11.0.14.1+1. #13869
Dissect Filter - 1.2.5
-
Fix: bad padding
→suffix with delimiter #84
Elasticsearch Filter - 3.11.1
-
Fix: hosts ⇒ "es_host:port" regression #156
Dead_letter_queue Input - 1.1.11
Elasticsearch Input - 4.12.2
-
Fix: hosts ⇒ "es_host:port" regression #168
Http_poller Input - 5.2.1
-
Deps: unpin rufus-scheduler dependency #132
Jdbc Integration - 5.2.4
Tcp Output - 6.0.1
-
Fix: logging fail retry to stdout #43
-
Fix: Use
reconnect_intervalwhen establish a connection
Uninstalling a plugin using bin/logtash-plugin uninstall may
result in an error:
Gem::LoadError: You have already activated jruby-openssl 0.12.2, but your Gemfile requires jruby-openssl 0.12.1. Prepending `bundle exec` to your command may solve this.Logstash should still run, and other plugin operations, such as update and install, should work as expected.
|
Note
|
The bin/logstash-plugin list command may fail with the same error after a failed uninstallation.
|
Resolution
A successful plugin update will resolve this issue, and allow subsequent uninstall and list operations to
work without issue.
The filter-dissect plugin has recent changes available for update.
Running bin/logstash-plugin update logstash-filter-dissect should mitigate this issue.
Http Filter - 1.3.0
-
Feat: support ssl_verification_mode option #37
Kv Filter - 4.6.0
-
Added
allow_empty_valuesoption #72
Http_poller Input - 5.2.0
-
Feat: support ssl_verification_mode option #131
Sqs Input - 3.2.0
-
Feature: Add
queue_owner_aws_account_idparameter for cross-account queues #60
Elastic_enterprise_search Integration - 2.2.1
-
Fix, change implementation of connectivity check method to be compatible with version
v8.0+of Workplace Search #16 -
Feature, switch the connection library to elastic-enterprise-search #3
-
[DOC] Added required parameters to Workplace Search example snippet and describe little better what’s expected in url parameter #11
Http_client Mixin - 7.1.0
-
Feat: add
ssl_verification_mode#39
Http Output - 5.3.0
-
Feat: support ssl_verification_mode option #126
-
Fixed monitoring incompatibility on Windows where the CPU metric was not available. #13727
-
Recently, users running
bin/logstash-pluginto install or update plugins stumbled upon an issue that would prevent Logstash from starting due a third-party dependency update. The dependency was pinned to an older version. #13777 -
Logstash startup and the
pqrepair/pqchecktools have been improved to handle corrupted files in case of an unexpected shutdown. #13692 #13721
Dissect Filter - 1.2.5
-
Fix bad padding
→suffix with delimiter #84
Elasticsearch Filter - 3.11.1
-
Fix: hosts ⇒ "es_host:port" regression #156
Beats Input - 6.2.6
-
Update guidance regarding the private key format and encoding #445
Dead_letter_queue Input - 1.1.10
Tcp Input - 6.2.7
-
Build: skip shadowing jar dependencies #187
-
plugin no longer shadows dependencies into its logstash-input-tcp.jar
-
log4j-api is now a provided dependency and is no longer packaged with the plugin
-
Jdbc Integration - 5.2.3
-
Performance: avoid contention on scheduler execution #103
Tcp Output - 6.0.1
-
Fixed logging fail retry to stdout #43
-
Fixed to use
reconnect_intervalwhen establish a connection
The following list are changes in 8.0.0 as compared to 7.17.0, and combines release notes from the 8.0.0-alpha1, -alpha2, -beta1, -rc1 and -rc2 releases.
-
Many plugins can now be run in a mode that avoids implicit conflict with the Elastic Common Schema (ECS). This mode is controlled individually with each plugin’s ecs_compatibility option, which defaults to the value of the Logstash pipeline.ecs_compatibility setting. In Logstash 8, this compatibility mode will be on-by-default for all pipelines. If you wish to lock in a pipeline’s behavior from Logstash 7.x before upgrading to Logstash 8, you can set
pipeline.ecs_compatibility: disabledto its definition inpipelines.yml(or globally inlogstash.yml). -
Starting from Logstash 8.0, the minimum required version of Java to run Logstash is Java 11. By default, Logstash will run with the bundled JDK, which has been verified to work with each specific version of Logstash, and generally provides the best performance and reliability.
-
Support for using
JAVA_HOMEto override the path to the JDK that Logstash runs with has been removed for this release. In the8.xrelease, users should set the value ofLS_JAVA_HOMEto the path of their preferred JDK if they wish to use a version other than the bundled JDK. The value ofJAVA_HOMEwill be ignored. -
The Java Execution Engine has been the default engine since Logstash 7.0, and works with plugins written in either Ruby or Java. Removal of the Ruby Execution Engine will not affect the ability to run existing pipelines. #12517
-
We have added support for UTF-16 and other multi-byte-character when reading log files. #9702
-
Setting
config.field_reference.parserhas been removed. The Field Reference parser interprets references to fields in your pipelines and plugins. Its behavior was configurable in 6.x, and since 7.0 allowed only a single option:strict. 8.0 no longer recognizes the setting, but maintains the same behavior as thestrictsetting. {ls} rejects ambiguous and illegal inputs as standard behavior. #12466
For a more detailed view of these changes please check [breaking-8.0].
-
As processing times speed up, millisecond granularity is not always enough. Inbound data increasingly has sub-millisecond granularity timestamps. The pull request #12797 allows the internal mechanisms of Logstash that hold moment-in-time data - such as the Logstash Event, the Persistent Queue, the Dead Letter Queue and JSON encoding/decoding - to have nanosecond granularity.
-
We have added another flag to the Benchmark CLI to allow passing a data file with previously captured data to the custom test case. This feature allows users to run the Benchmark CLI in a custom test case with a custom config and a custom dataset. #12437
Logstash 8.0.0 includes the same versions of all bundled plugins as Logstash 7.17.0. If you upgrade to 7.17 before upgrading to 8.0 (as recommended), you won’t see any changes to plugin versions.
Clone Filter - 4.2.0
-
Added support for ECS v8 as alias for ECS v1 #27
Geoip Filter - 7.2.11
-
Improved compatibility with the Elastic Common Schema #206
-
Added support for ECS’s composite
region_iso_code(US-WA), which replaces the non-ECSregion_code(WA) as a default field with City databases. To get the stand-aloneregion_codein ECS mode, you must include it in thefieldsdirective -
[DOC] Improve ECS-related documentation
-
-
[DOC] Air-gapped environment requires both ASN and City databases #204
Http Filter - 1.2.1
Ruby Filter - 3.1.8
-
[DOC] Added doc to describe the option `tag_with_exception_message`https://github.com/logstash-plugins/logstash-filter-ruby/pull/62[#62]
-
Fix SyntaxError handling so other pipelines can shut down gracefully #64
Useragent Filter - 3.3.3
-
Docs: mention added fields in 3.3 with a note #78
Exec Input - 3.4.0
-
Feat: adjust fields for ECS compatibility #28
-
Plugin will no longer override fields if they exist in the decoded payload (It no longer sets the
hostfield if decoded from the command’s output)
Gelf Input - 3.3.1
-
Fix: safely coerce the value of
_@timestampto avoid crashing the plugin #67
Generator Input - 3.1.0
-
Feat: adjusted fields for ECS compatibility #22
-
Fix: do not override the host field if it’s present in the generator line (after decoding)
-
Fix: codec flushing when closing input
Imap Input - 3.2.0
-
Feat: ECS compatibility #55
-
added (optional)
headers_targetconfiguration option -
added (optional)
attachments_targetconfiguration option -
Fix: plugin should not close
$stdin, while being stopped
Jms Input - 3.2.1
-
Fix: improve compatibility with MessageConsumer implementations #51, such as IBM MQ.
-
Test: Fix test failures due to ECS compatibility default changes in
8.xof logstash #53 -
Feat: event_factory support + targets to aid ECS #49
-
Fix: when configured to add JMS headers to the event, headers whose value is not set no longer result in nil entries on the event
-
Fix: when adding the
jms_reply_toheader to an event, a string representation is set instead of an opaque object.
Pipe Input - 3.1.0
-
Feat: adjust fields for ECS compatibility #19
S3 Input - 3.8.3
Snmp Input - 1.3.1
Snmptrap Input - 3.1.0
-
Feat: ecs_compatiblity support + (optional) target #37
Syslog Input - 3.6.0
-
Add support for ECS v8 as alias to v1 implementation #68
Twitter Input - 4.1.0
-
Feat: optional target + ecs_compatibility #72
Unix Input - 3.1.1
Jdbc Integration - 5.2.2
-
Feat: name scheduler threads + redirect error logging #102
-
Refactor: isolate paginated normal statement algorithm in a separate handler #101
-
Added
jdbc_paging_modeoption to choose if useexplicitpagination in statements and avoid the initial count query or useautoto delegate to the underlying library #95 -
Several improvements to Java driver loading
-
Refactor: to explicit Java (driver) class name loading #96. The change is expected to provide a more robust fix for the driver loading issue #83.
NOTE: A fatal driver error will no longer keep reloading the pipeline and now leads to a system exit.
-
Fix: regression due returning the Java driver class #98
-
Kafka Integration - 10.9.0
-
Refactor: leverage codec when using schema registry Previously using
schema_registry_urlparsed the payload as JSON even ifcodec ⇒ 'plain'was explicitly set, this is no longer the case. #106
Cloudwatch Output - 3.0.10
-
Fix: an old undefined method error which would surface with load (as queue fills up)
-
Deps: unpin rufus scheduler #20
Elasticsearch Output - 11.4.1
-
Feat: upgrade manticore (http-client) library #1063
-
the underlying changes include latest HttpClient (4.5.13)
-
resolves an old issue with
ssl_certificate_verification ⇒ falsestill doing some verification logic
-
-
Updates ECS templates #1062
-
Updates v1 templates to 1.12.1 for use with Elasticsearch 7.x and 8.x
-
Updates BETA preview of ECS v8 templates for Elasticsearch 7.x and 8.x
-
-
Feat: add support for 'traces' data stream type #1057
-
Refactor: review manticore error handling/logging, logging originating cause in case of connection related error when debug level is enabled. Java causes on connection related exceptions will now be extra logged when plugin is logging at debug level #1029
-
ECS-related fixes #1046
-
Data Streams requirement on ECS is properly enforced when running on Logstash 8, and warned about when running on Logstash 7.
-
ECS Compatibility v8 can now be selected
-
Core Patterns - 4.3.2
-
Fix: typo in
BIN9_QUERYLOGpattern (in ECS mode) #307
-
Fixed long-standing issue in which the
events.outcount incorrectly included events that had been dropped with the drop filter. Now the total out event count includes only events that reach the out stage. #13593 -
Reduced scope and impact of a memory leak that can be caused by using UUIDs or other high-cardinality field names #13642
-
Fixed an issue with the Azure input plugin that caused Logstash to crash when the input was used in a pipeline. #13603
Many plugins can now be run in a mode that avoids implicit conflict with the Elastic Common Schema (ECS). This mode is controlled individually with each plugin’s ecs_compatibility option, which defaults to the value of the Logstash pipeline.ecs_compatibility setting. In Logstash 8, this compatibility mode will be on-by-default for all pipelines.
If you wish to lock in a pipeline’s behavior from Logstash 7.x before upgrading to Logstash 8, you can set pipeline.ecs_compatibility: disabled to its definition in pipelines.yml (or globally in logstash.yml).
Starting from Logstash 8.0, the minimum required version of Java to run Logstash is Java 11. By default, Logstash will run with the bundled JDK, which has been verified to work with each specific version of Logstash, and generally provides the best performance and reliability.
See Breaking changes for a preview of additional breaking changes coming your way.
As processing times speed up, millisecond granularity is not always enough. Inbound data increasingly has sub-millisecond granularity timestamps. The pull request #12797 allows the internal mechanisms of Logstash that hold moment-in-time data - such as the Logstash Event, the Persistent Queue, the Dead Letter Queue and JSON encoding/decoding - to have nanosecond granularity.
Timestamp precision is limited to the JVM and Platform’s available granularity, which in many cases is microseconds.
This change also grants users access to Java time’s improved formatters, which include support fort ISO quarters, week-of-month, and a variety of timezone/offset-related format substitutions. For example:
filter {
mutate {
add_field => {"nanos" => "Nanos: %{{n}}" }
}
}Results in the following event:
{
"@timestamp" => 2021-10-31T22:32:34.747968Z,
"host" => "logstash.lan",
"nanos" => "Nanos: 747968000",
"message" => "test",
"type" => "stdin",
"@version" => "1"
}Support for using JAVA_HOME to override the path to the JDK that Logstash runs with has been removed for this release.
In the 8.x release, users should set the value of LS_JAVA_HOME to the path of their preferred JDK if they
wish to use a version other than the bundled JDK. The value of JAVA_HOME will be ignored.
The Java Execution Engine has been the default engine since Logstash 7.0, and works with plugins written in either Ruby or Java. Removal of the Ruby Execution Engine will not affect the ability to run existing pipelines. #12517
We have added support for UTF-16 and other multi-byte-character when reading log files. #9702
The Field Reference parser interprets references to fields in your pipelines and plugins. It was configurable in 7.x, with the default set to strict to reject inputs that are ambiguous or illegal. Configurability is removed in 8.0. Now {ls} rejects ambiguous and illegal inputs as standard behavior. #12466
Option to pass custom data to the benchmark CLI
We have added another flag to the Benchmark CLI to allow passing a data file with previously captured data to the custom test case. This feature allows users to run the Benchmark CLI in a custom test case with a custom config and a custom dataset. #12437