Skip to content

Latest commit

 

History

History
1025 lines (943 loc) · 81.2 KB

CHANGELOG.next.asciidoc

File metadata and controls

1025 lines (943 loc) · 81.2 KB
Raw

Beats version HEAD

Breaking changes

Affecting all Beats

  • The document id fields has been renamed from @metadata.id to @metadata._id 15859

  • Update to Golang 1.12.1. 11330

  • Disable Alibaba Cloud and Tencent Cloud metadata providers by default. 12812

  • Libbeat: Do not overwrite agent.*, ecs.version, and host.name. 14407

  • Libbeat: Cleanup the x-pack licenser code to use the new license endpoint and the new format. 15091

  • Refactor metadata generator to support adding metadata across resources 14875

  • Remove AddDockerMetadata and AddKubernetesMetadata processors from the script processor. They can still be used as normal processors in the configuration. 16349 16514

  • Introduce APM libbeat instrumentation, active when running the beat with ELASTIC_APM_ACTIVE=true. 17938

  • Remove the non-ECS agent.hostname field. Use the agent.name or agent.id fields for an identifier. 16377 18328

  • Make error message about locked data path actionable. 18667

  • Ensure dynamic template names are unique for the same field. 18849

  • Remove the deprecated xpack.monitoring. settings. Going forward only monitoring. settings may be used. 9424 18608

  • Added certificate TLS verification mode to ignore server name mismatch. 12283 20293

  • Autodiscover doesn’t generate any configuration when a variable is missing. Previously it generated an incomplete configuration. 20898

  • Remove redundant cloudfoundry.*.timestamp fields. This value is set in @timestamp. 21175

  • Allow embedding of CAs, Certificate of private keys for anything that support TLS in ouputs and inputs. 21179

  • Update to Golang 1.12.1. 11330

  • Disable Alibaba Cloud and Tencent Cloud metadata providers by default. 12812

  • API address is a required setting in add_cloudfoundry_metadata. 21759

  • Update to ECS 1.7.0. 22571

  • Add support for SCRAM-SHA-512 and SCRAM-SHA-256 in Kafka output. 12867

  • Remove id_field_data 25239

  • Removed beats central management 25696, 23908

  • MacOSX minimum supported version set to 10.14 https://github.com/elastic/beats/issues/24193{24193}

Auditbeat

  • File integrity dataset (macOS): Replace unnecessary file.origin.raw (type keyword) with file.origin.text (type text). 12423 15630

  • Change event.kind=error to event.kind=event to comply with ECS. 18870 20685

  • Change network.direction values to ECS recommended values (inbound, outbound). 12445 20695

  • Docker container needs to be explicitly run as user root for auditing. 21202

  • File integrity dataset no longer includes the leading dot in file.extension values (e.g. it will report "png" instead of ".png") to comply with ECS. 21644

  • Use ECS 1.7 ingress/egress network directions instead of inbound/outbound. 22991

  • Use ingress/egress instead of inbound/outbound for ECS 1.7 in auditd module. 23000

Auditbeat

Filebeat

  • Fix parsing of Elasticsearch node name by elasticsearch/slowlog fileset. 14547

  • Improve ECS field mappings in panw module. event.outcome now only contains success/failure per ECS specification. 16025 17910

  • Improve ECS categorization field mappings for nginx module. http.request.referrer only populated when nginx sets a value 16174 17844

  • Improve ECS field mappings in santa module. move hash.sha256 to process.hash.sha256 & move certificate fields to santa.certificate . 16180 17982

  • With the default configuration the cloud modules (aws, azure, googlecloud, o365, okta) will no longer send the host field that contains information about the host Filebeat is running on. This is because the host field specifies the host on which the event happened. 13920 18223

  • With the default configuration the following modules will no longer send the host field that contains information about the host on which Filebeat is running. You can revert this change by configuring tags for the module and omitting forwarded from the list. 13920

    • CEF 18223

    • PANW 18223 forwarded from the list. 13920

    • Cisco 18753

    • CrowdStrike 19132

    • Fortinet 19133

    • iptables 18756

    • Checkpoint 18754

    • Netflow 19087

    • Zeek 19113 (forwarded tag is not included by default)

    • Suricata 19107 (forwarded tag is not included by default)

    • CoreDNS 19134 (forwarded tag is not included by default)

    • Envoy Proxy 19134 (forwarded tag is not included by default)

  • Preserve case of http.request.method. ECS prior to 1.6 specified normalizing to lowercase, which lost information. Affects filesets: apache/access, elasticsearch/audit, iis/access, iis/error, nginx/access, nginx/ingress_controller, aws/elb, suricata/eve, zeek/http. 18154 18359

  • Adds check on <no value> config option value for the azure input resource_manager_endpoint. 18890

  • Okta module now requires objects instead of JSON strings for the http_headers, http_request_body, pagination, rate_limit, and ssl variables. 18953

  • With the default configuration the cloud modules (aws, azure, googlecloud, o365, okta)

  • With the default configuration the cef and panw modules will no longer send the host forwarded from the list. 13920 18223

  • Preserve case of http.request.method. ECS prior to 1.6 specified normalizing to lowercase, which lost information. Affects filesets: apache/access, elasticsearch/audit, iis/access, iis/error, nginx/access, nginx/ingress_controller, aws/elb, suricata/eve, zeek/http. 18154 18359

  • Adds oauth support for httpjson input. 18415 18892

  • Adds split_events_by option to httpjson input. 19246

  • Adds date_cursor option to httpjson input. 19483

  • Adds Gsuite module with SAML support. 19329

  • Adds Gsuite User Accounts support. 19329

  • Adds Gsuite Login audit support. 19702

  • Adds Gsuite Admin support. 19769

  • Adds Gsuite Drive support. 19704

  • Adds Gsuite Groups support. 19725

  • Move file metrics to dataset endpoint 19977

  • Add while_pattern type to multiline reader. 19662

  • Tracking session end reason in panw module. 18705

  • Fix PANW field spelling "veredict" to "verdict" on event.action 18808

  • Removed experimental modules citrix, kaspersky, rapid7 and tenable. 20706

  • Add support for GMT timezone offsets in decode_cef. 20993

  • Fix parsing of Elasticsearch node name by elasticsearch/slowlog fileset. 14547

  • API address and shard ID are required settings in the Cloud Foundry input. 21759

  • Remove suricata.eve.timestamp alias field. 10535 22095

  • Rename bad ECS field name tracing.trace.id to trace.id in aws elb fileset. 22571

  • Fix parsing issues with nested JSON payloads in Elasticsearch audit log fileset. 22975

  • Rename network.direction values in crowdstrike/falcon to ingress/egress. 23041

  • Change logging in logs input to structure logging. Some log message formats have changed. 25299

Heartbeat - Add support for screenshot blocks and use newer synthetics flags that only works in newer synthetics betas. 25808

Journalbeat

  • Improve parsing of syslog.pid in journalbeat to strip the username when present 16116

Metricbeat

  • Make use of secure port when accessing Kubelet API 16063

  • Add Tomcat overview dashboard 14026

  • Move service config under metrics and simplify metric types. 18691

  • Fix ECS compliance of user.id field in system/users metricset 19019

  • Remove "invalid zero" metrics on Windows and Darwin, don’t report linux-only memory and diskio metrics when running under agent. 21457

  • Change cloud.provider from googlecloud to gcp. 21775

  • API address and shard ID are required settings in the Cloud Foundry module. 21759

  • Rename googlecloud module to gcp module. 22246

  • Fix ECS compliance of user.id field in system/users metricset 19019

  • Use ingress/egress instead of inbound/outbound for system/socket metricset. 22992

  • Add new dashboard for VSphere host cluster and virtual machine 14135

  • kubernetes.container.cpu.limit.cores and kubernetes.container.cpu.requests.cores are now floats. 11975

  • Change types of numeric metrics from Kubelet summary api to double so as to cover big numbers. 23335

  • Add container.image.name and containe.name ECS fields for state_container. 23802

  • Add support for the MemoryPressure, DiskPressure, OutOfDisk and PIDPressure status conditions in state_node. 23905

  • Remove xpack enabled flag on ES, Logstash, Beats and Kibana 24427

  • Adjust host fields to adopt new names from 1.9.0 ECS. 24312

  • Add replicas.ready field to state_statefulset in Kubernetes modulehttps://github.com/elastic/pull/26088[26088]

Packetbeat

  • Redis: fix incorrectly handle with two-words redis command. 14872 14873

  • event.category no longer contains the value network_traffic because this is not a valid ECS event category value. 20556

  • Added redact_headers configuration option, to allow HTTP request headers to be redacted whilst keeping the header field included in the beat. 15353

  • Add dns.question.subdomain and dns.question.top_level_domain fields. 14578

Winlogbeat

  • Add support to Sysmon file delete events (event ID 23). 18094

  • Improve ECS field mappings in Sysmon module. related.hash, related.ip, and related.user are now populated. 18364

  • Improve ECS field mappings in Sysmon module. Hashes are now also populated to the corresponding process.hash, process.pe.imphash, file.hash, or file.pe.imphash. 18364

  • Improve ECS field mappings in Sysmon module. file.name, file.directory, and file.extension are now populated. 18364

  • Improve ECS field mappings in Sysmon module. rule.name is populated for all events when present. 18364

  • Add Powershell module. Support for event ID’s: 400, 403, 600, 800, 4103, 4014, 4105, 4106. 16262 18526

  • Fix Powershell processing of downgraded engine events. 18966

  • Fix unprefixed fields in fields.yml for Powershell module 18984

  • Remove top level hash property from sysmon events 20653

  • Use ECS 1.7 ingress/egress instead of inbound/outbound network.direction in sysmon. 22997

Functionbeat

Bugfixes

Affecting all Beats

  • Fix add_cloud_metadata to better support modifying sub-fields with other processors. 13808

  • TLS or Beats that accept connections over TLS and validate client certificates. 14146

  • Fix panics that could result from invalid TLS certificates. This can affect Beats that connect over TLS, or Beats that accept connections over TLS and validate client certificates. 14146

  • Fix panic in the Logstash output when trying to send events to closed connection. 15568

  • Fix a race condition with the Kafka pipeline client, it is possible that Close() get called before Connect() . 11945

  • Allow users to configure only cluster_uuid setting under monitoring namespace. 14338

  • Update replicaset group to apps/v1 15802

  • Fix missing output in dockerlogbeat 15719

  • Do not load dashboards where not available. 15802

  • Fix issue where TLS settings would be ignored when a forward proxy was in use. 15516

  • Update replicaset group to apps/v1 15802

  • Fix issue where default go logger is not discarded when either * or stdout is selected. 10251 15708

  • Upgrade go-ucfg to latest v0.8.1. 15937

  • Fix index names for indexing not always guaranteed to be lower case. 16081

  • Add ssl.ca_sha256 option to the supported TLS option, this allow to check that a specific certificate is used as part of the verified chain. 15717

  • Fix loading processors from annotation hints. 16348

  • Fix an issue that could cause redundant configuration reloads. 16440

  • Fix k8s pods labels broken schema. 16480

  • Fix k8s pods annotations broken schema. 16554

  • Upgrade go-ucfg to latest v0.8.3. 16450

  • Fix NewContainerMetadataEnricher to use default config for kubernetes module. 16857

  • Improve some logging messages for add_kubernetes_metadata processor 16866

  • Fix k8s metadata issue regarding node labels not shown up on root level of metadata. 16834

  • Fail to start if httpprof is used and it cannot be initialized. 17028

  • Fix concurrency issues in convert processor when used in the global context. 17032

  • Fix bug with monitoring.cluster_uuid setting not always being exposed via GET /state Beats API. 16732 17420

  • Fix building on FreeBSD by removing build flags from add_cloudfoundry_metadata processor. 17486

  • Improve some logging messages for add_kubernetes_metadata processor https://github.com/elastic/beats/pull/16866{16866}

  • Do not rotate log files on startup when interval is configured and rotateonstartup is disabled. 17613

  • Fix goroutine leak and Elasticsearch output file descriptor leak when output reloading is in use. 10491 17381

  • Fix setup.dashboards.index setting not working. 17749

  • Fix Elasticsearch license endpoint URL referenced in error message. 17880 18030

  • Fix panic when assigning a key to a nil value in an event. 18143

  • Change decode_json_fields processor, to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

  • Gives monitoring reporter hosts, if configured, total precedence over corresponding output hosts. 17937 17991

  • Change decode_json_fields processor, to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

  • [Autodiscover] Check if runner is already running before starting again. 18564

  • Fix keystore add hanging under Windows. 18649 18654

  • Fix an issue where error messages are not accurate in mapstriface. 18662 18663

  • Fix regression in add_kubernetes_metadata, so configured indexers and matchers are used if defaults are not disabled. 18481 18818

  • Fix the translate_sid processor’s handling of unconfigured target fields. 18990 18991

  • Fixed a service restart failure under Windows. 18914 18916

  • Fix metrics hints builder to avoid wrong container metadata usage when port is not exposed 18979

  • Server-side TLS config now validates certificate and key are both specified 19584

  • Fix terminating pod autodiscover issue. 20084

  • Fix seccomp policy for calls to chmod and chown. 20054

  • Output errors when Kibana index pattern setup fails. 20121

  • Fix issue in autodiscover that kept inputs stopped after config updates. 20305

  • Log debug message if the Kibana dashboard can not be imported from the archive because of the invalid archive directory structure 12211, 13387

  • Add service resource in k8s cluster role. 20546

  • [Metricbeat][Kubernetes] Change cluster_ip field from ip to keyword. 20571

  • Rename cloud.provider az value to azure inside the add_cloud_metadata processor. 20689

  • Add missing country_name geo field in add_host_metadata and add_observer_metadata processors. 20796 20811

  • [Autodiscover] Handle input-not-finished errors in config reload. 20915

  • Explicitly detect missing variables in autodiscover configuration, log them at the debug level. 20568 20898

  • Fix libbeat.output.write.bytes and libbeat.output.read.bytes metrics of the Elasticsearch output. 20752 21197

  • The o365input and o365 module now recover from an authentication problem or other fatal errors, instead of terminating. 21258

  • Orderly close processors when processing pipelines are not needed anymore to release their resources. 16349

  • Fix memory leak and events duplication in docker autodiscover and add_docker_metadata. 21851

  • Fixed documentation for commands in beats dev guide 22194

  • Fix parsing of expired licences. 21112 22180

  • Fix duplicated pod events in kubernetes autodiscover for pods with init or ephemeral containers. 22438

  • Fix FileVersion contained in Windows exe files. 22581

  • Fix index template loading when the new index format is selected. 22482 22682

  • Periodic metrics in logs will now report libbeat.output.events.active and beat.memstats.rss as gauges (rather than counters). 22877

  • Use PROGRAMDATA environment variable instead of C:\ProgramData for windows install service 22874

  • Fix reporting of cgroup metrics when running under Docker 22879

  • Fix typo in config docs 23185

  • Add FAQ entry for madvdontneed variable 23429

  • Fix panic due to unhandled DeletedFinalStateUnknown in k8s OnDelete 23419

  • Fix error loop with runaway CPU use when the Kafka output encounters some connection errors 23484

  • Fix ILM setup log reporting that a policy or an alias was created, even though the creation of any resource was disabled. 24046 24480

  • Fix ILM alias not being created if setup.ilm.check_exists: false and setup.ilm.overwrite: true has been configured. 24480

  • Fix issue discovering docker containers and metadata after reconnections 24318

  • Allow cgroup self-monitoring to see alternate hostfs paths 24334

  • Fix 'make setup' instructions for a new beat 24944

  • Fix out of date FreeBSD vagrantbox. 25652

  • Fix handling of file_selectors in aws-s3 input. 25792

  • Fix ILM alias creation when write alias exists and initial index does not exist 26143

  • Include date separator in the filename prefix of dateRotator to make sure nothing gets purged accidentally 26176

Auditbeat

  • system/socket: Fixed compatibility issue with kernel 5.x. 15771

  • system/package: Fix parsing of Installed-Size field of DEB packages. 16661 17188

  • system module: Fix panic during initialisation when /proc/stat can’t be read. 17569

  • system/package: Fix an error that can occur while trying to persist package metadata. 18536 18887

  • system/socket: Fix dataset using 100% CPU and becoming unresponsive in some scenarios. 19033 19764

  • system/socket: Fixed tracking of long-running connections. 19033

  • system/package: Fix librpm loading on Fedora 31/32. NNNN

  • file_integrity: Create fsnotify watcher only when starting file_integrity module 19505

  • auditd: Fix spelling of anomaly in event.category.

  • auditd: Fix typo in event.action of removed-user-role-from. 19300

  • auditd: Fix typo in event.action of used-suspicious-link. 19300

  • system/socket: Fix kprobe grouping to allow running more than one instance. 20325

  • system/socket: Fixed a crash due to concurrent map read and write. 21192 21690

  • file_integrity: stop monitoring excluded paths 21278 21282

  • auditd: Fix an error condition causing a lot of audit_send_reply kernel threads being created. 22673

  • system/socket: Fixed start failure when run under config reloader. 20851 21693

  • system/socket: Having some CPUs unavailable to Auditbeat could cause startup errors or event loss. 22827

  • Note incompatibility of system/socket on ARM. 23381

  • auditd: Fix kernel deadlock when netlink congestion causes "no buffer space available" errors. 26031 26032

Filebeat

  • Fix mapping of fortinet.firewall.mem as integer. 19335

  • Ensure all zeek timestamps include millisecond precision. 14599 16766

  • Fix s3 input hanging with GetObjectRequest API call by adding context_timeout config. 15502 15590

  • Add shared_credential_file to cloudtrail config 15652 15656

  • Fix typos in zeek notice fileset config file. 15764 15765

  • Fix mapping error when zeek weird logs do not contain IP addresses. 15906

  • Improve elasticsearch/audit fileset to handle timestamps correctly. 15942

  • Prevent Elasticsearch from spewing log warnings about redundant wildcards when setting up ingest pipelines for the elasticsearch module. 15840 15900

  • Fix mapping error for cloudtrail additionalEventData field 16088

  • Fix a connection error in httpjson input. 16123

  • Fix integer overflow in S3 offsets when collecting very large files. 22523

  • Fix CredentialsJSON unpacking for gcp-pubsub and httpjson inputs. 23277

  • Fix issue with m365_defender, when parsing incidents that has no alerts attached: 25421

Filebeat

  • cisco/asa fileset: Fix parsing of 302021 message code. 14519

  • Fix filebeat azure dashboards, event category should be Alert. 14668

  • Fixed dashboard for Cisco ASA Firewall. 15420 15553

  • Add shared_credential_file to cloudtrail config 15652 15656

  • Fix s3 input with cloudtrail fileset reading json file. 16374 16441

  • Rewrite azure filebeat dashboards, due to changes in kibana. 16466

  • Adding the var definitions in azure manifest files, fix for errors when executing command setup. 16270 16468

  • Fix merging of fileset inputs to replace paths and append processors. 16450

  • Add queue_url definition in manifest file for aws module. 16640

  • Fix issue where autodiscover hints default configuration was not being copied. 16987

  • Fix Elasticsearch _id field set by S3 and Google Pub/Sub inputs. 17026

  • Add queue_url definition in manifest file for aws module. https://github.com/elastic/beats/pull/16640{16640}

  • Fixed various Cisco FTD parsing issues. 16863 16889

  • Fix default index pattern in IBM MQ filebeat dashboard. 17146

  • Fix elasticsearch.gc fileset to not collect all logs when Elasticsearch is running in Docker. 13164 16583 17164

  • Fixed a mapping exception when ingesting CEF logs that used the spriv or dpriv extensions. 17216 17220

  • CEF: Fixed decoding errors caused by trailing spaces in messages. 17253

  • Fixed a mapping exception when ingesting Logstash plain logs (7.4+) with pipeline ids containing non alphanumeric chars. 17242 17243

  • Fixed MySQL slowlog module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. 17086 17156

  • Fix elasticsearch.audit data ingest pipeline to be more forgiving with date formats found in Elasticsearch audit logs. 17406

  • CEF: Fixed decoding errors caused by trailing spaces in messages. 17253

  • Fixed activemq module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. 17428

  • Remove migrationVersion map 7.7.0 reference from Kibana dashboard file to fix backward compatibility issues. 17425

  • Fix issue 17734 to retry on rate-limit error in the Filebeat httpjson input. 17734 17735

  • Fixed cloudfoundry.access to have the correct cloudfoundry.app.id contents. 17847

  • Fixing ingress_controller. fields to be of type keyword instead of text. 17834

  • Fixed typo in log message. 17897

  • Fix Cisco ASA ASA 3020** and 106023 messages 17964

  • Unescape file name from SQS message. 18370

  • Improve cisco asa and ftd pipelines' failure handler to avoid mapping temporary fields. 18391 18392

  • Fix source.address not being set for nginx ingress_controller 18511

  • Fix PANW module wrong mappings for bytes and packets counters. 18522 18525

  • Fixed ingestion of some Cisco ASA and FTD messages when a hostname was used instead of an IP for NAT fields. 14034 18376

  • Fix a rate limit related issue in httpjson input for Okta module. 18530 18534

  • Fix googlecloud.audit pipeline to only take in fields that are explicitly defined by the dataset. 18465 18472

  • Fix o365.audit failing to ingest events when ip address is surrounded by square brackets. 18587 18591

  • Fix Kubernetes Watcher goroutine leaks when input config is invalid and input.reload is enabled. 18629 18630

  • Okta module now sets the Elasticsearch _id field to the Okta UUID value contained in each system log to minimize the possibility of duplicating events. 18953

  • Fix improper nesting of session_issuer object in aws cloudtrail fileset. 18894 18915

  • Fix o365 module ignoring var.api settings. 18948

  • Fix netflow module to support 7 bytepad for IPFIX template. 18098

  • Fix Cisco ASA dissect pattern for 313008 & 313009 messages. 19149

  • Fix date and timestamp formats for fortigate module 19316

  • Fix memory leak in tcp and unix input sources. 19459

  • Add missing default_field: false to aws filesets fields.yml. 19568

  • Fix tls mapping in suricata module 19492 19494

  • Update container name for the azure filesets. 19899

  • Fix o365 module ignoring var.api settings. 18948

  • Fix improper nesting of session_issuer object in aws cloudtrail fileset. 18894 18915

  • Fix Cisco ASA ASA 3020** and 106023 messages 17964

  • Add missing default_field: false to aws filesets fields.yml. 19568

  • Fix bug with empty filter values in system/service 19812

  • Fix S3 input to trim delimiter /n from each log line. 19972

  • Ignore missing in Zeek module when dropping unecessary fields. 19984

  • Fix auditd module syscall table for ppc64 and ppc64le. 20052

  • Fix Filebeat OOMs on very long lines 19500, 19552

  • Fix s3 input parsing json file without expand_event_list_from_field. 19902 19962 20370

  • Fix millisecond timestamp normalization issues in CrowdStrike module 20035, 20138

  • Fix support for message code 106100 in Cisco ASA and FTD. 19350 20245

  • Fix event.outcome logic for azure/siginlogs fileset 20254

  • Fix fortinet setting event.timezone to the system one when no tz field present 20273

  • Fix okta geoip lookup in pipeline for destination.ip 20454

  • Fix mapping exception in the googlecloud/audit dataset pipeline. 18465 20465

  • Fix cisco asa and ftd parsing of messages 106102 and 106103. 20469

  • Fix event.kind for system/syslog pipeline 20365 20390

  • Clone value when copy fields in processors to avoid crash. 19206 20500

  • Fix event.type for zeek/ssl and duplicate event.category for zeek/connection 20696

  • Fix long registry migration times. 20717 20705

  • Fix event types and categories in auditd module to comply with ECS 20652

  • Update documentation in the azure module filebeat. 20815

  • Provide backwards compatibility for the set processor when Elasticsearch is less than 7.9.0. 20908

  • Remove wrongly mapped tls.client.server_name from fortinet/firewall fileset. 20983

  • Fix an error updating file size being logged when EOF is reached. 21048

  • Fix error when processing AWS Cloudtrail Digest logs. 21086 20943

  • Handle multiple upstreams in ingress-controller. 21215

  • Provide backwards compatibility for the append processor when Elasticsearch is less than 7.10.0. 21159

  • Fix checkpoint module when logs contain time field. 20567

  • Add field limit check for AWS Cloudtrail flattened fields. 21388 21382

  • Fix syslog RFC 5424 parsing in the CheckPoint module. 21854

  • Fix incorrect connection state mapping in zeek connection pipeline. 22151 22149

  • Fix handing missing eventtime and assignip field being set to N/A for fortinet module. 22361

  • Fix Zeek dashboard reference to zeek.ssl.server.name field. 21696

  • Fix for field [source] not present as part of path [source.ip] error in azure pipelines. 22377

  • Drop aws.vpcflow.pkt_srcaddr and aws.vpcflow.pkt_dstaddr when equal to "-". 22721 22716

  • Fix cisco umbrella module config by adding input variable. 22892

  • Fix network.direction logic in zeek connection fileset. 22967

  • Convert the o365 module’s client.port and source.port to numbers (from strings) in events. 22939

  • Fix Cisco ASA/FTD module’s parsing of WebVPN log message 716002. 22966

  • Fix aws s3 overview dashboard. 23045

  • Fix bad network.direction values in Fortinet/firewall fileset. 23072

  • Add support for organization and custom prefix in AWS/CloudTrail fileset. 23109 23126

  • Simplify regex for organization custom prefix in AWS/CloudTrail fileset. 23203 23204

  • Fix syslog header parsing in infoblox module. 23272 23273

  • Fix concurrent modification exception in Suricata ingest node pipeline. 23534

  • Fix handling of ModifiedProperties field in Office 365. 23777

  • Fix gcp/vpcflow module error where input type was defaulting to file. 24719

  • Change checkpoint.source_object from Long to Keyword. 25124 25145

  • Fix s3 input when there is a blank line in the log file. 25357

  • Fix Nginx module pipelines. 19088 24699

  • Remove space from field sophos.xg.trans_src_ ip. 25154 25250

  • Fix checkpoint.action_reason when its a string, not a Long. 25575 25609

  • Fix fortinet.firewall.addr when its a string, not an IP address. 25585 25608

  • Fix incorrect field name appending to related.hash in threatintel.abusechmalware ingest pipeline. 25151 25674

  • Add improvements to the azure activitylogs and platformlogs ingest pipelines. 26148

  • Fix kibana.log pipeline when event.duration calculation becomes a Long. 24556 25675

  • o365: Avoid mapping exception for Parameters and ExtendedProperties fields of string type. 26164

Heartbeat

  • Fixed excessive memory usage introduced in 7.5 due to over-allocating memory for HTTP checks. 15639

  • Fixed TCP TLS checks to properly validate hostnames, this broke in 7.x and only worked for IP SANs. 17549

  • Add Context to otherwise ambiguous HTTP body read errors. 25499

Journalbeat

Metricbeat

  • Fix checking tagsFilter using length in cloudwatch metricset. 14525

  • Fixed bug with elasticsearch/cluster_stats metricset not recording license expiration date correctly. 14541 14591

  • Log bulk failures from bulk API requests to monitoring cluster. 14303 14356

  • Fixed bug with elasticsearch/cluster_stats metricset not recording license ID in the correct field. 14592

  • Fix skipping protocol scheme by light modules. pull

  • Made logstash-xpack module once again have parity with internally-collected Logstash monitoring data. 16198

  • Revert changes in docker module: add size flag to docker.container. 16600

  • Fix detection and logging of some error cases with light modules. 14706

  • Fix imports after PR was merged before rebase. 16756

  • Add dashboard for redisenterprise module. 16752

  • Dynamically choose a method for the system/service metricset to support older linux distros. 16902

  • Reduce memory usage in elasticsearch/index metricset. 16503 16538

  • Check if CCR feature is available on Elasticsearch cluster before attempting to call CCR APIs from elasticsearch/ccr metricset. 16511 17073

  • Use max in k8s overview dashboard aggregations. 17015

  • Fix Disk Used and Disk Usage visualizations in the Metricbeat System dashboards. 12435 17272

  • Fix missing Accept header for Prometheus and OpenMetrics module. 16870 17291

  • Further revise check for bad data in docker/memory. 17400

  • Fix issue in Jolokia module when mbean contains multiple quoted properties. 17375 17374

  • Combine cloudwatch aggregated metrics into single event. 17345

  • Fix issue in Jolokia module when mbean contains multiple quoted properties. 17375 17374

  • Further revise check for bad data in docker/memory. 17400

  • Fix how we filter services by name in system/service 17400

  • Fix cloudwatch metricset missing tags collection. 17419 17424

  • check if cpuOptions field is nil in DescribeInstances output in ec2 metricset. 17418

  • Fix aws.s3.bucket.name terms_field in s3 overview dashboard. 17542

  • Fix Unix socket path in memcached. 17512

  • Fix azure storage dashboards. 17590

  • Metricbeat no longer needs to be started strictly after Logstash for logstash-xpack module to report correct data. 17261 17497

  • Fix pubsub metricset to collect all GA stage metrics from gcp stackdriver. 17154 17600

  • Add privileged option so as mb to access data dir in Openshift. 17606

  • Fix "ID" event generator of Google Cloud module 17160 17608

  • Add privileged option for Auditbeat in Openshift 17637

  • Fix storage metricset to allow config without region/zone. 17623 17624

  • Fix overflow on Prometheus rates when new buckets are added on the go. 17753

  • Remove specific win32 api errors from events in perfmon. 18292 18361

  • Fix application_pool metricset after pdh changes. 18477

  • Fix tags_filter for cloudwatch metricset in aws. 18524

  • Fix panic on metricbeat test modules when modules are configured in metricbeat.modules. 18789 18797

  • Fix getting gcp compute instance metadata with partial zone/region in config. 18757

  • Add missing network.sent_packets_count metric into compute metricset in googlecloud module. 18802

  • Fix compute and pubsub dashboard for googlecloud module. 18962 18980

  • Fix crash on vsphere module when Host information is not available. 18996 19078

  • Fix incorrect usage of hints builder when exposed port is a substring of the hint 19052

  • Remove dedot for tag values in aws module. 19112 19221

  • Stop counterCache only when already started 19103

  • Fix empty field name errors in the application pool metricset. 19537

  • Set tags correctly if the dimension value is ARN 19111 19433

  • Fix bug incorrect parsing of float numbers as integers in Couchbase module 18949 19055

  • Fix mapping of service start type in the service metricset, windows module. 19551

  • Fix config example in the perfmon configuration files. 19539

  • Add missing info about the rest of the azure metricsets in the documentation. 19601

  • Fix k8s scheduler compatibility issue. 19699

  • Fix SQL module mapping NULL values as string 18955 18898

  • Add support for azure light metricset app_stats. 20639

  • Fix ec2 disk and network metrics to use Sum statistic method. 20680

  • Fill cloud.account.name with accountID if account alias doesn’t exist. 20736

  • The Kibana collector applies backoff when errored at getting usage stats 20772

  • Update fields.yml in the azure module, missing metrics field. 20918

  • The elasticsearch/index metricset only requests wildcard expansion for hidden indices if the monitored Elasticsearch cluster supports it. 20938

  • Disable Kafka metricsets based on Jolokia by default. They require a different configuration. 20989

  • Fix panic index out of range error when getting AWS account name. 21101 21095

  • Handle missing counters in the application_pool metricset. 21071

  • Fix timestamp handling in remote_write. 21166

  • Fix remote_write flaky test. 21173

  • Visualization title fixes in aws, azure and googlecloud compute dashboards. 21098

  • Add a switch to the driver definition on SQL module to use pretty names 17378

  • Fix retrieving resources by ID for the azure module. 21711 21707

  • Use timestamp from CloudWatch API when creating events. 21498

  • Report the correct windows events for system/filesystem 21758

  • Fix regular expression in windows/permfon. 22146 21125

  • Fix azure storage event format. 21845

  • Fix panic in kubernetes autodiscover related to keystores 21843 21880

  • [Kubernetes] Remove redundant dockersock volume mount 22009

  • Revert change to report process.memory.rss as process.memory.wss on Windows. 22055

  • Add a switch to the driver definition on SQL module to use pretty names 17378

  • Fix instance name in perfmon metricset. 22218 22261

  • Remove io.time from windows 22237

  • Add interval information to monitor metricset in azure. 22152

  • Change Session ID type from int to string 22359

  • Fix filesystem types on Windows in filesystem metricset. 22531

  • Fix failiures caused by custom beat names with more than 15 characters 22550

  • Stop generating NaN values from Cloud Foundry module to avoid errors in outputs. 22634

  • Update NATS dashboards to leverage connection and route metricsets 22646

  • Fix rate metrics in Kafka broker metricset by using last minute rate instead of mean rate. 22733

  • Fix logstash module when xpack.enabled: true is set from emitting redundant events. 22808

  • Fix SQL module mapping NULL values as string 18955 elastic#18898[18898

  • Modify doc for app_insights metricset to contain example of config. 20185

  • Add required option for metrics in app_insights. 20406

  • Groups same timestamp metric values to one event in the app_insights metricset. 20403

  • Add support for azure light metricset app_stats. 20639

  • Fix remote_write flaky test. 21173

  • Remove io.time from windows 22237

  • Change vsphere.datastore.capacity.used.pct value to betweeen 0 and 1. 23148

  • Update config in windows.yml file. 23027https://github.com/elastic/beats/pull/23327[23327]

  • Fix metric grouping for windows/perfmon module 23489 23505

Packetbeat

Winlogbeat

  • Fix invalid IP addresses in DNS query results from Sysmon data. 18432 18436

  • Fields from Winlogbeat modules were not being included in index templates and patterns. 18983

  • Add source.ip validation for event ID 4778 in the Security module. 19627

  • Protect against accessing undefined variables in Sysmon module. 22219 22236

  • Protect against accessing an undefined variable in Security module. 22937

  • Fix related.ip field in renameCommonAuthFields 24892

Functionbeat

Elastic Logging Plugin

Added

Affecting all Beats

  • Decouple Debug logging from fail_on_error logic for rename, copy, truncate processors 12451

  • Allow a beat to ship monitoring data directly to an Elasticsearch monitoring cluster. 9260

  • Updated go-seccomp-bpf library to v1.1.0 which updates syscall lists for Linux v5.0. 11394

  • add_host_metadata is no GA. 13148

  • Add providers setting to add_cloud_metadata processor. 13812

  • Ensure that init containers are no longer tailed after they stop 14394

  • Fingerprint processor adds a new xxhash hashing algorithm 15418

  • Add configuration for APM instrumentation and expose the tracer trough the Beat object. 17938

  • Include network information by default on add_host_metadata and add_observer_metadata. 15347 16077

  • Add aws_ec2 provider for autodiscover. 12518 14823

  • Add support for multiple password in redis output. 16058 16206

  • Add support for Histogram type in fields.yml 16570

  • Remove experimental flag from setup.template.append_fields 16576

  • Add add_cloudfoundry_metadata processor to annotate events with Cloud Foundry application data. 16621

  • Add Kerberos support to Kafka input and output. 16781

  • Add add_cloudfoundry_metadata processor to annotate events with Cloud Foundry application data. 16621

  • Add support for kubernetes provider to recognize namespace level defaults 16321

  • Add translate_sid processor on Windows for converting Windows security identifier (SID) values to names. 7451 16013

  • Add capability of enrich container.id with process id in add_process_metadata processor 15947

  • Update RPM packages contained in Beat Docker images. 17035

  • Update supported versions of redis output. 17198

  • Update documentation for system.process.memory fields to include clarification on Windows os’s. 17268

  • Add replace processor for replacing string values of fields. 17342

  • Add optional regex based cid extractor to add_kubernetes_metadata processor. 17360

  • Add urldecode processor to for decoding URL-encoded fields. 17505

  • Add support for AWS IAM role_arn in credentials config. 17658 12464

  • Add keystore support for autodiscover static configurations. 16306

  • Add Kerberos support to Elasticsearch output. 17927

  • Add k8s keystore backend. 18096

  • Add support for fixed length extraction in dissect processor. 17191

  • Set agent.name to the hostname by default. 16377 18000

  • Add support for basic ECS logging. 17974

  • Add config example of how to skip the add_host_metadata processor when forwarding logs. 13920 18153

  • When using the decode_json_fields processor, decoded fields are now deep-merged into existing event. 17958

  • Add backoff configuration options for the Kafka output. 16777 17808

  • Update documentation for system.process.memory fields to include clarification on Windows os’s. 17268

  • Add urldecode processor to for decoding URL-encoded fields. 17505

  • Add keystore support for autodiscover static configurations. {pull]16306[16306]

  • When using the decode_json_fields processor, decoded fields are now deep-merged into existing event. 17958

  • Add keystore support for autodiscover static configurations. {pull]16306[16306]

  • Add TLS support to Kerberos authentication in Elasticsearch. 18607

  • Add support for multiple sets of hints on autodiscover 18883

  • Add config option rotate_on_startup to file output 19150 19347

  • Add a configurable delay between retries when an app metadata cannot be retrieved by add_cloudfoundry_metadata. 19181

  • Added the max_cached_sessions option to the script processor. 19562

  • Add support for DNS over TLS for the dns_processor. 19321

  • Add minimum cache TTL for successful DNS responses. 18986

  • Set index.max_docvalue_fields_search in index template to increase value to 200 fields. 20215

  • Add capability of enriching process metadata with contianer id also for non-privileged containers in add_process_metadata processor. 19767

  • Add replace_fields config option in add_host_metadata for replacing host fields. 20490 20464

  • Add option to select the type of index template to load: legacy, component, index. 21212

  • Add istiod metricset. 21519

  • Release add_cloudfoundry_metadata as GA. 21525

  • Add support for OpenStack SSL metadata APIs in add_cloud_metadata. 21590

  • Add cloud.account.id for GCP into add_cloud_metadata processor. 21776

  • Add proxy metricset for istio module. 21751

  • Add kubernetes.node.hostname metadata of Kubernetes node. 22189

  • Enable always add_resource_metadata for Pods and Services of kubernetes autodiscovery. 22189

  • Add add_resource_metadata option setting (always enabled) for add_kubernetes_metadata setting. 22189

  • Added Kafka version 2.2 to the list of supported versions. 22328

  • Add support for ephemeral containers in kubernetes autodiscover and add_kubernetes_metadata. 22389 22439

  • Added support for wildcard fields and keyword fallback in beats setup commands. 22521

  • Fix polling node when it is not ready and monitor by hostname 22666

  • Add expand_keys option to decode_json_fields processor and json input, to recusively de-dot and expand json keys into hierarchical object structures 22849

  • Update k8s client and release k8s leader lock gracefully 22919

  • Improve equals check. 22778

  • Added "detect_mime_type" processor for detecting mime types 22940

  • Improve event normalization performance 22974

  • Add tini as init system in docker images 22137

  • Added "add_network_direction" processor for determining perimeter-based network direction. 23076

  • Added new rate_limit processor for enforcing rate limits on event throughput. 22883

  • Allow node/namespace metadata to be disabled on kubernetes metagen and ensure add_kubernetes_metadata honors host 23012

  • Add support for defining explicitly named dynamic templates without path/type match criteria 25422

  • Improve ES output error insights. 25825

  • Libbeat: report beat version to monitoring. 26214

Auditbeat

  • Reference kubernetes manifests include configuration for auditd and enrichment with kubernetes metadata. 17431

  • Reference kubernetes manifests mount data directory from the host, so data persist between executions in the same node. 17429

  • Log to stderr when running using reference kubernetes manifests. 174443

  • Fix syscall kprobe arguments for 32-bit systems in socket module. 17500

  • Fix memory leak on when we miss socket close kprobe events. 17500

  • Add system module process dataset ECS categorization fields. 18032

  • Add system module socket dataset ECS categorization fields. 18036

  • Add ECS categories for system module host dataset. 18031

  • Add system module package dataset ECS categorization fields. 18033

  • Add system module login dataset ECS categorization fields. 18034

  • Add system module user dataset ECS categorization fields. 18035

  • Add file integrity module ECS categorization fields. 18012

  • Add file.mime_type, file.extension, and file.drive_letter for file integrity module. 18012

  • Add ECS categorization info for auditd module 18596

Filebeat

  • Set event.outcome field based on googlecloud audit log output. 15731

  • Add dashboard for AWS ELB fileset. 15804

  • Add dashboard for AWS vpcflow fileset. 16007

  • container and docker inputs now support reading of labels and env vars written by docker JSON file logging driver. 8358

  • Add index option to all inputs to directly set a per-input index value. 14010

  • Add ECS tls fields to zeek:smtp,rdp,ssl and aws:s3access,elb 15757 15936

  • Add custom string mapping to CEF module to support Forcepoint NGFW 14663 15910

  • Add ingress nginx controller fileset 16197

  • move create-[module,fileset,fields] to mage and enable in x-pack/filebeat 15836

  • Work on e2e ACK’s for the azure-eventhub input 15671 16215

  • Add MQTT input. 15602 16204

  • Add ECS categorization fields to activemq module. 16151 16201

  • Add a TLS test and more debug output to httpjson input 16315

  • Add an SSL config example in config.yml for filebeat MISP module. 16320

  • Improve ECS categorization, container & process field mappings in auditd module. 16153 16280

  • Improve ECS field mappings in aws module. 16154 16307

  • Improve ECS categorization field mappings in googlecloud module. 16030 16500

  • Improve ECS field mappings in haproxy module. 16162 16529

  • Add cloudwatch fileset and ec2 fileset in aws module. 13716 16579

  • Improve ECS categorization field mappings in kibana module. 16168 16652

  • Improve the decode_cef processor by reducing the number of memory allocations. 16587

  • Add cloudfoundry input to send events from Cloud Foundry. 16586

  • Improve ECS categorization field mappings in iis module. 16165 16618

  • Improve ECS categorization field mapping in kafka module. 16167 16645

  • Allow users to override pipeline ID in fileset input config. 9531 16561

  • Add o365audit input type for consuming events from Office 365 Management Activity API. 16196 16244

  • Improve ECS categorization field mappings in logstash module. 16169 16668

  • Update filebeat httpjson input to support pagination via Header and Okta module. 16354

  • Improve ECS categorization field mapping in icinga module. 16164 16533

  • Improve ECS categorization field mappings in ibmmq module. 16163 16532

  • Improve ECS categorization, host field mappings in elasticsearch module. 16160 16469

  • Add ECS related fields to CEF module 16157 16338

  • Improve ECS categorization field mappings in suricata module. 16181 16843

  • Release ActiveMQ module as GA. 17047 17049

  • Improve ECS categorization field mappings in iptables module. 16166 16637

  • Add Filebeat Okta module. 16362

  • Add custom string mapping to CEF module to support Check Point devices. 16041 16907

  • Add a TLS test and more debug output to httpjson input 16315

  • Add an SSL config example in config.yml for filebeat MISP module. 16320

  • Improve ECS categorization, container & process field mappings in auditd module. 16153 16280

  • Add cloudwatch fileset and ec2 fileset in aws module. 13716 16579

  • Improve the decode_cef processor by reducing the number of memory allocations. 16587

  • Add custom string mapping to CEF module to support Forcepoint NGFW 14663 15910

  • Add ECS related fields to CEF module 16157 16338

  • Improve ECS categorization, host field mappings in elasticsearch module. 16160 16469

  • Add pattern for Cisco ASA / FTD Message 734001 16212 16612

  • Added new module o365 for ingesting Office 365 management activity API events. 16196 16386

  • Add source field in k8s events 17209

  • Improve AWS cloudtrail field mappings 16086 16110 17155

  • Added new module crowdstrike for ingesting Crowdstrike Falcon streaming API endpoint event data. 16988

  • Added documentation for running Filebeat in Cloud Foundry. 17275

  • Move azure-eventhub input to GA. 15671 17313

  • Improve ECS categorization field mappings in mongodb module. 16170 17371

  • Improve ECS categorization field mappings for mssql module. 16171 17376

  • Added access_key_id, secret_access_key and session_token into aws module config. 17456

  • Add dashboard for Google Cloud Audit and AWS CloudTrail. 17379

  • Improve ECS categorization field mappings for mysql module. 16172 17491

  • Release Google Cloud module as GA. 17511

  • Add config option to select a different azure cloud env in the azure-eventhub input and azure module. 17649 17659

  • Added new Checkpoint Syslog filebeat module. 17682

  • Improve ECS categorization field mappings for nats module. 16173 17550

  • Add support for v10, v11 and v12 logs on Postgres 13810 17732

  • Enhance elasticsearch/server fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17714

  • Add support for Google Application Default Credentials to the Google Pub/Sub input and Google Cloud modules. 15668

  • Enhance elasticsearch/deprecation fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17728

  • Enhance elasticsearch/slowlog fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17729

  • Improve ECS categorization field mappings in misp module. 16026 17344

  • Added Unix stream socket support as an input source and a syslog input source. 17492

  • Added new Fortigate Syslog filebeat module. 17890

  • Improve ECS categorization field mappings in postgresql module. 16177 17914

  • Improve ECS categorization field mappings in rabbitmq module. 16178 17916

  • Make decode_cef processor GA. 17944

  • Improve ECS categorization field mappings in redis module. 16179 17918

  • Improve ECS categorization field mappings for zeek module. 16029 17738

  • Improve ECS categorization field mappings for netflow module. 16135 18108

  • Added documentation for running Filebeat in Cloud Foundry. 17275

  • Added access_key_id, secret_access_key and session_token into aws module config. 17456

  • Release Google Cloud module as GA. 17511

  • Update filebeat httpjson input to support pagination via Header and Okta module. 16354

  • Added new Checkpoint Syslog filebeat module. 17682

  • Added Unix stream socket support as an input source and a syslog input source. 17492

  • Added new Fortigate Syslog filebeat module. 17890

  • Change the json.* input settings implementation to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

  • Improve ECS categorization field mappings in osquery module. 16176 17881

  • Add http_endpoint input. 18298

  • Add support for array parsing in azure-eventhub input. 18585

  • Added observer.vendor, observer.product, and observer.type to PANW module events. 18223

  • The logstash module can now automatically detect the log file format (JSON or plaintext) and process it accordingly. 9964 18095

  • Added http_endpoint inputhttps://github.com/elastic/pull/18298[18298]

  • Add support for array parsing in azure-eventhub input. 18585

  • Added observer.vendor, observer.product, and observer.type to PANW module events. 18223

  • Improve ECS categorization field mappings in coredns module. 16159 18424

  • Improve ECS categorization field mappings in envoyproxy module. 16161 18395

  • Improve ECS categorization field mappings in coredns module. 16159 18424

  • Improve ECS categorization field mappings in cisco module. 16028 18537

  • Add geoip AS lookup & improve ECS categorization in aws cloudtrail fileset. 18644 18958

  • Improved performance of PANW sample dashboards. 19031 19032

  • Add support for v1 consumer API in Cloud Foundry input, use it by default. 19125

  • Explicitly set ECS version in all Filebeat modules. 19198

  • Add new mode to multiline reader to aggregate constant number of lines 18352

  • Add automatic retries and exponential backoff to httpjson input. 18956

  • Add awscloudwatch input. 19025

  • Add new mode to multiline reader to aggregate constant number of lines 18352

  • Changed the panw module to pass through (rather than drop) message types other than threat and traffic. 16815 19375

  • Improve ECS categorization field mappings in traefik module. 16183 19379

  • Improve ECS categorization field mappings in azure module. 16155 19376

  • Add text & flattened versions of fields with unknown subfields in aws cloudtrail fileset. 18866 19121

  • Added Microsoft Defender ATP Module. 17997 19197

  • Add experimental dataset tomcat/log for Apache TomCat logs 19713

  • Add experimental dataset netscout/sightline for Netscout Arbor Sightline logs 19713

  • Add experimental dataset barracuda/waf for Barracuda Web Application Firewall logs 19713

  • Add experimental dataset f5/bigipapm for F5 Big-IP Access Policy Manager logs 19713

  • Add experimental dataset bluecoat/director for Bluecoat Director logs 19713

  • Add experimental dataset cisco/nexus for Cisco Nexus logs 19713

  • Add experimental dataset citrix/virtualapps for Citrix Virtual Apps logs 19713

  • Add experimental dataset cylance/protect for Cylance Protect logs 19713

  • Add experimental dataset fortinet/clientendpoint for Fortinet FortiClient Endpoint Protection logs 19713

  • Add experimental dataset imperva/securesphere for Imperva Secure Sphere logs 19713

  • Add experimental dataset infoblox/nios for Infoblox Network Identity Operating System logs 19713

  • Add experimental dataset juniper/junos for Juniper Junos OS logs 19713

  • Add experimental dataset kaspersky/av for Kaspersky Anti-Virus logs 19713

  • Add experimental dataset microsoft/dhcp for Microsoft DHCP Server logs 19713

  • Add experimental dataset tenable/nessus_security for Tenable Nessus Security Scanner logs 19713

  • Add experimental dataset rapid7/nexpose for Rapid7 Nexpose logs 19713

  • Add experimental dataset radware/defensepro for Radware DefensePro logs 19713

  • Add experimental dataset sonicwall/firewall for Sonicwall Firewalls logs 19713

  • Add experimental dataset squid/log for Squid Proxy Server logs 19713

  • Add experimental dataset zscaler/zia for Zscaler Internet Access logs 19713

  • Add initial support for configurable file identity tracking. 18748

  • Add support for reading auditd logs that are prefixed with node=. 19659

  • Add event.ingested for CrowdStrike module 20138

  • Add support for additional fields and FirewallMatchEvent type events in CrowdStrike module 20138

  • Add event.ingested for Suricata module 20220

  • Add support for custom header and headersecret for filebeat http_endpoint input 20435

  • Add event.ingested to all Filebeat modules. 20386

  • Return error when log harvester tries to open a named pipe. 18682 20450

  • Avoid goroutine leaks in Filebeat readers. 19193 20455

  • Convert httpjson to v2 input 20226

  • Improve Zeek x509 module with x509 ECS mappings 20867

  • Improve Zeek SSL module with x509 ECS mappings 20927

  • Added new properties field support for event.outcome in azure module 20998

  • Add type and sub_type to panw panos fileset 20912

  • Add related.hosts ecs field to all modules 21160

  • Keep cursor state between httpjson input restarts 20751

  • New juniper.srx dataset for Juniper SRX logs. 20017

  • Adding support for Microsoft 365 Defender (Microsoft Threat Protection) 21446

  • Adding support for FIPS in s3 input 21446

  • Adding support for Oracle Database Audit Logs 21991

  • Add SSL option to checkpoint module 19560

  • Add max_number_of_messages config into s3 input. 21993

  • Update Okta documentation for new stateful restarts. 22091

  • Added support for MySQL Enterprise audit logs. 22273

  • Rename googlecloud module to gcp module. 22214

  • Rename awscloudwatch input to aws-cloudwatch. 22228

  • Rename google-pubsub input to gcp-pubsub. 22213

  • Copy tag names from MISP data into events. 21664

  • Added DNS response IP addresses to related.ip in Suricata module. 22291

  • Added TLS JA3 fingerprint, certificate not_before/not_after, certificate SHA1 hash, and certificate subject fields to Zeek SSL dataset. 21696

  • Add platform logs in the azure filebeat module. 22371

  • Added event.ingested field to data from the Netflow module. 22412

  • Improve panw ECS url fields mapping. 22481

  • Improve Nats filebeat dashboard. 22726

  • Add support for UNIX datagram sockets in unix input. {issues}18632[18632] 22699

  • Add new httpjson input features and mark old config ones for deprecation 22320

  • Add support for Snyk Vulnerability and Audit API. 22677

  • Add logic for external network.direction in sophos xg fileset 22973

  • Add http.request.mime_type for Elasticsearch audit log fileset. 22975

  • Add configuration option to set external and internal networks for panw panos fileset 22998

  • Add subbdomain fields for rsa2elk modules. 23035

  • Add subdomain enrichment for suricata/eve fileset. 23011

  • Add subdomain enrichment for zeek/dns fileset. 23011

  • Add event.category "configuration" to auditd module events. 23010

  • Add event.category "configuration" to gsuite module events. 23010

  • Add event.category "configuration" to o365 module events. 23010

  • Add event.category "configuration" to zoom module events. 23010

  • Add network.direction to auditd/log fileset. 23041

  • Preserve AWS CloudTrail eventCategory in aws.cloudtrail.event_category. 22776 22805

  • Migrate microsoft/defender_atp to httpjson v2 config 23017

  • Migrate microsoft/m365_defender to httpjson v2 config 23018

  • Add top_level_domain enrichment for suricata/eve fileset. 23046

  • Add top_level_domain enrichment for zeek/dns fileset. 23046

  • Add observer.egress.zone and observer.ingress.zone for cisco/asa and cisco/ftd filesets. 23068

  • Allow cisco/asa and cisco/ftd filesets to override network directionality based off of zones. 23068

  • Add network.direction to netflow/log fileset. 23052

  • Allow cef and checkpoint modules to override network directionality based off of zones 23066

  • Add the ability to override network.direction based on interfaces in Fortinet/firewall fileset. 23072

  • Add network.direction override by specifying internal_networks in gcp module. 23081

  • Migrate okta to httpjson v2 config 23059

  • Misp improvements: Migration to httpjson v2 config, pagination and deduplication ID 23070

  • Add Google Workspace module and mark Gsuite module as deprecated 22950

  • Mark m365 defender, defender atp, okta and google workspace modules as GA 23113

  • Added alternative_host option to google pubsub input 23215

  • Update PanOS module to parse Global Protect & User ID logs. 24722 24724 24927

  • Add HMAC signature validation support for http_endpoint input. 24918

  • Add new grok pattern for iptables module for Ubiquiti UDM 25615 25616

  • Add multiline support to aws-s3 input. 25249 25710

  • Add monitoring metrics to the aws-s3 input. 25711

  • Add Content-Type override to aws-s3 input. 25697 25772

  • In Cisco Umbrella fileset add users from cisco.umbrella.identities to related.user. 25776

  • Add fingerprint processor to generate fixed ids for google_workspace events. 25841

  • Update PanOS module to parse HIP Match logs. 24350 25686

  • Enhance GCP module to populate orchestrator.* fields for GKE / K8S logs 25368

  • http_endpoint: Support multiple documents in a single request by POSTing an array or NDJSON format. 25764

  • Make filestream input GA. 26127

  • Add new parser to filestream input: container. 26115

Heartbeat

  • Add mime type detection for http responses. 22976

Journalbeat

Metricbeat

  • Move the windows pdh implementation from perfmon to a shared location in order for future modules/metricsets to make use of. 15503

  • Add DynamoDB AWS Metricbeat light module 15097

  • Release elb module as GA. 15485

  • Add a system/network_summary metricset 15196

  • Add mesh metricset for Istio Metricbeat module 15535

  • Add IBM MQ light-weight Metricbeat module 15301

  • Add mixer metricset for Istio Metricbeat module 15696

  • Add pilot metricset for Istio Metricbeat module 15761

  • Make the system/cpu metricset collect normalized CPU metrics by default. 15618 15729

  • Add galley metricset for Istio Metricbeat module 15857

  • Add key/value mode for SQL module. 15770 15845

  • Add STAN dashboard 15654

  • Add support for Unix socket in Memcached metricbeat module. 13685 15822

  • Add up metric to prometheus metrics collected from host 15948

  • Add citadel metricset for Istio Metricbeat module 15990

  • Add collecting AuroraDB metrics in rds metricset. 14142 16004

  • Reuse connections in SQL module. 16001

  • Improve the logstash module (when xpack.enabled is set to true) to use the override cluster_uuid returned by Logstash APIs. 15772 15795

  • Add region parameter in googlecloud module. 15780 16203

  • Add kubernetes storage class support via kube-state-metrics. 16145

  • Add database_account azure metricset. 15758

  • Add support for NATS 2.1. 16317

  • Add Load Balancing metricset to GCP 15559

  • Add support for Dropwizard metrics 4.1. 16332

  • Add azure container metricset in order to monitor containers. 15751 16421

  • Improve the haproxy module to support metrics exposed via HTTPS. 14579 16333

  • Add filtering option for prometheus collector. 16420

  • Add metricsets based on Ceph Manager Daemon to the ceph module. 7723 16254

  • Release statsd module as GA. 16447 14280

  • Add collecting tags and tags_filter for rds metricset in aws module. 16605 16358

  • Add OpenMetrics Metricbeat module 16596

  • Add cloudfoundry module to send events from Cloud Foundry. 16671

  • Add redisenterprise module. 16482 15269

  • Add database_account azure metricset. 15758

  • Add Load Balancing metricset to GCP 15559

  • Add OpenMetrics Metricbeat module 16596

  • Add system/users metricset as beta 16569

  • Add additional cgroup fields to docker/diskiohttps://github.com/elastic/pull/16638[16638]

  • Add PubSub metricset to Google Cloud Platform module 15536

  • Add overview dashboard for googlecloud compute metricset. 16534 16819

  • Add Prometheus remote write endpoint 16609

  • Release STAN module as GA. 16980

  • Add query metricset for prometheus module. 17104

  • Release ActiveMQ module as GA. 17047 17049

  • Add Prometheus remote write endpoint 16609

  • Add support for CouchDB v2 16352 16455

  • Release Zookeeper/connection module as GA. 14281 17043

  • Add support for CouchDB v2 16352 16455

  • Add dashboard for pubsub metricset in googlecloud module. 17161

  • Replace vpc metricset into vpn, transitgateway and natgateway metricsets. 16892

  • Use Elasticsearch histogram type to store Prometheus histograms 17061

  • Allow to rate Prometheus counters when scraping them 17061

  • Release Oracle module as GA. 14279 16833

  • Release vsphere module as GA. 15798 17119

  • Add Storage metricsets to GCP module 15598

  • Added documentation for running Metricbeat in Cloud Foundry. 17275

  • Add test for documented fields check for metricsets without a http input. 17315 17334

  • Add final tests and move label to GA for the azure module in metricbeat. 17319

  • Refactor windows/perfmon metricset configuration options and event output. 17596

  • Add PubSub metricset to Google Cloud Platform module 15536

  • Add final tests and move label to GA for the azure module in metricbeat. 17319

  • Added documentation for running Metricbeat in Cloud Foundry. 17275

  • Reference kubernetes manifests mount data directory from the host when running metricbeat as daemonset, so data persist between executions in the same node. 17429

  • Stack Monitoring modules now auto-configure required metricsets when xpack.enabled: true is set. [16471 17609

  • Add aggregation aligner as a config parameter for googlecloud stackdriver metricset. [17141 17719

  • Move the perfmon metricset to GA. 16608 17879

  • Add static mapping for metricsets under aws module. 17614 17650

  • Add dashboard for googlecloud storage metricset. 18172

  • Stack Monitoring modules now auto-configure required metricsets when xpack.enabled: true is set. [16471 17609

  • Collect new bulk indexing metrics from Elasticsearch when xpack.enabled:true is set. https://github.com/elastic/beats/issues/ 17992

  • Remove requirement to connect as sysdba in Oracle module 15846 18182

  • Update MSSQL module to fix some SSPI authentication and add brackets to USE statements 17862]

  • Add client address to events from http server module 18336

  • Remove required for region/zone and make stackdriver a metricset in googlecloud. 16785 18398

  • Add memory metrics into compute googlecloud. 18802

  • Add Tomcat overview dashboard 14026

  • Accept prefix as metric_types config parameter in googlecloud stackdriver metricset. 19345

  • Update Couchbase to version 6.5 18595 19055

  • Add dashboards for googlecloud load balancing metricset. 18369

  • Add support for v1 consumer API in Cloud Foundry module, use it by default. 19268

  • The elasticsearch/index metricset now collects metrics for hidden indices as well. 18639 18703

  • Adds support for app insights metrics in the azure module. 18570 18940

  • Added cache and connection_errors metrics to status metricset of MySQL module 16955 19844

  • Update MySQL dashboard with connection errors and cache metrics 19913 16955

  • Add cloud.instance.name into aws ec2 metricset. 20077

  • Add scope setting for elasticsearch module, allowing it to monitor an Elasticsearch cluster behind a load-balancing proxy. 18539 18547

  • Add state_daemonset metricset for Kubernetes Metricbeat module 20649

  • Add host inventory metrics to azure compute_vm metricset. 20641

  • Add host inventory metrics to googlecloud compute metricset. 20391

  • Add host inventory metrics to system module. 20415

  • Add billing data collection from Cost Explorer into aws billing metricset. 20527 20103

  • Migrate compute_vm metricset to a light one, map cloud.instance.id field. 20889

  • Request prometheus endpoints to be gzipped by default 20766

  • Add latency config parameter into aws module. 20875

  • Release all kubernetes state metricsets as GA 20901

  • Add billing metricset into googlecloud module. 20812 20738

  • Move compute_vm_scaleset to light metricset. 21038 20985

  • Sanitize event.host. 21022

  • Add support for different Azure Cloud environments in the metricbeat azure module. 21044 20988

  • Add overview and platform health dashboards to Cloud Foundry module. 21124

  • Add dashboard for pubsub metricset in googlecloud module. 21326 17137

  • Move Prometheus query & remote_write to GA. 21507

  • Expand unsupported option from namespace to metrics in the azure module. 21486

  • Map cloud data filed cloud.account.id to azure subscription. 21483 21381

  • Move s3_daily_storage and s3_request metricsets to use cloudwatch input. 21703

  • Duplicate system.process.cmdline field with process.command_line ECS field name. 22325

  • Add awsfargate module task_stats metricset to monitor AWS ECS Fargate. 22034

  • Add connection and route metricsets for nats metricbeat module to collect metrics per connection/route. 22445

  • Add unit file states to system/service 22557

  • Add io.ops in fields exported by system.diskio. 22066

  • kibana module: stats metricset no-longer collects usage-related data. 22732

  • Adjust the Apache status fields in the fleet mode. 22821

  • Add AWS Fargate overview dashboard. 22941

  • Add process.state, process.cpu.pct, process.cpu.start_time and process.memory.pct. 22845

  • Move IIS module to GA and map fields. 22609 23024

  • Apache: convert status.total_kbytes to status.total_bytes in fleet mode. 23022

  • Release MSSQL as GA 23146

  • Refactor state_* metricsets to share response from endpoint. 25640

  • Add server id to zookeeper events. 25550

  • Add additional network metrics to docker/network 25354

  • Migrate ec2 metricsets to use cloudwatch input. 25924

  • Reduce number of requests done by kubernetes metricsets to kubelet. 25782

  • Migrate rds metricsets to use cloudwatch input. 26077

  • Migrate sqs metricsets to use cloudwatch input. 26117

Packetbeat

  • Add an example to packetbeat.yml of using the forwarded tag to disable host metadata fields when processing network data from network tap or mirror port. 19209

  • Add ECS fields for x509 certs, event categorization, and related IP info. 19167

  • Add 100-continue support 15830 19349

  • Add initial SIP protocol support 21221

  • Add support for overriding the published index on a per-protocol/flow basis. 22134

  • Change build process for x-pack distribution 21979

  • Tuned the internal queue size to reduce the chances of events being dropped. 22650

  • Add support for "http.request.mime_type" and "http.response.mime_type". 22940

  • Upgrade to ECS 1.8.0. 23783

  • Add event.type: [connection] to flow events and include end for final flows. 24564

Functionbeat

  • Add basic ECS categorization and cloud fields. 19174

  • Add support for parallelization factor for kinesis. 20727

  • Provide more ways to set AWS credentials. 12464 23344

  • Add support for multiple regions 21065

Heartbeat

Winlogbeat

  • Add more DNS error codes to the Sysmon module. 15685

  • Add experimental event log reader implementation that should be faster in most cases. 6585 16849

  • Set process.command_line and process.parent.command_line from Sysmon Event ID 1. 17327

  • Add support for event IDs 4673,4674,4697,4698,4699,4700,4701,4702,4768,4769,4770,4771,4776,4778,4779,4964 to the Security module 17517

  • Add registry and code signature information and ECS categorization fields for sysmon module 18058

  • Add new winlogbeat security dashboard 18775

  • Add event.outcome to events based on the audit success and audit failure keywords. 20564

  • Add file.pe and process.pe fields to ProcessCreate & LoadImage events in Sysmon module. 17335 22217

  • Add additional event categorization for security and sysmon modules. 22988

  • Add dns.question.subdomain fields for sysmon DNS events. 22999

  • Add dns.question.top_level_domain fields for sysmon DNS events. 23046

  • Changed the log level of the "Successfully published events" message from info to debug to reduce verbosity of the info logging level. To track event log reader activity use the published_events metric. 25617

Elastic Log Driver

  • Add support for docker logs command 19531

  • Fixed docs for hosts 23644

Deprecated

Affecting all Beats

Filebeat

Heartbeat

Journalbeat

Metricbeat

Packetbeat

Winlogbeat

Functionbeat

Known Issue

Journalbeat