Add decode_cef processor to Filebeat (elastic#13264)

This adds a decode_cef processor that decodes Common Event Format (CEF) messages. It is expected to be used in conjunction with the syslog input. The processor turns messages like

    "CEF:0|Trend Micro|Deep Security Manager|1.2.3|600|User Signed In|3|src=10.52.116.160 suser=admin target=admin msg=User signed in from 2001:db8::5"

into

    {
      "cef": {
        "device": {
          "event_class_id": "600",
          "product": "Deep Security Manager",
          "vendor": "Trend Micro",
          "version": "1.2.3"
        },
        "extensions": {
          "message": "User signed in from 2001:db8::5",
          "sourceAddress": "10.52.116.160",
          "sourceUserName": "admin",
          "target": "admin"
        },
        "name": "User Signed In",
        "severity": "3",
        "version": 0
      },
      "message": "User signed in from 2001:db8::5",
      "source": {
        "ip": "10.52.116.160",
        "user": {
          "name": "admin"
        }
      }
    }

The processor is built on top a CEF parser implemented using Ragel. The parser is a self contained Go package that's responsible for parsing the CEF message and optionally translating the standard short CEF extension names to their full names (e.g. src -> sourceAddress).

The processor uses the parser to get the data in a structured format, and then translates the extensions fields to ECS fields via copy. By copying rather than moving the original CEF fields are left intact.

The parser tries to be lenient in some aspects compared to the CEF specification.

- Extension key names can start with [a-zA-Z0-9_] and in addition contain dots (.), commas (,), left-brackets ([), and right-brackets (]).
- If an extension value contains an illegal unescaped equals sign, then the parser will attempt to jump the start of the next extension key can and continue parsing. It will return an error about this because it's possible that rest of the extensions may have been parsed incorrectly.

This is a benchmark of the cef parser package.

    BenchmarkEventUnpack-12         20000000               771 ns/op             498 B/op         16 allocs/op

.editorconfig

@@ -32,3 +32,7 @@ indent_style = tab
 [Vagrantfile]
 indent_size = 2
 indent_style = space
+
+[*.rl]
+indent_size = 4
+indent_style = space

filebeat/docs/fields.asciidoc

@@ -16,6 +16,7 @@ grouped in the following categories:
 * <<exported-fields-auditd>>
 * <<exported-fields-aws>>
 * <<exported-fields-beat-common>>
+* <<exported-fields-cef>>
 * <<exported-fields-cisco>>
 * <<exported-fields-cloud>>
 * <<exported-fields-coredns>>
@@ -1318,6 +1319,129 @@ type: keyword
 
 --
 
+[[exported-fields-cef]]
+== Decode CEF processor fields fields
+
+Common Event Format (CEF) data.
+
+
+
+[float]
+=== cef
+
+By default the `decode_cef` processor writes all data from the CEF message to this `cef` object. It contains the CEF header fields and the extension data.
+
+
+
+*`cef.version`*::
++
+--
+Version of the CEF specification used by the message.
+
+
+type: keyword
+
+--
+
+*`cef.device.vendor`*::
++
+--
+Vendor of the device that produced the message.
+
+
+type: keyword
+
+--
+
+*`cef.device.product`*::
++
+--
+Product of the device that produced the message.
+
+
+type: keyword
+
+--
+
+*`cef.device.version`*::
++
+--
+Version of the product that produced the message.
+
+
+type: keyword
+
+--
+
+*`cef.device.event_class_id`*::
++
+--
+Unique identifier of the event type.
+
+
+type: keyword
+
+--
+
+*`cef.severity`*::
++
+--
+Importance of the event. The valid string values are Unknown, Low, Medium, High, and Very-High. The valid integer values are 0-3=Low, 4-6=Medium, 7- 8=High, and 9-10=Very-High.
+
+
+type: keyword
+
+example: Very-High
+
+--
+
+*`cef.name`*::
++
+--
+Short description of the event.
+
+
+type: keyword
+
+--
+
+*`cef.extensions`*::
++
+--
+Collection of key-value pairs carried in the CEF extension field.
+
+
+type: object
+
+--
+
+*`observer.product`*::
++
+--
+Product name.
+
+type: keyword
+
+--
+
+*`source.service.name`*::
++
+--
+Service that is the source of the event.
+
+type: keyword
+
+--
+
+*`destination.service.name`*::
++
+--
+Service that is the target of the event.
+
+type: keyword
+
+--
+
 [[exported-fields-cisco]]
 == Cisco fields
 

filebeat/docs/index.asciidoc

@@ -18,6 +18,7 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[]
 :has_solutions:
 :ignores_max_retries:
 :has_docker_label_ex:
+:has_decode_cef_processor:
 :has_decode_csv_fields_processor:
 :has_script_processor:
 :has_timestamp_processor:

filebeat/magefile.go

@@ -148,6 +148,7 @@ func fieldDocs() error {
 		devtools.XPackBeatDir("module"),
 		devtools.OSSBeatDir("input"),
 		devtools.XPackBeatDir("input"),
+		devtools.XPackBeatDir("processors"),
 	}
 	output := devtools.CreateDir("build/fields/fields.all.yml")
 	if err := devtools.GenerateFieldsYAMLTo(output, inputs...); err != nil {

libbeat/docs/processors-using.asciidoc

@@ -211,6 +211,9 @@ The supported processors are:
  * <<community-id,`community_id`>>
  * <<convert,`convert`>>
  * <<decode-base64-field,`decode_base64_field`>>
+ifdef::has_decode_cef_processor[]
+* <<processor-decode-cef,`decode_cef`>>
+endif::[]
 ifdef::has_decode_csv_fields_processor[]
  * <<decode-csv-fields,`decode_csv_fields`>>
 endif::[]
@@ -793,6 +796,45 @@ Adds the environment field to every event:
 }
 -------------------------------------------------------------------------------
 
+ifdef::has_decode_cef_processor[]
+[[processor-decode-cef]]
+[role="xpack"]
+=== Decode CEF
+
+beta[]
+
+The `decode_cef` processor decodes Common Event Format (CEF) messages. This
+processor is available in Filebeat.
+
+Below is an example configuration that decodes the `message` field as CEF after
+renaming it to `event.original`. It is best to rename `message` to
+`event.original` because the decoded CEF data contains its own `message` field.
+
+[source,yaml]
+----
+processors:
+- rename:
+    fields:
+      - {from: "message", to: "event.original"}
+- decode_cef:
+    field: event.original
+----
+
+The `decode_cef` processor has the following configuration settings.
+
+.Decode CEF options
+[options="header"]
+|======
+| `field`          | no       | message | Source field containing the CEF message to be parsed.                        |
+| `target_field`   | no       | cef     | Target field where the parsed CEF object will be written.                    |
+| `ecs`            | no       | true    | Generate Elastic Common Schema (ECS) fields from the CEF data.
+                                          Certain CEF header and extension values will be used to populate ECS fields. |
+| `ignore_missing` | no       | false   | Ignore errors when the source field is missing.                              |
+| `ignore_failure` | no       | false   | Ignore failures when the source field does not contain a CEF message.        |
+| `id`             | no       |         | An identifier for this processor instance. Useful for debugging.             |
+|======
+endif::[]
+
 ifdef::has_decode_csv_fields_processor[]
 [[decode-csv-fields]]
 === Decode CSV fields

x-pack/filebeat/magefile.go

@@ -81,10 +81,10 @@ func TestPackages() error {
 	return devtools.TestPackages()
 }
 
-// Fields generates the fields.yml file and a fields.go for each module and
-// input.
+// Fields generates the fields.yml file and a fields.go for each module, input,
+// and processor.
 func Fields() {
-	mg.Deps(fieldsYML, moduleFieldsGo, inputFieldsGo)
+	mg.Deps(fieldsYML, moduleFieldsGo, inputFieldsGo, processorsFieldsGo)
 }
 
 func inputFieldsGo() error {
@@ -95,9 +95,13 @@ func moduleFieldsGo() error {
 	return devtools.GenerateModuleFieldsGo("module")
 }
 
+func processorsFieldsGo() error {
+	return devtools.GenerateModuleFieldsGo("processors")
+}
+
 // fieldsYML generates a fields.yml based on filebeat + x-pack/filebeat/modules.
 func fieldsYML() error {
-	return devtools.GenerateFieldsYAML(devtools.OSSBeatDir("module"), "module", "input")
+	return devtools.GenerateFieldsYAML(devtools.OSSBeatDir("module"), "module", "input", "processors")
 }
 
 // Dashboards collects all the dashboards and generates index patterns.
@@ -130,7 +134,7 @@ func Update() {
 }
 
 func includeList() error {
-	return devtools.GenerateIncludeListGo([]string{"input/*"}, []string{"module"})
+	return devtools.GenerateIncludeListGo([]string{"input/*", "processors/*"}, []string{"module"})
 }
 
 // IntegTest executes integration tests (it uses Docker to run the tests).

x-pack/filebeat/processors/decode_cef/_meta/fields.yml

@@ -0,0 +1,70 @@
+- key: cef
+  title: Decode CEF processor fields
+  description: >
+    Common Event Format (CEF) data.
+  fields:
+    - name: cef
+      type: group
+      description: >
+        By default the `decode_cef` processor writes all data from the CEF
+        message to this `cef` object. It contains the CEF header fields and the
+        extension data.
+      fields:
+        - name: version
+          type: keyword
+          description: >
+            Version of the CEF specification used by the message.
+
+        - name: device.vendor
+          type: keyword
+          description: >
+            Vendor of the device that produced the message.
+
+        - name: device.product
+          type: keyword
+          description: >
+            Product of the device that produced the message.
+
+        - name: device.version
+          type: keyword
+          description: >
+            Version of the product that produced the message.
+
+        - name: device.event_class_id
+          type: keyword
+          description: >
+            Unique identifier of the event type.
+
+        - name: severity
+          type: keyword
+          example: Very-High
+          description: >
+            Importance of the event. The valid string values are Unknown, Low,
+            Medium, High, and Very-High. The valid integer values are 0-3=Low,
+            4-6=Medium, 7- 8=High, and 9-10=Very-High.
+
+        - name: name
+          type: keyword
+          description: >
+            Short description of the event.
+
+        - name: extensions
+          type: object
+          object_type: keyword
+          description: >
+            Collection of key-value pairs carried in the CEF extension field.
+
+    - name: observer.product
+      type: keyword
+      description:
+        Product name.
+
+    - name: source.service.name
+      type: keyword
+      description:
+        Service that is the source of the event.
+
+    - name: destination.service.name
+      type: keyword
+      description:
+        Service that is the target of the event.

x-pack/filebeat/processors/decode_cef/cef/.gitignore

@@ -0,0 +1,2 @@
+cef.svg
+*.dot