ECS Mapper

Synopsis

This tool turns a field mapping CSV to roughly equivalent pipelines for:

• Beats
• Elasticsearch
• Logstash

The goal of this tool is to generate starter pipelines of each flavor, to help you get started quickly in mapping your event sources to ECS.

A "mapping CSV" is what you get, when you start planning how to map an event source to ECS in a spreadsheet.

Colleagues may collaborate on a spreadsheet that looks like this:

source_field	destination_field	notes
duration	event.duration	ECS supports nanoseconds precision
remoteip	source.ip	Hey @Jane do you agree with this one?
message		No need to change this field
...

You can export your spreadsheet to CSV, run it through the ECS mapper, and get your starter pipelines generated.

Note that this tool generates starter pipelines. They only do the rename/copy operations and some field format adjustments. It's up to you to integrate them in a complete pipeline that ingests and outputs the data however you need.

Scroll down to the Examples section below, to get right to a concrete example you can play with.

Maturity

This code is a proof of concept and is not officially supported. The pipelines generated by this tool are not meant to be complete, nor production-ready. They are simply meant to give you a head start in mapping various sources to ECS.

CSV Format

Here's more details on the CSV format supported by this tool. Since mapping spreadsheets are primarily used by humans, it's totally fine to have as many columns as you need in your spreadsheets/CSV. Only the following columns will be considered:

column name	required	allowed values	notes
source_field	required		A dotted Elasticsearch field name. Dots represent JSON nesting. Lines with empty "source_field" are skipped.
destination_field	required		A dotted Elasticsearch field name. Dots represent JSON nesting. Can be left empty if there's no rename (just a type conversion).
format_action	optional	to_float, to_integer, to_string, to_boolean, to_array, uppercase, lowercase, (empty)	Simple conversion to apply to the field value.
rename	optional	rename, copy, (empty)	What to do with the field. If left empty, default action is based on the --rename-action flag.

You can start from this spreadsheet template. Make a copy of it in your Google Docs account, or download it as an Excel file.

Usage and Dependencies

This is a simple Ruby program with no external dependencies, other than development dependencies.

Any modern version of Ruby should be sufficient. If you don't intend to run the tests or the rake tasks, you can skip right to usage tips.

Ruby Setup

If you want to tweak the code of this script, run the tests or use the rake tasks, you'll need to install the development dependencies.

Once you have Ruby installed for your platform, installing the dependencies is simply:

gem install bundler
bundle install

Run the tests:

rake test

Using the ECS Mapper

Help.

./ecs-mapper --help
Reads a CSV mapping of source field names to destination field names, and generates
Elastic pipelines to help perform the conversion.

You can have as many columns as you want in your CSV.
Only the following columns will be used by this tool:
destination_field, format_action, rename, source_field

Options:
    -f, --file FILE                  Input CSV file.
    -o, --output DIR                 Output directory. Defaults to parent dir of --file.
        --rename-action RENAME_ACTION
                                     Default action for field renames. Acceptable values are: copy, rename. Default is rename.
        --debug                      Shorthand for --log-level=debug
    -h, --help                       Display help

Process my.csv and output pipelines in the same directory as the csv.

./ecs-mapper --file my.csv

Process my.csv and output pipelines elsewhere.

./ecs-mapper --file my.csv --output pipelines/mine/

Process my.csv, fields with an empty value in the "rename" column are copied, instead of renamed (the default).

./ecs-mapper --file my.csv --rename copy

Examples

Look at an example CSV mapping and the pipelines generated from it:

• example/mapping.csv
• example/beats.yml
• example/elasticsearch.json
• example/logstash.conf

You can try each pipeline easily by following the instructions in example/README.md.

Caveats

• At this time, the Beats pipelines don't perform "to_array", "uppercase" nor "lowercase" transformations. They could be implemented via the "script" processor.