Creating a New Beat

This guide walks you through the steps for creating a new Elastic Beat. The Beats are a collection of lightweight daemons that collect operational data from your servers and ship it to Elasticsearch or Logstash. The common parts for all Beats are placed in the libbeat library, which contains packages for sending data to Elasticsearch and Logstash, for configuration file handling, for signal handling, for logging, and more. By using this common framework, you can ensure that all Beats behave consistently and that they are easy to package and run with common tools.

In this guide, you learn how to use the Beat generator to create source code for an example Beat called Countbeat. The Beat generator creates all the files required for a working Beat. To create your own Beat, you modify the generated files and implement the custom logic needed to collect the data you want to ship.

The following topics describe how to build a new Beat:

Getting Ready
Overview
Generating Your Beat
Fetching Dependencies and Setting up the Beat
Building and Running the Beat
The Beater Interface
Sharing Your Beat with the Community
[event-conventions]

The following topic provide information about moving generated Beats to go modules:

Migrate existing Beat to go modules

Getting Ready

All Beats are written in Go, so having Go installed and knowing the basics are prerequisites for understanding this guide.

Before you begin: Set up your Go environment as described under [setting-up-dev-environment] in [beats-contributing]. The minimum required Go version is {go-version}.

To build your Beat on a specific version of libbeat, check out the specific branch ({branch} in the example below):

cd ${GOPATH}/src/github.com/elastic/beats
git checkout {branch}

Overview

At the high level, a simple Beat has two main components:

a component that collects the actual data, and
a publisher that sends the data to the specified output, such as Elasticsearch or Logstash.

The publisher is already implemented in libbeat, so you typically only have to worry about the logic specific to your Beat (the code that creates the event and sends it to the publisher). Libbeat also offers common services like configuration management, logging, daemonzing, and Windows service handling, and data processing modules.

The event that you create is a JSON-like object (Go type map[string]interface{}) that contains the collected data to send to the publisher. At a minimum, the event object must contain a @timestamp field and a type field. Beyond that, events can contain any additional fields, and they can be created as often as necessary.

The following example shows an event object in Lsbeat:

{
  "@timestamp": "2016-07-13T21:33:58.355Z",
  "beat": {
    "hostname": "mar.local",
    "name": "mar.local"
  },
  "directory": false,
  "filename": "winlogbeat.yml",
  "filesize": 2895,
  "modtime": "2016-07-13T20:56:21.000Z",
  "path": "./vendor/github.com/elastic/beats/winlogbeat/winlogbeat.yml",
  "type": "lsbeat"
}

{
  "@timestamp": "2016-07-13T21:33:58.354Z",
  "beat": {
    "hostname": "mar.local",
    "name": "mar.local"
  },
  "directory": true,
  "filename": "system",
  "filesize": 238,
  "modtime": "2016-07-13T20:56:21.000Z",
  "path": "./vendor/github.com/elastic/beats/winlogbeat/tests/system",
  "type": "lsbeat"
}

Now that you have the big picture, let’s dig into the code.

Generating Your Beat

To generate your own Beat, you use the Beat generator available in the beats repo on GitHub. If you haven’t downloaded the Beats source code yet, follow the instructions in [setting-up-dev-environment].

Before running the Beat generator, you must decide on a name for your Beat. The name should be one word with the first letter capitalized. In our example, we use Countbeat.

Now create a directory under $GOPATH for your repository and change to the new directory:

mkdir ${GOPATH}/src/github.com/{user}
cd ${GOPATH}/src/github.com/{user}

Run the mage script to generate the custom beat:

mage GenerateCustomBeat

The mage script will prompt you to enter information about your Beat. For the project_name, enter Countbeat. For the github_name, enter your github id. The beat and beat_path are set to the correct values automatically (just press Enter to accept each default). For the full_name, enter your firstname and lastname. Finally, pick the revision of elastic/beats you would like to depend on.

Enter a project name [examplebeat]:  Countbeat
Enter a github name [your-github-name]: {username}
Enter a beat path [github.com/{username}/countbeat]:
Enter a full name [Firstname Lastname]: {Full Name}
Enter the github.com/elastic/beats revision [master]:

The Beat generator creates a directory called countbeat inside of your project folder (e.g. {project folder}/github.com/{github id}/countbeat).

Please note that it is not possible to use revisions of Beats which does not support go modules.

You now have a raw template of the Beat, but you still need to fetch dependencies and set up the Beat.

Fetching Dependencies and Setting up the Beat

First you need to install the following tools:

Python
Python venv module

To fetch dependencies and set up the Beat, run:

cd ${GOPATH}/src/github.com/{user}/countbeat
make setup

The Beat now contains the basic config file, countbeat.yml, and template files. The Beat is "complete" in the sense that you can compile and run it. However, to make it functionally complete, you need to add your custom logic (see The Beater Interface), along with any additional configuration parameters that your Beat requires.

Building and Running the Beat

To compile the Beat, make sure you are in the Beat directory ($GOPATH/src/github.com/{user}/countbeat) and run:

mage build

Note	we don’t support the `-j` option for make at the moment.

Running this command creates the binary called countbeat in $GOPATH/src/github.com/{user}/countbeat.

Now run the Beat:

./countbeat -e -d "*"

The command automatically loads the default config file, countbeat.yml, and sends debug output to the console.

You can stop the Beat by pressing Ctrl+C.

The Beater Interface

Each Beat needs to implement the Beater interface defined in libbeat.

// Beater is the interface that must be implemented by every Beat. A Beater
// provides the main Run-loop and a Stop method to break the Run-loop.
// Instantiation and Configuration is normally provided by a Beat-`Creator`.
//
// Once the beat is fully configured, the Run() method is invoked. The
// Run()-method implements the beat its run-loop. Once the Run()-method returns,
// the beat shuts down.
//
// The Stop() method is invoked the first time (and only the first time) a
// shutdown signal is received. The Stop()-method normally will stop the Run()-loop,
// such that the beat can gracefully shutdown.
type Beater interface {
	// The main event loop. This method should block until signalled to stop by an
	// invocation of the Stop() method.
	Run(b *Beat) error

	// Stop is invoked to signal that the Run method should finish its execution.
	// It will be invoked at most once.
	Stop()
}

To implement the Beater interface, you need to define a Beat object that implements two methods: Run() and Stop().

type Countbeat struct {
	done   chan struct{} (1)
	config config.Config (2)
	client publisher.Client (3)

	...
}

func (bt *Countbeat) Run(b *beat.Beat) error {
	...
}


func (bt *Countbeat) Stop() {
	...
}

By default, the Beat object contains the following:

done: Channel used by the Run() method to stop when the Stop() method is called.
config: Configuration options for the Beat
client: Publisher that takes care of sending the events to the defined output.

The Beat parameter received by the Run method contains data about the Beat, such as the name, version, and common configuration options.

Each Beat also needs to implement the New() function to create the Beat object. This means your Beat should implement the following functions:

New	Creates the Beat object
Run	Contains the main application loop that captures data and sends it to the defined output using the publisher
Stop	Contains logic that is called when the Beat is signaled to stop

When you run the Beat generator, it adds implementations for all these functions to the source code (see beater/countbeat.go). You can modify these implementations, as required, for your Beat.

We strongly recommend that you create a main package that contains only the main method (see main.go). All your Beat-specific code should go in a separate folder and package. This will allow other Beats to use the other parts of your Beat as a library, if needed.

Note	To be consistent with other Beats, you should append `beat` to your Beat name.

Let’s go through each of the methods in the Beater interface and look at a sample implementation.

New function

The New() function receives the configuration options defined for the Beat and creates a Beat object based on them. Here’s the New() function that’s generated in beater/countbeat.go when you run the Beat generator:

func New(b *beat.Beat, cfg *common.Config) (beat.Beater, error) {
	config := config.DefaultConfig
	if err := cfg.Unpack(&config); err != nil {
		return nil, fmt.Errorf("Error reading config file: %v", err)
	}

	bt := &Countbeat{
		done:   make(chan struct{}),
		config: config,
	}
	return bt, nil
}

Pointers are used to distinguish between when the setting is completely missing from the configuration file and when it has a value that matches the type’s default value.

The recommended way of handling the configuration (as shown in the code example) is to create a Config structure with the configuration options and a DefaultConfig with the default configuration options.

When you use the Beat generator, the Go structures for a basic config are added to config/config.go:

package config

import "time"

type Config struct {
	Period time.Duration `config:"period"`
}

var DefaultConfig = Config{
	Period: 1 * time.Second,
}

This mirrors the config options that are defined in the config file, countbeat.yml.

countbeat:
  # Defines how often an event is sent to the output
  period: 10s

period: Defines how often to send out events

The config file is generated when you run make setup to set up the beat. The file contains basic configuration information. To add configuration options to your Beat, you need to update the Go structures in config/config.go and add the corresponding config options to _meta/beat.yml.

For example, if you add a config option called path to the Go structures:

type Config struct {
    Period time.Duration `config:"period"`
    Path   string        `config:"path"`
}

var DefaultConfig = Config{
    Period: 1 * time.Second,
    Path:   ".",
}

You also need to add path to _meta/beat.yml:

countbeat:
  period: 10s
  path: "."

After modifying beat.yml, run the following command to apply your updates:

make update

Run Method

The Run method contains your main application loop.

func (bt *Countbeat) Run(b *beat.Beat) error {
	logp.Info("countbeat is running! Hit CTRL-C to stop it.")

	bt.client = b.Publisher.Connect()
	ticker := time.NewTicker(bt.config.Period)
	counter := 1
	for {
		select {
		case <-bt.done:
			return nil
		case <-ticker.C:
		}

		event := common.MapStr{ (1)
			"@timestamp": common.Time(time.Now()), (2)
			"type":       b.Name,
			"counter":    counter,
		}
		bt.client.PublishEvent(event) (3)
		logp.Info("Event sent")
		counter++
	}
}

Create the event object.
Specify a @timestamp field of time common.Time.
Use the publisher to send the event out to the defined output

Inside the loop, the Beat sleeps for a configurable period of time and then captures the required data and sends it to the publisher. The publisher client is available as part of the Beat object through the client variable.

The event := common.MapStr{} stores the event in a json format, and bt.client.PublishEvent(event) publishes data to Elasticsearch. In the generated Beat, there are three fields in the event: @timestamp, type, and counter.

When you add fields to the event object, you also need to add them to the _meta/fields.yml file:

- key: countbeat
  title: countbeat
  description:
  fields:
    - name: counter
      type: long
      required: true
      description: >
        PLEASE UPDATE DOCUMENTATION

Remember to run make update to apply your updates.

For more information on defining field mappings in _meta/fields.yml, see [event-fields-yml].

For more detail about naming the fields in an event, see [event-conventions].

Stop Method

The Stop method is called when the Beat is signaled to stop, for example through the SIGTERM signal on Unix systems or the service control interface on Windows. This method simply closes the channel which breaks the main loop.

func (bt *Countbeat) Stop() {
	bt.client.Close()
	close(bt.done)
}

The main Function

If you follow the Countbeat model and put your Beat-specific code in its own type that implements the Beater interface, the code from your main package is very simple:

package main

import (
	"os"

	"github.com/elastic/beats/libbeat/beat"
	"github.com/elastic/beats/libbeat/cmd"
	"github.com/elastic/beats/libbeat/cmd/instance"

	"github.com/kimjmin/countbeat/beater"
)

var RootCmd = cmd.GenRootCmdWithSettings(beater.New, instance.Settings{Name: "countbeat"})

func main() {
	if err := RootCmd.Execute(); err != nil {
		os.Exit(1)
	}
}

Sharing Your Beat with the Community

When you’re done with your new Beat, how about letting everyone know? Open a pull request to add your link to the {beats-ref}/community-beats.html[Community Beats] list.

Migrate existing Beat to go modules

Get started by making sure the contents of the vendored beats folder is pushed to a remote repository. Otherwise, go will overwrite the folder with the selected revision of github.com/elastic/beats.

The following command will initialize a new module in your repo.

go mod init

Make sure to add the approprite replace directives from the go.mod file of Beats: https://github.com/elastic/beats/tree/master/go.mod Unfortunately, this workaround is needed to make sure your Beat can be compiled.

To depend on the latest master of github.com/elastic/beats run the following command:

go get github.com/elastic/beats@master

We suggest you read the following section to learn about maintaining dependencies using go modules: * How to upgrade and downgrade dependencies

Please note that it is your choice whether you put dependencies in the folder vendor or not.

If you choose to abandon the vendor folder, you have to adjust the following things in your Beat:

add devtools.CrossBuildMountModcache = true to the magefile.go of your Beat
modify the path to ES_BEATS in the Makefile so it points to the folder under the module cache (go list -m -f '{{.Dir}}' $ES_BEATS_IMPORT_PATH)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

newbeat.asciidoc

newbeat.asciidoc

Creating a New Beat

Getting Ready

Overview

Generating Your Beat

Fetching Dependencies and Setting up the Beat

Building and Running the Beat

The Beater Interface

New function

Run Method

Stop Method

The main Function

Migrate existing Beat to go modules

Files

newbeat.asciidoc

Latest commit

History

newbeat.asciidoc

File metadata and controls

Creating a New Beat

Getting Ready

Overview

Generating Your Beat

Fetching Dependencies and Setting up the Beat

Building and Running the Beat

The Beater Interface

New function

Run Method

Stop Method

The main Function

Sharing Your Beat with the Community

Migrate existing Beat to go modules