[[new-beat]]
== Developer Guide: Creating a New Beat

This guide walks you through the steps for creating a new Elastic Beat.  The
Beats are a collection of lightweight daemons that collect operational data from
your servers and ship it to Elasticsearch or Logstash.  The common parts for
all Beats are placed in the libbeat library, which contains packages for sending
data to Elasticsearch and Logstash, for configuration file handling, for signal
handling, for logging, and more. By using this common framework, you can ensure
that all Beats behave consistently and that they are easy to package and run
with common tools.

In this guide, you learn how to use the Beat generator to create source code for
an example Beat called Countbeat. The Beat generator creates all the files required
for a working Beat. To create your own Beat, you modify the generated
files and implement the custom logic needed to collect the data you want to ship.

The following topics describe how to build a new Beat:

* <<newbeat-getting-ready>>
* <<newbeat-overview>>
* <<newbeat-generate>>
* <<setting-up-beat>>
* <<compiling-and-running>>
* <<beater-interface>>
* <<ls-beat>>
* <<newbeat-sharing>>
* <<event-conventions>>

[[newbeat-getting-ready]]
=== Getting Ready

All Beats are written in http://golang.org/[Go], so having Go installed and knowing
the basics are prerequisites for understanding this guide.
But don't worry if you aren't a Go expert. Go is a relatively new
language, and very few people are experts in it. In fact, several
people learned Go by contributing to Packetbeat and libbeat, including the
original Packetbeat authors.

For general information about contributing to Beats, see our
https://github.com/elastic/beats/blob/master/CONTRIBUTING.md[Contributing Guide].

After you have https://golang.org/doc/install[installed Go] and set up the
https://golang.org/doc/code.html#GOPATH[GOPATH] environment variable to point to
your preferred workspace location, a simple way of getting the source code for 
Beats, including libbeat and the Beat generator, is to do:

[source,shell]
----------------------------------------------------------------------
go get github.com/elastic/beats
----------------------------------------------------------------------

When you run the command, all source files are downloaded to the
`$GOPATH/src/github.com/elastic/beats` path. You can ignore the "no buildable Go source files" message because
you will build the source later.

[[newbeat-overview]]
=== Overview

At the high level, a simple Beat has two main components:

* a component that collects the actual data, and
* a publisher that sends the data to the specified output, such as Elasticsearch or
Logstash.

The publisher is already implemented in libbeat, so you typically only have to worry about the logic
specific to your Beat (the code that creates the event and sends it to the publisher).
Libbeat also offers common services like configuration management, logging,
daemonzing, and Windows service handling, and in the future, will offer data processing modules,
such as filtering or sampling.

image:./images/beat_overview.png[Beat overview architecture]

The event that you create is a JSON-like object (Golang type `map[string]interface{}`) that
contains the collected data to send to the publisher. At a minimum, the event object
must contain a `@timestamp` field and a `type` field. Beyond
that, events can contain any additional fields, and they can be created as often
as necessary.

The following example shows an event object in Lsbeat:

[source,json]
----------------------------------------------------------------------
{
  "@timestamp": "2016-07-13T21:33:58.355Z",
  "beat": {
    "hostname": "mar.local",
    "name": "mar.local"
  },
  "directory": false,
  "filename": "winlogbeat.yml",
  "filesize": 2895,
  "modtime": "2016-07-13T20:56:21.000Z",
  "path": "./vendor/github.com/elastic/beats/winlogbeat/winlogbeat.yml",
  "type": "lsbeat"
}

{
  "@timestamp": "2016-07-13T21:33:58.354Z",
  "beat": {
    "hostname": "mar.local",
    "name": "mar.local"
  },
  "directory": true,
  "filename": "system",
  "filesize": 238,
  "modtime": "2016-07-13T20:56:21.000Z",
  "path": "./vendor/github.com/elastic/beats/winlogbeat/tests/system",
  "type": "lsbeat"
}
----------------------------------------------------------------------

Now that you have the big picture, let's dig into the code.

[[newbeat-generate]]
=== Generating Your Beat

To generate your own Beat, you use the Beat generator available in the beats repo on GitHub. If you haven't
downloaded the Beats source code yet, follow the instructions in <<newbeat-getting-ready>>.

Before running the Beat generator, you must install https://github.com/audreyr/cookiecutter[cookiecutter], a
command-line utility that creates projects from project templates. Check out the 
http://cookiecutter.readthedocs.io/en/latest/installation.html[installation guide]. After installing cookiecutter,
decide on a name for your beat. The name should be one word with the first letter capitalized. In our example,
we use `Countbeat`.

Now create a directory under $GOPATH for your repository and change to the new directory:

[source,shell]
--------------------
mkdir $GOPATH/src/github.com/{user}
cd $GOPATH/src/github.com/{user}
--------------------

Run cookiecutter and specify the path to the Beat generator:

[source,shell]
--------------------
cookiecutter $GOPATH/src/github.com/elastic/beats/generate/beat
--------------------

Cookiecutter will prompt you to enter information about your Beat. For the `project_name`, enter `Countbeat`.
For the `github_name`, enter your github id. The `beat` and `beat_path` are set to the correct values automatically (just press Enter to accept each default). For the `full_name`, enter your firstname and lastname.

[source,shell]
---------
project_name [Examplebeat]: Countbeat
github_name [your-github-name]: {username}
beat [countbeat]:
beat_path [github.com/{github id}]:
full_name [Firstname Lastname]: {Full Name}
---------

The Beat generator creates a directory called `countbeat` inside of your project folder. 

You now have a raw template of the Beat, but you still need to <<setting-up-beat,fetch dependencies and set up the Beat>>.

[[setting-up-beat]]
=== Fetching Dependencies and Setting up the Beat

First you need to install the following tools: 

* https://www.python.org/downloads/[Python]
* https://virtualenv.pypa.io/en/stable/[virtualenv]

To fetch dependencies and set up the Beat, run:

[source,shell]
---------
cd $GOPATH/src/github.com/{user}/countbeat
make setup
---------

The Beat now contains the basic config file, `countbeat.yml`, and template files. The Beat is "complete" in the sense
that you can compile and run it. However, to make it functionally complete, you need to add your custom logic (see <<beater-interface>>), along with any additional configuration parameters that your Beat requires.

[[compiling-and-running]]
=== Building and Running the Beat

To compile the Beat, make sure you are in the Beat directory (`$GOPATH/src/github.com/{user}/countbeat`) and run:

[source,shell]
---------
make 
---------

Running this command creates the binary called `countbeat` in `$GOPATH/src/github.com/{user}/countbeat`.

Now run the Beat:

[source,shell]
---------
./countbeat -e -d "*" 
---------

The command automatically loads the default config file, `countbeat.yml`, and sends debug output to the console.

You can stop the Beat by pressing `Ctrl+C`. 

[[beater-interface]]
=== The Beater Interface

Each Beat needs to implement the Beater interface defined in libbeat.

[source,go]
----------------------------------------------------------------------
// Beater is the interface that must be implemented by every Beat. A Beater
// provides the main Run-loop and a Stop method to break the Run-loop.
// Instantiation and Configuration is normally provided by a Beat-`Creator`.
//
// Once the beat is fully configured, the Run() method is invoked. The
// Run()-method implements the beat its run-loop. Once the Run()-method returns,
// the beat shuts down.
//
// The Stop() method is invoked the first time (and only the first time) a
// shutdown signal is received. The Stop()-method normally will stop the Run()-loop,
// such that the beat can gracefully shutdown.
type Beater interface {
	// The main event loop. This method should block until signalled to stop by an
	// invocation of the Stop() method.
	Run(b *Beat) error

	// Stop is invoked to signal that the Run method should finish its execution.
	// It will be invoked at most once.
	Stop()
}
----------------------------------------------------------------------

To implement the Beater interface, you need to define a Beat object that
implements two methods: <<run-method,`Run()`>> and <<stop-method,`Stop()`>>. 

[source,go]
--------------
type Countbeat struct {
	done   chan struct{} <1>
	config config.Config <2>
	client publisher.Client <3>

	...
}

func (bt *Countbeat) Run(b *beat.Beat) error {
	...
}


func (bt *Countbeat) Stop() {
	...
}

--------------

By default, the Beat object contains the following:

<1> `done`: Channel used by the `Run()` method to stop when the `Stop()` method is called.
<2> `config`: Configuration options for the Beat
<3> `client`: Publisher that takes care of sending the events to the
  defined output.

The `Beat` parameter received by the `Run` method contains data about the
Beat, such as the name, version, and common configuration options.

Each Beat also needs to implement the <<new-function,`New()`>> function to create the Beat object. This means your
Beat should implement the following functions:

[horizontal]
<<new-function, New>>:: Creates the Beat object
<<run-method, Run>>:: Contains the main application loop that captures data
and sends it to the defined output using the publisher
<<stop-method, Stop>>:: Contains logic that is called when the Beat is signaled to stop

When you run the Beat generator, it adds implementations for all these functions to the source code (see
`beater/countbeat.go`). You can modify these implementations, as required, for your Beat.

We strongly recommend that you create a main package that contains only the main
method (see `main.go`). All your Beat-specific code should go in a separate folder and package.
This will allow other Beats to use the other parts of your Beat as a library, if
needed.

NOTE: To be consistent with other Beats, you should append `beat` to your Beat name.

Let's go through each of the methods in the `Beater` interface and look at a
sample implementation.

[[new-function]]
==== New function

The `New()` function receives the configuration options defined for the Beat and
creates a Beat object based on them. Here's the `New()` function that's generated in
`beater/countbeat.go` when you run the Beat generator:

[source,go]
----------------------------------------------------------------------
func New(b *beat.Beat, cfg *common.Config) (beat.Beater, error) {
	config := config.DefaultConfig
	if err := cfg.Unpack(&config); err != nil {
		return nil, fmt.Errorf("Error reading config file: %v", err)
	}

	bt := &Countbeat{
		done:   make(chan struct{}),
		config: config,
	}
	return bt, nil
}

----------------------------------------------------------------------

Pointers are used to distinguish between when the setting is completely
missing from the configuration file and when it has a value that matches the
type's default value.

The recommended way of handling the configuration (as shown in the code example)
is to create a `Config` structure with the configuration options and a `DefaultConfig` with
the default configuration options. 

When you use the Beat generator, the Go structures for a basic config are added to `config/config.go`:

[source,go]
----------------------------------------------------------------------
package config

import "time"

type Config struct {
	Period time.Duration `config:"period"`
}

var DefaultConfig = Config{
	Period: 1 * time.Second,
}

----------------------------------------------------------------------


This mirrors the config options that are defined in the config file, `countbeat.yml`. 

[source,yaml]
------------
countbeat:
  # Defines how often an event is sent to the output
  period: 10s
------------

 - `period`: Defines how often to send out events
 
The config file is generated when you run `make setup` to <<setting-up-beat,set up the beat>>. The file contains
basic configuration information. To add configuration options to your Beat, you need to update the Go structures in
`config/config.go` and add the corresponding config options to `etc/beat.yml`.

For example, if you add a config option called `path` to the Go structures: 

[source,go]
----------------------------------------------------------------------
type Config struct {
    Period time.Duration `config:"period"`
    Path   string        `config:"path"`
}

var DefaultConfig = Config{
    Period: 1 * time.Second,
    Path:   ".",
}

----------------------------------------------------------------------

You also need to add `path` to `etc/beat.yml`: 

[source,yml]
----------------------------------------------------------------------
countbeat:
  period: 10s
  path: "."
----------------------------------------------------------------------

After modifying `beat.yml`, run the following command to apply your updates:

[source,shell]
----------------------------------------------------------------------
make update
----------------------------------------------------------------------

[[run-method]]
==== Run Method

The `Run` method contains your main application loop. 

[source,go]
----------------------------------------------------------------------
func (bt *Countbeat) Run(b *beat.Beat) error {
	logp.Info("countbeat is running! Hit CTRL-C to stop it.")

	bt.client = b.Publisher.Connect()
	ticker := time.NewTicker(bt.config.Period)
	counter := 1
	for {
		select {
		case <-bt.done:
			return nil
		case <-ticker.C:
		}

		event := common.MapStr{ <1>
			"@timestamp": common.Time(time.Now()), <2>
			"type":       b.Name,
			"counter":    counter,
		}
		bt.client.PublishEvent(event) <3>
		logp.Info("Event sent")
		counter++
	}
}
----------------------------------------------------------------------
<1> Create the event object.
<2> Specify a `@timestamp` field of time `common.Time`.
<3> Use the publisher to send the event out to the defined output

Inside the loop, the Beat sleeps for a configurable period of time and then
captures the required data and sends it to the publisher. The publisher client is available as part of the Beat object
through the `client` variable.

The `event := common.MapStr{}` stores the event in a json format, and `bt.client.PublishEvent(event)` publishes data to Elasticsearch. 
In the generated Beat, there are three fields in the event: @timestamp, type, and counter. 

When you add fields to the event object, you also need to add them to the `etc/fields.yml` file:

[source,yaml]
----------------------------------------------------------------------
- key: countbeat
  title: countbeat
  description:
  fields:
    - name: counter
      type: long
      required: true
      description: >
        PLEASE UPDATE DOCUMENTATION
----------------------------------------------------------------------

Remember to run `make update` to apply your updates.

For more detail about naming the fields in an event, see <<event-conventions>>.

[[stop-method]]
==== Stop Method

The `Stop` method is called when the Beat is signaled to stop, for
example through the SIGTERM signal on Unix systems or the service control
interface on Windows. This method simply closes the channel 
which breaks the main loop.

[source,go]
----------------------------------------------------------------------
func (bt *Countbeat) Stop() {
	bt.client.Close()
	close(bt.done)
}
----------------------------------------------------------------------

==== The main Function

If you follow the `Countbeat` model and put your Beat-specific code in its own type
that implements the `Beater` interface, the code from your main package is
very simple:

[source,go]
----------------------------------------------------------------------
package main

import (
	"os"

	"github.com/elastic/beats/libbeat/beat"

	"github.com/kimjmin/countbeat/beater"
)

func main() {
	err := beat.Run("countbeat", "", beater.New)
	if err != nil {
		os.Exit(1)
	}
}
----------------------------------------------------------------------


[[ls-beat]]
=== Example: Building Lsbeat from Scratch

https://github.com/kimjmin/lsbeat[Lsbeat] is similar to the `ls` command-line tool, but instead of printing the files and subdirectories to the screen,
Lsbeat periodically ships them to Elasticsearch for storage. 

To help you learn how to build a Beat, we've created this http://elastic.co//blog/build-your-own-beat[blog post]
that describes how to build Lsbeat from scratch. You can refer to the Lsbeat implementation as a working example.

[[newbeat-sharing]]
=== Sharing Your Beat with the Community

When you're done with your new Beat, how about letting everyone know? Open
a pull request to add your link <<community-beats, here>>.
