[[config-file-format]]
== Config file format

Beats config files are based on http://www.yaml.org[YAML], a file format that is
easier to read and write than other common data formats like XML or JSON.

In beats all YAML files start with a dictionary, an unordered collection of
name/value pairs. In addition to dictionaries, YAML also supports lists, numbers,
strings, and many other data types. All members of the same list or dictionary must
have the same indentation level.

Dictionaries are represented by simple `key: value` pairs all having the same
indentation level. The colon after `key` must be followed by a space.

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
name: John Doe
age: 34
country: Canada
------------------------------------------------------------------------------

Lists are introduced by dashes `- `. All list members will be lines beginning
with `- ` at the same indentation level.

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
- Red
- Green
- Blue
------------------------------------------------------------------------------

Lists and dictionaries are used in beats to build structured configurations.

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
filebeat:
  prospectors:
    - input_type: log
      paths:
        - /var/log/*.log
      multiline:
        pattern: '^['
        match: after
------------------------------------------------------------------------------

Lists and dictionaries can also be represented in abbreviated form. Abbreviated
form is somewhat similar to JSON using `{}` for dictionaries and `[]` for lists:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
person: \{name: "John Doe", age: 34, country: "Canada"}
colors: ["Red", "Green", "Blue"]
------------------------------------------------------------------------------

The following topics provide more detail to help you understand and work with config files in YAML:

* <<config-file-format-namespacing>>
* <<config-file-format-type>>
* <<config-file-format-env-vars>>
* <<config-gile-format-refs>>
* <<config-file-format-cli>>
* <<config-file-format-tips>>

[[config-file-format-namespacing]]
=== Namespacing

All settings are structured using dictionaries and lists. Those are collapsed
into "namespaced" settings, by creating a setting using the full path of the
settings name and it's parent structures names, when reading the configuration
file.

For example this setting:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------

output:
  elasticsearch:
    index: 'beat-%{+yyyy.MM.dd}'

------------------------------------------------------------------------------

gets collapsed into `output.elasticsearch.index: 'beat-%{+yyyy.MM.dd}'`. The
full name of a setting is based on all parent structures involved.

Lists create numeric names starting with 0.

For example this filebeat setting:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------

filebeat:
  prospectors:
    - input_type: log

------------------------------------------------------------------------------

Gets collapsed into `filebeat.prospectors.0.input_type: log`.

Alternatively to using indentation, setting names can be used in collapsed form too.

Note: having two settings with same fully collapsed path is invalid.

Simple filebeat example with partially collapsed setting names and use of compact form:


["source","yaml",subs="attributes"]
------------------------------------------------------------------------------

filebeat.prospectors:
- input_type: log
  paths: ["/var/log/*.log"] 
  multiline.pattern: '^['
  multiline.match: after

output.elasticsearch.hosts: ["http://localhost:9200"]

------------------------------------------------------------------------------

[[config-file-format-type]]
=== Config file Data Types

Values of configuration settings are interpreted as required by beats.
If a value can not be correctly interpreted as the required type - for example a
string is given when a number is required - the beat will fail to start up.

==== Boolean

Boolean values can be either `true` or `false`. Alternative names for `true` are
`yes` and `on`. Instead of `false` the values `no` and `off` can be used.

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
enabled: true
disabled: false
------------------------------------------------------------------------------

==== Number

Number values require you to enter the number to use without using single or
double quotes. Some settings only support a restricted number range though.

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
integer: 123
negative: -1
float: 5.4
------------------------------------------------------------------------------

==== String

In YAML[http://www.yaml.org], multiple styles of string definitions are supported:
 double-quoted, single-quoted, unquoted.

The double-quoted style is specified by surrounding the string with `"`. This
style provides support for escaping unprintable characters using `\`, but comes
at the cost of having to escape `\` and `"` characters.

The single-quoted style is specified by surrounding the string with `'`. This
style supports no escaping (use `''` to quote a single quote). Only printable
characters can be used when using this form.

Unquoted style requires no quotes, but does not support any escaping plus care
needs to be taken to not use any symbol that has a special meaning in YAML.

Note: Single-quoted style is recommended when defining regular expressions,
event format strings, windows file paths, or non-alphabetical symbolic characters.

==== Duration

Durations require a numeric value with optional fraction and required unit.
Valid time units are `ns`, `us`, `ms`, `s`, `m`, `h`. Sometimes features based
on durations can be disabled by using zero or negative durations.

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
duration1: 2.5s
duration2: 6h
duration_disabled: -1s
------------------------------------------------------------------------------

==== Regular Expression

Regular expressions are special strings getting compiled into regular
expressions at load time.

As regular expressions and YAML use `\` for escaping
characters in strings, it's highly recommended to use single quoted strings when
defining regular expressions. When single quoted strings are used, `\` character
is not interpreted by YAML parser as escape symbol.

==== Format String (sprintf)

Format strings enable you to refer to event field values creating a string based
on the current event being processed. Variable expansions are enclosed in
expansion braces `%{<accessor>:default value}`. Event fields are accessed using
field references `[fieldname]`. Optional default values can be specified in case the
field name is missing from the event.

You can also format time stored in the
`@timestamp` field using the `+FORMAT` syntax where FORMAT is a valid (time
format)[https://godoc.org/github.com/elastic/beats/libbeat/common/dtfmt].

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
constant-format-string: 'constant string'
field-format-string: '%{[fieldname]} string'
format-string-with-date: '%{[fieldname]}-%{+yyyy.MM.dd}'
------------------------------------------------------------------------------


[[config-file-format-env-vars]]
=== Environment Variables

Beats support use of environment variables in config files to set values that
need to be configurable during deployment. Environment variable expansion is
introduced using `${VAR}`, where `VAR` is the name of the environment variable.

Note: Only values can be set using environment variables. Environment variables
usage in namespace and setting names are not supported.

Variable references are replaced when settings are read by beats. The
replacement is case-sensitive and occurs after the YAML file itself has been
parsed. References to undefined variables will lead to errors when dereferenced
and no default value is specified. To specify a default value, use:

`${VAR:default_value}`

Where `default_value` is the value to use if the environment variable is
undefined.

If you need to use a literal `${` in your configuration file then you can write
`$${` to escape the expansion. The `$` symbol can be used to escape other
characters in the default_value like using `$}` in order to generate a `}`
character without closing the variable expansion.

After changing the value of an environment variable, the beat needs to be
restarted to pick up the new value.

[float]
==== Examples

Here are some examples of configurations that use environment variables
and what each configuration looks like after replacement:

[options="header"]
|==================================
|Config source	       |Environment setting   |Config after replacement
|`name: ${NAME}`       |`export NAME=elastic` |`name: elastic`
|`name: ${NAME:beats}` |no setting            |`name: beats`
|`name: ${NAME:beats}` |`export NAME=elastic` |`name: elastic`
|==================================

[[config-gile-format-refs]]
=== Reference Variables

Beats settings can reference other settings splicing multiple optionally custom
named settings into new values. References use the same syntax as
<<config-file-format-env-vars>> do. Only fully collapsed setting names can be
referenced to.

For example the filebeat registry file defaults to:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
filebeat.registry: $\{path.data}/registry
------------------------------------------------------------------------------

With `path.data` being an implicit config setting, that is overwritable from
command line, as well as in the configuration file.

Example referencing `es.host` in `output.elasticsearch.hosts`:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
es.host: '$\{ES_HOST:localhost}'

output.elasticsearch:
  hosts: ['http://$\{es.host}:9200']
------------------------------------------------------------------------------

Introducing `es.host`, the host can be overwritten from command line using
`-E es.host=another-host`.

Plain references, having no default value and are not spliced with other
references or strings can reference complete namespaces.

These setting with duplicate content:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
namespace1:
  subnamespace:
    host: localhost
    sleep: 1s

namespace2:
  subnamespace:
    host: localhost
    sleep: 1s
------------------------------------------------------------------------------

can be rewritten to

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
namespace1: $\{shared}
namespace2: $\{shared}

shared:
  subnamespace:
    host: localhost
    sleep: 1s
------------------------------------------------------------------------------

when using plain references.


[[config-file-format-cli]]
=== Command line arguments

Config files to load are set using the `-c` flag on command line. If no flag is
given, a beat and OS-specific default file path will be assumed.

You can specify multiple configuration files by repeating the `-c` flag. You can
use this, for example, for setting defaults in a base configuration file, and
overwrite settings via local configuration files.

In addition to overwriting settings using multiple configuration files,
individual settings can be overwritten using `-E <setting>=<value>`.

[[config-file-format-tips]]
=== YAML Tips and Gotchas

When you edit the configuration file, there are a few things that you should know.

[float]
==== Use Spaces for Indentation

Indentation is meaningful in YAML. Make sure that you use spaces, rather than
tab characters, to indent sections.

In the default configuration files and in all the examples in the documentation,
we use 2 spaces per indentation level. We recommend you do the same.

[float]
==== Look at the Default Config File for Structure

The best way to understand where to define a configuration option is by looking
at the provided sample configuration files. The configuration files contain most
of the default configurations that are available per beat. To change a setting,
simply uncomment the line and change the values.

[float]
==== Test Your Config File

You can test your configuration file to verify that the structure is valid. Simply run
your in the foreground with the `-configtest` flag specified. For example:

["source","yaml",subs="attributes,callouts"]
----------------------------------------------------------------------
filebeat -c filebeat.yml -configtest
----------------------------------------------------------------------

You'll see a message if an error in the configuration file is found.

[float]
==== Wrap Regular Expressions in Single Quotation Marks

If you need to specify a regular expression in a YAML file, it's a good idea to
wrap the regular expression in single quotation marks to work around YAML's
tricky rules for string escaping.

For more information about YAML, see http://yaml.org/.
