Logstash

From Leo's Notes
Revision as of 05:13, 29 March 2015 by Leo (talk | contribs)
This page was last edited on 29 March 2015, at 05:13.

Logstash is the open source copy of splunk - a log capturing, indexing, and searching service - using ElasticSearch as its search engine.

Installation

For detailed information, consult logstash's tutorial (http://logstash.net/docs/). Prior to logstash 1.4.0, the logstash package comes as a monolithic .jar file. To get started, install java and run the jar file.

If you are using logstash >= 1.4.0, it's probably easier to just install logstash and ElasticSearch from their repository: 

[logstash-1.4]
name=logstash repository for 1.4.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1


[elasticsearch-1.3]
name=Elasticsearch repository for 1.3.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.3/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

Then run yum install logstash elasticsearch

Configuration

The configuration files for logstash are located at:

  1. /etc/sysconfig/logstash
  2. /etc/logstash/conf.d/

The files in the conf.d directory will be treated as a single configuration file in sequence by their filenames by logstash.

You may want to change the DATA_DIR path in the logstash configuration. The actual input/parse/output configurations will be placed in the conf.d directory. More on this below.

ElasticSearch's files are at:

  1. /etc/elasticsearch/elasticsearch.yml

You will need to configure ElasticSearch based on how you want to set up your search. For replication/sharding, you should ideally have more than one server. If you do have more than one server, make sure you have the node names set and have the autodiscoverer configured.


Configuration

The configuration you provide logstash defines how logstash deals with incoming messages. There are three main parts to the configuration:

  1. Input
  2. Filter
  3. Output

Input defines what ports logstash listens on and how to tag the incoming messages. Filter defines what logstash needs to do on the incoming messages, based on the tags defined from the input step. Output defines how these messages are stored.

For example, my current configuration is: 

input {

	# Import syslog messages
	tcp {
		type => syslog_import
		port => 4401
	}

	# Accept syslog messages from hosts
	syslog {
		type => syslog
		port => 5544
	}
}



filter {

	if [type] == "syslog" {
		# Does the syslog parsing.
		syslog_pri { }

		mutate {
			replace => [ "@source", "%{logsource}" ]
			replace => [ "@message", "%{message}" ]
			replace => [ "@program", "%{program}" ]
			replace => [ "@type", "syslog" ]
		}

		# Date is parsed and placed into @timstamp.
		date {
			match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ]
		}

		# Clean up the extra syslog_ fields generated above from grok.
		mutate {
			remove_field => [ "syslog_hostname", "syslog_message", "syslog_program", "syslog_timestamp", "type", "message", "logsource", "program"]
		}
	}

	# For imported syslog messages...
	if [type] == "syslog_import" {
		
		if [message] =~ /last message repeated.*/ {
			drop {
			}
		}
		
		if [message] == "" {
			drop {
			}
		}


		# Parse with grok
		grok {
			# Use the custom SYSLOGYEARTIMESTAMP pattern from the patterns
			# directory. We need this to define year.
			patterns_dir => "./patterns"

			# The pattern to match.
			# This is the standard syslog pattern.
			match => { "message" => "%{SYSLOGYEARTIMESTAMP:syslog_timestamp} (%{USER:syslog_user}\@)?%{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

			# Add a few intermediate fields
			add_field => [ "received_at", "%{@timestamp}" ]
			add_field => [ "received_from", "%{host}" ]
		}
		
		# When the above grok parsing fails, a '_grokparsefailure' tag gets
		# added to the message. In that case, we attempt to update some fields.
		# Why? Beats me.
		if !("_grokparsefailure" in [tags]) {
			mutate {
				replace => [ "@source", "%{syslog_hostname}" ]
				replace => [ "@message", "%{syslog_message}" ]
				replace => [ "@program", "%{syslog_program}" ]
				replace => [ "@type", "syslog imported" ]
			}
		}

		# Parse the date. This puts it into the @timestamp field on a successful
		# parse.
		date {
			match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "YYYY MMM  d HH:mm:ss", "YYYY MMM dd HH:mm:ss" ]
		}

		# Clean up the extra syslog_ fields generated above from grok.
		mutate {
			remove_field => [ "syslog_hostname", "syslog_message", "syslog_program", "syslog_timestamp", "type", "message", "host" ]
		}

	}

}


output {
	# Debugging
	# stdout {
	# 	codec => json
	# }

	elasticsearch {
		# Define our own... hosted on my computer
		# bind_host => "leo-linux"
		# bind_port => 9200
		host => "127.0.0.1"
		port => 9300

		cluster => "logstash_es"
		node_name => "logstash_0"

		# Index defaults to 'logstash-%{+YYYY.MM.dd}'
		# The templates being used can be defined using:
		template => "/etc/logstash/template/logstash.json"
		
	}
}

The major sections of the configuration file are:

  1. Inputs
  2. Parsing/Filtering
  3. Outputs

You can see the entire list of available plugins for each of these sections at: http://logstash.net/docs/1.4.2/

Inputs

The inputs define what logstash will listen to for information. The configuration above listens on a tcp port (for backlog imports) for raw text and another port for syslog.

Parsing / Filtering

For each of the inputs, certain things are done in order to parse the input data into variables which are then passed to the output section.

Operations to the data are done through more sets of plugins (think: functions). For example, text parsing is done through grok. Parameters to these 'functions' are passed as parameters in the grok block. When grok fails at parsing a certain string, it will add an additional tag (ie: variable) called _grokparsefailure which can be used later on in the parse section.

Variables starting with a '@' are used by ElasticSearch to denote mandatory fields (... I think?) which are defined in the ElasticSearch template file.

Example

grok {
	# Use the custom SYSLOGYEARTIMESTAMP pattern from the patterns
	# directory. We need this to define year.
	patterns_dir => "./patterns"

	# The pattern to match.
	# This is the standard syslog pattern.
	match => { "message" => "%{SYSLOGYEARTIMESTAMP:syslog_timestamp} (%{USER:syslog_user}\@)?%{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

	# Add a few intermediate fields
	add_field => [ "received_at", "%{@timestamp}" ]
	add_field => [ "received_from", "%{host}" ]
}

This grok instance attempts to match the incoming message to the defined pattern. The syntax defining matched strings is %{PATTERN_NAME:variable_name} where PATTERN_NAME is a grok-pattern defined in /patterns/* in the .jar file and also in the directory defined in the patterns_dir directory, and variable_name is the name that can be used to referenced the matched value later on in the grok instance.

The SYSLOGYEARTIMESTAMP pattern is a custom pattern defined in my ./patterns directory. 

cat patterns/extra 
SYSLOGYEARTIMESTAMP %{YEAR} %{MONTH} +%{MONTHDAY} %{TIME}

In the case above, syslog messages being imported whose date field matches the format given in SYSLOGYEARTIMESTAMP will be placed in the variable syslog_timestamp.

Outputs

The outputs section defines what LogStash will do with the variables generated from the parsing/filtering section.

To debug the inputs/filtering section, you can do: 

stdout {
	codec => json
}

Variables / tags generated can be seen as part of a json object.

ElasticSearch takes in a template which defines the schema of indexes generated by logstash. This template is optional, since logstash will use a default template by default.

In the configuration example above, a template was defined for the ElasticSearch output. 

{
    "template": "logstash-*",
    "settings" : {
        "index.query.default_field" : "@message"
    },
    "mappings": {
        "_default_": {
            "_all": { "enabled": false },
            "_source": { "compress": false },
            "dynamic_templates": [
                {
                    "fields_template" : {
                        "mapping": { "type": "string", "index": "not_analyzed" },
                        "path_match": "@fields.*"
                    }
                },
                {
                    "tags_template" : {
                        "mapping": { "type": "string", "index": "not_analyzed" },
                        "path_match": "@tags.*"
                    }
                }
            ],
            "properties" : {
                "@fields": { "type": "object", "dynamic": true, "path": "full" },
                "@timestamp" : { "type" : "date", "index" : "not_analyzed" },
                "@program" : { "type" : "string", "index" : "not_analyzed" },
                "@source" : { "type" : "string", "index" : "not_analyzed" },
                "@message" : { "type" : "string", "analyzer" : "whitespace" },
                "@type" : { "type" : "string", "index" : "not_analyzed" }
             }
        }
    }
}

The variables/tags that were generated from the parse field should match the property names defined in the template file. Depending on what you want out of ElasticSearch, you may or may not want to have every field analyzed.

Be careful with templates though. If the properties defined in the template file are not provided by the filtering/parsing section, the log entry will not be added to ElasticSearch.

See Also

Linux