Logstash - Revision history

Leo at 18:18, 8 November 2023

2023-11-08T18:18:27Z

@@ Line 474: / Line 474: @@
 One '<code>@</code>' denotes UDP. Use '<code>@@</code>' to have logs shipped via TCP. TCP transfers might be a bad idea since your rsyslog server might get bogged down trying to resend logs if the syslog server dies.
-==See Also==
 *[https://groups.google.com/forum/#!msg/logstash-users/X6kNHU0alBg/j95HZkTLo-EJ https://groups.google.com/forum/#!msg/ Logstash-users/X6kNHU0alBg/j95HZkTLo-EJ]

Leo: /* Example */

2021-02-05T23:34:02Z

Example

← Older revision		Revision as of 23:34, 5 February 2021
Line 276:		Line 276:
	<br />		<br />

	== Example ==		==Example==

	=== Rsyslog Logging ===		===Rsyslog Logging===
	Use a pipeline similar to the following:		Use a pipeline similar to the following:
	{{Highlight		{{Highlight
Line 467:		Line 467:
	SYSLOG %{SYSLOG5424}{{!}}%{SYSLOG3164}		SYSLOG %{SYSLOG5424}{{!}}%{SYSLOG3164}
	\| lang = text		\| lang = text
	}}Start Logstash. Configure rsyslog so that all logs are sent to the logstash server by appending to rsyslog.conf:		}}Start Logstash. Configure rsyslog so that all logs are sent to the logstash server by appending to <code>rsyslog.conf</code>:
	{{Highlight		{{Highlight
	\| code = . @10.1.2.3:5144		\| code = . @10.1.2.3:5144
Line 473:		Line 473:
	}}		}}

			One '<code>@</code>' denotes UDP. Use '<code>@@</code>' to have logs shipped via TCP. TCP transfers might be a bad idea since your rsyslog server might get bogged down trying to resend logs if the syslog server dies.
	==See Also==		==See Also==

Leo: /* Inputs */

2021-02-05T23:32:33Z

Inputs

Show changes

Leo at 23:28, 5 February 2021

2021-02-05T23:28:57Z

Show changes

Leo: Text replacement - "Category:Linux{{Navbox Linux}}" to "{{Navbox Linux}}Category:Linux"

2019-09-01T06:21:22Z

Text replacement - "Category:Linux{{Navbox Linux}}" to "{{Navbox Linux}}Category:Linux"

← Older revision		Revision as of 06:21, 1 September 2019
Line 284:		Line 284:


	[[Category:Linux]]~~{{Navbox Linux}}~~		{{Navbox Linux}}[[Category:Linux]]

	{{Navbox Services}}		{{Navbox Services}}

Leo: Text replacement - "{{Navbox Linux}}{{Navbox Linux}}" to "{{Navbox Linux}}"

2019-09-01T05:10:58Z

Text replacement - "{{Navbox Linux}}{{Navbox Linux}}" to "{{Navbox Linux}}"

← Older revision		Revision as of 05:10, 1 September 2019
Line 284:		Line 284:


	[[Category:Linux]]~~{{Navbox Linux}}~~{{Navbox Linux}}		[[Category:Linux]]{{Navbox Linux}}

	{{Navbox Services}}		{{Navbox Services}}

Leo: Text replacement - "Category:Linux" to "Category:Linux{{Navbox Linux}}"

2019-09-01T05:07:41Z

Text replacement - "Category:Linux" to "Category:Linux{{Navbox Linux}}"

← Older revision		Revision as of 05:07, 1 September 2019
Line 284:		Line 284:


	[[Category:Linux]]{{Navbox Linux}}		[[Category:Linux]]{{Navbox Linux}}{{Navbox Linux}}

	{{Navbox Services}}		{{Navbox Services}}

Leo: Text replacement - "Category:Linux " to "Category:Linux{{Navbox Linux}} "

2019-09-01T03:44:16Z

Text replacement - "Category:Linux " to "Category:Linux{{Navbox Linux}} "

← Older revision		Revision as of 03:44, 1 September 2019
Line 284:		Line 284:


	[[Category:Linux]]		[[Category:Linux]]{{Navbox Linux}}

	{{Navbox Services}}		{{Navbox Services}}

Leo: Text replacement - "" to "}}"

2017-02-24T19:14:24Z

Text replacement - "</code>" to "}}"

← Older revision		Revision as of 19:14, 24 February 2017
Line 24:		Line 24:
	</syntaxhighlight>		</syntaxhighlight>

	Then run {{code\|yum install logstash elasticsearch~~</code>~~		Then run {{code\|yum install logstash elasticsearch}}

	=== Configuration ===		=== Configuration ===

	The configuration files for logstash are located at:		The configuration files for logstash are located at:
	# {{code\|/etc/sysconfig/logstash~~</code>~~		# {{code\|/etc/sysconfig/logstash}}
	# {{code\|/etc/logstash/conf.d/~~</code>~~		# {{code\|/etc/logstash/conf.d/}}

	The files in the conf.d directory will be treated as a single configuration file in sequence by their filenames by logstash.		The files in the conf.d directory will be treated as a single configuration file in sequence by their filenames by logstash.
Line 37:		Line 37:

	ElasticSearch's files are at:		ElasticSearch's files are at:
	# {{code\|/etc/elasticsearch/elasticsearch.yml~~</code>~~		# {{code\|/etc/elasticsearch/elasticsearch.yml}}

	You will need to configure ElasticSearch based on how you want to set up your search. For replication/sharding, you should ideally have more than one server. If you do have more than one server, make sure you have the node names set and have the autodiscoverer configured.		You will need to configure ElasticSearch based on how you want to set up your search. For replication/sharding, you should ideally have more than one server. If you do have more than one server, make sure you have the node names set and have the autodiscoverer configured.
Line 190:		Line 190:
	For each of the inputs, certain things are done in order to parse the input data into variables which are then passed to the output section.		For each of the inputs, certain things are done in order to parse the input data into variables which are then passed to the output section.

	Operations to the data are done through more sets of plugins (think: functions). For example, text parsing is done through {{code\|grok~~</code>~~. Parameters to these 'functions' are passed as parameters in the {{code\|grok~~</code>~~ block. When {{code\|grok~~</code>~~ fails at parsing a certain string, it will add an additional tag (ie: variable) called {{code\|_grokparsefailure~~</code>~~ which can be used later on in the parse section.		Operations to the data are done through more sets of plugins (think: functions). For example, text parsing is done through {{code\|grok}}. Parameters to these 'functions' are passed as parameters in the {{code\|grok}} block. When {{code\|grok}} fails at parsing a certain string, it will add an additional tag (ie: variable) called {{code\|_grokparsefailure}} which can be used later on in the parse section.

	Variables starting with a '@' are used by ElasticSearch to denote mandatory fields (... I think?) which are defined in the ElasticSearch template file.		Variables starting with a '@' are used by ElasticSearch to denote mandatory fields (... I think?) which are defined in the ElasticSearch template file.
Line 211:		Line 211:
	</syntaxhighlight >		</syntaxhighlight >

	This grok instance attempts to match the incoming message to the defined pattern. The syntax defining matched strings is {{code\|%{PATTERN_NAME:variable_name}~~</code>~~ where {{code\|PATTERN_NAME~~</code>~~ is a grok-pattern defined in {{code\|/patterns/*~~</code>~~ in the .jar file and also in the directory defined in the {{code\|patterns_dir~~</code>~~ directory, and {{code\|variable_name~~</code>~~ is the name that can be used to referenced the matched value later on in the grok instance.		This grok instance attempts to match the incoming message to the defined pattern. The syntax defining matched strings is {{code\|%{PATTERN_NAME:variable_name}}} where {{code\|PATTERN_NAME}} is a grok-pattern defined in {{code\|/patterns/*}} in the .jar file and also in the directory defined in the {{code\|patterns_dir}} directory, and {{code\|variable_name}} is the name that can be used to referenced the matched value later on in the grok instance.

	The {{code\|SYSLOGYEARTIMESTAMP~~</code>~~ pattern is a custom pattern defined in my {{code\|./patterns~~</code>~~ directory.		The {{code\|SYSLOGYEARTIMESTAMP}} pattern is a custom pattern defined in my {{code\|./patterns}} directory.
	<syntaxhighlight lang="text" line start="1" enclose="div">		<syntaxhighlight lang="text" line start="1" enclose="div">
	cat patterns/extra		cat patterns/extra
Line 219:		Line 219:
	</syntaxhighlight >		</syntaxhighlight >

	In the case above, syslog messages being imported whose date field matches the format given in {{code\|SYSLOGYEARTIMESTAMP~~</code>~~ will be placed in the variable {{code\|syslog_timestamp~~</code>~~.		In the case above, syslog messages being imported whose date field matches the format given in {{code\|SYSLOGYEARTIMESTAMP}} will be placed in the variable {{code\|syslog_timestamp}}.

	=== Outputs ===		=== Outputs ===

Leo: Text replacement - "`" to "{{code|"`


2017-02-24T19:12:55Z
Text replacement - "<code>" to "{{code|"

				
				
				
				
				
				← Older revision
				Revision as of 19:12, 24 February 2017
				
Line 24:
Line 24:
</syntaxhighlight> </syntaxhighlight>



Then run <code>yum install logstash elasticsearch</code> Then run {{code|yum install logstash elasticsearch</code>



=== Configuration === === Configuration ===



The configuration files for logstash are located at: The configuration files for logstash are located at:
# <code>/etc/sysconfig/logstash</code> # {{code|/etc/sysconfig/logstash</code>
# <code>/etc/logstash/conf.d/</code> # {{code|/etc/logstash/conf.d/</code>



The files in the conf.d directory will be treated as a single configuration file in sequence by their filenames by logstash. The files in the conf.d directory will be treated as a single configuration file in sequence by their filenames by logstash.
Line 37:
Line 37:



ElasticSearch's files are at: ElasticSearch's files are at:
# <code>/etc/elasticsearch/elasticsearch.yml</code> # {{code|/etc/elasticsearch/elasticsearch.yml</code>



You will need to configure ElasticSearch based on how you want to set up your search. For replication/sharding, you should ideally have more than one server. If you do have more than one server, make sure you have the node names set and have the autodiscoverer configured. You will need to configure ElasticSearch based on how you want to set up your search. For replication/sharding, you should ideally have more than one server. If you do have more than one server, make sure you have the node names set and have the autodiscoverer configured.
Line 190:
Line 190:
For each of the inputs, certain things are done in order to parse the input data into variables which are then passed to the output section. For each of the inputs, certain things are done in order to parse the input data into variables which are then passed to the output section.



Operations to the data are done through more sets of plugins (think: functions). For example, text parsing is done through <code>grok</code>. Parameters to these 'functions' are passed as parameters in the <code>grok</code> block. When <code>grok</code> fails at parsing a certain string, it will add an additional tag (ie: variable) called <code>_grokparsefailure</code> which can be used later on in the parse section. Operations to the data are done through more sets of plugins (think: functions). For example, text parsing is done through {{code|grok</code>. Parameters to these 'functions' are passed as parameters in the {{code|grok</code> block. When {{code|grok</code> fails at parsing a certain string, it will add an additional tag (ie: variable) called {{code|_grokparsefailure</code> which can be used later on in the parse section.



Variables starting with a '@' are used by ElasticSearch to denote mandatory fields (... I think?) which are defined in the ElasticSearch template file. Variables starting with a '@' are used by ElasticSearch to denote mandatory fields (... I think?) which are defined in the ElasticSearch template file.
Line 211:
Line 211:
</syntaxhighlight > </syntaxhighlight >



This grok instance attempts to match the incoming message to the defined pattern. The syntax defining matched strings is <code>%{PATTERN_NAME:variable_name}</code> where <code>PATTERN_NAME</code> is a grok-pattern defined in <code>/patterns/*</code> in the .jar file and also in the directory defined in the <code>patterns_dir</code> directory, and <code>variable_name</code> is the name that can be used to referenced the matched value later on in the grok instance.   This grok instance attempts to match the incoming message to the defined pattern. The syntax defining matched strings is {{code|%{PATTERN_NAME:variable_name}</code> where {{code|PATTERN_NAME</code> is a grok-pattern defined in {{code|/patterns/*</code> in the .jar file and also in the directory defined in the {{code|patterns_dir</code> directory, and {{code|variable_name</code> is the name that can be used to referenced the matched value later on in the grok instance.  



The <code>SYSLOGYEARTIMESTAMP</code> pattern is a custom pattern defined in my <code>./patterns</code> directory. The {{code|SYSLOGYEARTIMESTAMP</code> pattern is a custom pattern defined in my {{code|./patterns</code> directory.
<syntaxhighlight lang="text" line start="1" enclose="div"> <syntaxhighlight lang="text" line start="1" enclose="div">
cat patterns/extra   cat patterns/extra  
Line 219:
Line 219:
</syntaxhighlight > </syntaxhighlight >



In the case above, syslog messages being imported whose date field matches the format given in <code>SYSLOGYEARTIMESTAMP</code> will be placed in the variable <code>syslog_timestamp</code>. In the case above, syslog messages being imported whose date field matches the format given in {{code|SYSLOGYEARTIMESTAMP</code> will be placed in the variable {{code|syslog_timestamp</code>.



=== Outputs === === Outputs ===

← Older revision		Revision as of 19:12, 24 February 2017
Line 24:		Line 24:
	</syntaxhighlight>		</syntaxhighlight>

	Then run <code>yum install logstash elasticsearch</code>		Then run {{code\|yum install logstash elasticsearch</code>

	=== Configuration ===		=== Configuration ===

	The configuration files for logstash are located at:		The configuration files for logstash are located at:
	# <code>/etc/sysconfig/logstash</code>		# {{code\|/etc/sysconfig/logstash</code>
	# <code>/etc/logstash/conf.d/</code>		# {{code\|/etc/logstash/conf.d/</code>

	The files in the conf.d directory will be treated as a single configuration file in sequence by their filenames by logstash.		The files in the conf.d directory will be treated as a single configuration file in sequence by their filenames by logstash.
Line 37:		Line 37:

	ElasticSearch's files are at:		ElasticSearch's files are at:
	# <code>/etc/elasticsearch/elasticsearch.yml</code>		# {{code\|/etc/elasticsearch/elasticsearch.yml</code>

	You will need to configure ElasticSearch based on how you want to set up your search. For replication/sharding, you should ideally have more than one server. If you do have more than one server, make sure you have the node names set and have the autodiscoverer configured.		You will need to configure ElasticSearch based on how you want to set up your search. For replication/sharding, you should ideally have more than one server. If you do have more than one server, make sure you have the node names set and have the autodiscoverer configured.
Line 190:		Line 190:
	For each of the inputs, certain things are done in order to parse the input data into variables which are then passed to the output section.		For each of the inputs, certain things are done in order to parse the input data into variables which are then passed to the output section.

	Operations to the data are done through more sets of plugins (think: functions). For example, text parsing is done through <code>grok</code>. Parameters to these 'functions' are passed as parameters in the <code>grok</code> block. When <code>grok</code> fails at parsing a certain string, it will add an additional tag (ie: variable) called <code>_grokparsefailure</code> which can be used later on in the parse section.		Operations to the data are done through more sets of plugins (think: functions). For example, text parsing is done through {{code\|grok</code>. Parameters to these 'functions' are passed as parameters in the {{code\|grok</code> block. When {{code\|grok</code> fails at parsing a certain string, it will add an additional tag (ie: variable) called {{code\|_grokparsefailure</code> which can be used later on in the parse section.

	Variables starting with a '@' are used by ElasticSearch to denote mandatory fields (... I think?) which are defined in the ElasticSearch template file.		Variables starting with a '@' are used by ElasticSearch to denote mandatory fields (... I think?) which are defined in the ElasticSearch template file.
Line 211:		Line 211:
	</syntaxhighlight >		</syntaxhighlight >

	This grok instance attempts to match the incoming message to the defined pattern. The syntax defining matched strings is <code>%{PATTERN_NAME:variable_name}</code> where <code>PATTERN_NAME</code> is a grok-pattern defined in <code>/patterns/*</code> in the .jar file and also in the directory defined in the <code>patterns_dir</code> directory, and <code>variable_name</code> is the name that can be used to referenced the matched value later on in the grok instance.		This grok instance attempts to match the incoming message to the defined pattern. The syntax defining matched strings is {{code\|%{PATTERN_NAME:variable_name}</code> where {{code\|PATTERN_NAME</code> is a grok-pattern defined in {{code\|/patterns/*</code> in the .jar file and also in the directory defined in the {{code\|patterns_dir</code> directory, and {{code\|variable_name</code> is the name that can be used to referenced the matched value later on in the grok instance.

	The <code>SYSLOGYEARTIMESTAMP</code> pattern is a custom pattern defined in my <code>./patterns</code> directory.		The {{code\|SYSLOGYEARTIMESTAMP</code> pattern is a custom pattern defined in my {{code\|./patterns</code> directory.
	<syntaxhighlight lang="text" line start="1" enclose="div">		<syntaxhighlight lang="text" line start="1" enclose="div">
	cat patterns/extra		cat patterns/extra
Line 219:		Line 219:
	</syntaxhighlight >		</syntaxhighlight >

	In the case above, syslog messages being imported whose date field matches the format given in <code>SYSLOGYEARTIMESTAMP</code> will be placed in the variable <code>syslog_timestamp</code>.		In the case above, syslog messages being imported whose date field matches the format given in {{code\|SYSLOGYEARTIMESTAMP</code> will be placed in the variable {{code\|syslog_timestamp</code>.

	=== Outputs ===		=== Outputs ===