Awk - Revision history

Leo: Leo moved page Awk Tips & Tricks to Awk

2021-07-03T03:56:33Z

← Older revision		Revision as of 03:56, 3 July 2021
(No difference)

Leo at 03:53, 3 July 2021

2021-07-03T03:53:52Z

@@ Line 1: / Line 1: @@
-==Random Notes==
+Awk is an excellent text parser and scripting language. It can be run in-line as part of a Unix command pipeline which makes it extremely useful when needing to add more complicated behavior in a shell script.
-===Insert a single quote within single quotes===
+== Introduction to Awk ==
-Awk script is almost always passed to awk using single quotes since it allows us to use <code>$1 ... $NF</code> variables without needing to escape the <code>$</code>. However, this makes inserting a single quote tricky. To add a single quote, you will need to use <code>'"'"'</code> which inserts a single <code>'</code> by switching from single to double quotes. This is useful if you need to insert a single quote in order to trigger a subshell call, for instance.
+Awk's scripting language is structed as a sequence of patterns and actions:
-===Variables===
+==== Variables ====
 Awk has built-in variables:
@@ Line 13: / Line 69: @@
 *<code>RS</code> - record separator
-You can also pass your own with the <code>-v var=value</code> parameter. Eg:
+You can also pass your own custom variables with the <code>-v var=value</code> parameter. Eg:
 {{Highlight
 | code = $ echo {{!}} awk \
@@ Line 23: / Line 79: @@
+'
 | lang = terminal
 }}
 ==Tasks==
 ===Line Matching===
 With Awk, it's easy to do something for each matching line using the regex matching operator.
@@ Line 88: / Line 155: @@
 }}
-=== String Matching ===
+===String Matching===
 Use the <code>match(string, regex, output_array)</code> command to parse out specific values with regex.
 {{Highlight

Leo at 23:33, 23 October 2020

2020-10-23T23:33:13Z

@@ Line 86: / Line 86: @@
+}
 </nowiki>
 }}

Leo: /* Random Quick Notes */

2020-08-26T01:07:30Z

Random Quick Notes

← Older revision		Revision as of 01:07, 26 August 2020
Line 1:		Line 1:
	== Random ~~Quick~~ Notes ==		==Random Notes==

	=== Insert a single quote within single quotes ===		===Insert a single quote within single quotes===
	Awk script is almost always passed to awk using single quotes since it allows us to use <code>$1 ... $NF</code> variables without needing to escape the <code>$</code>. However, this makes inserting a single quote tricky. To add a single quote, you will need to use <code>'"'"'</code> which inserts a single <code>'</code> by switching from single to double quotes. This is useful if you need to insert a single quote in order to trigger a subshell call, for instance.		Awk script is almost always passed to awk using single quotes since it allows us to use <code>$1 ... $NF</code> variables without needing to escape the <code>$</code>. However, this makes inserting a single quote tricky. To add a single quote, you will need to use <code>'"'"'</code> which inserts a single <code>'</code> by switching from single to double quotes. This is useful if you need to insert a single quote in order to trigger a subshell call, for instance.

	==== Variables ====		===Variables===
	Awk has built-in variables:		Awk has built-in variables:

	* <code>NR</code> - current line number		*<code>NR</code> - current line number
	* <code>NF</code> - number of fields in current line		*<code>NF</code> - number of fields in current line
	* <code>OFS</code> - output field separator		*<code>OFS</code> - output field separator
	* <code>FS</code> - field separator, specified with -F		*<code>FS</code> - field separator, specified with -F
	* <code>RS</code> - record separator		*<code>RS</code> - record separator

	You can also pass your own with the <code>-v var=value</code> parameter. Eg:		You can also pass your own with the <code>-v var=value</code> parameter. Eg:

Leo: /* Tasks */

2020-08-26T01:06:46Z

Tasks

← Older revision		Revision as of 01:06, 26 August 2020
Line 1:		Line 1:
	~~<br />~~		== Random Quick Notes ==

	== Tasks ==		=== Insert a single quote within single quotes ===
			Awk script is almost always passed to awk using single quotes since it allows us to use <code>$1 ... $NF</code> variables without needing to escape the <code>$</code>. However, this makes inserting a single quote tricky. To add a single quote, you will need to use <code>'"'"'</code> which inserts a single <code>'</code> by switching from single to double quotes. This is useful if you need to insert a single quote in order to trigger a subshell call, for instance.

			==== Variables ====
			Awk has built-in variables:

			* <code>NR</code> - current line number
			* <code>NF</code> - number of fields in current line
			* <code>OFS</code> - output field separator
			* <code>FS</code> - field separator, specified with -F
			* <code>RS</code> - record separator

			You can also pass your own with the <code>-v var=value</code> parameter. Eg:
			{{Highlight
			\| code = $ echo {{!}} awk \
			-v hostname=`hostname` \
			-v timestamp=`date +%s` '
			{
			printf("Hostname is %s at time %s", hostname, timestamp")
			}
			'
			\| lang = terminal
			}}

			==Tasks==

	===Line Matching===		===Line Matching===
Line 32:		Line 56:
	}}		}}

	==== Retrieve a section of text with line matching ====		====Retrieve a section of text with line matching====
	To grab a specific section from a .spec file (where sections begin with {{code\|%}}):		To grab a specific section from a .spec file (where sections begin with {{code\|%}}):
	{{highlight\|lang=bash\|code=		{{highlight\|lang=bash\|code=
Line 64:		Line 88:
	}}		}}

	==Converting Values==		===Converting Values===

	==== Convert Number as Bytes to Human Readable Value ====		====Convert Number as Bytes to Human Readable Value====
	I wanted to sort size in reverse order, but in order to do that properly, the value from du needs to be in kilobytes. I also didn't want to run this through du again just to get the human readable value. So, I did this:		I wanted to sort size in reverse order, but in order to do that properly, the value from du needs to be in kilobytes. I also didn't want to run this through du again just to get the human readable value. So, I did this:

Line 107:		Line 131:
	For example, a process running for {{code\|66-00:12:58}} has been running for 5703178 seconds. The Awk command will match how many columns were found and then do the proper calculation.		For example, a process running for {{code\|66-00:12:58}} has been running for 5703178 seconds. The Awk command will match how many columns were found and then do the proper calculation.

	=== String Manipulation ===		===String Manipulation===

	====String Upper/Lower Casing====		====String Upper/Lower Casing====

Leo: /* String Upper/Lower Casing */

2020-08-26T00:50:32Z

String Upper/Lower Casing

← Older revision		Revision as of 00:50, 26 August 2020
Line 115:		Line 115:
	{{highlight\|lang=terminal\|code=		{{highlight\|lang=terminal\|code=
	## Lower case every file in the current directory		## Lower case every file in the current directory
	$ for i in `ls` ; do mv $i `echo $i \| awk '{print tolower($1)}'`; done		$ for i in `ls` ; do mv $i `echo $i {{!}} awk '{print tolower($1)}'`; done

	## Alternatively, use `tr 'A-Z' 'a-z'` to do the lowercasing.		## Alternatively, use `tr 'A-Z' 'a-z'` to do the lowercasing.
	$ for i in `ls` ; do mv $i `echo $i \| tr 'A-Z 'a-z'`; done		$ for i in `ls` ; do mv $i `echo $i {{!}} tr 'A-Z 'a-z'`; done
	}}		}}

Leo: /* Do something until a line match */

2020-08-26T00:50:09Z

Do something until a line match

Show changes

Leo: /* Executing Commands */

2020-08-14T20:46:21Z

Executing Commands

← Older revision		Revision as of 20:46, 14 August 2020
Line 1:		Line 1:

	== Do something until a line match ==		==Do something until a line match==
	With Awk, it's easy to do something for each matching line using the regex matching operator.		With Awk, it's easy to do something for each matching line using the regex matching operator.


	=== Print every line before a line match ===		===Print every line before a line match===
	Simple awk code:		Simple awk code:
	{{highlight\|lang=terminal\|code=		{{highlight\|lang=terminal\|code=
Line 12:		Line 12:
	Basically, the code does: match against the given pattern. If it matches, awk exits. Otherwise, print the line and continue.		Basically, the code does: match against the given pattern. If it matches, awk exits. Otherwise, print the line and continue.

	=== Print line number on matching lines ===		===Print line number on matching lines===
	To print the number of lines up to a matching line, we do something similar to the previous example but now we keep an accumulator (n):		To print the number of lines up to a matching line, we do something similar to the previous example but now we keep an accumulator (n):
	{{highlight\|lang=terminal\|code=		{{highlight\|lang=terminal\|code=
Line 32:		Line 32:


			==Convert Number as Bytes to Human Readable Value==
	== Convert Number as Bytes to Human Readable Value ==
	I wanted to sort size in reverse order, but in order to do that properly, the value from du needs to be in kilobytes. I also didn't want to run this through du again just to get the human readable value. So, I did this:		I wanted to sort size in reverse order, but in order to do that properly, the value from du needs to be in kilobytes. I also didn't want to run this through du again just to get the human readable value. So, I did this:

Line 56:		Line 55:


	== Convert {{code\|ps}} Elapsed Time to Seconds ==		==Convert {{code\|ps}} Elapsed Time to Seconds==
	To convert the elapsed time (in [http://pubs.opengroup.org/onlinepubs/009696799/utilities/ps.html POSIX locale] formatted as {{code\|<nowiki>[[dd-]hh:]mm:ss</nowiki>}}) to seconds using awk:		To convert the elapsed time (in [http://pubs.opengroup.org/onlinepubs/009696799/utilities/ps.html POSIX locale] formatted as {{code\|<nowiki>[[dd-]hh:]mm:ss</nowiki>}}) to seconds using awk:

Line 76:		Line 75:
	For example, a process running for {{code\|66-00:12:58}} has been running for 5703178 seconds. The Awk command will match how many columns were found and then do the proper calculation.		For example, a process running for {{code\|66-00:12:58}} has been running for 5703178 seconds. The Awk command will match how many columns were found and then do the proper calculation.

	== Get a Text Section ==		==Get a Text Section==
	To grab a specific section from a .spec file (where sections begin with {{code\|%}}):		To grab a specific section from a .spec file (where sections begin with {{code\|%}}):

Line 110:		Line 109:
	}}		}}

	== String Upper/Lower Casing ==		==String Upper/Lower Casing==
	There is a {{code\|tolower()}} function that can lowercase an entire string.		There is a {{code\|tolower()}} function that can lowercase an entire string.

Line 122:		Line 121:
	}}		}}

	== Executing Commands ==		==Executing Commands==
	If your awk script needs to call an external process, pass the command to {{code\|getline}} followed by the variable name used to store stdout output.		If your awk script needs to call an external process, pass the command to {{code\|getline}} followed by the variable name used to store stdout output.

Line 136:		Line 135:
	}}		}}

	{{Navbox Linux}}[[Category:Linux]]		== Retrieving all columns after column N ==
			You can retrieve a specific column using <code>$N</code>, where <code>N</code> is the column number. If you wish to get all values after a specific column, you can use this function:
			{{Highlight
			\| code = function after(x) {
			out=""
			for (i=x; i<=NF; i++) out=out" "$i
			return out
			}

			printf("After column 11: %s\n", after(11))
			\| lang = awk
			}}{{Navbox Linux}}
			[[Category:Linux]]
	[[Category:TipsnTricks]]		[[Category:TipsnTricks]]
	[[Category:LinuxUtilities]]		[[Category:LinuxUtilities]]

Leo: /* Executing Commands */

2020-05-22T00:33:20Z

Executing Commands

← Older revision		Revision as of 00:33, 22 May 2020
Line 131:		Line 131:
	}}		}}

	If you intend to run many commands, you should close the pipe or else you will get a "Too many files ~~open"~~		If you intend to run many commands, you should close the pipe or else you will get a {{code\|fatal: cannot open pipe 'xyz' (Too many open files)}} error. Do so by using the close function:

	{{highlight\|lang=text\|code=		{{highlight\|lang=text\|code=
	cmd="date -d\""$1" "$2"\" \"+%s\""; cmd {{!}} getline timestamp; close(cmd)		cmd="date -d\""$1" "$2"\" \"+%s\""; cmd {{!}} getline timestamp; close(cmd)

Leo: /* Executing Commands */

2020-05-22T00:32:25Z

Executing Commands

← Older revision		Revision as of 00:32, 22 May 2020
Line 131:		Line 131:
	}}		}}

			If you intend to run many commands, you should close the pipe or else you will get a "Too many files open"

			{{highlight\|lang=text\|code=
			cmd="date -d\""$1" "$2"\" \"+%s\""; cmd {{!}} getline timestamp; close(cmd)
			}}

	{{Navbox Linux}}[[Category:Linux]]		{{Navbox Linux}}[[Category:Linux]]
	[[Category:TipsnTricks]]		[[Category:TipsnTricks]]
	[[Category:LinuxUtilities]]		[[Category:LinuxUtilities]]