SED and AWK for scripting configs

By Manny Fernandez

October 27, 2019

SED and AWK for scripting configs

Let me start by saying that I did not write this code, but I have used it for many years. Essentially, you can get a CSV file loaded with information and then iterate through the CSV and replace variables with content from CSV. To give you an example, I provided the networking and security for the 2012 Presidential debate. At the time, I was a Cisco junky. We needed to configure hundreds of access switches. Instead of connecting to each switch and adding the configs, I created a template file containing the basics and used variables for: VLAN IDs, 3rd Octet for IP Addresses, Network statements on BGP, hostnames, etc. I will break down the scrip here and will post on my github for you to use.

Although this script show a basic Fortinet Fortigate address object template, I have personally used this same script, modified obviously, for very, very complex configs.

You can also check out my buddies blog site that also has a powershell version of a similar script.

Source File

The source file is a CSV format that will contain different information used to create the config file (e.g. hostnames, VLAN IDs, etc.). Although I tend to NOT use extensions for the file names, you may want to add an extension (e.g. .csv, .txt, etc). Below is the contents of my example.

Host-92.241.165.127,92.241.165.127 
Host-219.94.198.78,219.94.198.78 
Host-69.16.180.34,69.16.180.34 
Host-74.92.135.109,74.92.135.109 
Host-80.177.183.2,80.177.183.2 
Host-60.234.251.122,60.234.251.122

Template File

This is what the config output SHOULD look like but with VAR labels that will come into play when you see the script. Below is the contents of my example.

 edit "VAR_HOSTNAME"
set subnet VAR_IPADDR 255.255.255.255
next

Output

This is the ready-to-use output once the script has successfully run. (Shown later in the post).

The Script

This is a bash script and it used sed and awk to do the heavy lifting.

#!/bin/bash
IMPORT="/Users/mannyfernandez/Desktop/Script/source"
TEMPLATE="/Users/mannyfernandez/Desktop/Script/template"

for i in `cat ${IMPORT}`
do 
VAR_HOSTNAME=`echo $i | awk -F, '{print $1}'`
VAR_IPADDR=`echo $i | awk -F, '{print $2}'`
cat $TEMPLATE | sed -e s/VAR_HOSTNAME/$VAR_HOSTNAME/g \
-e s/VAR_IPADDR/$VAR_IPADDR/g \
| tee -a ///Users/mannyfernandez/Desktop/Script/output 1>/dev/null
done

In the script, you will see the IMPORT line that shows the source file to use. The second line or the TEMPLATE line shows the file that will be used as a template.

The other important piece to remember is the section in the curly brackets or {} . This print $1 statement tell the script what column to use to populate the corresponding VAR_ variable.

NOTE: To have it create a separate file for each iteration, remove the -a from the

| tee -a ///Users/mannyfernandez/Desktop/Script/output 1>/dev/null done

command above.

A little bit about SED and AWK. Sed

SED

sed refers to Stream Editor. It can perform text transformations on a given file or an input stream.

AWK

awk is a text pattern scanning and processing language, which is created by Aho, Weinberger & Kernighan. awk is mostly used for data extraction and reporting (dealing with .csv files).

I got a great PDF eBook from a great resource www.thegeekstuff.com . He has a great ebook on SED and AWK.

Making Script Executable

By default, when you create a script file, it will not be executable, meaning you cannot run the command from the command line with the ./ pre-seeding command.

Notice the rights associated with the script file.

We will now make the file executable from the cli. To do this, we are going to go back to the CLI and enter the following command:

chmod +x import-script.sh

Here we can see that we ran the chmod command and when we ran the ls -all we can now see that there is a x in the output.

2019-10-27_16-23-26

Running the Script

To run the script, simply shell out to the CLI again and type:

./import-script.sh

When you run the script, you should not see any errors. If you do, you should make sure that you are escaping any . or other special characters.

The Output

Here we see the output

edit "Host-92.241.165.127"
set subnet 92.241.165.127 255.255.255.255
next

edit "Host-219.94.198.78"
set subnet 219.94.198.78 255.255.255.255
next

edit "Host-69.16.180.34"
set subnet 69.16.180.34 255.255.255.255
next

edit "Host-74.92.135.109"
set subnet 74.92.135.109 255.255.255.255
next

edit "Host-80.177.183.2"
set subnet 80.177.183.2 255.255.255.255
next

edit "Host-60.234.251.122"
set subnet 60.234.251.122 255.255.255.255
next

SED and AWK for scripting configs

Source File

Template File

Output

The Script

SED

AWK

Making Script Executable

Running the Script

Recent posts