Bash Programming Interactive lecture#

We recommend using your own choice of text editor#

* Everyone should know how to exit and edit text in vim

Vim basics#

On some supercomputers (e.g. SAGA), vim may be the only option for text editing.

  • Open vim and a textfile hello_world.sh with $ vim hello_world.sh

  • To edit text in vim we have to enter INSERT mode by typing “i”

  • The cursor can not be moved with mouse clicks. Use arrows and “PgUp”, “End” etc.

  • Inside INSERT mode text can be edited as normal

  • To exit INSERT mode press Esc

  • To save changes type :w

  • To exit vim (important!), type :q

  • Save and exit :wq

Reminder: Combining bash commands: Pipes#

The bash pipe | connects STDOUT of one command to STDIN of another.

Live programming: Oslo’s weather report.#

⚠️ Unix commands used here:

  • curl: downloads a website

  • grep: filter lines with a given keyword

  • head: limit the output to the first X lines

  • cut: select a specific column defined by a deliminter

  • cowsay: let a cow say something

Interactive programming#

Solve the following exercises:

  1. Implement a weather forcast script that outputs the current weather in Bergen (i suggest using https://pent.no or https://wttr.in/your_location). Feel free to vary the layout and amount of details in the output.

  2. (Optional) Implement a script that outputs headlines of nrk.no.

  3. (Optional) Implement a weather forecast script as in 1) using www.yr.no.

useful: curl, grep, head, cut, (cowsay)

Time: 15 minutes

More pipe examples#

  • Sort all files in a directory tree, with the largest files appearing first, and equip the output with paging functionality:

du -a lectures | sort -rn | less

⚠️ du shows the disk usages of files
⚠️ sorts sorts the input
⚠️ less paginates the output for easier reading

  • Save output to disk and display it on the console:

./weather.sh | tee weather.log

⚠️ tee writes STDIN to STDOUT and file ../../_images/tee.png

Quiz: Can you simplify this code?#

ls > files.txt ; grep weather < files.txt ; rm files.txt

⚠️ CMD1 ; CMD2 executes CMD2 after CMD1.

The code above implements an (inefficient) pipe.

Better:

ls | grep weather

Functions#

Bash has simple support for functions.

function geturl {
    
    # function arguments: $1 $2 $3 and so on
    
	if [ $1 == "Oslo" ]; then
		url="https://www.yr.no/place/Norway/Oslo/Oslo/Oslo/"
	fi
}


# call:
geturl "Oslo"
echo $url       # Bash functions have no return values, but all variables are global

Interactive programming#

Implement a “geturl” function that takes a city as input and stores the assoicated weather report url as output.

The function should work for Bergen, Oslo and Stavanger.

If an invalid city is provided, the function should exit the script and print an error message.

Optional: Print out the current weather instead of the link to the website

Time: 10 minutes

Convenient debugging tool: -x#

Each source code line is printed prior to its execution if you add -x as option to /bin/sh or /bin/bash

Either in the header

#!/bin/bash -x

or on the command line:

bash -x scripts/weather_functions.sh

Live example

!scripts/weather_functions.sh

Reminder: command line arguments#

Bash provides the special (array) variable $ for accessing these arguments:

  • $0 contains the script name

  • $1 contains the first command line option

  • $2 contains the second command line option

  • …

Interactive programming#

  1. Extend your weather forecast script so that it takes the city as a command line option:

Example usage:

weather.sh Oslo   # Prints out Oslo's weather

weather.sh Mars   # Prints out an error message (not a city)

  1. (Optional): Print a short usage documentation for the script when no command line options are provided.

Time: 10 minutes

Running an application in the foreground#

Syntax:

  • cmd & executes a program in the background.

  • Use wait to block a script until all background processes are completed.

Example:

myprog -c file.1 -p -f -q < my_input_file &

# any commands here will be executed while myprog is still running

Live demo: scripts/background_test.sh

File globbing, for loop on the command line#

List all .ps and .gif files using wildcard notation:

files=$(ls *.png *.eps)

# compress and move the files:
gzip $files
for file in $files; do
  cp ${file}.gz $HOME/Pictures
done

⚠️ gzip compresses a file, cp copies a file, $HOME is a string with the path to your home directory.

The find command#

Very useful command!

⚠️ find visits all files in a directory tree and can execute one or more commands for every file

Basic example: find the weather codes

find -name 'weather*'

Or find all log and PDF files

find $HOME -name '*.log' -o -name '*.pdf'

That is it for today#

Applications of find (1)#

Find all files larger than 2000 blocks a 512 bytes (=1Mb):

find $HOME -name '*' -type f -size +2000 -exec ls -s {} \;

Remove all these files:

find $HOME -name '*' -type f -size +2000 \
           -exec ls -s {} \; -exec rm -f {} \;

or ask the user for permission to remove:

find $HOME -name '*' -type f -size +2000 \
           -exec ls -s {} \; -ok rm -f {} \;

Backup slides#

Redirects and pipes can be combined#

View STDOUT and STDERR with paging functionality:

./compile 2>&1 | tee compile.log | less

Example: bundle files#

Pack a series of files into one file

Executing this single file as a Bash script packs out all the individual files again

Usage:

bundle file1 file2 file3 > onefile  # pack
bash onefile # unpack

Writing bundle is easy:

#/bin/sh
for i in $@; do
    echo "echo unpacking file $i"
    echo "cat > $i <<EOF"
    cat $i
    echo "EOF"
done

The bundle output file#

Consider 2 fake files: file1:

Hello, World!
No sine computations today

and file2:

1.0 2.0 4.0
0.1 0.2 0.4

Running bundle file1 file2 yields the output

echo unpacking file file1
cat > file1 <<EOF
Hello, World!
No sine computations today
EOF
echo unpacking file file2
cat > file2 <<EOF
1.0 2.0 4.0
0.1 0.2 0.4
EOF

Note: In the terminal, you can send a foreground job into the background by pressing Ctrl-Z followed by the bg command, and retrieve it again with fg.

How to return a value from a function? Define a new variable within the function - all functions are global!

Applications of find (2)#

Find all files not being accessed for the last 90 days:

find $HOME -name '*' -atime +90 -print

and move these to /tmp/trash:

find $HOME -name '*' -atime +90 -print \
           -exec mv -f {} /tmp/trash \;

Tar and gzip#

⚠️ The tar command can pack single files or all files in a directory tree into one file, which can be unpacked later

tar -cvf myfiles.tar mytree file1 file2

# options:
# c: pack, v: list name of files, f: pack into file

# unpack the mytree tree and the files file1 and file2:
tar -xvf myfiles.tar

# options:
# x: extract (unpack)

The tarfile can be compressed:

gzip mytar.tar
# result: mytar.tar.gz

A find/tar/gzip example#

Pack all PostScript figures:

tar -cvf ps.tar `find $HOME -name '*.ps' -print`
gzip ps.tar