13. Text manipulation¶

13.1. Review¶

You’ve come a long way already. You know how to use the terminal to start and stop processes. You’re aware of how to operate with files, list, move, copy and view them. You can redirect command output to files. You know how to create chains of commands, feeding the output of one as the input of the other.

13.2. Searching¶

In fact, we have some tools which do exactly that.

grep 'ssh' /etc/services

The grep command searches for a string (ssh) in a file (/etc/services) and prints you all matching lines. The quotes are optional but a good practice. You’ll need them in any case if you’re search text includes whitespaces.

This example was very simple, trivial even. Quite handy but we can do more. Much more. Very often you’ll use the following form of the command.

cat /etc/services | grep 'ssh'

Still not extremely interesting in this case but let’s be clear about the implication: You can easily and quickly search for text in any command’s output. Which is way more fast and comfortable than using less or scanning through the output yourself. You need the PID of your bash? No problem:

ps -ef | grep bash

But read on to see why bash is such a popular tool.

13.3. Text manipulation tools¶

In fact, there’s several commands which focus on text processing and text manipulation. They all provide some very basic functionality. In itself, the program won’t do much. But combined with input/outupt redirection and pipelining, they can be very powerful.

Just for fun, maybe you can appreciate the sequence below.

echo "Hello world" | cut -d ' ' -f 1 | tee /tmp/teaout | tr -c -d '\ne[:upper:]' | cat -n | tr -d ' \t' | tr 1H By | cat - /tmp/teaout | sort -r | cat -n

The man pages of these commands explains the exact meaning of the various options. Some of the tools are mainly useful in scripts (cut, tr, tee) where you cannot (or don’t want to) process a program’s output yourself. Others are very handy to manipulate text files (uniq, sort), also ones not related to commands or program output. Especially sort is quite good to scan through unsorted program output more quickly.

The diff tool reaches very deep into the linux community and is used in many other places (patches, git). Any system changes with time. With diff you get a quick overview over what changed. You can either have a side-by-side view (-y) of the whole document or show the differences (plus context, -C NUM). And you can also see the difference between all files of two directories (-r) or just the summary of what files are different (-q -r).

13.4. Regular expressions¶

What makes grep so powerful? So far you’ve only given a simple keyword to search for. But this section is called Regular expression (or regex for short), so what is this and how is it related to grep?

Consider an example you’ve seen before:

ls /dev/[sh]d[a-z]

The device name follows a certain scheme. The first two letters are either sd or hd. Then comes one additional letter (which identifies the device number). We don’t know yet what files will be there so we want to show any following this scheme. So we formalize the scheme as follows:

1. Either s or h
2. d
3. Any (lowercase) letter from the latin alphabet (a-z)

What’s in between the brackets ([, ]) gives you a choice over characters. You can either define single characters ([ahxY4]) or ranges ([0-9], [a-k]). But you can go further. Let’s talk quantities:

ls /dev/[sh]d[a-z][0-9]*

Now the last part demands any number (0-9). You know the selection with the brackets, but the star (*) was added. Does this look familiar? You’ve seen this one a couple of times and actually know it to be the wildcard. It’s like saying that the number [0-9] may occur an arbitrary number of times. So /dev/sda0, /dev/sda00, /dev/sda012 are all valid. But the wildcard also includes zero occurrences, so /dev/sda is listed also. When you put a wildcard (or another quantity specification) it is applied to the character or character choice given before. Like the range [0-9] in the example.

But in fact, with ls you don’t use regular expressions, they copied just the range and wildcard. Regex goes way further. It’s a language to specify a search term where you know some structure but not everything. It basically goes like this: You put a character (or a list thereof) first, then its quantity. The character is either a letter or a list, as seen above. The quantity can be the wildcard (*, zero or more occurences), a plus (+, at least one occurence), a question mark (?, zero or one occurence) or a specific number of occurrences in curly braces ({4}). If the quantity is one, you don’t put anything.

hello
h[a-z]llo
hel{2}o
hel?lo
hel+o
help*lo
h[el]*lo

All of the examples above match “hello”, but some of them also match other words. In the last example, the range choice and the quantity was combined, leading to much more interesting words.

Let’s continue a little further. Often you don’t know a single character or searching words where a single character changes. Then, you can just put a dot (.) instead of that character. It basically means “a single arbitrary character”. And I write character on purpose, because it’s not limited to a letter but can also be a digit or a whitespace.

he.lo
h..lo
.ell.
h.*llo

In the last example, the quantifier was again combined with the dot. Meaning the word has to start with ‘h’ and end with ‘llo’, but there can be anything in between.

OK, so that was a very short introduction into regex. You can even do more and it can become pretty complicated. None the less, I encourage everyone to experiment with the regex and learn it thoroughly. For example by reading the regex-tutorial. It’s an extremely powerful tool and well supported by many terminal applications. For example, you can use the same syntax in less! After you’ve studied the tutorial, also keep the regex-cheat-sheet close.

13.5. Summary¶

Looking for the needle in the haystack isn’t a problem for you anymore - as long as you’re looking for text. You’ve accepted the superiority of the terminal over mouse-driven programs. Using grep comes natural to you. Sorting and manipulating text helps you scanning through files quickly and finding crutial information more easily. You know how to state a search term more generally than just a simple word. You can mix these techniques with everyday commands.

13.6. Cheatsheet¶

• grep KEYWORD FILE