1.4 Four Hurdles to Mastering sed and awk

There are a number of introductory UNIX books that will acquaint you with sed and awk. The goal of this book is to take you much further - to help you master sed and awk and to reduce the amount of time and effort that it takes you to reach that goal.

There are four hurdles on the way to mastering sed and awk. You must learn:

1. How to use sed and awk. This is a relatively low hurdle to clear because, fortunately, sed and awk work in a very similar manner, based on the line editor ed. Chapter 2, Understanding Basic Operations, covers the mechanics of using sed and awk.

2. To apply UNIX regular expression syntax. Using UNIX regular expression syntax for pattern matching is common to both sed and awk, and many other UNIX programs. This can be a difficult hurdle for two reasons: the syntax is arcane, and though many people have some experience using regular expressions, few have persevered to master the complete syntax. The more facile you are in using this syntax, the easier sed and awk are to use. That is why we spend a good deal of time covering regular expressions in Chapter 3, Understanding Regular Expression Syntax.

3. How to interact with the shell. While not directly related to sed and awk themselves, managing the interaction with the command shell is often a frustrating problem, since the shell shares a number of special characters with both programs. If you can, avoid the problem by putting your script in a separate file. If not, use a Bourne-compatible shell for your scripts (the quoting rules are more straightforward), and use single quotes to contain your script. If you are using csh as your interactive shell, remember to escape any exclamation points with a backslash ("\!"). There is no other way to get csh to leave the exclamation point alone.[3]

   [3] Well, you can set the histchars variable. See the csh man page.

4. The knack of script writing. This is the most difficult, rather like a series of high hurdles. Because of this, the bulk of the book is devoted to script writing. With sed, you have to learn a set of single-letter commands. With awk, you have to learn the statements of a programming language. To get the knack of script writing, though, you simply must pore over lots of examples and, of course, must try your hand at writing scripts yourself.

If you were running the high hurdles, the ability to clear the hurdles does not win the race - clearing them swiftly does. In writing scripts, learning the scripting command set or language is simply clearing the hurdle. Acquiring the ability to attack interesting problems with your scripts is running fast enough to compete.