Shell utilities

grep

grep                    search for a string

grep string file

(output of prog) | grep string | grep otherstring

-i    ignore case
-v    return all lines that do NOT match

grep
DCU Solaris - no recursive grep
DCU Linux - recursive grep - grep -r
Odd syntax:
grep -r string .
grep -ir --include="*html" string .

string matching / regular expressions

^                       start-of-line
$                       end-of-line
.                       any character

where "c" stands for the character:

c*			0 or more instances of c
cc*			1 or more instances of c
grep "  *"		1 or more spaces
.*                  any sequence of characters



where "c" has a special meaning, e.g. is $ or ., etc:

\c                  the character itself
grep "\."           the '.' character itself

recall the two forms of quote:

grep '\$'	works (searches for the "$" char instead of end-of-line)
grep "\$"	fails (double quote treatment of $ is different to single quote treatment of $)
grep "\\$"	works

Regular expressions
- POSIX syntax
- POSIX character classes

cut

cut     extract columns of text on command-line

e.g. To extract the RHS columns of the ls listing (by character or by field):

  ls -l | cut -c28-

  ls -l | cut -f6- -d' '

sed


sed     substitute text on the command-line

sed 's|oldstring|newstring'     change first match on each line
sed 's|oldstring|newstring|g'   change all matches 

e.g. To do an ls that highlights the HTML files:

 ls -l | sed 's|\.html|  [Hypertext HTML file]|g'

sed takes a single string (enclosed in quotes) as an argument.
The '|' inside the string above are used just as separators.
We can actually use any character.

sed examples

sed examples (and more)
sed FAQ
To insert a new line:
- Some versions of sed interpret "\n" in the RHS pattern as "newline".
- If "\n" is not recognised, you can do a multi-line sed:
  Put a new line in front of every string "www":
```
sed 's|www|\
www|g'
```
  Put a new line in front of every HTML tag:
```
sed 's|<|\
<|g'
```
  Put a new line after every HTML tag:
```
sed 's|>|>\
|g'
```

To substitute back in the wildcard pattern we matched:


# \( ... \) to mark a pattern
# \1 to reference it later

# e.g. change:
# (start of line)file.html: ...
# to:
# <a href=file.html>file.html</a>: ...

# search for:
# ^\(.*\.html\):
# change to:
# <a href=\1>\1</a>:


grep -i $1 *html |

	sed -e "s|^\(.*\.html\):| <a href=\1>\1</a>: |g"

tr, awk


tr    - character substitutions

change spaces to new lines:

cat file | tr ' ' '\n'

man tr
uses character classes (numeric, alphanumeric, etc.)

awk - a powerful pattern scanning and processing language

dirname, basename

useful cutting and pasting with filenames

dirname	  

basename	  


$ echo $HOME
/users/group/me

$ dirname $HOME
/users/group

$ basename $HOME
me

$ dirname `dirname $HOME`
/users

date

date                         looks like: "Tue Feb 17 16:28:33 GMT 2009"
CURRENTDATE=`date`           remember backquotes
echo $CURRENTDATE             

date "+%b %e"                looks like: "Jan 21" (see "man date")
CURRENTDATE=`date "+%b %e"`    
echo $CURRENTDATE

e.g.

We want a random temporary filename:
```
filename="/tmp/random.`date +%H%M%S`.txt"
```
Perhaps not random enough. e.g. 2 clients accessing server at same second. A totally unique, random temporary name can be got from $$ which is the process ID of this invocation of the shell script:
```
filename=/tmp/random.$$.txt
```
Or want a different seed each time for random-number generator.

On bash (but not csh), see also the following strange environment variable. It does not exist until you try to access it. Then it exists!

set | grep -i random
echo $RANDOM  
set | grep -i random
echo $RANDOM            
echo $RANDOM