Dr. Mark Humphrys

School of Computing. Dublin City University.

Home      Blog      Teaching      Research      Contact

Search:

CA216      CA249      CA318

CA400      CA651      CA668


Shell utilities



grep

grep                    search for a string

grep string file

(output of prog) | grep string | grep otherstring

-i    ignore case
-v    return all lines that do NOT match


  
More complex solution to long-line problem:
  1. Use egrep to display from 0 to max 40 chars each side of the string:
    egrep -o ".{0,40}string.{0,40}" file
    . means any char
    0 to max n of any char is .{0,n}



string matching / regular expressions

^                       start-of-line
$                       end-of-line
.                       any character

where "c" stands for the character:

c*			0 or more instances of c
cc*			1 or more instances of c
grep "  *"		1 or more spaces
.*                  any sequence of characters



where "c" has a special meaning, e.g. is $ or ., etc:

\c                  the character itself
grep "\."           the '.' character itself

recall the two forms of quote:

grep '\$'	works (searches for the "$" char instead of end-of-line)
grep "\$"	fails (double quote treatment of $ is different to single quote treatment of $)
grep "\\$"	works



  


From xkcd.




cut

cut     extract columns or fields of text on command-line

To extract columns  30  to end of line of the ls listing:

  ls -l | cut -c30-

In grep output, extract the 1st field, with delimiter ":"

 grep string *html | cut -f1 -d':' 

Extract the 2nd to end fields, with delimiter ":"

 grep string *html | cut -f2- -d':' 


Q. Why "-f2-" ?
Why not "-f2" ?


sed


sed     "stream editor" - find and replace text on command-line (and other things)


sed 's|oldstring|newstring|'    change first match on each line
sed 's|oldstring|newstring|g'   change all matches 

'|' is just my choice of a separator.
Other people like '/'
We can actually use any character as a separator (whatever comes first after "s").


e.g. ls listing that highlights web pages:

 ls -l 	| sed "s|\.html| [Web page]|"


e.g. ls listing that changes how my username appears:

 ls -l 	| sed "s|$USER|ME|"






sed examples


To insert a new line


To put new lines in front of and after every HTML tag


To substitute back in the pattern we matched


# \( ... \) to mark a pattern
# \1 to reference it later

# e.g. change:
# (start of line)file.html: ...
# to:
# <a href=file.html>file.html</a>: ...

# search for:
# ^\(.*\.html\):
# change to:
# <a href=\1>\1</a>:


grep -i $1 *html |

	sed -e "s|^\(.*\.html\):| <a href=\1>\1</a>: |g"





tr, awk


tr    - character substitutions

change spaces to new lines:

cat file | tr ' ' '\n' 


awk - a powerful pattern scanning and processing language


dirname, basename

useful cutting and pasting with filenames
dirname	  

basename	  

$ echo $HOME /users/group/me $ dirname $HOME /users/group $ basename $HOME me $ dirname `dirname $HOME` /users




date

date                         looks like: "Tue Feb 17 16:28:33 GMT 2009"
CURRENTDATE=`date`           remember backquotes
echo $CURRENTDATE             

date "+%b %e"                looks like: "Jan 21" (see "man date")

date "+%b.%e.log"            can add things to the string 

file=`date "+%b.%e.log"`    
echo $file             


Using date to get unique filename

Say web server in response to client needs to make a temporary file.
Use date to get a new filename that is unique to the current second:
timenow=`date +%H%M%S`
filename="/tmp/random.$timenow.txt"
Unique to second and nanosecond:
date "+%H.%M.%S.%N"


Alternative ways of getting unique filename


About random-number generators:

A pseudo-random-number generator typically generates a series of random-looking numbers based on a start seed.
If the seed is identical, the algorithm will generate the same numbers.
So we want a different start seed each time we run.
Seed could be based on the current time (second and nanosecond).




JavaScript random number generation demo.
Usage: Freeze the seed key. Set parameters. Then generate random numbers.
Can view JavaScript code to see how it works.



Feeds      w2mind.org

On Internet since 1987.