linux unix xml

extract xml from the command line

I just discovered this morning this cool utility in my /bin directory : xmllint

You can use it to extract values from your xml files within your shell scripts

$ cat foo.xml



$ echo 'cat //emplist/emp[@no="1"]/ename/text()'|
xmllint --shell foo.xml |
sed -n 3p

I like this !

linux unix

read without Enter

A small unix tip today.

Do you want to continue ?

If you are expecting “y” or “n” but do not want to enforce the user to type y[Enter] but simply y, you can use the -n option in bash.

Within a ksh script:

yorn=$(bash -c 'read -p "Do you want to continue ? " -n 1 ans;echo "$ans"')

Within bash:

read -n 1 yorn

linux unix


Do you know cd ? I thought I did until this afternoon …

OK, let’s start some basic.

I create two directories
$ echo $SHELL
$ mkdir /tmp/foo
$ mkdir /tmp/bar

create a symlink /tmp/bar/baz pointing to /tmp/foo
$ ln -s /tmp/foo /tmp/bar/baz

create a file foo1 in foo
$ touch /tmp/foo/foo1

change to the symbolic link baz
$ cd /tmp/bar/baz

Ok, so far so good. Let’s check what is in ../foo
$ ls ../foo

From the symbolic baz, .. point to /tmp/foo. This is because ls and most command line utilities use the physical path.

To print the Logical [default] and Physical working directories, use pwd -L and pwd -P
$ pwd -L
$ pwd -P

to change directory relatively to the logical path, use cd -L … (default), for physical, use cd -P … !
$ pwd -L
$ cd -L ../foo
ksh: ../foo: not found

Obviously /tmp/bar/foo does not exist

$ pwd -L
$ pwd -P
$ cd -P ../foo
$ pwd

Obviously /tmp/foo/../foo is /tmp/foo

So far so good, some of you may know that already.

Let’s bring some devil element in play
$ bash

Arghh!!! Ôôôôôôôôôôôôôô râge, Ôôôôôôôôôôôôôô désespoir, I switched to a non-working shell!

$ cd /tmp/bar/baz
$ cd -L ../foo
$ pwd -L

Even if I switched to a not working directory, bash cd -L weirdly decided to switch to the physical path instead of the logical path.

Let’s retry
$ cd /tmp/bar/baz
$ mkdir /tmp/bar/foo
$ cd -L ../foo
$ pwd

This time bash cd -L changed to the logical path. So if you use bash and cd, you cannot possibly know where you are landing without checking first if the directory exist !

BTW, I just discovered Digger HD , unrelated to this post of course …

11gR2 news unix

11.2 Sparc

Good news, go there 🙂

linux unix


Something I like in ksh is to change from /my/old/directory/path to /my/new/directory/path by typing cd old new. This does not work in bash

So I had to find a workaround 😉

$ ksh
$ cd /app/oracle/product/
$ cd 6 7
$ pwd
$ bash
$ cd 7 6
bash: cd: 7: No such file or directory
$ c() { cd ${PWD/$1/$2}; }
$ c 7 6
$ pwd

linux unix

chmod -R 777 .

This is one of the thing I hate to see, recursively changing everything to 777 👿

If you want to give read access to all, then 644 is enough for files and 755 for directories. If you want to give execution permission too, you could give 755 to executable files.

Also sometimes you have files with special permission like suid or guid bit (to run as the owner instead of to run as the run user), so it would be best to use relative (go+r) to make the file readable to group and others.

Therefore I prefer relative change. Still there is one thing I do not want, is making every file executable…

Ok, here it is

chmod -R a+rX .

note the big X 🙂 we change recursively all files and dirs to be readable, and we set the executable flag ONLY IF the file is executable for the owner !

linux security unix

to ftp or to sftp

Ftp is seen as an old-time unsecure protocol. Many shops nowadays have switched or are switching to sftp. I will try to point out some differences :

Compatibility: none. the protocol is completly different. Multiple graphical clients however do support both mode. But the basic “ftp” client will not work with sftp.

Ascii mode: only in ftp. In sftp, it is always binary so there will be no conversion. Also no blocksize, recordlength or primary/secondary space for your OS/390 connections.

Interactive mode: similar. you enter your username and password, do cd, put and get. But to quit, by will not work in sftp 😉 Use quit or exit instead

Batch mode: different. Most probably you will end up setting a private/public key infrastructure for your ssh connection and use scp (secure copy). If you are using a ssh client like putty, it is possible to do something like pscp -l user -pw password server:file .

Security: sftp is secure, ftp is not.

Speed: ftp is fast, sftp is slow 🙁 !

Oh NOOOOOOO!!!!! What’s the point is bringing something new if it is slower !!!

Ok, let’s try to download a 100m file:
$ time (echo "open dbsrv01
user oracle secret
get 100m"|ftp -n )

real 0m24.673s
user 0m0.030s
sys 0m0.016s
$ time scp -q oracle@dbsrv01:100m .

real 1m46.978s
user 0m0.108s
sys 0m0.202s

it is about 4x slower! Is there anything we could do about it?

Well, maybe :

$ time scp -q -o Compression=yes oracle@dbsrv01:100m .

real 0m18.634s
user 0m0.748s
sys 0m0.452s

ssh/scp/sftp have a compression mode. If you are transferring your large files across a slow network, this may be an interesting option to consider !

OpenSSH homepage :

dba unix

How to reuse connection in shell

It is Friday, I wanted to give my readers some stuff for the week-end 😉

Imagine that piece of code :

countlines() {
c=$(sqlplus -s /nolog <

I can run this

time ./script1
there are 14 lines in EMP
there are 4 lines in DEPT

real 0m0.46s
user 0m0.06s
sys 0m0.09s

Sounds like a regular shell script. How could we optimize the countlines function? Well, we could create the connection only once and use coprocess pipe (with |& that is pipe ampersand)

sqlplus -s /nolog |&

print -p "connect scott/tiger"

read -p line
if [ $line != Connected. ]
exit 1

print -p "set feed off head off"

countlines() {
print -p "select count(*) from $1;"
read -p c
echo "there is $c lines in $1"

countlines EMP
countlines DEPT

print -p disconnect

A two-ways pipe is opened with sqlplus. There is only one connect, and one disconnect.

Let's check the performance :

$ time ./script2
there is 14 lines in EMP
there is 4 lines in DEPT

real 0m0.23s
user 0m0.00s
sys 0m0.01s

About twice as fast! Note the "Connected" output may not exist in recent version of sqlplus in silent mode. If you have a script that generates hundreds of connections, or which create a connection every 5 seconds or so, think about it 🙂

Enjoy your week-end

dba linux unix

return code before grep

In my previous post hide-password-from-ps-output-sql-loader I mentioned a way to pass the password to the loader thru a parameter file. As correctly suggested by Brian Tkatch, the password could be passed as standard input

sqlldr control=x.ctl silent=header,feedback <

The Username: prompt is displayed 🙁   🙁

How do we get rid of this ?

sqlldr control=x.ctl silent=header,feedback <

There is no output. But what's the error code
echo $?

The return code is 1 🙁

This is not the error code from sqlldr, but the error code from grep !

Ok, here is the trick, a bit cryptic if you are not familiar with file descriptors

( ( (sqlldr control=x <&3) |grep -v "^Username:" >&4 ) 3>&1 |(read x;exit $x) )4>&1
echo $?

The return code is 0 🙂

linux unix

To bash or not to bash

I have been inspired by Chen to talk about bash…

I have been using ksh for many years, and I mean ksh88 not ksh93. The main reason is, I want my script to run the same way in any Unix flavor.

ksh93 has never been too much popular. I used it a few time to sleep half a second

echo sleep 0.5| /usr/dt/bin/dtksh

ksh has a lot of nice features. I just used one of them in my script :

$ typeset -u name
$ read name?"Enter your name : "
Enter your name : Laurent
$ echo $name

Way easier to force a variable to be uppercase rather than using echo|tr etc

Bash has some nice features too, but unfortunately every OS release come with a different bash version, which is the same pain as perl when you want to write a script that last for a decade or two.

Ok, just4fun

$ mkdir -p {a..z}/{1..9}
... create directories a/1 a/2 ... z/8 z /9
$ [[ text =~ t..t ]]
... check if text matches regular expression t..t
$ echo ${text/pattern/string}
... replace pattern by string

The first two commands require bash3, the last is just fine with bash2.

Have fun shell-scripting 🙂

linux unix

echo does not accept end of arguments operator

Let’s start with an example :

$ cat AI
while :
echo "What's your name ?"
read a
if [ ! $a ]
echo "Your name is :"
echo $a
echo "Bye"

$ ./AI
What's your name ?
Your name is :

What's your name ?
Your name is :

What's your name ?
Your name is :

What's your name ?


This artificial intelligence is not very intelligent, it cannot recognize me if I am called “-e” (it is Friday, have a look at Little Bobby Tables ).

Most unix tools consider
-- signals the end of options and disables further option processing

But not echo 🙁
$ touch -e
touch: invalid option -- e
Try `touch --help' for more information.
$ touch -- -e
$ ls -l -e
ls: invalid option -- e
Try `ls --help' for more information.
$ ls -l -- -e
-rw-r--r-- 1 lsc dba 0 Oct 31 15:44 -e
$ rm -e
rm: invalid option -- e
Try `rm ./-e' to remove the file `-e'.
Try `rm --help' for more information.
$ rm -- -e
$ echo -e

$ echo -- -e
-- -e

So, what’s the solution? well, probably not using “echo”, for example printf

$ (echo “What’s your name ?”
read a
echo “Your name is :”
printf “%s\n” “$a”)
What’s your name ?
Your name is :

cygwin linux unix

updatedb does not work with samba drives and cygwin

updatedb and locate are parts of the findutils package, which exists since a long time on most unix / linux flavors.

Something I never get worked is to search for files on network drives with Cygwin, I always got :

find: /cygdrive/y/Favorites/Links/del changed during execution of find (old inode number -474324984, new inode number -44545478
4, filesystem type is system) [ref 1114]
find: /cygdrive/y/Favorites/Links/del changed during execution of find (old inode number -513303800, new inode number -47432498
4, filesystem type is system) [ref 1114]

which is because the file system is mounted and unmounted on demand and get new inodes.

So I wrote a hack in /usr/bin/updatedb

# lschnei2
for d in c: y:
cd $d
/cygdrive/c/OS/system32/attrib /s
) |
sed ‘s,………..\(.\):,/cygdrive/\1,;y,\\,/,’ |
tr ‘\r’ ‘\0’ | tr -d ‘\n’

Then I can search efficiently

$ time updatedb

real 0m18.273s
user 0m3.806s
sys 0m7.332s

$ time locate hosts

real 0m0.261s
user 0m0.265s
sys 0m0.015s

dba linux unix

How to cron?

RTFM is not the best answer …

man crontab
SunOS 5.10 Last change: 10 Nov 2005

User Commands crontab(1)
A crontab file consists of lines of six fields each. The
fields are separated by spaces or tabs. The first five are
integer patterns that specify the following:

minute (0-59),
hour (0-23),
day of the month (1-31),
month of the year (1-12),
day of the week (0-6 with 0=Sunday).

so far so good. But read this

Each of these patterns can be either an asterisk (meaning
all legal values) or a list of elements separated by commas.
An element is either a number or two numbers separated by a
minus sign (meaning an inclusive range). Time specified here
is interpreted in the timezone of the cron(1M) daemon, which
is set system-wide in /etc/default/init. Entries do not use
the invoking user's timezone. The specification of days can
be made by two fields (day of the month and day of the
week). Both are adhered to if specified as a list of ele-
ments. See .

See .

Let’s imagine you want to run a job today, Thursday May 15, 2008 at 2:15pm
You write:
15 14 15 05 4 /tmp/run-my-job

The job will be run Today at 2:15, next year Friday May 15, 2009 at 2:15pm and next week Thursday May 22, 2008 at 2:15pm…

The correct method is to use
15 14 15 05 * /tmp/run-my-job

Specifying both day of week and day of month is not what I expected 😈