Don’t call it test

There are quite a few names to avoid in your scripts. Even if there are not reserved-words, keep away !

I’ll start with test

cd $HOME/bin
vi test
  echo hello world
chmod +x test
  hello world

The problem is that it may break your other scripts

$ ssh localhost test 1 = 2 && echo WHAT???
hello world

And it may break sooner or later, depending on your OS / version / default shell / default path / others.

There are quite a few filenames you should not use, like test, date, time, hostname, mail, view, touch, sort and make. The command type lists some of those as reserved word, shell builtin, tracked alias, shell keyword. But again it is not consistent over Unix flavors.

$ uname -sr; type date
SunOS 5.10
date is /usr/bin/date
$ uname -sr; type date
Linux 2.6
date is a tracked alias for /bin/date

Your sysadmin may also alias things for colors and safety in the common profile: for instance vi, ls, rm. But if it annoys you, then use \vi instead of vi.

Untrusted X11 forwarding

I wrote a while ago about my security concerns regarding

xhost +
xterm -display mypc:0

Way back then, I suggested ssh tunnel. SSH is pretty easy to set up, by enabling the X11Forwarding option.

In OpenSSH 3.8 release note, 2004, there was a new default .

ssh(1) now uses untrusted cookies for X11-Forwarding

In the man ssh_config page, it’s still documented as being the default

ForwardX11Trusted The default is ‘no’

But it actually isn’t on most *ix derivates, e.g. RedHat /etc/ssh/ssh_config

# If this option is set to yes then
# remote X11 clients will have full access
# to the original X11 display. As virtually
# no X11 client supports the untrusted
# mode correctly we set this to yes.
ForwardX11Trusted yes

Who is we?

Okay, let’s go back.

If you use the unsafest method, xhost + and xterm -display pc:0, then you grant everybody the right to manipulate X.

If you use trusted ssh, which is the _undocumented_ default in Linux, then you grant this right only to anyone with access to your authority, most probably located in the file $HOME/.Xauthority. So root and yourself, at least.

If you trust neither yourself nor root, you could restrict access to your resource, preventing one hacker from switching your mouse buttons or doing a screenshot. But this is probably going to prevent most of your applications from working. Also, it probably won’t work at all if you use putty, reflection and (virtually any?) other client tools.

If you want to force Trusted mode, use -Y or -o ForwardX11Trusted=yes.

If you want to force Untrusted mode, use -X and -o ForwardX11Trusted=no.

If you use only -X, it may transparently defaults to the more convenient but less secure -Y. Sometimes. At least on Linux OpenSSH. But if you use different Unix / SSH flavours, the -X may ends with an error message like connection to “localhost:10.0″ refused by server. In that case, simply use -Y. Actually, always use -Y if you want Trusted.

run sudo, ssh, password, su in simulated interactive mode

Some commands do not like non-interactive mode

$ passwd <<EOF
> oldpassword
> newpassword
> newpassword
Changing password for user lsc.
Current password for passwd: Authentication token manipulation error
$ echo oraclepassword | su - oracle
standard in must be a tty
$ echo sudopassword | sudo su - oracle
[sudo] password for lsc:
sudo: sorry, you must have a tty to run sudo

But ok, if you really want to run those in a script, you have plenty of clean (no hack there) ways of doing this.

For instance, let’s use a screen called BAR.

$ xterm -e "screen -S BAR" &
[1]     31732

Now we have an interactive terminal. It could be redirected to a frame buffer device if no x server is started.

Not really a human device, but an interactive terminal.

Now let’s send stuff

$ CR="$(echo '\r')"
$ screen -S BAR -X stuff "sudo su - oracle$CR"
$ screen -S BAR -X stuff "sudopassword$CR"
$ screen -S BAR -X stuff "id > /tmp/xxx$CR"
$ screen -S BAR -X stuff "exit$CR"
$ screen -S BAR -X stuff "exit$CR"
[1] +  Done                    xterm -e "screen -S BAR" &
$ cat /tmp/xxx
uid=100(oracle) gid=100(dba) groups=100(dba)

Usual disclaimer: it is a bad security practice to hardcode your passwords in scripts. Use this only if you really understand security. Read man openssl about how to use openssl to encrypt your password. Ask your security friends before trying

How to *really* send a script to the background

Let’s check this small script

echo foo.1:`date` | tee $HOME/tmp/foo.txt
sleep 3
echo foo.2:`date` | tee -a $HOME/tmp/foo.txt

$ $HOME/tmp/
foo.1:Thu Nov 27 17:34:53 CET 2014
foo.2:Thu Nov 27 17:34:56 CET 2014

Very obvious, I write to the console, wait three seconds, then write to the console.

Ok, let’s take another script that would call this script in the background using &

echo bar.1:`date`
$HOME/tmp/ &
echo bar.2:`date`

$ $HOME/tmp/
bar.1:Thu Nov 27 17:36:32 CET 2014
bar.2:Thu Nov 27 17:36:32 CET 2014
foo.1:Thu Nov 27 17:36:32 CET 2014
foo.2:Thu Nov 27 17:36:35 CET 2014

bar is printing the date, calling foo in the background, then printing the date, then it returns to you, and foo is still running.

BUT this is only in a relative background …

Let’s try this

$ time $HOME/tmp/ > /dev/null

real    0m0.01s
user    0m0.00s
sys     0m0.00s

So it takes no time to run bar you believe ?

Let’s try, for instance, over ssh (or cron or whatever)

$ time ssh localhost $HOME/tmp/ > /dev/null
real    0m3.81s
user    0m0.01s
sys     0m0.01s

running bar suddenly waits 3 seconds for foo to finish.

To be sure the script is sent to the farest background, you need to close the file descriptors, stdin, stdout, stderr

I rewrote it as

echo bar.1:`date`
$HOME/tmp/ <&- >&- 2>&- &
echo bar.2:`date`

$ time ssh localhost $HOME/tmp/ >/dev/null
real    0m0.44s
user    0m0.00s
sys     0m0.00s

Now the script baz is immediately finished and does not wait for foo to complete

fun with cron

Today I find out that my scheduler was too busy to execute all jobs in my crontab !?

* * * * * (while :;do ssh :; done)
59 23 19 06 * touch /tmp/bang

my while loop is going to produce so much hangs on the cron deamon that it may not be able to read the crontab once a minute. If it reads it at 23:58 and at 00:00, the 23:59 won’t be run.

This is actually the first time I see this behaviour. And -believe me- it’s annoying!

Testing for (non-)empty string in shell

One way to test for (non-)empty string is to use test and -z (-n)

$ x=foo
$ test -z "$x"
$ echo $?

This is mostly seen with an if and [ -z … ] syntax

$ y=bar
$ if [ -n "$y" ];
  then echo non-empty; 

Instead of a variable, it could be the output of a script.


if [ -n "$(grep ORA- alertDB01.log)" ] 
  echo there is an error in the alert log
  echo "fine :)"

This will work for years until one day you get :

ksh: no space

Why that? This is the way the shell works. Your shell (here ksh on AIX) starts having errors as soon as your subshell (here the grep) is exhausting the space.

$ wc -l alertDB01.log
       2 alertDB01.log
$ if [ -n "$(grep ORA- alertDB01.log)" ]; 
  then echo non-empty; 
  else echo "fine :)"; 
$ wc -l alertDB01.log
   75025 alertDB01.log
$ if [ -n "$(grep ORA- alertDB01.log)" ]; 
  then echo non-empty; 
  else echo "fine :)"; 
ksh: no space

You got a memory error, how the shell will react is random (core dump, errors, continue, crashes). It will just bug and you do not want this.

There is more than one to circumvent this. For instance you could use the return code of grep

$ if grep ORA- alertDB01.log >/dev/null;
  then echo non-empty;
  else echo "fine :)";

Different shells (Bash / Bourne) and different OSs (Linux / AIX / HPUX) may react differently. If AIX crashed with a 50’000 lines, it may scale up to millions of lines in recent Linux’s – but still use trucks of memory

hot to bypass requiretty in sudo

You can execute it a command without password from the commande line

$ sudo -l
User lsc may run the following commands on this host:
    (root) NOPASSWD: /usr/local/bin/helloworld
$ sudo /usr/local/bin/helloworld
Hello World!

Now you try to run it via cron and you get

sudo: sorry, you must have a tty to run sudo

The message is clear, you need a terminal.

Either you edit your sudoers files to disable requiretty, or you just get yourself a terminal.

Maybe you tried to assign a pseudo terminal with ssh -t, but you may get an error if ssh has no local tty

Pseudo-terminal will not be allocated because stdin is not a terminal.

Don’t despair, read man ssh

Multiple -t options force tty allocation, even if ssh has no local tty.

Let’s try

* * * * * ssh -t -t sudo /usr/local/bin/helloworld >> /tmp/txt

This should work, providing you configured ssh keys :)

use cron to schedule a job only once

I wrote about not using DAY OF MONTH and DAY OF WEEK simultanously in how to cron

The correct method is to use
15 14 15 05 * /tmp/run-my-job

But… I wrote this five years ago. Hmmm ! Not that correct then since it would run every year 馃槈

Ok, periodically I check for jobs are scheduled to run a specific date only

$ crontab -l|awk '$1!~/#/&&$3*$4'
15 14 15 05 * /tmp/run-my-job

I have 9 more days to remove this before it runs for the fifth time :)

return code and sqlplus

Calling a shell script from within sqlplus is buggy…

I have reported bug 3798918 in (back in 2004) and bug 13349119 in because some metalink guru closed 3798918 as not reproducible.

As written in return code, host does not return the correct code

SQL> host exit 7
SQL> def _RC
DEFINE _RC             = "0" (CHAR)

If you never use _RC, you may believe you are safe. But watch this :

SQL> get list
  1  #!/bin/sh
  2  if /bin/false
  3  then
  4    echo this is wrong
  5* fi
SQL> host ./
this is wrong

The return code not being set is not only affecting the _RC variable, but it is also affecting all subshells !

Note this is not reproducable with SQLPLUS /NOLOG

SQL> host false
SQL> def _rc
DEFINE _RC             = "1" (CHAR)
SQL> conn x/x
ORA-01017: invalid username/password; logon denied
SQL> host false
SQL> def _rc
DEFINE _RC             = "0" (CHAR)

After my (failed or successfull) tentative to connect as x/x, it is reproducible again

How to quit crontab -e without overwritting cron

Imagine my crontab

* * * * * /usr/bin/date > /tmp/foo

I am writing the date to /tmp/foo every minute
$  cat /tmp/foo
Thu Jul  5 08:45:01 CEST 2012

Now I want to view my crontab in my EDITOR (vi).

$ crontab -e

I do not quit yet.

In the meantime, my colleague modify the crontab.

* * * * * /usr/bin/date > /tmp/bar

Later, I quit vi with :q!

O Surprise, the crontab is * * * * * /usr/bin/date > /tmp/foo again

According to the doc :

When you finish creating entries and exit the file, the crontab command
copies it into the /var/spool/cron/crontabs directory

Even if you did not make change, you overwrite the content of your crontab !

If you want to exit your crontab editor really without overwritting the crontab, you need to kill yourself.


[2] + Stopped (SIGTSTP)        crontab -e
$ kill %2
[2] + Stopped (SIGTTOU)        crontab -e

Thanks to Colin comment, I realized I could not kill with kill, let’s kill with -9

$ kill -9 %2
[2] + Killed                   crontab -e

xhost+ security hole part 2

Five years ago I wrote xhost+ is a huge security hole, I turned out red this morning when my neighbour sent me a smiley via X.

Do I really want everyone to have full access to my screen? No, I don’t. And I don’t do xhost+.

So why did it happen to me ???

I am using X-Window Attachmate aka Reflection X. And in this tool, according to the doc, the default X policy is unrestricted. This is in my opinion a huge flaw in the security design. Make sure you always change this to something more secure.

In Reflection X Manager Settings, Category Security, choose for instance User-based security and Prompt. Configuring X Cookies is probably more cumbersome.

Then when you or someone else will start an XTERM on your desktop, you will get a nice dialog box :

[Reflection X]
Client could not successfully authenticate itself to Reflection X server. Would you like Reflection X to connect to this client as an UNTRUSTED client ? Client originated from (RX1303)

Ok, I have to click one more button, but at least I can deny access to my screen :)

[alert] AIX Posix Timezone issue

Maybe you did get or you will get an issue with the date command in AIX.

expected behavior, Linux

$ TZ=NZST-12NZDT,M10.1.0/2,M3.3.0/3 date
Sat Mar 17 00:14:54 NZDT 2012
$ TZ=Pacific/Auckland date
Sat Mar 17 00:14:58 NZDT 2012

unexpected behavior, AIX

$ TZ=Pacific/Auckland date
Sat Mar 17 00:15:50 GMT+13:00 2012
$ TZ=NZST-12NZDT,M10.1.0/2,M3.3.0/3 date
Fri Mar 16 23:15:52 NZST 2012

The consequence : date, and all other unix commands like ls, who, ps that display the date in human readable format, and all programs that use ctime are affected

$ TZ=NZST-12NZDT,M10.1.0/2,M3.3.0/3 perl -e 'use POSIX;print ctime(time)'
Fri Mar 16 23:19:51 2012

Reference and link to the fixes :

delete unused shared memory segments from an Oracle instance

Once upon a time, a dba issues some kill -9 to clean up dying database processes. Or the database instance crashes. This will left some shared memory segments. Note 68281.1 describe how to remove them on a server with multiple databases.

First, list the ipc process

$ ipcs
IPC status from /dev/mem as of Mon Nov 14 11:28:58 CET 2011
T        ID     KEY        MODE       OWNER    GROUP
Message Queues:
q         0 0x4107001c -Rrw-rw----     root   printq

Shared Memory:
m         0 0x7800006f --rw-rw-rw- itmuser1 itmusers
m         1 0x78000070 --rw-rw-rw- itmuser1 itmusers
m   5242882 0x41d2ba80 --rw-r-----   oracle      dba
m  99614723 0xb0d4d164 --rw-rw----   oracle      dba
m  12582917 0xb84cbc28 --rw-rw----   oracle      dba
m  79691782 0x1058873f --rw-------   oracle      dba
m 638582792 0x78000382 --rw-rw-rw-     root   system
m 218103817 0x780003b7 --rw-rw-rw-     root   system
s         1 0x6202c477 --ra-r--r--     root   system
s   6291461 0x0102c2d8 --ra-------     root   system
s         6 0xa100004b --ra-ra-ra-     root   system

Get a list of the running databases

$ ps -ef | grep pmon | grep -v grep
  oracle  483334       1   1   Aug 16      -  6:46 ora_pmon_db03
  oracle 1253476       1   0   Oct 31      -  2:00 ora_pmon_db01
  oracle 2298042       1   0   Sep 05      - 11:07 ora_pmon_db02

Then, for each database, get the ipc information
$ export ORACLE_SID=db01
$ sqlplus / as sysdba
SQL> oradebug setmypid
Statement processed.
SQL> oradebug ipc
Information written to trace file.
SQL> oradebug tracefile_name
$ awk '/Shared Memory:/{getline;getline;print}' /u01/app/oracle/admin/db01/udump/db01_ora_2625574.trc
5242882         0x41d2ba80
$ export ORACLE_SID=db02
$ sqlplus / as sysdba
SQL> oradebug setmypid
Statement processed.
SQL> oradebug ipc
Information written to trace file.
SQL> oradebug tracefile_name
$ awk '/Shared Memory:/{getline;getline;print}' /u01/app/oracle/diag/rdbms/db02a/db02/trace/db02_ora_2441408.trc
99614723        0xb0d4d164
$ export ORACLE_SID=db03
$ sqlplus / as sysdba
SQL> oradebug setmypid
Statement processed.
SQL> oradebug ipc
Information written to trace file.
SQL> oradebug tracefile_name
$ awk '/Shared Memory:/{getline;getline;print}' /u01/app/oracle/diag/rdbms/db03b/db03/trace/db03_ora_2617416.trc
12582917        0xb84cbc28

Compare it with the first list, and if you are absolutely sure to do what you are doing, remove the oracle segments that are not assigned to any database instance with ipcrm. If possible try first to figure out where they come from and do a shutdown abort of the not-correctly-stopped database.

Ok, with ipcrm

$ ipcrm -m 79691782

I removed the segment that apparently does not relate to any running instance

This could help you if you are really forced to remove some shared memory segments and you cannot afford shutting down other databases.

Check if it a program is already running in Unix

There is more than one way to do it, the safe is probably to check if /home/lsc/OH_YES_I_AM_RUNNING exists and believe it. This is called the file.PID method and is widely used (Apache used to use it since a long long time). It needs file. It needs cleanup if you reboot your server in the middle of something (and surely you do not want to delete old pid files yourself)

Ok, often you see this :

ps -ef | grep program

There you list all processes and check the lines that contain program. So some does a vi program or anything worse (emacs?), you will get more rows than needed.

Maybe it is fine to run program with different arguments, this must be decided.

Well, take a simple test case : and :

while :
  date  > /dev/null

let’s try to use ps

$ nohup ./ &
$ nohup ./ &
$ jobs
[2] +  Running                 nohup ./ &
[1] -  Running                 nohup ./ &
$ ps -ef | egrep 'x[12]'
  u22  9240796  6226164  30 14:56:52  pts/2  0:00 /bin/ksh ./
  u22 20840608  6226164  31 14:56:48  pts/2  0:01 /bin/ksh ./

So fine so good, I see I have one instance of each program.

Let’s try to see if the results are consistent over time :

 $ n=9999;while :
    ps -ef | 
      egrep 'x[12].sh'>f
    if [ $(wc -l <f) != $n ]
      n=$(wc -l <f)
      cat f
      echo "==> $n"

Fri Oct 28 15:01:01 CEST 2011
  u22  9240796  6226164  32 14:56:52  pts/2  0:14 /bin/ksh ./
  u22 20840608  6226164  28 14:56:48  pts/2  0:14 /bin/ksh ./
==>        2

Fri Oct 28 15:01:08 CEST 2011
  u22  9240796  6226164  50 14:56:52  pts/2  0:14 /bin/ksh ./
==>        1

Fri Oct 28 15:01:09 CEST 2011
  u22  9240796  6226164  52 14:56:52  pts/2  0:14 /bin/ksh ./
  u22 20840608  6226164  53 14:56:48  pts/2  0:15 /bin/ksh ./
==>        2

Fri Oct 28 15:01:17 CEST 2011
  u22  9240796  6226164  40 14:56:52  pts/2  0:15 /bin/ksh ./
  u22 10944520  9240796   0 15:01:17  pts/2  0:00 /bin/ksh ./
  u22 20840608  6226164  31 14:56:48  pts/2  0:16 /bin/ksh ./
==>        3

the fact that a subshell (pid 10944520 ) of x2 appear is not a problem for me. I have much more of a problem at 15:01:08 where x1 disappeared !

Conclusion : you cannot trust ps

shell and list of files

How do you loop thru a list of files?

For instance you want to archive than delete all pdf documents in the current directory :

Bad practice :

tar cvf f.tar *.pdf
rm *.pdf

There are multiple issue with the command above

1) new files could come during the tar, so the rm will delete files that have not been archived

filelist=$(ls *.pdf)
tar cvf f.tar $filelist
rm $filelist

2) if there is no file, tar and rm will return an error

filelist=$(ls|grep '\.pdf')
if [ -n "$filelist" ]
  tar cvf f.tar $filelist
  rm $filelist

3) this will not work for long list (above 100k documents)

filelist=/tmp/filelist.$(date "+%Y%m%d%H%M%S").$$.$RANDOM
ls|grep '\.pdf' > $filelist
if [ -s "$filelist" ]
  tar cvfL f.tar $filelist
  for f in $(<filelist)
    rm $f

As you see, this require special handling. tar for instance use the -L option to accept a list of files, rm could delete files one by one (or in bunches with xargs -L).

This 100’000 limit (the limit may vary for your shell/os) is something that often gets forgotten.

Typical error that could occur are

ksh: no space
bash: Arg list too long

pstree in AIX

For those who do not want to download some linuxlike freeware on your aix box, use ps -T :)

ps -fT 2412672
     UID     PID    PPID   C    STIME    TTY  TIME CMD
  oracle 2412672       1   0   Sep 05      -  0:00 /u01/app/oracle/product/OAS
  oracle  630956 2412672   0   Sep 05      -  6:11     \--/u01/app/oracle/prod
  oracle 1347672  630956   0   Sep 05      - 15:32        |\--/u01/app/oracle/
  oracle 1437836  630956   0   Sep 05      -  1:02        |\--/u01/app/oracle/
  oracle  880820 1437836   0   Sep 05      -  0:32        |   |\--/u01/app/ora
  oracle 1036532 1437836   0   Sep 05      -  0:00        |   |\--/u01/app/ora
  oracle 1134796 1437836   0   Sep 05      -  0:01        |   |\--/u01/app/ora
  oracle 1343712 1437836   0   Sep 05      -  0:33        |   |\--/u01/app/ora
  oracle 1368166 1437836   0   Sep 05      -  1:11        |   |\--/u01/app/ora
  oracle 1384684 1437836   0   Sep 05      -  0:33        |   |\--/u01/app/ora
  oracle 1392862 1437836   0   Sep 05      -  0:32        |   |\--/u01/app/ora
  oracle 1396898 1437836   0   Sep 05      -  0:33        |   |\--/u01/app/ora
  oracle 1482978 1437836   0   Sep 05      -  0:32        |   |\--/u01/app/ora
  oracle 1527890 1437836   0   Sep 05      -  0:00        |   |\--/u01/app/ora
  oracle 1781798 1437836   0   Sep 05      -  0:32        |   |\--/u01/app/ora
  oracle 2195474 1437836   0   Sep 26      -  0:13        |    \--/u01/app/ora
  oracle 1626296  630956   0   Sep 05      - 13:49         \--/u01/app/oracle/

Generate network graph from command line

I recently wrote on gnuplot, today I tried another command line utility to generate graphs, graphviz, version 2.24.0 on AIX5L.

Pretty straightforward syntax :

  echo "digraph Emp {"
    sqlplus -s -L scott/tiger << EOF
      set pages 0 lin 120 hea off feed off
        ename ||'->'|| 
        (select ename from emp where empno=e.mgr) || ';' 
      from emp e where mgr is not null;
  echo "}"
)| neato -Tpng | uuencode Emp.png | mailx

(or neato -Tpng -o Emp.png to save locally, or -Tps|lp, etc…)

send graph per mail from sqlplus

How to send a graph with a single command from your database to your mail in Unix?

I tried this (gnuplot is available for Solaris, AIX and most Unix derivates) :

echo '
  set hea off pages 0 feed off
  prom set title "salaries of EMP"
  prom unset key
  prom unset xtics
  prom unset xlabel
  prom set term png
  prom plot "-" with circle 
  select row_number() over (order by sal),sal from emp;
  prom e
  prom quit
' | 
  sqlplus -s scott/tiger | 
    gnuplot | 
      uuencode emp.png | 

The picture is sent as attachment.

If you want to embed your png in an HTML mail, use sendmail

/usr/sbin/sendmail -t <<EOF
Subject: EMP
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="FILEBOUNDARY"

Content-Type: multipart/alternative; boundary="MSGBOUNDARY"

Content-Type: text/html
Content-Disposition: inline

<html><body>Below a graph...<br/>
<img src="cid:png"/><br/></body></html>
Content-Type: image/png
Content-Disposition: inline; filename="png.png"
Content-Transfer-Encoding: base64
Content-Id: <png>


Sendmail syntax and more tips on : Sending Emails With Sendmail – Part 3

vi large files

Once upon a time a colleague asked me if there is a better editor than vi installed on my db server. Well, I was not really about arguing the benefit of ed (less memory usage, no useless error message). But one advantage of ed was (I believed) the ability to read large files

$ vi alert_DB01.log
Out of memory saving lines for undo - try using ed
"alert_DB01.log" 612637 lines, 20458891 characters
Thu Aug 25 16:08:20 2011
LNS: Standby redo logfile selected for thread 1 sequence 6898 for destination LOG_ARCHIVE_DEST_2

of course most of readers are using some clicky fancy tools with colors and/or unlimited undo.

Well, for those who use vi on AIX, check this

$ vi -y 9999999 alert_DB01.log
Tue Oct 26 13:59:12 2010
Starting ORACLE instance (normal)
sskgpgetexecname failed to get name
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
ILAT =27
"alert_DB01.log" 612662 lines, 20459538 characters

Oh yes !!!

scp tuning

I twitted yesterday :

copying 1TB over ssh sucks. How do you fastcopy in Unix without installing Software and without root privilege?

I got plenty of expert answers. I have not gone to far in recompile ssh and I did not try plain ftp.

Ok, let’s try first to transfer 10 files of 100M from srv001 to srv002 with scp :

time scp 100M* srv002:
100M1    100%   95MB   4.5MB/s   00:21
100M10   100%   95MB   6.4MB/s   00:15
100M2    100%   95MB   6.0MB/s   00:16
100M3    100%   95MB   4.2MB/s   00:23
100M4    100%   95MB   3.4MB/s   00:28
100M5    100%   95MB   4.2MB/s   00:23
100M6    100%   95MB   6.4MB/s   00:15
100M7    100%   95MB   6.8MB/s   00:14
100M8    100%   95MB   6.8MB/s   00:14
100M9    100%   95MB   6.4MB/s   00:15

real    3m4.50s
user    0m27.07s
sys     0m21.56s

more than 3 minutes for 1G.

I got hints about the buffer size, about SFTP, about the cipher algorythm, and about parallelizing. I did not install new software and I have a pretty old openssh client (3.8). Thanks to all my contributors tmuth, Ik_zelf, TanelPoder, fritshoogland, jcnars, aejes, surachart, syd_oracle and the ones the will answer after the writting of this blog post…

Ok, let’s try a faster algorythm, with sftp (instead of scp), a higher buffer and in parallel

$ cat batch.ksh
echo "progress\nput 100M1" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M2" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M3" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M4" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M5" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M6" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M7" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M8" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M9" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M10" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
$ time batch.ksh
real    0m19.07s
user    0m12.08s
sys     0m5.86s

This is a 1000% speed enhancement :-)

What is the current setting of NLS_LANG in sqlplus?

I just learnt a neat trick from Oracle Support.

How do you see the current value of NLS_LANG in SQLPLUS ?

HOST is not the right answer.


SQL> host echo $NLS_LANG



The correct setting is revealed by @.[%NLS_LANG%]

SP2-0310: unable to open file ".[AMERICAN_AMERICA.WE8ISO8859P1]"


SQL>  @.[%NLS_LANG%]
SP2-0310: unable to open file ".[AMERICAN_AMERICA.WE8ISO8859P1]"

It could well be that both return the same answer, but not necessarly, as shown above.

The unix discrepancy is related to the subshell created by HOST. The subshell may read some .profile and overwrite the value of NLS_LANG

In Windows, the NLS_LANG setting may be set by sqlplus according to some registry entries

Time offset in Unix

What is the time offset of the current date in Unix?

perl -e '
print ($d."\n")'

Am I in summer (DST)?

perl -e 'if((localtime)[8]){print"yes"}else{print "no"}'

to cvs or to subversion

First surprise, after migration, the size of my subversion folder is double the size of my cvs folder. With a bunch of 2Gb disks shared amoung dozens of unix persons, and regular reminders the current usage reached 100%, you will feel the pain of having each developers doublesizing its home directory…
The reason is a .svn/test-base directory containing a duplicate of your local copy.
The benefit there is that it reduces network usage, for instance when diffing with the working version.

Second surprise, not sure if it is a generic issue or related to the one above, a full checkout take ages :( .

svn and cvs command line interfaces almost have the same options, a bit like vi and vim -vi is my favorite- but the “tagging” massively differs. The is no cvs tag command in subversion, you use a different syntax.
cvs :

$ cd /home/lsc/CVS/proj1/source/pkg
$ cvs tag MYTAG helloworld.pkb helloworld.pks

now enjoy the power of subversion! [updated as I found the –parents option]

$ cd /home/lsc/SVN/proj1/source/pkg
$ svn copy --parents helloworld.pkb -m "my first svn tag"
$ svn copy --parents helloworld.pks -m "my first svn tag"

I always loved the cvs rename command

$ cvs rename pkg1.pkb pkg2.pkb
cvs [rename aborted]: Remote server does not support rename

Joke apart, in CVS I used to physically logon to the server and manually move the pkg1.pkb,v to pkg2.pkb,v
Renaming works in subversion :

$ svn move pkg1.pkb pkg2.pkb
A         pkg2.pkb
D         pkg1.pkb

Update: One more annoyance, you cannot checkout a single file :(
subversion faq

extract xml from the command line

I just discovered this morning this cool utility in my /bin directory : xmllint

You can use it to extract values from your xml files within your shell scripts

$ cat foo.xml
  <emp no="1">
  <emp no="2">
$ echo 'cat //emplist/emp[@no="1"]/ename/text()'|
  xmllint --shell foo.xml |
  sed -n 3p

I like this !

read without Enter

A small unix tip today.

Do you want to continue ?

If you are expecting “y” or “n” but do not want to enforce the user to type y[Enter] but simply y, you can use the -n option in bash.

Within a ksh script:

yorn=$(bash -c 'read -p "Do you want to continue ? " -n 1 ans;echo "$ans"')

Within bash:

read -n 1 yorn


Do you know cd ? I thought I did until this afternoon …

OK, let’s start some basic.

I create two directories

$ echo $SHELL
$ mkdir /tmp/foo
$ mkdir /tmp/bar

create a symlink /tmp/bar/baz pointing to /tmp/foo

$ ln -s /tmp/foo /tmp/bar/baz

create a file foo1 in foo

$ touch /tmp/foo/foo1

change to the symbolic link baz

$ cd /tmp/bar/baz

Ok, so far so good. Let’s check what is in ../foo

$ ls ../foo

From the symbolic baz, .. point to /tmp/foo. This is because ls and most command line utilities use the physical path.

To print the Logical [default] and Physical working directories, use pwd -L and pwd -P

$ pwd -L
$ pwd -P

to change directory relatively to the logical path, use cd -L … (default), for physical, use cd -P … !

$ pwd -L
$ cd -L ../foo
ksh: ../foo:  not found

Obviously /tmp/bar/foo does not exist

$ pwd -L
$ pwd -P
$ cd -P ../foo
$ pwd

Obviously /tmp/foo/../foo is /tmp/foo

So far so good, some of you may know that already.

Let’s bring some devil element in play

$ bash

Arghh!!! 脭么么么么么么么么么么么么么 r芒ge, 脭么么么么么么么么么么么么么 d茅sespoir, I switched to a non-working shell!

$ cd /tmp/bar/baz
$ cd -L ../foo
$ pwd -L

Even if I switched to a not working directory, bash cd -L weirdly decided to switch to the physical path instead of the logical path.

Let’s retry

$ cd /tmp/bar/baz
$ mkdir /tmp/bar/foo
$ cd -L ../foo
$ pwd

This time bash cd -L changed to the logical path. So if you use bash and cd, you cannot possibly know where you are landing without checking first if the directory exist !

BTW, I just discovered Digger HD , unrelated to this post of course …


Something I like in ksh is to change from /my/old/directory/path to /my/new/directory/path by typing cd old new. This does not work in bash

So I had to find a workaround 馃槈

$ ksh
$ cd /app/oracle/product/
$ cd 6 7
$ pwd 
$ bash
$ cd 7 6
bash: cd: 7: No such file or directory
$ c() { cd ${PWD/$1/$2}; }
$ c 7 6
$ pwd

chmod -R 777 .

This is one of the thing I hate to see, recursively changing everything to 777 馃懣

If you want to give read access to all, then 644 is enough for files and 755 for directories. If you want to give execution permission too, you could give 755 to executable files.

Also sometimes you have files with special permission like suid or guid bit (to run as the owner instead of to run as the run user), so it would be best to use relative (go+r) to make the file readable to group and others.

Therefore I prefer relative change. Still there is one thing I do not want, is making every file executable…

Ok, here it is

chmod -R a+rX .

note the big X :) we change recursively all files and dirs to be readable, and we set the executable flag ONLY IF the file is executable for the owner !

to ftp or to sftp

Ftp is seen as an old-time unsecure protocol. Many shops nowadays have switched or are switching to sftp. I will try to point out some differences :

Compatibility: none. the protocol is completly different. Multiple graphical clients however do support both mode. But the basic “ftp” client will not work with sftp.

Ascii mode: only in ftp. In sftp, it is always binary so there will be no conversion. Also no blocksize, recordlength or primary/secondary space for your OS/390 connections.

Interactive mode: similar. you enter your username and password, do cd, put and get. But to quit, by will not work in sftp 馃槈 Use quit or exit instead

Batch mode: different. Most probably you will end up setting a private/public key infrastructure for your ssh connection and use scp (secure copy). If you are using a ssh client like putty, it is possible to do something like pscp -l user -pw password server:file .

Security: sftp is secure, ftp is not.

Speed: ftp is fast, sftp is slow :( !

Oh NOOOOOOO!!!!! What’s the point is bringing something new if it is slower !!!

Ok, let’s try to download a 100m file:

$ time (echo "open dbsrv01
user oracle secret
get 100m"|ftp -n )

real    0m24.673s
user    0m0.030s
sys     0m0.016s
$ time scp -q oracle@dbsrv01:100m .

real    1m46.978s
user    0m0.108s
sys     0m0.202s

it is about 4x slower! Is there anything we could do about it?

Well, maybe :

$ time scp -q -o Compression=yes oracle@dbsrv01:100m .

real    0m18.634s
user    0m0.748s
sys     0m0.452s

ssh/scp/sftp have a compression mode. If you are transferring your large files across a slow network, this may be an interesting option to consider !

OpenSSH homepage :

How to reuse connection in shell

It is Friday, I wanted to give my readers some stuff for the week-end 馃槈

Imagine that piece of code :

countlines() {
  c=$(sqlplus -s /nolog <<EOF | grep -v Connected
connect scott/tiger
set feed off head off
select count(*) from $1;
  echo there are $c lines in $1

countlines EMP
countlines DEPT

I can run this

time ./script1
there are 14 lines in EMP
there are 4 lines in DEPT

real    0m0.46s
user    0m0.06s
sys     0m0.09s

Sounds like a regular shell script. How could we optimize the countlines function? Well, we could create the connection only once and use coprocess pipe (with |& that is pipe ampersand)

sqlplus -s /nolog |&

print -p "connect scott/tiger"

read -p line
if [ $line != Connected. ]
  exit 1

print -p "set feed off head off"

countlines() {
  print -p "select count(*) from $1;"
  read -p c
  echo "there is $c lines in $1"

countlines EMP
countlines DEPT

print -p disconnect

A two-ways pipe is opened with sqlplus. There is only one connect, and one disconnect.

Let’s check the performance :

$ time ./script2
there is 14 lines in EMP
there is 4 lines in DEPT

real    0m0.23s
user    0m0.00s
sys     0m0.01s

About twice as fast! Note the “Connected” output may not exist in recent version of sqlplus in silent mode. If you have a script that generates hundreds of connections, or which create a connection every 5 seconds or so, think about it :)

Enjoy your week-end