Posted on

Don’t call it test

There are quite a few names to avoid in your scripts. Even if there are not reserved-words, keep away !

I’ll start with test


cd $HOME/bin
vi test
  echo hello world
chmod +x test
./test
  hello world

The problem is that it may break your other scripts


$ ssh localhost test 1 = 2 && echo WHAT???
hello world
WHAT???

And it may break sooner or later, depending on your OS / version / default shell / default path / others.

There are quite a few filenames you should not use, like test, date, time, hostname, mail, view, touch, sort and make. The command type lists some of those as reserved word, shell builtin, tracked alias, shell keyword. But again it is not consistent over Unix flavors. 


$ uname -sr; type date
SunOS 5.10
date is /usr/bin/date
$ uname -sr; type date
Linux 2.6
date is a tracked alias for /bin/date

Your sysadmin may also alias things for colors and safety in the common profile: for instance vi, ls, rm. But if it annoys you, then use \vi instead of vi.

Posted on

Untrusted X11 forwarding

I wrote a while ago about my security concerns regarding


xhost +
xterm -display mypc:0

Way back then, I suggested ssh tunnel. SSH is pretty easy to set up, by enabling the X11Forwarding option.

In OpenSSH 3.8 release note, 2004, there was a new default .

ssh(1) now uses untrusted cookies for X11-Forwarding

In the man ssh_config page, it’s still documented as being the default

ForwardX11Trusted The default is ‘no’

But it actually isn’t on most *ix derivates, e.g. RedHat /etc/ssh/ssh_config

# If this option is set to yes then
# remote X11 clients will have full access
# to the original X11 display. As virtually
# no X11 client supports the untrusted
# mode correctly we set this to yes.
ForwardX11Trusted yes

Who is we?

Okay, let’s go back.

If you use the unsafest method, xhost + and xterm -display pc:0, then you grant everybody the right to manipulate X.

If you use trusted ssh, which is the _undocumented_ default in Linux, then you grant this right only to anyone with access to your authority, most probably located in the file $HOME/.Xauthority. So root and yourself, at least.

If you trust neither yourself nor root, you could restrict access to your resource, preventing one hacker from switching your mouse buttons or doing a screenshot. But this is probably going to prevent most of your applications from working. Also, it probably won’t work at all if you use putty, reflection and (virtually any?) other client tools.

If you want to force Trusted mode, use -Y or -o ForwardX11Trusted=yes.

If you want to force Untrusted mode, use -X and -o ForwardX11Trusted=no.

If you use only -X, it may transparently defaults to the more convenient but less secure -Y. Sometimes. At least on Linux OpenSSH. But if you use different Unix / SSH flavours, the -X may ends with an error message like connection to “localhost:10.0” refused by server. In that case, simply use -Y. Actually, always use -Y if you want Trusted.

Posted on

run sudo, ssh, password, su in simulated interactive mode

Some commands do not like non-interactive mode


$ passwd <<EOF
> oldpassword
> newpassword
> newpassword
> EOF
Changing password for user lsc.
Current password for lsc@example.com: passwd: Authentication token manipulation error
$ echo oraclepassword | su - oracle
standard in must be a tty
$ echo sudopassword | sudo su - oracle
[sudo] password for lsc:
sudo: sorry, you must have a tty to run sudo

But ok, if you really want to run those in a script, you have plenty of clean (no hack there) ways of doing this.

For instance, let’s use a screen called BAR.


$ xterm -e "screen -S BAR" &
[1]     31732

Now we have an interactive terminal. It could be redirected to a frame buffer device if no x server is started.

Not really a human device, but an interactive terminal.

Now let’s send stuff


$ CR="$(echo '\r')"
$ screen -S BAR -X stuff "sudo su - oracle$CR"
$ screen -S BAR -X stuff "sudopassword$CR"
$ screen -S BAR -X stuff "id > /tmp/xxx$CR"
$ screen -S BAR -X stuff "exit$CR"
$ screen -S BAR -X stuff "exit$CR"
[1] +  Done                    xterm -e "screen -S BAR" &
$ cat /tmp/xxx
uid=100(oracle) gid=100(dba) groups=100(dba)

Usual disclaimer: it is a bad security practice to hardcode your passwords in scripts. Use this only if you really understand security. Read man openssl about how to use openssl to encrypt your password. Ask your security friends before trying

Posted on

How to *really* send a script to the background

Let’s check this small script

– foo.sh


#!/bin/sh
echo foo.1:`date` | tee $HOME/tmp/foo.txt
sleep 3
echo foo.2:`date` | tee -a $HOME/tmp/foo.txt



$ $HOME/tmp/foo.sh
foo.1:Thu Nov 27 17:34:53 CET 2014
foo.2:Thu Nov 27 17:34:56 CET 2014

Very obvious, I write to the console, wait three seconds, then write to the console.

Ok, let’s take another script that would call this script in the background using &

– bar.sh


#!/bin/sh
echo bar.1:`date`
$HOME/tmp/foo.sh &
echo bar.2:`date`



$ $HOME/tmp/bar.sh
bar.1:Thu Nov 27 17:36:32 CET 2014
bar.2:Thu Nov 27 17:36:32 CET 2014
$ 
foo.1:Thu Nov 27 17:36:32 CET 2014
foo.2:Thu Nov 27 17:36:35 CET 2014

bar is printing the date, calling foo in the background, then printing the date, then it returns to you, and foo is still running.

BUT this is only in a relative background …

Let’s try this


$ time $HOME/tmp/bar.sh > /dev/null

real    0m0.01s
user    0m0.00s
sys     0m0.00s

So it takes no time to run bar you believe ?

Let’s try, for instance, over ssh (or cron or whatever)


$ time ssh localhost $HOME/tmp/bar.sh > /dev/null
real    0m3.81s
user    0m0.01s
sys     0m0.01s

running bar suddenly waits 3 seconds for foo to finish.

To be sure the script is sent to the farest background, you need to close the file descriptors, stdin, stdout, stderr

I rewrote it as

– baz.sh


#!/bin/sh
echo bar.1:`date`
$HOME/tmp/foo.sh <&- >&- 2>&- &
echo bar.2:`date`



$ time ssh localhost $HOME/tmp/baz.sh >/dev/null
real    0m0.44s
user    0m0.00s
sys     0m0.00s

Now the script baz is immediately finished and does not wait for foo to complete

Posted on

fun with cron

Today I find out that my scheduler was too busy to execute all jobs in my crontab !?


* * * * * (while :;do ssh example.com :; done)
59 23 19 06 * touch /tmp/bang

my while loop is going to produce so much hangs on the cron deamon that it may not be able to read the crontab once a minute. If it reads it at 23:58 and at 00:00, the 23:59 won’t be run.

This is actually the first time I see this behaviour. And -believe me- it’s annoying!

Posted on

Testing for (non-)empty string in shell

One way to test for (non-)empty string is to use test and -z (-n)


$ x=foo
$ test -z "$x"
$ echo $?
1

This is mostly seen with an if and [ -z … ] syntax


$ y=bar
$ if [ -n "$y" ];
  then echo non-empty; 
  fi
non-empty

Instead of a variable, it could be the output of a script.

Like


if [ -n "$(grep ORA- alertDB01.log)" ] 
then 
  echo there is an error in the alert log
else
  echo "fine :)"
fi

This will work for years until one day you get :


ksh: no space

Why that? This is the way the shell works. Your shell (here ksh on AIX) starts having errors as soon as your subshell (here the grep) is exhausting the space.


$ wc -l alertDB01.log
       2 alertDB01.log
$ if [ -n "$(grep ORA- alertDB01.log)" ]; 
  then echo non-empty; 
  else echo "fine :)"; 
  fi
non-empty
$ wc -l alertDB01.log
   75025 alertDB01.log
$ if [ -n "$(grep ORA- alertDB01.log)" ]; 
  then echo non-empty; 
  else echo "fine :)"; 
  fi
ksh: no space

You got a memory error, how the shell will react is random (core dump, errors, continue, crashes). It will just bug and you do not want this.

There is more than one to circumvent this. For instance you could use the return code of grep


$ if grep ORA- alertDB01.log >/dev/null;
  then echo non-empty;
  else echo "fine :)";
  fi
non-empty

Different shells (Bash / Bourne) and different OSs (Linux / AIX / HPUX) may react differently. If AIX crashed with a 50’000 lines, it may scale up to millions of lines in recent Linux’s – but still use trucks of memory

Posted on

hot to bypass requiretty in sudo

You can execute it a command without password from the commande line


$ sudo -l
User lsc may run the following commands on this host:
    (root) NOPASSWD: /usr/local/bin/helloworld
$ sudo /usr/local/bin/helloworld
Hello World!

Now you try to run it via cron and you get


sudo: sorry, you must have a tty to run sudo

The message is clear, you need a terminal.

Either you edit your sudoers files to disable requiretty, or you just get yourself a terminal.

Maybe you tried to assign a pseudo terminal with ssh -t, but you may get an error if ssh has no local tty


Pseudo-terminal will not be allocated because stdin is not a terminal.

Don’t despair, read man ssh


Multiple -t options force tty allocation, even if ssh has no local tty.

Let’s try


* * * * * ssh -t -t sudo /usr/local/bin/helloworld >> /tmp/txt

This should work, providing you configured ssh keys 🙂

Posted on

use cron to schedule a job only once

I wrote about not using DAY OF MONTH and DAY OF WEEK simultanously in how to cron

The correct method is to use
15 14 15 05 * /tmp/run-my-job

But… I wrote this five years ago. Hmmm ! Not that correct then since it would run every year 😉

Ok, periodically I check for jobs are scheduled to run a specific date only

$ crontab -l|awk '$1!~/#/&&$3*$4'
15 14 15 05 * /tmp/run-my-job

I have 9 more days to remove this before it runs for the fifth time 🙂

Posted on

return code and sqlplus

Calling a shell script from within sqlplus is buggy…

I have reported bug 3798918 in 10.1.0.2 (back in 2004) and bug 13349119 in 11.2.0.3 because some metalink guru closed 3798918 as not reproducible.

As written in return code, host does not return the correct code


SQL> host exit 7
SQL> def _RC
DEFINE _RC             = "0" (CHAR)

If you never use _RC, you may believe you are safe. But watch this :


SQL> get foo.sh list
  1  #!/bin/sh
  2  if /bin/false
  3  then
  4    echo this is wrong
  5* fi
SQL> host ./foo.sh
this is wrong

The return code not being set is not only affecting the _RC variable, but it is also affecting all subshells !

Note this is not reproducable with SQLPLUS /NOLOG


SQL> host false
SQL> def _rc
DEFINE _RC             = "1" (CHAR)
SQL> conn x/x
ERROR:
ORA-01017: invalid username/password; logon denied
SQL> host false
SQL> def _rc
DEFINE _RC             = "0" (CHAR)

After my (failed or successfull) tentative to connect as x/x, it is reproducible again

Posted on

How to quit crontab -e without overwritting cron

Imagine my crontab


* * * * * /usr/bin/date > /tmp/foo

I am writing the date to /tmp/foo every minute
$  cat /tmp/foo
Thu Jul  5 08:45:01 CEST 2012

Now I want to view my crontab in my EDITOR (vi).


$ crontab -e

I do not quit yet.

In the meantime, my colleague modify the crontab.


* * * * * /usr/bin/date > /tmp/bar

Later, I quit vi with :q!

O Surprise, the crontab is * * * * * /usr/bin/date > /tmp/foo again

According to the doc :

When you finish creating entries and exit the file, the crontab command
copies it into the /var/spool/cron/crontabs directory

Even if you did not make change, you overwrite the content of your crontab !

If you want to exit your crontab editor really without overwritting the crontab, you need to kill yourself.

Ex:


CTRL-Z
[2] + Stopped (SIGTSTP)        crontab -e
$ kill %2
$
[2] + Stopped (SIGTTOU)        crontab -e

Thanks to Colin comment, I realized I could not kill with kill, let’s kill with -9


$ kill -9 %2
$
[2] + Killed                   crontab -e

Posted on

xhost+ security hole part 2

Five years ago I wrote xhost+ is a huge security hole, I turned out red this morning when my neighbour sent me a smiley via X.

Do I really want everyone to have full access to my screen? No, I don’t. And I don’t do xhost+.

So why did it happen to me ???

I am using X-Window Attachmate aka Reflection X. And in this tool, according to the doc, the default X policy is unrestricted. This is in my opinion a huge flaw in the security design. Make sure you always change this to something more secure.

In Reflection X Manager Settings, Category Security, choose for instance User-based security and Prompt. Configuring X Cookies is probably more cumbersome.

Then when you or someone else will start an XTERM on your desktop, you will get a nice dialog box :

[Reflection X]
Client could not successfully authenticate itself to Reflection X server. Would you like Reflection X to connect to this client as an UNTRUSTED client ? Client originated from 192.168.0.1 (RX1303)
[Yes][No]

Ok, I have to click one more button, but at least I can deny access to my screen 🙂

Posted on

[alert] AIX Posix Timezone issue

Maybe you did get or you will get an issue with the date command in AIX.

expected behavior, Linux


$ TZ=NZST-12NZDT,M10.1.0/2,M3.3.0/3 date
Sat Mar 17 00:14:54 NZDT 2012
$ TZ=Pacific/Auckland date
Sat Mar 17 00:14:58 NZDT 2012

unexpected behavior, AIX


$ TZ=Pacific/Auckland date
Sat Mar 17 00:15:50 GMT+13:00 2012
$ TZ=NZST-12NZDT,M10.1.0/2,M3.3.0/3 date
Fri Mar 16 23:15:52 NZST 2012

The consequence : date, and all other unix commands like ls, who, ps that display the date in human readable format, and all programs that use ctime are affected


$ TZ=NZST-12NZDT,M10.1.0/2,M3.3.0/3 perl -e 'use POSIX;print ctime(time)'
Fri Mar 16 23:19:51 2012

Reference and link to the fixes : www-01.ibm.com/support/docview.wss?uid=isg3T1013017

Posted on

delete unused shared memory segments from an Oracle instance

Once upon a time, a dba issues some kill -9 to clean up dying database processes. Or the database instance crashes. This will left some shared memory segments. Note 68281.1 describe how to remove them on a server with multiple databases.

First, list the ipc process


$ ipcs
IPC status from /dev/mem as of Mon Nov 14 11:28:58 CET 2011
T        ID     KEY        MODE       OWNER    GROUP
Message Queues:
q         0 0x4107001c -Rrw-rw----     root   printq

Shared Memory:
m         0 0x7800006f --rw-rw-rw- itmuser1 itmusers
m         1 0x78000070 --rw-rw-rw- itmuser1 itmusers
m   5242882 0x41d2ba80 --rw-r-----   oracle      dba
m  99614723 0xb0d4d164 --rw-rw----   oracle      dba
m  12582917 0xb84cbc28 --rw-rw----   oracle      dba
m  79691782 0x1058873f --rw-------   oracle      dba
m 638582792 0x78000382 --rw-rw-rw-     root   system
m 218103817 0x780003b7 --rw-rw-rw-     root   system
Semaphores:
s         1 0x6202c477 --ra-r--r--     root   system
s   6291461 0x0102c2d8 --ra-------     root   system
s         6 0xa100004b --ra-ra-ra-     root   system

Get a list of the running databases


$ ps -ef | grep pmon | grep -v grep
  oracle  483334       1   1   Aug 16      -  6:46 ora_pmon_db03
  oracle 1253476       1   0   Oct 31      -  2:00 ora_pmon_db01
  oracle 2298042       1   0   Sep 05      - 11:07 ora_pmon_db02

Then, for each database, get the ipc information
$ export ORACLE_SID=db01
$ sqlplus / as sysdba
SQL> oradebug setmypid
Statement processed.
SQL> oradebug ipc
Information written to trace file.
SQL> oradebug tracefile_name
/u01/app/oracle/admin/db01/udump/db01_ora_2625574.trc
$ awk '/Shared Memory:/{getline;getline;print}' /u01/app/oracle/admin/db01/udump/db01_ora_2625574.trc
5242882         0x41d2ba80
$ export ORACLE_SID=db02
$ sqlplus / as sysdba
SQL> oradebug setmypid
Statement processed.
SQL> oradebug ipc
Information written to trace file.
SQL> oradebug tracefile_name
/u01/app/oracle/diag/rdbms/db02a/db02/trace/db02_ora_2441408.trc
$ awk '/Shared Memory:/{getline;getline;print}' /u01/app/oracle/diag/rdbms/db02a/db02/trace/db02_ora_2441408.trc
99614723        0xb0d4d164
$ export ORACLE_SID=db03
$ sqlplus / as sysdba
SQL> oradebug setmypid
Statement processed.
SQL> oradebug ipc
Information written to trace file.
SQL> oradebug tracefile_name
/u01/app/oracle/diag/rdbms/db03b/db03/trace/db03_ora_2617416.trc
$ awk '/Shared Memory:/{getline;getline;print}' /u01/app/oracle/diag/rdbms/db03b/db03/trace/db03_ora_2617416.trc
12582917        0xb84cbc28

Compare it with the first list, and if you are absolutely sure to do what you are doing, remove the oracle segments that are not assigned to any database instance with ipcrm. If possible try first to figure out where they come from and do a shutdown abort of the not-correctly-stopped database.

Ok, with ipcrm 


$ ipcrm -m 79691782

I removed the segment that apparently does not relate to any running instance

This could help you if you are really forced to remove some shared memory segments and you cannot afford shutting down other databases.

Posted on

Check if it a program is already running in Unix

There is more than one way to do it, the safe is probably to check if /home/lsc/OH_YES_I_AM_RUNNING exists and believe it. This is called the file.PID method and is widely used (Apache used to use it since a long long time). It needs file. It needs cleanup if you reboot your server in the middle of something (and surely you do not want to delete old pid files yourself)

Ok, often you see this :


ps -ef | grep program

There you list all processes and check the lines that contain program. So some does a vi program or anything worse (emacs?), you will get more rows than needed.

Maybe it is fine to run program with different arguments, this must be decided.

Well, take a simple test case :
x1.sh and x2.sh :

#!/bin/ksh
while :
do
  date  > /dev/null
done

let’s try to use ps


$ nohup ./x1.sh &
$ nohup ./x2.sh &
$ jobs
[2] +  Running                 nohup ./x2.sh &
[1] -  Running                 nohup ./x1.sh &
$ ps -ef | egrep 'x[12]'
  u22  9240796  6226164  30 14:56:52  pts/2  0:00 /bin/ksh ./x2.sh
  u22 20840608  6226164  31 14:56:48  pts/2  0:01 /bin/ksh ./x1.sh

So fine so good, I see I have one instance of each program.

Let’s try to see if the results are consistent over time :

 $ n=9999;while :
  do 
    ps -ef | 
      egrep 'x[12].sh'>f
    if [ $(wc -l <f) != $n ]
    then 
      n=$(wc -l <f)
      echo
      date
      cat f
      echo "==> $n"
    fi
  done

Fri Oct 28 15:01:01 CEST 2011
  u22  9240796  6226164  32 14:56:52  pts/2  0:14 /bin/ksh ./x2.sh
  u22 20840608  6226164  28 14:56:48  pts/2  0:14 /bin/ksh ./x1.sh
==>        2

Fri Oct 28 15:01:08 CEST 2011
  u22  9240796  6226164  50 14:56:52  pts/2  0:14 /bin/ksh ./x2.sh
==>        1

Fri Oct 28 15:01:09 CEST 2011
  u22  9240796  6226164  52 14:56:52  pts/2  0:14 /bin/ksh ./x2.sh
  u22 20840608  6226164  53 14:56:48  pts/2  0:15 /bin/ksh ./x1.sh
==>        2

Fri Oct 28 15:01:17 CEST 2011
  u22  9240796  6226164  40 14:56:52  pts/2  0:15 /bin/ksh ./x2.sh
  u22 10944520  9240796   0 15:01:17  pts/2  0:00 /bin/ksh ./x2.sh
  u22 20840608  6226164  31 14:56:48  pts/2  0:16 /bin/ksh ./x1.sh
==>        3

the fact that a subshell (pid 10944520 ) of x2 appear is not a problem for me. I have much more of a problem at 15:01:08 where x1 disappeared !

Conclusion : you cannot trust ps

Posted on

shell and list of files

How do you loop thru a list of files?

For instance you want to archive than delete all pdf documents in the current directory :

Bad practice :


tar cvf f.tar *.pdf
rm *.pdf

There are multiple issue with the command above

1) new files could come during the tar, so the rm will delete files that have not been archived


filelist=$(ls *.pdf)
tar cvf f.tar $filelist
rm $filelist

2) if there is no file, tar and rm will return an error


filelist=$(ls|grep '\.pdf')
if [ -n "$filelist" ]
then
  tar cvf f.tar $filelist
  rm $filelist
fi

3) this will not work for long list (above 100k documents)


filelist=/tmp/filelist.$(date "+%Y%m%d%H%M%S").$$.$RANDOM
ls|grep '\.pdf' > $filelist
if [ -s "$filelist" ]
then
  tar cvfL f.tar $filelist
  for f in $(<filelist)
  do
    rm $f
  done
fi

As you see, this require special handling. tar for instance use the -L option to accept a list of files, rm could delete files one by one (or in bunches with xargs -L).

This 100’000 limit (the limit may vary for your shell/os) is something that often gets forgotten.

Typical error that could occur are


ksh: no space
bash: Arg list too long

Posted on

pstree in AIX

For those who do not want to download some linuxlike freeware on your aix box, use ps -T 🙂


ps -fT 2412672
     UID     PID    PPID   C    STIME    TTY  TIME CMD
  oracle 2412672       1   0   Sep 05      -  0:00 /u01/app/oracle/product/OAS
  oracle  630956 2412672   0   Sep 05      -  6:11     \--/u01/app/oracle/prod
  oracle 1347672  630956   0   Sep 05      - 15:32        |\--/u01/app/oracle/
  oracle 1437836  630956   0   Sep 05      -  1:02        |\--/u01/app/oracle/
  oracle  880820 1437836   0   Sep 05      -  0:32        |   |\--/u01/app/ora
  oracle 1036532 1437836   0   Sep 05      -  0:00        |   |\--/u01/app/ora
  oracle 1134796 1437836   0   Sep 05      -  0:01        |   |\--/u01/app/ora
  oracle 1343712 1437836   0   Sep 05      -  0:33        |   |\--/u01/app/ora
  oracle 1368166 1437836   0   Sep 05      -  1:11        |   |\--/u01/app/ora
  oracle 1384684 1437836   0   Sep 05      -  0:33        |   |\--/u01/app/ora
  oracle 1392862 1437836   0   Sep 05      -  0:32        |   |\--/u01/app/ora
  oracle 1396898 1437836   0   Sep 05      -  0:33        |   |\--/u01/app/ora
  oracle 1482978 1437836   0   Sep 05      -  0:32        |   |\--/u01/app/ora
  oracle 1527890 1437836   0   Sep 05      -  0:00        |   |\--/u01/app/ora
  oracle 1781798 1437836   0   Sep 05      -  0:32        |   |\--/u01/app/ora
  oracle 2195474 1437836   0   Sep 26      -  0:13        |    \--/u01/app/ora
  oracle 1626296  630956   0   Sep 05      - 13:49         \--/u01/app/oracle/

Posted on

Generate network graph from command line

I recently wrote on gnuplot, today I tried another command line utility to generate graphs, graphviz, version 2.24.0 on AIX5L.

Pretty straightforward syntax :

(
  echo "digraph Emp {"
    sqlplus -s -L scott/tiger << EOF
      set pages 0 lin 120 hea off feed off
      select 
        ename ||'->'|| 
        (select ename from emp where empno=e.mgr) || ';' 
      from emp e where mgr is not null;
EOF
  echo "}"
)| neato -Tpng | uuencode Emp.png | mailx laurentschneider@example.com

(or neato -Tpng -o Emp.png to save locally, or -Tps|lp, etc…)

Posted on

send graph per mail from sqlplus

How to send a graph with a single command from your database to your mail in Unix?

I tried this (gnuplot is available for Solaris, AIX and most Unix derivates) :

echo '
  set hea off pages 0 feed off
  prom set title "salaries of EMP"
  prom unset key
  prom unset xtics
  prom unset xlabel
  prom set term png
  prom 
  prom plot "-" with circle 
  select row_number() over (order by sal),sal from emp;
  prom e
  prom quit
' | 
  sqlplus -s scott/tiger | 
    gnuplot | 
      uuencode emp.png | 
        mailx email@example.com

The picture is sent as attachment.

If you want to embed your png in an HTML mail, use sendmail 

/usr/sbin/sendmail -t <<EOF
To: email@example.com
From: email@example.com
Subject: EMP
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="FILEBOUNDARY"

--FILEBOUNDARY
Content-Type: multipart/alternative; boundary="MSGBOUNDARY"

--MSGBOUNDARY
Content-Type: text/html
Content-Disposition: inline

<html><body>Below a graph...<br/>
<img src="cid:png"/><br/></body></html>
--MSGBOUNDARY--
--FILEBOUNDARY
Content-Type: image/png
Content-Disposition: inline; filename="png.png"
Content-Transfer-Encoding: base64
Content-Id: <png>

iVBORw0KGgoAAAANSUhEUgAAAoAAAAHgCAMAAAACDyzWAAABIFBMVEX///8AAACgoKD/AAAAwAAA
gP/AAP8A7u7AQADIyABBaeH/wCAAgEDAgP8wYICLAABAgAD/gP9//9SlKir//wBA4NAAAAAaGhoz
MzNNTU1mZmZ/f3+ZmZmzs7PAwMDMzMzl5eX////wMjKQ7pCt2ObwVfDg///u3YL/tsGv7u7/1wAA
/wAAZAAA/38iiyIui1cAAP8AAIsZGXAAAIAAAM2HzusA////AP8AztH/FJP/f1DwgID/RQD6gHLp
lnrw5oy9t2u4hgv19dyggCD/pQDugu6UANPdoN2QUEBVay+AFACAFBSAQBSAQICAYMCAYP+AgAD/
gED/oED/oGD/oHD/wMD//4D//8DNt57w//Cgts3B/8HNwLB8/0Cg/yC+vr7MAY9LAAAKJElEQVR4
nO3d0XaiSBSGUV15/3ee7unEoKLWwYIfrL0vJhNDmLn4FgQoPacTAAAAAAAAAAAAbOl8Ps++uvAX
oWhhgH+2+N3o/L/T/9+f/33X8f+Qj7awlatf+/7m75fvMBXII9/Hp5/D1M8//2Vz+dnMlje/frXL
y48EyHOTPK5LmqZz+cddT5cXb6M8TwJ0Cuax6eHqND0Cnq9ivPR0f7CbffF0VSw8dhXR7cHussXN
YW/ud29+LkBeuz2xPg/w0SlYgCw1ubQ4T06zk1Pw9MXfX7j99Ztd/qYqQAAAAAAAAACGMP983cpJ
NjL7rgVrx9nKZHGvANneb3Lnuy+XTaCsFuF5LsKfH3ctnuP5Kr38TznAx6dgAY5uzQBnyxMgV2ZT
e9pfezXfp+vZL8Vd8anWDPA1AQ5PgGTNxPa8PwHSkwDJusvtRX8CpKuvr+ff3xEgfU2Te5mfAOnv
6193Xw35CZBVfLXVdxIgYQIkSoBECZAoARIlQKIESJQAiRIgUQIkSoBECZAoARIlQKIESJQAiRIg
UQIkSoBECZAoARIlQKIESJQAiRIgUQIkSoBECZAoARIlQKIESFR9VNf5fmprfVfwT6GayWQ4k5Lo
pL2a8/nnCChAuimM6roeEjcZmHnZojZ8k9FVipkc7IxrpZ/2AC+1mpZJP+WrYAHSUzFA41rpy41o
ogRIlACJEiBRAiRKgEQJkCgBEiVAogRIlACJEiBRAiRKgEQJkCgBEiVAogRIlACJEiBRAiRKgEQJ
kCgBEiVAogRIlACJEiBRAiRKgEQJkCgBEiVAogRIlACJEiBRAiRKgEQJkCgBEmVQDVEL5gUb1UU/
5XnBAqSn+rzg892XyxbmBVOxYF7wfX2OgLyjOC/YKZi+zAsmym0YotyIJkqARAmQKAESJUCiBEiU
AIkSIFECJEqARAmQKAESJUCiBEiUAIkSIFECJEqARAmQKAESJUCiBEiUAIkSIFECJEqARAmQKAES
JUCiBEiUAIkSIFECJEqARAmQKAESJUCiBEiUAIkqzIr7GRNyPy6kuCu4KM+Ku/3OpCTeUR7Vdf6p
ToB0UD4F/w5uNa6V91SL+T3/GtdKJ0sDdAqmiwUXIQKkn+LfgMa10pcb0UQJkCgBEiVAogRIlACJ
EiBRAiRKgEQJkCgBEiVAogRIlACJEiBRAiRKgEQJkCgBEiVAogRIlACJEiBRAiRKgEQJkCgBEiVA
ogRIlACJEiBRAuR0+vqj53YFAhzed1Qv22rdrkaAg5vm9Cyt1u2qBPjBXp8xb3/+aPvW7eoE+Kma
zph3P3sUYON2dQL8TG1nzJkfzG7but0CAvxEzWfWppcESFHjGfOt2DoVaE7IB3qroZ0GaFLSgXxi
gP9vK8BDePPPuMVXxovUx7VeT8o0rnV/DhPggnGt9/U5Au7OYQL8yyn44zT3cqgAXYQcxkb3/NyG
GdfzZ7wfGuCmu+Kxl894Bch6Wp7xbnLPz6O4EbU94xUgK2m7QC2cMS3HoqD/AcuCVArWOGNakk+r
N863T1dFe1MSTQoBFs+Y3pZJg8rtuLXOmN0J8DCKj2TXOWN2J8DDqK8JWOGM2Z0AD2PVRSkxAjwM
AW63K+asuSYgRoDHIcDNdsUcAW62K2atuCglRoAHIsCtdsW89VZFxQjwSNZbFRUjwGM5zDPeVgI8
moM8420lwAM6wjPeVgIkSoBECZAoARIlQKIESJQAiRIgUQIkSoBECZAoARIlQKIEuB+ftMilmQB3
4sOW+TUT4C583ELnZgLcgQ98q0ez8rDC36+XF6u74sYHvtmt2YJRXdffGdX1tk98u2+zcoA/U1sF
2I0AW7edTik0L7iXj/zIlya1Yi7bmhfc17gB/lW4CJn8m1NwRwJs2m7u6kOAHXzmB582aw9w7v6L
2zDvE+AOdzUSAe5wVyMR4A53NRQXIfvb1VAEuL9dDUWA+9vVWDyK292uxiLA3e1qMJZj7W1Xg7Eg
dW+7Go4l+fva1YC8KWlPuxqTt2XuZlcMQ4BECZAoARIlQKIESJQAiRIgUQIkSoBECZAoARIlQKIE
SJQAiRIgUQIkSoBECZAoARIlQKIESJQAiRIgUQIkSoBECZCo4rhWc0Loa8G4VpOS6EeARNXHtZ7v
vlx+bFwrFQvGtd7X5wjIO4rjWp2C6cu4VqKMayXKjWiiBEiUAIkSIFECJEqARAmQKAESJUCiBEiU
AIkSIFECJEqARAmQKAESJUCiBEiUAIkSIFECJEqARAlwfV9/pP8fdkuAK/uOT4MPCHBV0+wkOEeA
K7pNToL3BLiiu94EeEeA65nJTYG3BLgeATYQ4GpmY1PgDQGuRoAtBLjcixvMAmwhwIVe3mB+8LoC
rwlwkYYbzAJsIsAFmm4wC7CJABdousEswCYCrGu8v+cipIUA6wTYkQDLWsMSYIvKqK7To3Eh1V0d
W3NYHsU1KIzq+lfc9zcDT0oSYE/Nk5ImR8CxAyxc3VqO9Vr5FDwztfWywRDjWisBWpD6VHVc6++/
jDyutXR/z5L8V5YG6BTc9rI3Jb2w+BQ8bID12yvelvmEca1l7u/15EZ0mQB7EmCd+3sdCbBOgB0J
8M7rawY3mPsR4LWmuyZuMPcjwKnm+8ZuMPciwF+lA5sbzH0I8Ff1Tzs3mDs4foDNGbza0MVtwsED
bD4RNmwowIRDB9j1msEDjogDB9h8zdC2oQAjjhzgyxdKGwow4rgBNv/J1rah95FnCPDp7wpwbYcN
8J33ps29KMAMAT7bTICrE2B5h/R01ACbD1jvbSjAtQnw6YYCXJsAiy/RlwCLL9HXfgPs9Rn0bzz2
0N/6dhrg68UrawRopf32dhlg18UrlYsLK+03t8MAGw9EnR/FXf5jVtpvao8Bvnzh0asdAjxZab+t
/QW4PKz3lmMRceQA+y5IJWJ3Aa5yzeDiYrcOHWDhmsHFxU4dPMBT4ZrBxcUe7S1Ay/IGI0CiBEiU
AInaPMAFn9Civw+2bYDLPqFFgB9sywCbbgcLcCzbBdj6QMzC5KFUR3UtH1TTuiRAgENZMC942aiu
9q4sXhlJeV7wBgFavDKQ+rDC61GtrfOCS9cWFq+MYcG84Pv6Go+AxYtbi1dGsdW41vrdFYtXhrBR
gJ6wMW+jecECZN5GN6IFyDwBErXVoziPeJnVM8AunyXEWLoeAXt8lhBj6X0Kriyz0h8r/A347mcJ
MZQVLkIscqHdGlfBb36UCyPZLkCLXJixyn3A9z/KhVFsG+DJIheubR4gTK3zKE6BNBIgUQIkSoBE
uQghSoBECZCoLR/FwR0BErXdciyYsdmCVJiz2ZJ8mLPVm5Jg1lZvy4RZe5sTwmAESJQAiRIgUQIk
SoBECZAoARIlQKIESJQAiRIgUcVq5seFLNoVnBYEePlSnJYJc5YcAQVINwuOgE/HtULNgmYfjGuF
TTyalgmrezauFdb3fda+/gIAAAAAAAC0+Q9+0RhW4FaMYwAAAABJRU5ErkJggg==
--FILEBOUNDARY--
EOF

Sendmail syntax and more tips on : Sending Emails With Sendmail – Part 3

Posted on

vi large files

Once upon a time a colleague asked me if there is a better editor than vi installed on my db server. Well, I was not really about arguing the benefit of ed (less memory usage, no useless error message). But one advantage of ed was (I believed) the ability to read large files 


$ vi alert_DB01.log
Out of memory saving lines for undo - try using ed
:ed
"alert_DB01.log" 612637 lines, 20458891 characters
:612636
Thu Aug 25 16:08:20 2011
LNS: Standby redo logfile selected for thread 1 sequence 6898 for destination LOG_ARCHIVE_DEST_2
:q

of course most of readers are using some clicky fancy tools with colors and/or unlimited undo.

Well, for those who use vi on AIX, check this


$ vi -y 9999999 alert_DB01.log
Tue Oct 26 13:59:12 2010
Starting ORACLE instance (normal)
sskgpgetexecname failed to get name
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
IMODE=BR
ILAT =27
LICENSE_MAX_USERS = 0
"alert_DB01.log" 612662 lines, 20459538 characters

Oh yes !!!

Posted on

scp tuning

I twitted yesterday :

laurentsch
copying 1TB over ssh sucks. How do you fastcopy in Unix without installing Software and without root privilege?

I got plenty of expert answers. I have not gone to far in recompile ssh and I did not try plain ftp.

Ok, let’s try first to transfer 10 files of 100M from srv001 to srv002 with scp :

time scp 100M* srv002:
100M1    100%   95MB   4.5MB/s   00:21
100M10   100%   95MB   6.4MB/s   00:15
100M2    100%   95MB   6.0MB/s   00:16
100M3    100%   95MB   4.2MB/s   00:23
100M4    100%   95MB   3.4MB/s   00:28
100M5    100%   95MB   4.2MB/s   00:23
100M6    100%   95MB   6.4MB/s   00:15
100M7    100%   95MB   6.8MB/s   00:14
100M8    100%   95MB   6.8MB/s   00:14
100M9    100%   95MB   6.4MB/s   00:15

real    3m4.50s
user    0m27.07s
sys     0m21.56s

more than 3 minutes for 1G.

I got hints about the buffer size, about SFTP, about the cipher algorythm, and about parallelizing. I did not install new software and I have a pretty old openssh client (3.8). Thanks to all my contributors tmuth, Ik_zelf, TanelPoder, fritshoogland, jcnars, aejes, surachart, and the ones the will answer after the writting of this blog post…

Ok, let’s try a faster algorythm, with sftp (instead of scp), a higher buffer and in parallel

$ cat batch.ksh
echo "progress\nput 100M1" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M2" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M3" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M4" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M5" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M6" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M7" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M8" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M9" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
echo "progress\nput 100M10" | sftp -B 260000 -o Ciphers=arcfour -R 512 srv002&
wait
$ time batch.ksh
real    0m19.07s
user    0m12.08s
sys     0m5.86s

This is a 1000% speed enhancement 🙂

Posted on

What is the current setting of NLS_LANG in sqlplus?

I just learnt a neat trick from Oracle Support.

How do you see the current value of NLS_LANG in SQLPLUS ?

HOST is not the right answer.

E.g.:
Unix:


SQL> host echo $NLS_LANG
AMERICAN_SWITZERLAND

Windows:

SQL> HOST ECHO %NLS_LANG%
%NLS_LANG%

The correct setting is revealed by @.[%NLS_LANG%]
E.g.:
Unix:


SQL> @.[$NLS_LANG]
SP2-0310: unable to open file ".[AMERICAN_AMERICA.WE8ISO8859P1]"

Windows:

SQL>  @.[%NLS_LANG%]
SP2-0310: unable to open file ".[AMERICAN_AMERICA.WE8ISO8859P1]"

It could well be that both return the same answer, but not necessarly, as shown above.

The unix discrepancy is related to the subshell created by HOST. The subshell may read some .profile and overwrite the value of NLS_LANG

In Windows, the NLS_LANG setting may be set by sqlplus according to some registry entries

Posted on

Time offset in Unix

What is the time offset of the current date in Unix?

perl -e '
  $t=time;
  @l=localtime($t);
  @g=gmtime($t);
  $d=$l[2]-$g[2]+($l[1]-$g[1])/60;
  $gd=$g[3]+$g[4]*31+$g[5]*365;
  $ld=$l[3]+$l[4]*31+$l[5]*365;
  if($gd<$ld){$d+=24};
  if($gd>$ld){$d-=24}
print ($d."\n")'
2

Am I in summer (DST)?

perl -e 'if((localtime)[8]){print"yes"}else{print "no"}'
yes

Posted on

to cvs or to subversion

First surprise, after migration, the size of my subversion folder is double the size of my cvs folder. With a bunch of 2Gb disks shared amoung dozens of unix persons, and regular reminders the current usage reached 100%, you will feel the pain of having each developers doublesizing its home directory…
The reason is a .svn/test-base directory containing a duplicate of your local copy.
The benefit there is that it reduces network usage, for instance when diffing with the working version.

Second surprise, not sure if it is a generic issue or related to the one above, a full checkout take ages 🙁 .

svn and cvs command line interfaces almost have the same options, a bit like vi and vim -vi is my favorite- but the “tagging” massively differs. The is no cvs tag command in subversion, you use a different syntax.
cvs :


$ cd /home/lsc/CVS/proj1/source/pkg
$ cvs tag MYTAG helloworld.pkb helloworld.pks

now enjoy the power of subversion! [updated as I found the –parents option]

$ cd /home/lsc/SVN/proj1/source/pkg
$ svn copy --parents helloworld.pkb http://myserver.domain.com/svn/REPOS1/proj1/tags/MYTAG/proj1/source/pkg -m "my first svn tag"
$ svn copy --parents helloworld.pks http://myserver.domain.com/svn/REPOS1/proj1/tags/MYTAG/proj1/source/pkg -m "my first svn tag"

I always loved the cvs rename command

$ cvs rename pkg1.pkb pkg2.pkb
cvs [rename aborted]: Remote server does not support rename

Joke apart, in CVS I used to physically logon to the server and manually move the pkg1.pkb,v to pkg2.pkb,v
Renaming works in subversion :

$ svn move pkg1.pkb pkg2.pkb
A         pkg2.pkb
D         pkg1.pkb

Update: One more annoyance, you cannot checkout a single file 🙁
subversion faq

Posted on

extract xml from the command line

I just discovered this morning this cool utility in my /bin directory : xmllint

You can use it to extract values from your xml files within your shell scripts


$ cat foo.xml
<emplist>
  <emp no="1">
    <ename>John</ename>
  </emp>
  <emp no="2">
    <ename>Jack</ename>
  </emp>
</emplist>
$ echo 'cat //emplist/emp[@no="1"]/ename/text()'|
  xmllint --shell foo.xml |
  sed -n 3p
John

I like this !

Posted on

read without Enter

A small unix tip today.

Do you want to continue ?

If you are expecting “y” or “n” but do not want to enforce the user to type y[Enter] but simply y, you can use the -n option in bash.

Within a ksh script:


yorn=$(bash -c 'read -p "Do you want to continue ? " -n 1 ans;echo "$ans"')

Within bash:


read -n 1 yorn

Posted on

cd

Do you know cd ? I thought I did until this afternoon …

OK, let’s start some basic.

I create two directories

$ echo $SHELL
/bin/ksh
$ mkdir /tmp/foo
$ mkdir /tmp/bar

create a symlink /tmp/bar/baz pointing to /tmp/foo

$ ln -s /tmp/foo /tmp/bar/baz

create a file foo1 in foo

$ touch /tmp/foo/foo1

change to the symbolic link baz

$ cd /tmp/bar/baz

Ok, so far so good. Let’s check what is in ../foo

$ ls ../foo
foo1

From the symbolic baz, .. point to /tmp/foo. This is because ls and most command line utilities use the physical path.

To print the Logical [default] and Physical working directories, use pwd -L and pwd -P

$ pwd -L
/tmp/bar/baz
$ pwd -P
/tmp/foo

to change directory relatively to the logical path, use cd -L … (default), for physical, use cd -P … !

$ pwd -L
/tmp/bar/baz
$ cd -L ../foo
ksh: ../foo:  not found

Obviously /tmp/bar/foo does not exist

$ pwd -L
/tmp/bar/baz
$ pwd -P
/tmp/foo
$ cd -P ../foo
$ pwd
/tmp/foo

Obviously /tmp/foo/../foo is /tmp/foo

So far so good, some of you may know that already.

Let’s bring some devil element in play

$ bash

Arghh!!! Ôôôôôôôôôôôôôô râge, Ôôôôôôôôôôôôôô désespoir, I switched to a non-working shell!

$ cd /tmp/bar/baz
$ cd -L ../foo
$ pwd -L
/tmp/foo

Even if I switched to a not working directory, bash cd -L weirdly decided to switch to the physical path instead of the logical path.

Let’s retry

$ cd /tmp/bar/baz
$ mkdir /tmp/bar/foo
$ cd -L ../foo
$ pwd
/tmp/bar/foo

This time bash cd -L changed to the logical path. So if you use bash and cd, you cannot possibly know where you are landing without checking first if the directory exist !

BTW, I just discovered Digger HD , unrelated to this post of course …

Posted on

cd OLD NEW

Something I like in ksh is to change from /my/old/directory/path to /my/new/directory/path by typing cd old new. This does not work in bash

So I had to find a workaround 😉


$ ksh
$ cd /app/oracle/product/11.1.0.6/database/rdbms/admin
$ cd 6 7
$ pwd 
/app/oracle/product/11.1.0.7/database/rdbms/admin
$ bash
$ cd 7 6
bash: cd: 7: No such file or directory
$ c() { cd ${PWD/$1/$2}; }
$ c 7 6
$ pwd
/app/oracle/product/11.1.0.6/database/rdbms/admin

Posted on

chmod -R 777 .

This is one of the thing I hate to see, recursively changing everything to 777 👿

If you want to give read access to all, then 644 is enough for files and 755 for directories. If you want to give execution permission too, you could give 755 to executable files.

Also sometimes you have files with special permission like suid or guid bit (to run as the owner instead of to run as the run user), so it would be best to use relative (go+r) to make the file readable to group and others.

Therefore I prefer relative change. Still there is one thing I do not want, is making every file executable…

Ok, here it is


chmod -R a+rX .

note the big X 🙂 we change recursively all files and dirs to be readable, and we set the executable flag ONLY IF the file is executable for the owner !

Posted on

to ftp or to sftp

Ftp is seen as an old-time unsecure protocol. Many shops nowadays have switched or are switching to sftp. I will try to point out some differences :

Compatibility: none. the protocol is completly different. Multiple graphical clients however do support both mode. But the basic “ftp” client will not work with sftp.

Ascii mode: only in ftp. In sftp, it is always binary so there will be no conversion. Also no blocksize, recordlength or primary/secondary space for your OS/390 connections.

Interactive mode: similar. you enter your username and password, do cd, put and get. But to quit, by will not work in sftp 😉 Use quit or exit instead

Batch mode: different. Most probably you will end up setting a private/public key infrastructure for your ssh connection and use scp (secure copy). If you are using a ssh client like putty, it is possible to do something like pscp -l user -pw password server:file .

Security: sftp is secure, ftp is not.

Speed: ftp is fast, sftp is slow 🙁 !

Oh NOOOOOOO!!!!! What’s the point is bringing something new if it is slower !!!

Ok, let’s try to download a 100m file:

$ time (echo "open dbsrv01
user oracle secret
bin
get 100m"|ftp -n )

real    0m24.673s
user    0m0.030s
sys     0m0.016s
$ time scp -q oracle@dbsrv01:100m .

real    1m46.978s
user    0m0.108s
sys     0m0.202s

it is about 4x slower! Is there anything we could do about it?

Well, maybe : 

$ time scp -q -o Compression=yes oracle@dbsrv01:100m .

real    0m18.634s
user    0m0.748s
sys     0m0.452s

ssh/scp/sftp have a compression mode. If you are transferring your large files across a slow network, this may be an interesting option to consider !

OpenSSH homepage : http://www.openssh.org

Posted on

How to reuse connection in shell

It is Friday, I wanted to give my readers some stuff for the week-end 😉

Imagine that piece of code :

countlines() {
  c=$(sqlplus -s /nolog <<EOF | grep -v Connected
connect scott/tiger
set feed off head off
select count(*) from $1;
EOF
)
  echo there are $c lines in $1
}

countlines EMP
countlines DEPT

I can run this 

time ./script1
there are 14 lines in EMP
there are 4 lines in DEPT

real    0m0.46s
user    0m0.06s
sys     0m0.09s

Sounds like a regular shell script. How could we optimize the countlines function? Well, we could create the connection only once and use coprocess pipe (with |& that is pipe ampersand)

sqlplus -s /nolog |&

print -p "connect scott/tiger"

read -p line
if [ $line != Connected. ]
then
  exit 1
fi

print -p "set feed off head off"

countlines() {
  print -p "select count(*) from $1;"
  read -p c
  echo "there is $c lines in $1"
}

countlines EMP
countlines DEPT

print -p disconnect

A two-ways pipe is opened with sqlplus. There is only one connect, and one disconnect.

Let’s check the performance :

$ time ./script2
there is 14 lines in EMP
there is 4 lines in DEPT

real    0m0.23s
user    0m0.00s
sys     0m0.01s

About twice as fast! Note the “Connected” output may not exist in recent version of sqlplus in silent mode. If you have a script that generates hundreds of connections, or which create a connection every 5 seconds or so, think about it 🙂

Enjoy your week-end