Introduction

There is already a standard Linux utility called cut that has some of scut’s functionality bundled with every Linux distro (try man cut), however, it’s fairly stupid and has no regular expression (aka regex) capability.

scut’s cut functions

scut (code here) was written originally to find similar lines in 2 different files and output a combined output, sort of like the join command but as I worked on it, I realized that what I was doing was reiterating an SQL engine. Since there were already great SQL engines available, I stopped work on that aspect of it (tho it still works) but it’s still a better cut than cut (named as a contraction of super-cut, or perhaps the util that does your “scut” work for you. Anyway, it’s designed to cut and re-order columns of data from a file. The original functions are still intact, so if you need to look for lines in one file that exist in another file, that should still work as well and if join is unable to do what you need, you might want to try scut.

Consider for example, a long (11 million lines) gene expression data file named OMG that looks like this:

chr10:100018403-100020903;CHR10FS100020403;1    chr10_0 100020403       100020452       0.01
chr10:100018403-100020903;CHR10FS100019203;1    chr10_1 100019203       100019256       0.22
chr10:100018403-100020903;CHR10FS100019503;1    chr10_2 100019503       100019559       0.08
chr10:100018403-100020903;CHR10FS100019903;1    chr10_3 100019903       100019952       0.66
chr10:100018403-100020903;CHR10FS100020203;1    chr10_4 100020203       100020259       0.26
chr10:100018403-100020903;CHR10FS100019803;1    chr10_5 100019803       100019853       0.50
chr10:100018403-100020903;CHR10FS100018703;1    chr10_6 100018703       100018752       0.03
chr10:100018403-100020903;CHR10FS100018403;1    chr10_7 100018403       100018466       -0.12
chr10:100018403-100020903;CHR10FS100018903;1    chr10_8 100018903       100018952       -1.05
chr10:100018403-100020903;CHR10FS100020303;1    chr10_9 100020303       100020364       -0.76
[continues for 11,353,343 lines]

and you wanted to break it on both ; and whitespace (any contiguous number of spaces and and tabs), writing fields 1 3 4 but in the order 3 1 4 , and have the output data fields separated by ” ^ “, you could do this with scut in a single pass:

scut --id1=';|\s+'  --od=' ^ ' --c1='3 1 4'  < OMG

Out would spring:

chr10_0 ^ CHR10FS100020403 ^ 100020403
chr10_1 ^ CHR10FS100019203 ^ 100019203
chr10_2 ^ CHR10FS100019503 ^ 100019503
chr10_3 ^ CHR10FS100019903 ^ 100019903
chr10_4 ^ CHR10FS100020203 ^ 100020203
chr10_5 ^ CHR10FS100019803 ^ 100019803
chr10_6 ^ CHR10FS100018703 ^ 100018703
chr10_7 ^ CHR10FS100018403 ^ 100018403
chr10_8 ^ CHR10FS100018903 ^ 100018903
chr10_9 ^ CHR10FS100020303 ^ 100020303
[etc]

The secret here is that scut allows the use of Perl Regular Expressions that can encode just about any pattern. Regular Expressions are both fearsomely confusing and insanely powerful. Wikipedia has a good description.

Because scut can break data on arbitrary regexes, you can tame most text data in a single pass, certainly in multiple passes by piping the output of one operation into the input of another.

scut –help will give you the help pages, and the internal code is fairly clear, tho not tremendously well-documented. It’s also not entirely free of bugs – some of the routines were written for specific tasks and were not generalized well, but I still find myself using it a lot.

Here’s another example of scut helping me find out the size distribution of all files in a directory:

ls -l | scut --c1='4' |stats

(note that the GNU ls command shipped with Linux produces a slightly different format than other ls variants. This one works with GNU ls).

The above command takes the output of ls -l, pipes it into scut which extracts field 4, then pipes that data into stats, another Perl utility that generates descriptive stats of whatever goes into it.

it produces: (with [my comments added])

Sum       193109146  [total of 193MB)
Number    18 [in 18 files]
Mean      10728285.8888889  [mean size of 10.7MB]
Median    27761 [but obviously due to some large files]
Mode      FLAT
NModes    No # was represented more than once
Min       0
Max       73690201 [here's a 73.6MB  reason for the skew]
Range     73690201
Variance  401893120826683  [and the rest]
Std_Dev   20047272.1542529
SEM       4725187.36152149
Skew      2.14197169376939
Std_Skew  3.71000380198295
Kurtosis  2.96898153032708

scut’s join feature

The join feature of scut works a little differently than does join itself. For a good example of how join works, see this IBM DeveloperWorks tutorial called Simplify data extraction using Linux text utilities

In scut, like join, you have to specify 2 filenames, the key fields, and what fields you want output, but with scut, you can specify different input and output delimiters (which can be multicharacter instead of single character), the fields in any order, and a host of other options.

If we have a file of needles (called needles) that need to be found in a haystack file (called haystack), and needles looks like this:

apbA
carA+pyrA
cdsA+cds
codA
ddlA
dinJ
dnaX+dnaZ
fepB
fhuB
fixC
folK
<etc>

and the haystack file looks like this:

      1     0                 1                2            3            4            5
      2     b              gene               EC         C1.0         C1.1         C2.0
      3 b0001              thrL      0.000423101   0.00046544  0.000429262  0.000433869 ...
      4 b0002  thrA+thrA1+thrA2  1.1.1.3+2.7.2.4  0.001018277  0.001268078  0.001312524 ...
      5 b0003              thrB         2.7.1.39  0.000517967  0.000457605  0.000582354 ...
      6 b0004              thrC         4.2.99.2  0.000670075  0.000558063  0.000789501 ...
      <etc>

and you wanted to match the fields in the single column of the needles file with the gene field in the haystack file, and you wanted the output to include:

• haystack columns 1, 0, 2, 5 in that order
• the output delimiter being | instead of <TAB>
• written to a file called hits

the command to do so would be:

scut --f1='needles' --f2='haystack' --k1=0 --k2=1  --c2='1 0 2 5'  --od='|' >hits

and the hits file would look like this:

apbA|b0425|1.55E-05|1.34E-05|
carA+pyrA|b0032|6.3.5.5|0.000155796|
cdsA+cds|b0175|2.7.7.41|2.91E-05|
codA|b0337|3.5.4.1|3.20E-05|
ddlA|b0381|6.3.2.4|0|
dinJ|b0226|0|0|
dnaX+dnaZ|b0470|2.7.7.7|4.48E-05|
<etc>

I’ve just modified scut to handle column specifiers more efficiently. It’s spelled out in the help section. ie:

      --c1='# # ..'   - the numbers of the columns from file1 that you want
                         printed out in the order in which you want them.  If
                         you DON'T want any columns from the file, just
                         omit the --c1 option completely.
                         If you want the whole line, type --c1='ALL'.

                         You can also use discontinous ranges like '2:4 8:10'
                         to print [2 3 4 8 9 10] and decreasing ranges like
                         '8:4' to print cols [8 7 6 5 4].  You can also negate
                         columns to remove them from a larger range '9:12' -11'
                         to print [9 10 12] or 12:1 -7:-4 to print
                         [12 11 10 9 8 3]. You can also use the 'ALL' keyword
                         to print all cols and negate the ones you don't
                         want with negative ranges - 'ALL -8:-14' to print all
                         columns EXCEPT 8-14.

                         Notes:
                         1) #s are split on whitespace, not commas.
                         2) scut also supports Excel-style column specifiers such as:
         or
      --c1='A C F ..'    (A B F AD BG etc) for up to 78 columns (->BZ)  If you want
                         more, add them to the %excel_ids hash in the code or create an
                         algorithm that does it right.

The cols utility

In my work, the input to scut is often a horrendously wide data file that overflows even the widest terminal screens with the tiniest fonts. And even if it didn’t overflow the terminal, when the data is printed to the terminal, the columns are almost always mismatched due to tab skips (if a field exceeds a tab boundary, it will skip to the next tab). Trying to figure out if a floating point number is the 21st or 22nd field can be trying if you have to do it all day.

cols (code here) was developed to take the such very “wide” files that have 10s of columns and present them in a terminal window so that you don’t have to import them into a GUI spreadsheet app to check that the parsing operation has gone well. The output of the scut operation is typically piped to cols and then to less -S to promote easier viewing of long lines.

An example is probably the best way to describe this. The file ”DataSet” is a long file of TAB-separated identifiers and data that has a header line of column IDs prefixed with a hash mark to mark it as a comment in the standard unixy way.

Would you rather try to decipher the column values from this output :

      $ head  -15 DataSet | less -S
          1 #b      gene    EC      C1.0    C1.1    C2.0    C2.1    C3.0    C3.1    E1.0    E1.1    E2.0    E2.1
          2 b0001   thrL    0.000423101     0.00046544      0.000429262     0.000433869     0.000250998     0.000
          3 b0002   thrA+thrA1+thrA2        1.1.1.3+2.7.2.4 0.001018277     0.001268078     0.001312524     0.001
          4 b0003   thrB    2.7.1.39        0.000517967     0.000457605     0.000582354     0.000640462     0.000
          5 b0004   thrC    4.2.99.2        0.000670075     0.000558063     0.000789501     0.000801508     0.000
          6 b0005   0       0       0       2.64E-07        0       0       0       0       0       0       0
          7 b0006   yaaA    0       0       0       0       0       0       0       0       0       0       0
          8 b0007   yaaJ    8.52E-06        8.87E-06        1.54E-05        2.74E-05        0       0       0
          9 b0008   talB    2.2.1.2 0.001160911     0.00118164      0.001263549     0.001345351     0.001103703
         10 b0009   mog+chlG        1.87E-05        1.91E-05        1.95E-05        1.70E-05        0       0
         11 b0010   yaaH    0       0       0       0       0       0       0       0       0       0       0
         12 b0011   0       0       0       0       0       0       0       0       0       2.86E-05        0

or from this output?

     $ head  -15 DataSet | cols | less -S
          1     0                 1                2            3            4            5            6
          2    #b              gene               EC         C1.0         C1.1         C2.0         C2.1
          3 b0001              thrL      0.000423101   0.00046544  0.000429262  0.000433869  0.000250998  0.00026
          4 b0002  thrA+thrA1+thrA2  1.1.1.3+2.7.2.4  0.001018277  0.001268078  0.001312524  0.001398845  0.00078
          5 b0003              thrB         2.7.1.39  0.000517967  0.000457605  0.000582354  0.000640462   0.0003
          6 b0004              thrC         4.2.99.2  0.000670075  0.000558063  0.000789501  0.000801508  0.00055
          7 b0005                 0                0            0     2.64E-07            0            0
          8 b0006              yaaA                0            0            0            0            0
          9 b0007              yaaJ         8.52E-06     8.87E-06     1.54E-05     2.74E-05            0
         10 b0008              talB          2.2.1.2  0.001160911   0.00118164  0.001263549  0.001345351  0.00110
         11 b0009          mog+chlG         1.87E-05     1.91E-05     1.95E-05     1.70E-05            0
         12 b0010              yaaH                0            0            0            0            0

(output of both is from the less pager to avoid wrap artefacts.)

They both looks nice and columnar, but if you look closely, you’ll see some variants that will drive you straight to the Advil after 30 minutes of trying to figure it out, especially if you’ve side-scrolled enough to miss one of the tab-skips. For example: most of the gene names are short enough to fit in the tab space, but in line 4, ”thrA+thrA1+thrA2” is wide enough to cause a tab-skip which will then throw off the rest of the line. Similarly, in line 9 of the top listing, does the value ”0.001345351” belong to the ”C3.0” or to the ”C3.1” column? Actually, it belongs to the ”C2.1” column as is shown in the second listing, which has columnized it correctly.

Also note that the lower one has column headers inserted (line 1) which give you the 0-based column count (change to 1-based numbering with –ch=1

Note that this utility is meant for visualizing, not actual processing, except in edge cases, as the padding is all spaces. Also the utlity aborts after reading 22 lines, unless told not to.

Here is the help output from ”cols –help”

    cols is a small Perl-based utility to view columns of data to help
    programmers check that the columns correspond to what they want.
    It strips tabs from the input and pads columns with spaces so it's
    NOT meant to be used as a pipeline processing tool, only as a checking
    tool.

    usage: pipe or redirect X-delimited tabular data (where X=TAB by default,
    but can be set to any Perl regex) to 'cols' with the following options:

    --mw=#  set max width to this many chars or the max per-col width if smaller.
            Defaults to 20.

    --ml=#  process this many lines of input. (Defaults to 22)

    --ch=#  add a line of column headers (starting at # - defaults to 0)
             to the output to tell where you are in very wide output
             (very useful)

    --delim=s the delimiter to use to split the fields (Defaults to TAB)
              Use 'ws' for whitespace (but you can use '\s+' if you want).

    --help  dumps this help

    Pipe output to 'less -S' to view long lines without wrap and arrow keys to
    scroll around.

    example:
              cols --mw=11 --ch=1 <huge.data |less -S

Introduction

FORTRAN has the reputation for being old, crufty, quite hard to use and stuck in some very old programming paradigms. However, recent versions of FORTRAN include very modern abilities and many people are still using it, especially for pure number crunching as the compilers are still among the best for doing so. FORTRAN does not provide easy access to GUI’s, relational databases, or methods for handling options (AFAIK – please correct), while many scripting languages, such as Python & Perl do.

This is how I converted a very sophisticated, but fairly UI-ugly (and hard-to-modify) FORTRAN program to one that uses Python as the application glue. Python was used to do the scut work of command-line user interface and configuration file management. It was also used to add an optional GUI to it and record some usage to a relational database. I used the f2py module of the scientific Python package numpy to do the FORTRAN compilation and generation of the shared lib, and then used Qt-designer and the Qt widget set to draw the GUI and then PyQt to convert it to Python. That sounds quite complex, but as you’ll see, it’s not especially if you use a recent version of Linux as all the required packages are available for free.

This was the 1st time I’ve used numpy for this and it worked much better than I had expected. My previous experience had been with SWIG (the Rosetta Stone for mixing computer languages), and while SWIG allows you to do tremendous magic, it was a slog to get it to work. With numpy and f2py, it just worked.

The end result is a more easily maintainable program that separates the FORTRAN math engine from the user interface, provides a more standard option-handling and configuration file capabilities, provides the (optional) GUI, and also adds reporting to a remote relational database for use and platform tracking. All this in about 300 lines of code which includes much debugging.

The Problem

The initial problem was that a researcher had a great Magnetic Resonance (MR) analysis program for proteins that he wanted to be more user-friendly. It was written in FORTRAN and ran quite efficiently, but it was difficult to use.

The last time I wrote anything in FORTRAN was in the 70’s but I got the code and figured out approximately what it did. I then ran it thru a profiler (oprofile) and was able to tell where it spent its time (90% in 1 nested set of functions):

$ opreport --exclude-dependent --demangle=smart --symbols /home/hjm/shaka/1D-Mangalam-py/fd_rrt1d.so
CPU: Core 2, speed 1667 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples  %        symbol name
1107170  61.4356  cqzhes_
345869   19.1919  cqzvec_
179710    9.9719  cqz_
144523    8.0194  fd_rrt1d_
6700      0.3718  _g95_exp_z8
4593      0.2549  _g95_power_z8_i4
4171      0.2314  umatrix1d_ms0_
3231      0.1793  fd_1d_
1336      0.0741  .plt
1091      0.0605  cqzval_
...

I was initially going to try to improve the efficiency but for a number of reasons that was not a priority at this time. Ease of use, ability of others to help improve it, and multi-platform ability were higher priority for the researcher.

Since I had some experience with Python, I decided that this would be a good time to try out the f2py functionality of numpy.

The original FORTRAN code I was given included 3 FORTRAN source files totalling 1880 lines and some associated configuration and support files.

Converting the FORTRAN to a shared lib

The first thing that I did was to convert the FORTRAN main() to a function so it could be called from Python. Since I wasn’t re-writing the application, I just needed a Python front-end to set everything up and then kick off the run by passing all the required variables to the native FORTRAN routines. This takes only a few lines of code – primarily to add the subroutine call with all the variables that were being set from the calling Python:

      subroutine fd_rrt1d(signal,theta,method,Nsig,wmino,wmaxo,par,
     &     threshhold,ReSp,ImSp,AbsSp,rho,Nb0,Nbc,Nsp,Gamm,cheat,
     &     cheatmore,ros)

that’s really all it took. Besides that single change, there were few changes to the FORTRAN code besides inserting some debugging variables and comments to myself to clarify the code a bit more. Here’s a diff view:

To compile the whole thing into a shared lib that can be called from Python took little more work:

f2py --opt="-O3" -c -m fd_rrt1d --fcompiler=gnu95  --link-lapack_opt *.f

The above line uses the gfortran compiler (aka gnu95) which seems to both generate marginally faster code and is also more compatible with MacOSX than the g95 compiler I 1st tried (g95 worked fine on Linux, but had problems on MacOSX due to API incompatibilities with numpy on MacOSX). The end result of this command was a shared lib fd_rrt1d.so which is callable by subroutine name by both Python and FORTRAN (the FORTRAN code calls several subroutines spread over those 3 files). That is one of the magic things about the numpy package; it all works the way it’s supposed hiding the considerable magic.

$ ./1d.py
Traceback (most recent call last):
  File "./1d.py", line 27, in <module>
    from fd_rrt1d import *
ImportError: /home/hjm/shaka/1D-Mangalam-py/fd_rrt1d.so: undefined
symbol: zgemm_

$ nm fd_rrt1d.so |tail
000065b0 t string_from_pyobj
         U strlen@@GLIBC_2.0
         U strncpy@@GLIBC_2.0
0001df40 b u.1294
00013bee T umatrix1d_dms0_
000128c3 T umatrix1d_dms1_
00015458 T umatrix1d_ms0_
         U zgemm_            <---
         U zgemv_
         U zgesv_

I wrote a skeleton Python program that assigned the variables and called the FORTRAN code. Astonishingly, it worked on the 1st try, so I continued to expand the skeleton to add the commandline option-handling.

Commandline option-handling

Python has a standard way of providing commandline option handling this via its getopt package. It’s probably not the best, but there’s a lot to be said for doing it in a semi-standard way. It’s also very easy to implement.

The following is the entire option-handling code for the MANY options that it supports and reads any defined commandline option in, then either calls a function (such as –gui) or does some munging of the variable (–wmaxo) and sticks it in a dictionary (aka hash) for easy lookup and passing to other functions. If the option is unrecognized, it just calls the usage() function to let the user figure out the error of his ways.

import getopt
 ...
try:
    opts, args = getopt.getopt(sys.argv[1:], 'hD', ['help', 'debug', 'help1d', 'gui', 'nodb', 'paramfile=', 'signal=', 'theta=', 'method=', 'Nsig=', 'wmino=', 'wmaxo=', 'par=', 'threshhold=', 'ReSp=', 'ImSp=', 'AbsSp=', 'rho=', 'Nb0=', 'Nbc=', 'Nsp=', 'Gamm=', 'cheat=', 'cheatmore=', 'ros='])
except getopt.GetoptError:
    # print help information and exit:
    print "There was an error specifying an option flag.  Here's the correct usage:"
    usage(1)
# set up the options required
for opt, arg in opts:
    if opt in ('-h', '--help'):   usage(1)
    elif opt in ('-D', '--debug'):      DEBUG = 1
    elif opt in ('--help1d'):     usage1d(1)
    elif opt in ('--gui'):        gui()
    elif opt in ('--nodb'):       USEDB = 0
    elif opt in ('--paramfile'):  paramfile = arg
    elif opt in ('--signal'):     clcfg['signal = arg']         # file name
    elif opt in ('--theta'):      clcfg['theta'] = float(arg)   # float
    elif opt in ('--method'):     clcfg['nmr_method'] = arg         # FDM, RRT or DFT
    elif opt in ('--Nsig'):       clcfg['Nsig'] = int(round(float(arg)))      #int
    elif opt in ('--wmino'):      clcfg['wmino'] = int(round(float(arg)))   #int
    elif opt in ('--wmaxo'):      clcfg['wmaxo'] = int(round(float(arg)))   #int
    elif opt in ('--par'):        clcfg['par'] =  arg           # linelist output file
    elif opt in ('--threshhold'): clcfg['threshhold'] = float(arg)
    elif opt in ('--ReSp'):       clcfg['ReSp'] = arg           # file name
    elif opt in ('--ImSp'):       clcfg['ImSp'] = arg           # file name
    elif opt in ('--AbsSp'):      clcfg['AbsSp'] = arg          # file name
    elif opt in ('--rho'):        clcfg['rho'] = float(arg)   #
    elif opt in ('--Nb0'):        clcfg['Nb0'] = int(round(float(arg)))   #
    elif opt in ('--Nbc'):        clcfg['Nbc'] = int(round(float(arg)))   #
    elif opt in ('--Nsp'):        clcfg['Nsp'] = int(round(float(arg)))   #
    elif opt in ('--Gamm'):       clcfg['Gamm'] = float(arg)   #
    elif opt in ('--cheat'):
        clcfg['cheat'] = float(arg)   # float
        if clcfg['cheat'] != 1 or clcfg['cheat'] != 0:
            print >> sys.stderr, "cheat must be '1' or '0'"
            sys.exit(1)
    elif opt in ('--cheatmore'):
        clcfg['cheatmore'] = arg   # T or F
        if clcfg['cheatmore'] != 'T' or clcfg['cheatmore'] != 'F':
            print >> sys.stderr, "cheatmore must be 'T' or 'F'"
            sys.exit(1)
    elif opt in ('--ros'):        clcfg['ros'] = float(arg)   #

Configuration File Handling

The original FORTRAN program supported a custom-written configuration file input of options that had this form:

'2p-no-noise.txt'                               /signal
1.5708                                          /theta
'FDM'                                   /method (FDM/RRT/DFT)
4                                       /Nsig
-9000 4500                              /wmin wmax
'par',  1d-4                            /parameters, output threshhold
'fdm','none','none'                     /ReSpectrum,ImSpectrum,AbsSpectrum
1., 512, -20                            /rho, Nb0, Nbc
20000, 5d-2                             /Npower, Gamm
1 F                                     /cheat, cheatmore
1d-8                                    /ros

User-written FORTRAN code then parsed this to set the variables. In providing the Python front-end, it was extremely easy to provide a more sophisticated way of doing this using the configobj module which is designed to do just this.

from configobj import ConfigObj  # for the configuration module
 ...
if paramfile != "": # If there's a param file named, try to get params from it.
    fcfg = ConfigObj(file(paramfile))  # reads all variables in file as strings

    # now have to coerce everything from the param file that is not a
    # string to the correct type
    # following members used to iterate over to coerce into int or float
    int_params = ("Nsig", "Nsp", "wmino", "wmaxo", "Nb0", "Nbc")
    float_params = ("cheat", "theta", "threshhold", "rho", "Gamm", "ros")

    for ip in int_params: fcfg[ip]=int(fcfg[ip])
    for fp in float_params: fcfg[fp]=float(fcfg[fp])

I had to add some logic to allow options entered at the commandline to override those in the configuration file, but essentially the code above was all that was needed to support a configuration file that allows key = value pairs that can be nested into arbitrary stanzas.

Here’s an extract of the config file showing assignment of strings, ints, and floats. Note that all values are interpreted as strings and have to be coerced into the appropriate type – see above in the option-handling section. The config file included in the tarball includes a summary explanation of how the file is structured and the URL to the home page of the configobj module.

# signal is the file that contains the signal data; if no leading path, then it is
# assumed to be in the current directory.
signal = "2p-no-noise.txt"   # file containing the signal data
theta = 1.5708               # fl pt var returned as string, conv in wrapper code
method = "FDM"               # can be one of (FDM/RRT/DFT)
Nsig = 4                     # int var returned as string, conv in wrapper code

Graphical User Interface

The addition of a GUI used to be stuff of wizards and black arts. It’s still not trivial but it’s considerably easier using the Designer approach, in which you use an application that allows you to drag control widgets to a canvas, arranging them as you like. I used Trolltech’s Qt widget library and their VERY easy-to-use Designer app to mock up an interface and then converted the interface XML description to Python using Riverbank’s PyQt toolkit. Here’s a screenshot of the Qt4 Designer being used to design the fd_rr1d GUI:

After the UI is drawn and saved as GUI_1D.ui (an XML representation of the design), the UI file is converted to Python code using the PyQt utility pyuic4.

$ pyuic4 GUI_1D.ui > UI.py

The autogenerated code (UI.py) is then wired to functionality using conventional programming techniques or by using Qt’s system of Signals & Slots that can be mostly done using their Designer application

The code required for making the proof of concept (a GUI that pops up and allows the user to set all the options graphically) is actually quite concise (the complexity is hidden in the Qt lib that you link to).

from PyQt4.QtCore import * # PyQt core libs
from PyQt4.QtGui import *  # PyQy GUI components
from GUI_1D import *       # the interface definition file converted to Python code
...

# the class multiply inherits from the library prototype and the specific interface
# class defined in the designer ui -> py
class Form1D(QDialog,Ui_Dialog):
    # to pop it up, it only needs to __init__ itself, declare its parent (itself) as
    # it's a top-level dialog, and then call the designer -> pyuic4-generated setupUi()
    # to make it do anything useful, I have to write all the glue code to pass
    # the params, connect signals & slots, do error-checking, etc. but his pops it up
    # don't forget to erase the no-longer needed class and defs when finished.
    def __init__(self, parent=None):
        super(Form1D, self).__init__(parent)
        self.setupUi(self)

def gui():
    """To pop up the designer-built form, it only needs to declare an instance of the
    QtApplication, ditto the form itself, and show it.
    To make it do anything useful, still have to write all the glue code to pass
    the params, connect signals & slots, do error-checking, etc. but this pops it up.
    """

    app = QApplication(sys.argv)
    form = Form1D()
    form.show()
    app.exec_()
...

# the above class is referenced from option-handling stanza:
for opt, arg in opts:
    if opt in ('-h', '--help'):   usage(1)
...
    elif opt in ('--gui'):        gui()

So if you started the app with the –gui option, the GUI window would pop up and allow you fill out all the variables via the mouse. [This section still incomplete]

Relational Database connectivity

When releasing a piece of academic software into the wild, it is often useful to the author to figure out how it’s being used so that she can rewrite instructions, concentrate on most-used features, find out the platform distribution, etc. This mechanism can be exploited trivially using Python’s Relational DataBase (RDB) connection module. During the run of this program, the Python wrapper provides all the variables, times the execution of the run, and can provide some network information to the author. This information is presented at the end of the run with a request to send the info back to the author. If the user agrees, the Python wrapper attempts to contact a pre-defined database server and send back the information.

The mechanism is straightforward:

• collect the information
• compose an INSERT command to the RDB
• show that information and ask the user’s permission to return it.
• if granted, connect to the remote RDB and execute the INSERT command.

The information returned includes the date, the hostname, IP #, and OS of the computer, all program variables, the program run-time, and some platform information about the machine that ran the program.

Here’s the info collected. Note that the sysinfo string is the full output of lshw -short and should be trimmed considerably.

date:   Tue Jul 29 15:20:41 2008
user:   hjm
host:   bongo
ipnbr:  128.200.34.98
OS:     Linux
sysinfo:        H/W path       Device    Class       Description
================================================
                         system      Computer
/0                       bus         Motherboard
/0/0                     memory      3041MiB System memory
/0/1                     processor   Intel(R) Core(TM)2 CPU         T5500  @ 1.66GHz
/0/1/0.1                 processor   Logical CPU
/0/1/0.2                 processor   Logical CPU
/0/100                   bridge      Mobile 945GM/PM/GMS, 943/940GML and 945GT Express Memory Controller Hub
/0/100/1                 bridge      Mobile 945GM/PM/GMS, 943/940GML and 945GT Express PCI Express Root Port
/0/100/1/0               display     Radeon Mobility X1400
/0/100/1b                multimedia  82801G (ICH7 Family) High Definition Audio Controller
/0/100/1c                bridge      82801G (ICH7 Family) PCI Express Port 1
/0/100/1c/0    eth0      network     82573L Gigabit Ethernet Controller
/0/100/1c.1              bridge      82801G (ICH7 Family) PCI Express Port 2
/0/100/1c.1/0  wmaster0  network     PRO/Wireless 3945ABG Network Connection
/0/100/1c.2              bridge      82801G (ICH7 Family) PCI Express Port 3
/0/100/1c.3              bridge      82801G (ICH7 Family) PCI Express Port 4
/0/100/1d                bus         82801G (ICH7 Family) USB UHCI Controller #1
/0/100/1d.1              bus         82801G (ICH7 Family) USB UHCI Controller #2
/0/100/1d.2              bus         82801G (ICH7 Family) USB UHCI Controller #3
/0/100/1d.3              bus         82801G (ICH7 Family) USB UHCI Controller #4
/0/100/1d.7              bus         82801G (ICH7 Family) USB2 EHCI Controller
/0/100/1e                bridge      82801 Mobile PCI Bridge
/0/100/1e/0              bridge      PCI1510 PC card Cardbus Controller
/0/100/1f                bridge      82801GBM (ICH7-M) LPC Interface Bridge
/0/100/1f.1              storage     82801G (ICH7 Family) IDE Controller
/0/100/1f.2              storage     82801GBM/GHM (ICH7 Family) SATA AHCI Controller
/0/100/1f.3              bus         82801G (ICH7 Family) SMBus Controller

runtime:        5.47741103172
par :   FDM_par.out
Nb0 :   100
Nsig :  40960
ImSp :  ImSp_spectra.data
cheat : 1.0
signal :        2p-no-noise.txt
nmr_method :    FDM
cheatmore :     F
Nsp :   20000
ReSp :  ReSp_spectra.data
wmaxo : 4500
rho :   2.0
threshhold :    0.0001
Nbc :   -20
theta : 1.5
Gamm :  0.05
ros :   1e-08
AbsSp : AbsSp_spectra.data
wmino : -9000

The entire data to be returned (formatted as above) is presented to the user just prior to sending it, so they have the opportunity to refuse sending the information.

Additional useful hints

Presenting help files in an easily navigable way usually requires a hypertext browser or a custom screen pager. Python offers a very easy way to present any text file via any pager application on the system. Since most *nix-like systems have the less pager, I just called that pager on the help file. Here’s the entire function that presents pagable help text.

from pydoc import pipepager
...
def usage1d(code):
    try:
        help_fp = file("1d_orig_help.txt", "r")
        help_txt = help_fp.read()                # read in any text from a text file.
    except:
        print "Can't find the help file - should be called '1d_orig_help.txt' - Did you rename it?"
        sys.exit(code)
    pipepager(help_txt, '/usr/bin/less -NS') # pipe help text into 'less -NS'
    sys.exit(code)

Download the entire code tree

The entire code tree can be downloaded The from here. The File Manifest is included; those files not explicitly named are probably not required.

Abstract

Free/Libre/Open Source Software (FLOSS) represents a huge pool of ready-made, well-designed, pre-integrated software that that we can use to improve IT support. And not only is it free, but it promotes sharing, scalability, reduces licensing and support costs, and relieves its users of legal liability more than it encumbers them.

There’s every reason to be careful in IT spending, especially (but not only) in these unsettled times. Just as energy conservation is the most efficient mechanism for saving energy (see NegaWatts), the most efficient mechanism for saving money in Information Technology is by NOT paying for that which you do not need to pay for. I call them NegaBIT$ (as in B undles of IT $ you don’t have to spend).

While the direct saving on software costs relative to UC’s deficit is tiny, it can have a multiplier effect, as the use of FLOSS not only has an immediate effect in reducing some costs, but has a much greater effect in the long term as a way of approaching IT and reducing other costs.

An example may be useful. I don’t know the cost of the “Enterprise Monitoring system” we use here (Netreo), but I do know that Google reports ~33,000 hits on it. Nagios, a similar, but Open Source system, has >5,000,000 hits. This doesn’t mean that Netreo is bad software, nor that Nagios is good software, but it certainly should cause ears to perk up and to examine Nagios in closer detail to see why there are 156 times more Google refs to Nagios than to Netreo. Another way of evaluating mindshare is to check how many pages link to the site. Using Google’s advanced search, about 1400 pages link to www.nagios.org, while only 2 pages link to www.netreo.com.

One of the complaints I’ve heard from people charged with evaluating software is that they can’t obtain comparable information about a competitive FLOSS package as from a commercial vendor. This is an inherent problem with FLOSS. Since the software is free, how can the producers also be expected to provide salespeople and promotional kits that would compete with those from a commercial vendor? If we are going to exploit FLOSS, we have to be prepared to dig a little harder. Or not – would you be willing to accept at face value whatever a commercial vendor tells you? In some cases, especially for the more mature FLOSS systems, they do provide comparable propaganda, but regardless, we should certainly do our own homework.

The solution is to nominate someone to evaluate the best FLOSS products. This is actually often much easier than doing the evaluation on commercial products as the FLOSS packages are freely available in unrestricted form (and the best known packages, are already in the FLOSS repositories) so that installation is often as simple as (for Nagios on a Debian-based system):

   sudo apt-get install nagios

There are an additional ~20 optional (and free) packages that work with Nagios that are available via the same mechanism.

There are extensive methodologies for doing such evaluations. For examples, one of the most best documented is David Wheeler’s IRCA approach:

• I dentify candidates
• R ead existing reviews
• C ompare the leading programs’ basic attributes to your needs, and then
• A nalyze the top candidates in more depth.

It would be very helpful if such evaluations were also published so that others at UC could make use of them or use them as a starting point for their own evaluations.

Paying for software does not assure quality, and not paying for software does not assure catastrophy. One of the best-selling software titles of 1995 was Syncronys Software’s SoftRAM, a Memory Optimizer program for Windows that was eventually shown to do … nothing. This exemplifies just one problem with closed source software – short of actually disassembling the binary executable code, you can’t see how it works so you can’t tell if it’s doing what it says it does. Not that most of us would be going thru the code line by line, but you can bet that there would be geeks out there that were.

There is also the problem of vendor lock-in and transitioning. If you have dedicated an entire infrastructure around a particular commercial software, the vendor obviously knows this and can then increase the renewal or support costs to just shy of lethal levels, knowing that they can. You now have no choice but to pay their ransom or face a huge transition cost. Sometimes this transition is unexpectedly forced on you, as when rivals buy each other and kill or neglect the competing software lines offered by their now-subsidiary.

Why do people choose commercial software over the OSS equivalent? The feature set is a typical explanation of why an organization chooses a commercial product over an OSS one – there are often additional options and features in commercial software packages that OSS ones lack. There is good reason for this. Vendors have to offer something beyond what the OSS has, and often the reason the FLOSS equivalent lacks a feature is that no one uses or has requested it.

Besides the unit costs, commercial software has an additional hidden cost when provided by a large organization – that of negotiating license agreements, packaging and distributing the software, tracking the use and leakage of such software, and providing the accounting reports of such use. For a UCI-sized campus, there are probably 3-4 FTE-equivalents that are involved in this process.

This is not to suggest that FLOSS has no costs – FLOSS packages have the same kinds of costs for configuration that non-FLOSS packages have. However when you gain that functional knowledge, it can be used to roll out the package in bulk for little additional cost.

Once you start down this path of exploiting FLOSS, it’s a slippery slope to saving. You can find all kinds of compatible software that can be mixed and matched and mashed up to provide additional functionality for no $ down. And since much FLOSS uses the same underlying libraries, it tends to have a greater overall compatibility.

And when you find a FLOSS solution that works well, you can roll it out to all 100,000 of your closest friends. Now THAT is scalability.

NegaBIT$ indeed.

Note

Executive Summary

If you have to transfer data, transfer only that which is necessary. If you unavoidably have lots of data to transfer, consider having your institution set up a GridFTP node.

If GridFTP is not available, the fastest, easiest, user-mode, node-to-node method to move data for Linux and MacOSX is with bbcp.

If you use Windows, fdt is Java-based and will run there as well.

Note that bbcp and the similar bbftp can require considerable tuning to extract maximum bandwidth. If these applications do not work at expected rates, ESNet’s Guide to Bulk Data Transfer over a WAN is an excellent summary of the deeper network issues.

We all need to transfer data, and the amount of that data is increasing as the world gets more digital. If it’s not climate model data from the IPCC, it’s high energy particle physics data from the LHC, or audio & video streams from a performance recording.

The usual methods of transferring data (scp, http and ftp utilities (such as curl or wget) work fine when your data is in the MB range, but when you have very large collections of data there are some tricks that are worth mentioning.

Compression & Encryption

Whether to compress and/or encrypt your data in transit depends on the cost of doing so. For a modern desktop or laptop computer, the CPU(s) are usually not doing much of anything so the cost incurred in doing the compression/encryption is generally not even noticed. However on an otherwise loaded machine, it can be significant, so it depends on what has to be done at the same time. Compression can reduce the amount of data that needs to be transmitted considerably if the data is of a type that is compressible (text, XML, uncompressed images and music), however progressively such data is already compressed on the disk (in the form of jpeg or mp3 compression), and compressing already compressed data yields little improvement. Some compression utilities try to detect already-compressed data and skip it, so there’s often no penalty in requesting compression, but some utilities (like the popular Linux archiving tar) will not detect it correctly and waste lots of time trying.

As an extreme example, here’s the timing of making a tar archive of a large directory that consists of mostly already compressed data, using compression or not.

Using compression:

$ time tar -czpf /bduc/data.tar.gz /data
tar: Removing leading `/' from member names

real    201m38.540s
user    95m32.114s
sys     7m13.807s

tar file = 84,284,016,900 bytes

NOT using compression:

$ time tar -cpf /bduc/data.tar /data
tar: Removing leading `/' from member names

real    127m13.404s
user    0m43.579s
sys     5m35.437s

tar file = 86,237,952,000

It took more than 74 minutes (about 58%) longer using compression which gained us about 2GB less storage (2.3% decrease in size.) YMMV.

Similarly, there is a computational cost to encrypting and decrypting a text, but less so than with compression. scp uses ssh to do the underlying encryption and it does a very good job, but like the other single-TCP-stream utilities like curl and wget, it will only be able to push so much thru a connection.

Avoiding data transfer

The most efficient way to transfer data is not to transfer it at all. There are a number of utilities that can be used to assist in NOT transferring data. Some of them are listed below.

kdirstat

The elegant, open source kdirstat (and it’s ports to MacOSX Disk Inventory X and Windows Windirstat) are quick ways to visualize what’s taking up space on your disk so you can either exclude the unwanted data that needs to be copied or delete it to make more space. All of these are fully native GUI applications that show disk space utilization by file type and directory structure.

rsync

rsync, from the fertile mind of Andrew (samba) Tridgell, is an application that will synchronize 2 directory trees, transferring only blocks which are different.

For example, if you had recently added some songs to your 120 GB MP3 collection and you wanted to refresh the collection to your backup machine, instead of sending the entire collection over the network, rsync would detect and send only the new songs.

For example, the first time rsync is used to transfer a directory tree, there will be no speedup.

$ rsync -av ~/FF moo:~
building file list ... done
FF/
FF/6vxd7_10_2.pdf
FF/Advanced_Networking_SDSC_Feb_1_minutes_HJM_fw.doc
FF/Amazon Logitech $30 MIR MX Revolution mouse.pdf
FF/Atbatt.com_receipt.gif
FF/BAG_bicycle_advisory_group.letter.doc
FF/BAG_bicycle_advisory_group.letter.odt
 ...

sent 355001628 bytes  received 10070 bytes  11270212.63 bytes/sec
total size is 354923169  speedup is 1.00

but a few minutes later after adding danish_wind_industry.html to the FF directory

$ rsync -av ~/FF moo:~
building file list ... done
FF/
FF/danish_wind_industry.html

sent 63294 bytes  received 48 bytes  126684.00 bytes/sec
total size is 354971578  speedup is 5604.05

So the synchronization has a speedup of 5600-fold relative to the initial transfer.

Even more efficiently, if you had a huge database to back up and you had recently modified it so that most of the bits were identical, rsync would send only the blocks that contained the differences.

Here’s a modest example using a small binary database file:

$ rsync -av mlocate.db moo:~
building file list ... done
mlocate.db

sent 13580195 bytes  received 42 bytes  9053491.33 bytes/sec
total size is 13578416  speedup is 1.00

After the transfer, I update the database and rsync it again:

$ rsync -av mlocate.db moo:~
building file list ... done
mlocate.db

sent 632641 bytes  received 22182 bytes  1309646.00 bytes/sec
total size is 13614982  speedup is 20.79

There are many utilities based on rsync that are used to synchronize data on 2 sides of a connection by only transmitting the differences. The backup utility BackupPC is one.

The open source rsync is included by default with almost all Linux distributions. Versions of rsync exist for Windows as well, via Cygwin and DeltaCopy

Unison

Unison is a slightly different take on transmitting only changes. It uses a bi-directional sync algorithm to unify filesystems across a network. Native versions exist for Windows as well as Linux/Unix and it is usually available from the standard Linux repositories.

From a Ubuntu or Debian machine, to install it would require:

$ sudo apt-get install unison

Streaming Data Transfer

bbcp

bbcp seems to be a very similar utility to bbftp below, with the exception that it does not require a remote server running. In this behavior, it’s much more like scp in that data transfer requires only user-executable copies on both sides of the connection. Short of access to a GridFTP site, this appears to be the fastest, most convenient single-node method for transferring data.

The code compiled & installed easily with one manual intervention

curl http://www.slac.stanford.edu/~abh/bbcp/bbcp.tar.Z |tar -xZf -
cd bbcp
# edit Makefile to change line 18 to: LIBZ       =  /usr/lib/libz.a
make
# there is no *install* stanza in the distributed 'Makefile'
cp bin/your_arch/bbcp ~/bin   # if that's where you store your personal bins.
hash -r   # or 'rehash' if using cshrc
# bbcp now ready to use.

bbcp can act very much like scp for simple usage:

$ time bbcp  file.633M   user@remotehost.subnet.uci.edu:/high/perf/raid/file
real    0m9.023s

The file transferred in under 10s for a 633MB file, giving >63MB/s on a Gb net. Note that this is over our very fast internal campus backbone. That’s pretty good, but the transfer rate is sensitive to a number of things and can be tuned considerably. If you look at all the bbcp options, it’s obvious that bbcp was written to handle lots of exceptions.

If you increase the number of streams (-s) from the default 4 (as above), you can squeeze a bit more bandwidth from it as well:

$ bbcp -P 10 -w 2M -s 10 file.4.2G hjm@remotehost.subnet.uci.edu:/userdata/hjm/
bbcp: Creating /userdata/hjm/file.4.2G
bbcp: At 081210 12:48:18 copy 20% complete; 89998.2 KB/s
bbcp: At 081210 12:48:28 copy 41% complete; 89910.4 KB/s
bbcp: At 081210 12:48:38 copy 61% complete; 89802.5 KB/s
bbcp: At 081210 12:48:48 copy 80% complete; 88499.3 KB/s
bbcp: At 081210 12:48:58 copy 96% complete; 84571.9 KB/s

or almost 85MB/s for 4.2GB which is very good sustained transfer.

Even traversing the CENIC net from UCI to SDSC is fairly good:

$ time bbcp -P 2 -w 2M -s 10 file.633M   user@machine.sdsc.edu:~/test.file

bbcp: Source I/O buffers (61440K) > 25% of available free memory (200268K); copy may be slow
bbcp: Creating ./test.file
bbcp: At 081205 14:24:28 copy 3% complete; 23009.8 KB/s
bbcp: At 081205 14:24:30 copy 11% complete; 22767.8 KB/s
bbcp: At 081205 14:24:32 copy 20% complete; 25707.1 KB/s
bbcp: At 081205 14:24:34 copy 33% complete; 29374.4 KB/s
bbcp: At 081205 14:24:36 copy 41% complete; 28721.4 KB/s
bbcp: At 081205 14:24:38 copy 52% complete; 29320.0 KB/s
bbcp: At 081205 14:24:40 copy 61% complete; 29318.4 KB/s
bbcp: At 081205 14:24:42 copy 72% complete; 29824.6 KB/s
bbcp: At 081205 14:24:44 copy 81% complete; 29467.3 KB/s
bbcp: At 081205 14:24:46 copy 89% complete; 29225.5 KB/s
bbcp: At 081205 14:24:48 copy 96% complete; 28454.3 KB/s

real    0m26.965s

or almost 30MB/s.

When making the above test, I noticed the disks to and from which the data was being written can have a large effect on the transfer rate. If the data is not (or cannot be) cached in RAM, the transfer will eventually require the data to be read from or written to the disk. Depending on the storage system, this may slow the eventual transfer if the disk I/O cannot keep up with the the network. On the systems that I used in the example above, I saw this effect when I transferred the data to the /home partition (on a slow IDE disk – see below) rather than the higher performance RAID system that I used above.

$ time bbcp -P 2  file.633M  user@remotehost.subnet.uci.edu:/home/user/nother.big.file
bbcp: Creating /home/user/nother.big.file
bbcp: At 081205 13:59:57 copy 19% complete; 76545.0 KB/s
bbcp: At 081205 13:59:59 copy 43% complete; 75107.7 KB/s
bbcp: At 081205 14:00:01 copy 58% complete; 64599.1 KB/s
bbcp: At 081205 14:00:03 copy 59% complete; 48997.5 KB/s
bbcp: At 081205 14:00:05 copy 61% complete; 39994.1 KB/s
bbcp: At 081205 14:00:07 copy 64% complete; 34459.0 KB/s
bbcp: At 081205 14:00:09 copy 66% complete; 30397.3 KB/s
bbcp: At 081205 14:00:11 copy 69% complete; 27536.1 KB/s
bbcp: At 081205 14:00:13 copy 71% complete; 25206.3 KB/s
bbcp: At 081205 14:00:15 copy 72% complete; 23011.2 KB/s
bbcp: At 081205 14:00:17 copy 74% complete; 21472.9 KB/s
bbcp: At 081205 14:00:19 copy 77% complete; 20206.7 KB/s
bbcp: At 081205 14:00:21 copy 79% complete; 19188.7 KB/s
bbcp: At 081205 14:00:23 copy 81% complete; 18376.6 KB/s
bbcp: At 081205 14:00:25 copy 83% complete; 17447.1 KB/s
bbcp: At 081205 14:00:27 copy 84% complete; 16572.5 KB/s
bbcp: At 081205 14:00:29 copy 86% complete; 15929.9 KB/s
bbcp: At 081205 14:00:31 copy 88% complete; 15449.6 KB/s
bbcp: At 081205 14:00:33 copy 91% complete; 15039.3 KB/s
bbcp: At 081205 14:00:35 copy 93% complete; 14616.6 KB/s
bbcp: At 081205 14:00:37 copy 95% complete; 14278.2 KB/s
bbcp: At 081205 14:00:39 copy 98% complete; 13982.9 KB/s

real    0m46.103s

You can see how the transfer rate decays as it approaches the write capacity of the /home disk.

bbftp

bbftp is a modification of the FTP protocol that enables you to open multiple simultaneous TCP streams to transfer data. It therefore allows you to sometimes bypass per-TCP restrictions that result from badly configured intervening machines.

In order to use it, you ‘ll need a bbftp client and server. Most places that recieve large amounts of data (SDSC, NCAR, other supercomputer centers, teragrid nodes) will already have a bbftp server running, but you can also compile and run the server yourself.

The more usual case is to run only the client. It builds very easily on Linux with just the typical curl/untar, cd, ./configure, make, make install dance:

$ curl http://doc.in2p3.fr/bbftp/dist/bbftp-client-3.2.0.tar.gz |tar -xzvf -
$ cd bbftp-client-3.2.0/bbftpc/
$ ./configure --prefix=/usr/local
$ make -j3
$ sudo make install

Using bbftp is more complicated than the usual ftp client because it has its own syntax:

To send data to a server:

$ bbftp -s -e 'put file.154M  /gpfs/mangalam/big.file' -u mangalam -p 10 -V tg-login1.sdsc.teragrid.org
Password:
>> COMMAND : put file.154M /gpfs/mangalam/big.file
<< OK
160923648 bytes send in 7.32 secs (2.15e+04 Kbytes/sec or 168 Mbits/s)

the arguments mean:
-s  use ssh encryption
-e  'local command'
-E  'remote command' (not used above, but often used to cd on the remote system)
-u  'user_login'
-p  # use # parallel TCP streams
-V  be verbose

The data was sent at 21MB/s to SDSC thru 10 parallel TCP streams (but well below the peak bandwidth of about 90MB/s on a Gb network)

To get data from a server:

$ bbftp -s -e 'get /gpfs/mangalam/big.file from.sdsc' -u mangalam -p 10 -V tg-login1.sdsc.teragrid.org
Password:
>> COMMAND : get /gpfs/mangalam/big.file from.sdsc
<< OK
160923648 bytes got in 3.46 secs (4.54e+04 Kbytes/sec or 354 Mbits/s)

I was able to get the data at 45MB/s, about half of the theoretical maximum.

As a comparison, because the remote reciever is running an old (2.4) kernel which does not handle dynamic TCP window scaling, scp is only able to manage 2.2MB/s to this server:

$ scp  file.154M mangalam@tg-login1.sdsc.teragrid.org:/gpfs/mangalam/junk
Password:
file.154M                                  100%  153MB   2.2MB/s   01:10

Fast Data Transfer (fdt)

Fast Data Transfer is an application for moving data quickly writ in Java so it can theoretically run on any platform. The performance results on the web page are very impressive, but in local tests, it was slower than bbcp and the startup time for Java (as well as its failure to work in scp mode (couldn’t find the fdt.jar, even tho it was in the CLASSPATH, required you to explicitly start the receiving FDT server (not hard – see below, but another step)) argue somewhat against it.

Starting the server is easy; it starts by default in server mode:

java -jar ./fdt.jar
# usual Java verbosity omitted

The client uses the same jarfile but a different syntax:

java -jar ./fdt.jar -ss 1M -P 10 -c remotehost.domain.uci.edu  ~/file.633M  -d /userdata/hjm

# where
# -ss 1M  ..... sets the TCP SO_SND_BUFFER size to 1 MB
# -P 10 ....... uses 10 parallel streams (default is 1)
# -c host ..... defines the remote host
# -d dir ...... sets the remote dir

The speed is certainly impressive. Much more than scp:

# scp done over the same net, about the same time

$ scp file.4.2G  remotehost.domain.uci.edu:~
hjm@remotehost's password: ***********
 file.4.2G                   100% 4271MB  25.3MB/s   02:49
                                          ^^^^^^^^

# using the default 1 stream:
$ java -jar fdt.jar -c remotehost.domain.uci.edu ../file.4.2G -d /userdata/hjm/
[transferred in 86s for *53MB/s* ]

# with 10 streams and a larger buffer:
$ java -jar fdt.jar -P 10 -bs 1M -c remotehost.domain.uci.edu ../file.4.2G -d /userdata/hjm/
[transferred in 68s for *66MB/s* with 10 streams]

But fdt is slower than bbcp. The following test was done at about the same time between the same hosts:

bbcp -P 10 -w 2M -s 10 file.4.2G hjm@remotehost.domain.uci.edu:/userdata/hjm/
bbcp: Creating /userdata/hjm/file.4.2G
bbcp: At 081210 12:48:18 copy 20% complete; 89998.2 KB/s
bbcp: At 081210 12:48:28 copy 41% complete; 89910.4 KB/s
bbcp: At 081210 12:48:38 copy 61% complete; 89802.5 KB/s
bbcp: At 081210 12:48:48 copy 80% complete; 88499.3 KB/s
bbcp: At 081210 12:48:58 copy 96% complete; 84571.9 KB/s

GridFTP

If you and your colleagues have to transfer data in the range of multiple GBs and you have to do it regularly, it’s probably worth setting up a GridFTP site. Because it allows multipoint, multi-stream TCP connections, it can transfer data at mulitple GB/s. However, it’s beyond the scope of this simple doc to describe its setup and use, so if this sounds useful, bother your local network guru/sysadmin.

netcat

netcat (aka nc) is installed by default on most Linux and MacOSX systems. It provides a way of opening TCP or UDP network connections between nodes, acting as an open pipe thru which you can send any data as fast as the connection will allow, imposing no additional protocol load on the transfer. Because of its widespread availability and it’s speed, it can be used to transmit data between 2 points relatively quickly, especially if the data doesn’t need to be encrypted or compressed (or if it already is).

However, to use netcat, you have to have login privs on both ends of the connection and you need to explicitly set up a sender that waits for a connection request on a specific port from the receiver. This is less convenient to do than simply initiating an scp or rsync connection from one end, but may be worth the effort if the size of the data transfer is very large. To monitor the transfer, you also have to use something like pv (pipeviewer); netcat itself is quite laconic.

How it works: On the sending end, you need to set up a listening port:

[send_host]: $ pv -pet honkin.big.file | nc -q 1 -l -p 1234 <enter>

This sends the honkin.big.file thru pv -pet which will display progress, ETA, and time taken. The command will hang, listening (-l) for a connection from the other end. The -q 1 option tells the sender to wait 1s after getting the EOF and then quit.

On the receiving end, you connect to the nc listener

[receive_host] $ nc host.domain.uci.edu 1234 |pv -b > honkin.big.file <enter>

(note: no -p to indicate port on the receiving side). The -b option to pv shows only bytes received.

Once the receive_host command is inititated, the transfer starts, as can be seen by the pv output on the sending side and the bytecount on the receiving side. When it finishes, both sides terminate the connection 1s after getting the EOF.

This arrangement is slightly arcane, but supports the unix tools philosophy which allows you to chain various small tools together to perform a task. While the above example shows the case for a sinle large file, it can also be modified only slightly to do recursive transfers, using tar, shown here recursively copying the local sge directory to the remote host.

[send_host]: $ tar -czvf - sge | nc -q 1 -l -p 1234

[receive_host] $  nc host.domain.uci.edu 1234 |tar -xzvf -

In this case, I’ve added the verbose flag (-v) to the tar command so using pv is redundant. It also uses tar’s built-in compression flag (-c) to compress as it transmits.

You could also bundle the 2 together in a script, using ssh to execute the remote command. etc, etc, etc, etc.

Latest version of this Document

The latest version of this document should always be here.

R’s operational modes

R is a programming language designed explicitly for statistical computation. As such, it has many of the characteristics of a general-purpose language including iterators, control loops, network and database operations, many of which are useful, but in general not as easy to use as the more general Python or Perl

R can operate in 2 modes, as an interactive interpreter (the R shell) and as a scripting language, much like Perl or Python. Typically the R shell is used to try things out and the serial commands written in the R language and saved in a file are used to automate operations once the sequence is well-defined and debugged.

While it is not a great language for procedural programming, it does excel at mathematical and statistical manipulations. However, it does so in an odd way, especially for those who have done procedural programming before. R is quite object-oriented in that you tend to deal with data objects rather than with individual integers, floats, arrays, etc. The best way to think of data if you have programmed in C is to think of R data (typically termed tables or frames) as C structs, arbitrary constructs that can be dealt with by name. If you haven’t programmed in a procedural language, it may actually be easier for you. R manipulates chunks of data similar to how you might think of them. For example, “multiply that column by 3.54″ or “pivot that spreadsheet”.

For a still-brief but more complete overview of R, see Wikipedia’s entry for R.

Graphics and Graphical User Interfaces for R

R was developed as a commandline language. However, it has gained progressively more graphics capabilities and graphical user interfaces (GUIs). Some notable examples are the de facto standard R GUI R Commander, the elegant and powerful interactive graphics utility ggobi, and the fully graphical statistics package gretl which was developed external to R for time-series econometrics but which now supports R as an external module.

In addition, many routines in R are packaged with their own GUIs so that when called from the commandline interface, the GUI will pop up and allow the user to interact with a mouse rather than the keyboard.

As opposed to fully integrated commercial applications which have a cohesive interface, these packages differ slightly in the way that they approach getting things done, but in general the mechanisms follow generally accepted user interface conventions.

Getting help on R

After starting R, you can usually start R’s internal help with:

bash %  R  # more comments on R's startup state below
# R startup messages deleted
> help.start().

This will start a browser page with a lot of links to R documentation, both introductory and advanced.

If that help is not installed, most R help can be found in PDF and HTML at the R documentation page.

There is a very useful 100 page PDF book called: An Introduction to R (also in HTML). Especially note Chapter 1 and Appendix A on page 78, which is a more sophisticated, (but less gentle) tutorial than this.

Note also that if you have experience in SAS or SPSS, there is a good introduction written especially with this experience in mind. It has been expanded into a large book (to be released imminently), but much of the core is still available for free as R for SAS and SPSS Users

There is another nice Web page that provides a quick, broad intro to R appropriately called Using R for psychological research: A simple guide to an elegant package which manages to compress a great deal of what you need to know about R into about 30 screens of well-cross-linked HTML.

There is also a website that expands on simple tutorials and examples, so after buzzing thru this very simple example, please continue to the QuickR website

Thomas Girke at UC Riverside has a very nice HTML Introduction to R and Bioconductor as part of his Bioinformatics Core, which also runs frequent R-related courses which may be of interest for the Southern California area.

Start the R interpreter

bash %  R  # now that was simple, no?

# if you get the line shown below:

[Previously saved workspace restored]

you had saved the previous data environment and if you type ls() you’ll see all the previously created variables:

> ls()
 [1] "aa"          "data.matrix" "fil_ss"      "i"           "ma5"
 [6] "mn"          "mn_tss"      "norm.data"   "sg5"         "ss"
[11] "ss_mn"       "sss"         "tss"         "t_ss"        "tss_mn"
[16] "x"

If the previous session was not saved, there will be no saved variables in the environment

> ls()
character(0) # denotes an empty character (= nothing there)

NB: In many cases the data objects can be manipulated as you would files in the Unix shell, with the takes-some-getting-used-to of adding the almost-always-required parens afterwards:

> ls() # lists all data objects in view
> ls(someframe) # may give additional inforamtion on the frame
> rm(someframe) # deletes the 'someframe' data object

To determine what an object is, you can use a variety of methods, but one of the most useful is str (get the structure of an abject).

> str(aa)  # aa is a table or data.frame resulting from reading in a 25MB file.
'data.frame':   385238 obs. of  6 variables:
 $ ID    : Factor w/ 1649 levels "ENSXETG00000000007_SCAFFOLD_1356_47264_67263",..: 1 1 1 1 1 1 1 1 1 1 ...
 $ BEGIN : int  1 70 139 232 301 370 439 508 577 658 ...
 $ END   : int  50 119 188 281 350 419 488 557 626 707 ...
 $ Blue  : num  -0.67 -0.47 0.36 0.42 0.09 0.22 -0.42 0.61 0.71 1.61 ...
 $ Red   : num  2.8 2.56 2.96 -0.76 -2.6 -1.52 -0.36 -3.24 -1.8 -3.36 ...
 $ RedNeg: num  -0.7 -0.64 -0.74 0.19 0.65 0.38 0.09 0.81 0.45 0.84 ...

> str(y) # y is a vector of size 50
 num [1:50] -0.202 -0.418  0.325 -0.377 -0.405 ...

You can also use dim() to get the bare dimensions of the data structure. Always a good idea before you try to view it by referencing the data directly without limits.

> dim (aa)  # aa is a table or data.frame resulting from reading in a 25MB file.
[1] 385238      6

Some basic R data types

Before we get too far, note that R has a variety of data structures that may be required for different functions.

Some of the main ones are:

• numeric – (aka double) the default double-precision (64bit) float representation for a number.
• integer – a single-precision (32bit) integer.
• single – a single precision (32bit) float
• string – any character string (“harry”, “r824_09_hust”, “888764″). Note the last is a number defined as a string so you couldn’t add “888764″ + 765.
• vector – a series of data types of the same type. (3, 5, 6, 2, 15) is a vector of numerics. (“have”, “third”, “start”, “grep) is a vector of strings.
• matrix – an array of identical data types – all numerics, all strings, all booleans, etc.
• data.frame – a table (another name for it) or array of mixed data types. A column of strings, a column of integers, a column of booleans, 3 columns of numerics.
• list – a concatenated series of data objects.

Loading data into R

This process is described in much more detail in the document R Data Import/Export, but this will give you a short example of one of the most popular ways of loading data into R.

The following reads the example data file above (red+blue_all.txt) into an R table called aa; note the ,header=TRUE option which uses the file header line to name the columns. In order for this to work, the column headers have to be separated by the same delimiter as the data (usually a space or a tab).

> aa <- read.table(file="red+blue_all.txt",header=TRUE) # try it

Note that in R, you can work left to right or right to left, altho the left pointing arrow is the usual syntax. The above command could also have been given as :

> read.table(file="red+blue_all.txt",header=TRUE)  -> aa   # try it

Did we get the col names labeled correctly?

> names(aa)  # try it
[1] "ID"    "BEGIN" "END"   "Blue"  "Red"

Many functions will require the data as a matrix (a data structure whose components are identical data types, usually strings. The following will load the file into a matrix, doing the conversions along the way. This command also shows the use of the sep=”\t” option which explicitly sets the data delimiter to the TAB character.

Data <- as.matrix(read.table("red+blue_all.txt", sep="\t", header=TRUE))

Viewing data

To view some of the table, we only need to reference it. R assumes that a naked variable is a request to view it. An R table is referenced in [rows,column] order. That is, the index is [row indices, col indices], so that a reference like aa[1:6,1:5] will show a square window of the table; rows 1-6, columns 1-5

Also: R parameters are 1-indexed, not 0-indexed, tho it seems pretty forgiving (if you ask for [0-6,0-5], R will assume you meant [1-6,1-5]. also note that R requires acknowledging that the data structures are n-dimensional: to see rows 1-11, you can’t just type:

> aa[1:11] # try it!

but must instead type:

> aa[1:11,]   # another dimension implied by the ','

To slice the table even further, you can view only those elements that interest you

  # R will provide row and column headers if they were provided.
  # if they weren't provided, it will provide column indices.
> aa[3:12,4:5]  # try it.

Visualizing Data with ggobi

While there are many ways in R to plot your data, one unique benefit that R provides is its interface to ggobi, an advanced visualization tool for multivariate data, which for the sake of argument is more than 3 variables.

Operations on Data

To operate on a column, ie: to multiply it by some value, you only need to reference the column ID, not each element. This is a key idea behind R’s object-oriented approach. The following multiplies the column Red of the table aa by -1 and stores the result in the column ‘RedNeg. There’s no need to pre-define the RedNeg column; R will allocate space for it as needed.

> aa$RedNeg = aa$Red * -1   # try it

# the table now has a new column called 'RedNeg'
> aa[1:4,]      # try it

If you want to have the col overwritten by the new value, simply use the original col name as the left-hand value. You can also combine statements by separating them with a ;

> aa$Red = aa$Red * -2; aa[1:4,] # mul aa$Red by -2 and print the 1st 4 lines

To transpose (aka pivot) a table:

> aacopy <- aa # copy the table

> aac_t <- t(aacopy) # transpose the table

# show the all the rows & 1st 3 columns of the transposed table
> aac_t[,1:3]

Basic Statistics

Now, let’s get some basic descriptive stats out of the original table. 1st, load the lib that has the functions we’ll need

> library(pastecs)  # load the lib that has the correct functions
> attach(aa) # make the Red & Blue columns usable as variables.

# following line combines an R data structure by column.
# type 'help(cbind)' or '?cbind' for more info on cbind, rbind

> redblue <- cbind(Red,Blue)

# the following calculates a table of descriptive stats for both the Red & Blue variables
> stat.desc(redblue)
                       Red          Blue
nbr.val       3.852380e+05  3.852380e+05
nbr.null      2.486000e+03  2.188000e+03
nbr.na        0.000000e+00  0.000000e+00
min          -1.504000e+01 -5.130000e+00
max           2.256000e+01  5.340000e+00
range         3.760000e+01  1.047000e+01
sum          -6.929548e+04  3.060450e+03
median        4.000000e-02  0.000000e+00
mean         -1.798771e-01  7.944310e-03
SE.mean       3.826047e-03  1.115451e-03
CI.mean.0.95  7.498938e-03  2.186250e-03
var           5.639358e+00  4.793247e-01
std.dev       2.374733e+00  6.923328e-01
coef.var     -1.320198e+01  8.714826e+01

There is also a basic function called summary, which will give summary stats on all components of a dataframe.

> summary(aa)
                                               ID             BEGIN
 ENSXETG00000004313_SCAFFOLD_317_890727_910726  :   289   Min.   :    1
 ENSXETG00000010135_SCAFFOLD_646_44207_64206    :   289   1st Qu.: 4875
 ENSXETG00000017293_SCAFFOLD_132_1100669_1120668:   289   Median : 9722
 ENSXETG00000020596_SCAFFOLD_474_677230_697229  :   289   Mean   : 9781
 ENSXETG00000021861_SCAFFOLD_101_1701655_1721654:   289   3rd Qu.:14639
 ENSXETG00000023499_SCAFFOLD_486_295358_315357  :   289   Max.   :19950
 (Other)                                        :383504
      END             Blue                Red               RedNeg
 Min.   :   50   Min.   :-5.130000   Min.   :-15.0400   Min.   :-5.64000
 1st Qu.: 4924   1st Qu.:-0.470000   1st Qu.: -1.7600   1st Qu.:-0.39000
 Median : 9771   Median : 0.000000   Median :  0.0400   Median :-0.01000
 Mean   : 9830   Mean   : 0.007944   Mean   : -0.1799   Mean   : 0.04497
 3rd Qu.:14688   3rd Qu.: 0.480000   3rd Qu.:  1.5600   3rd Qu.: 0.44000
 Max.   :19999   Max.   : 5.340000   Max.   : 22.5600   Max.   : 3.76000

Introduction

Just as energy conservation is the most efficient mechanism for decreasing energy costs (see NegaWatts), the most efficient mechanism for saving money in the IT world is NOT paying for that which you do not need to pay for. You might call this saving NegaBIT$ (as in B undles of IT $ you don’t have to pay for).

One of the most obvious ways of not spending $ on software is by not paying for it – legally. Using Free/Libre/Open Source Software (FLOSS) has a number of advantages personally and organizationally, for both the short and especially the long term.

You don’t need to run Linux to take advantage of FLOSS; Windows and MacOSX are surprisingly amenable to running FLOSS, and with free Virtualization technology such as VirtualBox, you can run whatever Operating System (OS) you need to run whatever application you need (albeit a bit slower, and you may need to pay for the other OS).

The use of FLOSS makes using a computer cheaper, easier to maintain, less legally problematic, more friendly, and .. did I mention cheaper? Read on..

FLOSS on Different Operating Systems

Regardless of the Operating System you use you use (Mac, Windows, Linux), you can exploit FLOSS.

Windows

Despite what you might think, most FLOSS is used on Windows, the least free OS. That’s because there are more people using Windows than all other OS’s combined, creating a huge “market” for FLOSS and therefore programmers have started to address it. There are even web sites dedicated to enumerating Windows FLOSS. Among the standouts are the Firefox browser and Thunderbird email client, Pidgin IM, VLC and Miro video players, the OpenOffice Suite, Audacity for simple audio editing, GIMP for photo-editing, Blender for 3D modeling and animation are all examples of FLOSS that can be run on Windows.

MacOS X

The MacOS X is based on FLOSS and in fact you can download the basic OS (Darwin) for free. The GUI add-on bits and proprietary applications are what Apple charges for. Besides stand-alone FLOSS applications for the Mac, there are 2 systems that allow you easy access to thousands of the same FLOSS applications that run on Linux: the Ports system and fink.. These 2 systems use the same kind of installation architecture that the Linux apt system uses to handle and resolve the complications and dependencies of installing thousands of applications. Note however, that most of these are apps of the geeky variety: commandline, powerful and user-surly. There are some native Aqua apps such as NeoOffice (a Mac-native port of OpenOffice), the Smultron text editor, Cyberduck, as well as a number of the same apps as on Windows: Firefox/Thunderbird, Audacity, Blender, Gimp, VLC, etc (see above for links).

Linux

On Linux, as opposed to Windows and the Mac, you have to go looking for applications to pay for. Almost everything is free, and most of the default applications are of very high quality. I use Linux on my laptop and with the KDE desktop, I can point’n’click’n’drag’n’drop or use the commandline to do what I need. I use Firefox, Konqueror, and Opera for browsing , nedit for text editing, kate or jedit (Java) as a programming editor, Kontact for a combined POPmail client / calendar / RSS reader / contact manager, OpenOffice for when I have to view MS Office docs (works about as well as one version of MS Office reading another version of MSOffice), Skype & Gizmo for VOIPing, VNC (RealVNC or TightVNC) for remote control and desktop sharing, kpdf or Okular for viewing PDF docs (Adobe makes a Linux client, but the free ones are smaller, faster), digiKam or Picasa for direct downloading & handling the zillion photos I have, VLC or Mplayer and flash for video, Audacity for audio editing and Amarok for handling my music collection and podcasts.

Virtualization and Universal Applications

With free Virtualization technology, you can run any combination of Windows, MacOS X, and Linux simultaneously on the same computer. If you haven’t guessed, I use Linux. However, when I have to use Windows to debug a problem, I fire up the free VirtualBox virtual machine and run WinXP simultaneously. Windows in VirtualBox actually boots faster and seems faster than Windows running native on my laptop, with the exception that the graphics are noticably slower). Note that you do have to pay for the Windows or Mac OS that you run virtualized, altho you can run the WINE Windows environment for free.

There are also interpreted languages which allow programs written in that language to be run fairly easily on all platforms. The most widely known of such languages is Java, but Perl, Python, Ruby, and PHP all can be used across platforms fairly well.

Other Advantages of FLOSS

Scaling

You don’t pay for the software..at all. It’s free to download, to use, to give to your friends, and in many cases to modify or include in your own software if you’re so inclined (tho with caveats; the individual FLOSS licenses differ considerably). Whether you support your family or a large organization, the scaling advantages of FLOSS are hard to overstate.

If you learn to use FLOSS, you’ll find that you can do without a lot of commercial software. Those NegaBIT$ add up, not only from the initial non-payment, but especially if totaled up over a decade, or .. your lifetime. And for many such systems, even if they are initially primitive to the senses (vi/vim and emacs for examples), once you learn them, the interface doesn’t change (see Interface Changes & Featuritis below), they’re available on almost every system you’ll try and they are enormously powerful and robust. That is a tremendous advantage.

Upgrading the OS & Application Software

Upgrading is much easier. On my Linux system, a system-wide upgrade can be done with 2 lines in the terminal:

# the following updates all the versioning info

apt-get update

# the following upgrades the entire system, including kernel,
# kernel modules, applications, utilities, and all the supporting
# libraries, documentation, source code, and other software.

sudo apt-get dist-upgrade

And if you don’t like typing, the Synaptic GUI will let you do it with a few clicks.

Note that I didn’t have to go to individual web sites for applications, disable my anti-virus software, spend hours on the phone to Bangalore convincing some guy that I really did pay for my copy of Planet Blortfarg, carve chunks out of my credit card, have to spend hours checking why my drivers didn’t work anymore, or any of the things a Windows upgrade often entails. It just pretty much works.

Interface Changes & Featuritis

Often, I find that the interfaces (commandline or GUI) of FLOSS does not change as often as with commercial software. Like car manufacturers, proprietary SW Vendors have to make visual changes in their product at each release cycle to make you perceive that their product is changing for the better. But as anyone who has searched for an extra 10 minutes to find out where the subscript button has moved to can tell you, change is not always for the better. When the interface is changed semi-randomly like this, the vendor is telling you “We don’t care how your productivity may suffer. We have a new way of doing things and you’re going to learn it.” In FLOSS, quite often the interface will stay the same or quite similar to the previous version, but the guts will be improved (and those improvements will typically be listed in their accompanying change log).

Legality and Risk

If you use FLOSS, you escape forever the risk that a large, hungry, deep-pocketed software company will come looking to audit you for piracy or license abuse. This alone is a powerful argument for using FLOSS.

Note also that if you are involved in supporting software for a large organization, there are additional costs of managing a large roll-out of commercial software besides the unit costs. There are also the costs of negotiating license agreements, packaging and distributing the software, tracking the use and leakage of such software, and providing the accounting reports of such use. For a UCI-sized campus, there are probably 3-4 FTE-equivalents that are involved in this process.

Some Cautions

This may paint a rosier picture for FLOSS than I intend. Good software is always hard to write and maintain. That a software package is free does not assure high quality. However, if that software makes it into the Linux repositories, it has at least gotten numerous recommendations from users, has received a cursory evaluation for the distribution team and it has passed a compilation and compatibility test with the rest of the software. It may well not be up to your standards, but if that’s the case, you haven’t wasted anything but a few minutes in installing and testing it.

And uninstalling it is just as easy:

sudo apt-get remove [package name]

Where does FLOSS not work well?

FLOSS fails when there are software packages that you depend on that simply do not exist in the FLOSS world. The GIMP and Krita are both very good digital image editing programs, but overall not of the same quality or depth of Adobe Photoshop. So if you depend on the extra features or formats of Photoshop, you have no choice. However, like many such programs, the FLOSS equivalent is designed to address the features requested by most of its users and for most users, I bet that the GIMP is as much as they need. Or more than they need – most users would probably be happy with Picasa, a free (but not Open Source) application from Google.

While there are many excellent mathematics applications in the FLOSS world (Octave, Scilab, R, and sage, and many more) there’s nothing like Mathematica. It is one of the best arguments for commercial software. (But also note that Mathematica and MATLAB have Linux versions).

Integration can be more difficult. FLOSS is often written without regard for what other software an organization needs and therefore some cutting and glueing at the interface may be necessary. This is less troublesome than previously because of the near-universal use of XML which allows output formats to be parsed more easily. Similarly, connection to standard relational databases and the use of SQL can often help resolve integration problems.

Questions? Comments? Updates?

If you have comments, questions, criticisms, please let me know. I’m <harry.mangalam@uci.edu>

The latest version of this document can be found here.

Note

Assumptions

I’m assuming that you’re logged into a bash shell on a Linux system with most of the usual Linux utilities installed as well as R. You should create a directory for this excercise – name it anything you want, but I’ll refer to it as $DDIR for DataDir. You can as well by assigning the real name to the shell variable DDIR:

export DDIR=/the/name/you/gave/it

Shell commands are prefixed by bash > and can be moused into your own shell to test including the embedded comments (prefixed by #; they will be ignored.) Do not, of course include the bash > prefix.

Also, all the utilities described here will be available on the interactive BDUC cluster nodes at UC Irvine. Unless otherwise stated, they are also freely available for any distribution of Linux.

Introduction

If you’re coming from Windows, the world of the Linux command line can be perplexing – you have to know what you want before you can do anything – there’s nothing to click, no wizards, few hints. So let me supply a few…

I assume you’ve been forced to the Linux shell prompt somewhat against your will and you have no burning desire to learn the cryptic and agonizing commands that form the basis of xkcd and other insider jokes. You want get your work done and you want to get it done fast.

However, there are some very good reasons for using the commandline for doing your data processing. With more instruments providing digital output and new technologies providing LOTS (TBs) of digital data, trying to handle this data with Excel is just not going to work. And Comma Separated Value (CSV) files are probably not going to be much help either in a bit. But we can deal with all of these on Linux using some fairly simple, free utilities. The overwhelming majority of tools that you’ll use on Linux are free. There are also some good proprietary tools that have been ported to Linux (MATLAB, Mathematica, SAS, etc), but I’m going to concentrate on the free ones until I hit a roadblock. This is also not to say that you can’t do much of this on Windows, using native utilities and applications, but it’s second nature on Linux and it works better as well. The additional point is that it will ALWAYS work like this on Linux. No learning a new interface every 6 months because Microsoft or Apple need to bump their profit by releasing yet another pointless upgrade that you have to pay for in time and money.

There is a great, free, self-paced tutorial called Software Carpentry that examines much of what I’ll be zooming thru in better detail. The title refers to the general approach of Unix and Linux: use of simple, well-designed tools that tend to do one job very well (think saw or hammer). Unlike the physical tools, tho, the output of one can be piped into the input of another to form a suprisingly effective (if simple) workflow for many needs.

Showmedo is another very useful website, sort of like YouTube for Computer tools. It has tutorial videos covering Linux, Python, Perl, Ruby, the bash shell, Web tools, etc. And especially for beginners, it has a section related specifically to the above-referenced Software Carpentry series

OK, let’s start.

Getting your data to and from the Linux host.

This has been covered in many such tutorials, and the short version is to use WinSCP on Windows and Cyberduck on the Mac. I’ve written more on this, so go and take a look if you want the short and sweet version or the longer, more sophisticated version. Also, see the MacOSX note above – you can do all of this on the Mac as well, altho there are some Linux-specific details that I’ll try to mention.

OK, you’ve got your data to the Linux host… Now what..?

Simple data files – examination and slicing

What kind of file is it?

First, even tho you might have generated the data in your lab, you might not know what kind of data it is. While it’s not foolproof, a tool that may help is called file. It tries to answer the question: “What kind of file is it?” Unlike the Windows approach that maps file name endings to a particular type (filename.typ), file actually peeks inside the file to see if there are any diagnostic characteristics, so especially if the file has been renamed or name-mangled in translation, it can be very helpful.

ie

bash > file /home/hjm/z/netcdf.tar.gz
/home/hjm/z/netcdf.tar.gz: gzip compressed data, from Unix, last modified: Thu Feb 17 13:37:35 2005

# now I'll copy that file to one called 'anonymous'

bash > cp /home/hjm/z/netcdf.tar.gz  anonymous

bash > file anonymous
anonymous: gzip compressed data, from Unix, last modified: Thu Feb 17 13:37:35 2005

# see - it still works.

How big is the file?

We’re going to use a 25MB tab-delimited data file called red+blue_all.txt. Download it by clicking on the previous link. Save it to the $DDIR directory you’ve created for this excercise.

We can get the total bytes with ls

bash > mkdir ~/where/you/want/the/DDIR   # make the DDIR
bash > export DDIR=/where/you/made/the/DDIR  # create a shell variable to store it
bash >  cd $DDIR               # cd into that data dir
bash >  ls -l red+blue_all.txt
-rw-r--r-- 1 hjm hjm 26213442 2008-09-10 16:49 red+blue_all.txt
                     ^^^^^^^^

or in human-readable form with:

bash >  ls -lh red+blue_all.txt
#            ^
-rw-r--r-- 1 hjm hjm 25M 2008-09-10 16:49 red+blue_all.txt
#                    ^^^ (25 Megabytes)

We can get a little more information using wc (wordcount)

bash >  wc red+blue_all.txt
  385239  1926195 26213442 red+blue_all.txt

wc shows that it’s 385,239 lines 1,926,195 words and 26,213,442 characters

Native Spreadsheet programs for Linux

In some cases, you’ll want to use a spreadsheet application to review a spreadsheet. (For the few of you who don’t know how to use spreadsheets for data analysis (as opposed to just looking at columns of data), there is a Linux-oriented tutorial at the oft-referenced Software Carpentry site.)

While Excel is the acknowledged leader in this application area, there are some very good native free spreadsheets available on Linux that behave very similarly to Excel. There’s a good exposition on spreadsheet history as well as links to free and commercial spreadsheets for Linux here and a good comparison of various spreadsheets.

For normal use, I’d suggest either:

• OpenOffice calc (oocalc)
• Gnumeric (gumeric).

The links will give you more information on them and either will let you view and edit most Excel spreadsheets. In addition, there are 2 Mac-native version of OpenOffice: one a port from the OpenOffice group called:

• OpenOffice Aqua the X11 port of OpenOffice
• NeoOffice – a native fork of OpenOffice.

A NeoOffice-supplied comparison chart may be worth reviewing.

While both OpenOffice and MS Office have bulked up considerably in their officially stated capacity, often spreadsheets are not appropriate to the kind of data processing we want to do. So how to extract the data?

The first way is the most direct and may in the end be the easiest – open the files (oowriter for Word files, oocalc for Excel files) and export the data as plain text

For Word files:

File Menu > Save as... > set Filter to text (.txt), specify directory and name > [OK]

For Excel files:

File Menu > Save As... > set Filter: to text CSV (.csv), specify directory and name > [OK] >
   set 'Field delimiter' and 'Text delimiter' > [OK]

Or use the much faster method below.

Extracting data from MS Excel and MS Word files files

The above method of extracting data from a binary MS file requires a fair amount of clicking and mousing. The Linux Way would be to use a commandline utility to do it in one line.

Converting a Word file to text

For MS Word documents, there is a utility, appropriately called… antiword

antiword does what the name implies; it takes a Word document and inverts it – turns it into a plain text document:

bash > time antiword some_MS_Word.doc > some_MS_Word.txt

real    0m0.004s
user    0m0.004s
sys     0m0.000s

# took 0.004s!

bash > /usr/bin/time antiword some_MS_Word.doc > some_MS_Word.txt
0.00user 0.00system 0:00.00elapsed 80%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+16outputs (0major+262minor)pagefaults 0swaps

# or even more verbosely:
bash > /usr/bin/time -v  antiword some_MS_Word.doc > some_MS_Word.txt
        Command being timed: "antiword some_MS_Word.doc > some_MS_Word.txt"
        User time (seconds): 0.00
        System time (seconds): 0.00
        Percent of CPU this job got: 0%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 0
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 263
        Voluntary context switches: 1
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 16
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Extracting an Excel spreadsheet

There’s an excellent Excel extractor called py_xls2csv, part of the free python-excelerator package (on Ubuntu). It works similarly to antiword:

bash > py_xls2csv BodaciouslyHugeSpreadsheet.xls > BodaciouslyHugeSpreadsheet.csv

py_xls2csv takes no options and saves output with commas as the only separator, which is generally what you want.

If you want to do further data mangling in Python, the xlrd module is a very good Excel reader, but is not an entire utility by itself.

If you are Perl-oriented and have many Excel spreadsheets to manipulate, extract, spindle and mutilate, the Perl Excel-handling modules are also very good. Here is a good description of how to do this.

Viewing and Manipulating the data

Whether the file has been saved to a text format, or whether it’s still in a binary format, you will often want to examine it to determine the columnar layout if you haven’t already. You can do this either with a commandline tool or via the native application, often a spreadsheet. If the latter, the OpenOffice spreadsheet app oocalc is the most popular and arguably the most capable and compatible Linux spreadsheet application. It will allow you to view Excel data in native format so you can determine which cols are relevant to continued analysis.

If the file is in text format or has been converted to it, it may be easier to use a text-mode utility to view the columns. There are a few text-mode spreadsheet programs available, but they are overkill for simply viewing the layout. Instead, consider using either a simple editor or the text mode utilities that can be used to view it.

Editors for Data

Common, free GUI editors that are a good choice for viewing such tabular data are nedit, jedit, and kate. All have unlimited horizontal scroll and rectangular copy and paste, which makes them useful for copying rectangular chunks of data. Nedit and jedit are also easily scriptable and can record keystrokes to replay for repeated actions. Nedit has a backlighting feature that is sometimes helpful in debugging a data file, which can visually differ between tabs and spaces. Jedit is written in Java so it’s portable across platforms and has tremendous support in the form of various plugins.

Also, while it is despised and adored in equal measure, the xemacs editor can also do just about anything you want to do, if you learn enough about it. It’s as much a lifestyle as an editor.

Text-mode data manipulation utilities

The grep family

Possibly the most used utilities in the Unix/Linux world. These elegant utilities are used to search files for patterns of text called regular expressions (aka regex) and can select or omit a line based on the matching of the regex. The most popular of these is the basic grep and it, along with some of its bretheren are described well in Wikipedia. Another variant which behaves similarly but with one big difference is agrep, or approximate grep which can search for patterns with variable numbers of errors, such as might be expected in a file resulting from an optical scan or typo’s. Baeza-Yates and Navarro’s nrgrep is even faster, if not as flexible (or differently flexible), as agrep.

A grep variant would be used to extract all the lines from a file that had a particular phrase or pattern embedded in it.

For example:

bash > wc /home/hjm/FF/CyberT_C+E_DataSet
  600  9000 52673 /home/hjm/FF/CyberT_C+E_DataSet
# so the file has 600 lines.  If we wanted only the lines that had the identifier 'mur', followed by anything, we could extract it:

# (passed thru 'scut' (link:#scut[see below]) to trim the extraneous cols.)
bash > grep mur /home/hjm/FF/CyberT_C+E_DataSet | scut --c1='0 1 2 3 4'
b0085   murE    6.3.2.13        0.000193129     0.000204041
b0086   murF+mra        6.3.2.15        0.000154382     0.000168569
b0087   mraY+murX       2.7.8.13        1.41E-05        1.89E-05
b0088   murD    6.3.2.9 0.000117098     0.000113005
b0090   murG    2.4.1.- 0.000239323     0.000247582
b0091   murC    6.3.2.8 0.000245371     0.00024733

# if we wanted only the id's murD and murG:
bash > grep mur[DG] /home/hjm/FF/CyberT_C+E_DataSet | scut --c1='0 1 2 3 4'
b0088   murD    6.3.2.9 0.000117098     0.000113005
b0090   murG    2.4.1.- 0.000239323     0.000247582

cat

cat (short for concatenate) is one of the simplest Linux text utilities. It simply dumps the contents of the named file (or files) to STDOUT, normally the terminal screen. However, because it dumps the file(s) to STDOUT, it can also be used to concatenate multiple files into one.

bash > cat greens.txt
turquoise
putting
sea
leafy
emerald
bottle

bash > cat blues.txt
cerulean
sky
murky
democrat
delta

# now concatenate the files
bash > cat greens.txt blues.txt >greensNblues.txt

# and dump the concatenated file
bash > cat greensNblues.txt
turquoise
putting
sea
leafy
emerald
bottle
cerulean
sky
murky
democrat
delta

more & less

These critters are called pagers – utilities that allow you to page thru text files in the terminal window. less is more than more in my view, but your mileage may vary. These pagers allow you to queue up a series of files to view, can scroll sideways, allow search by regular expression, show progression thru a file, spawn editors, and many more things. Video example

head & tail

These two utilities perform similar functions – they allow you view the beginning (head) or end (tail) of a file. Both can be used to select contiguous ends of a file and pipe it to another file or pager. tail -f can also be used to view the end of a file continuously (as when you have a program continuously generating output to a file and you want to watch the progress). These are also described in more detail near the end of the IBM DeveloperWorks tutorial .

cut & scut

These are columnar slicing utilities, which allow you to slice vertical columns of characters or fields out of a file, based on character offset or column delimiters. cut is on every Linux system and works very quickly, but is fairly primitive in its ability to select and separate data. scut is a Perl utility which trades some speed for much more flexibility, allowing you to select data not only by character column and single character delimiters, but also by data fields identified by any delimiter that a regular expression (aka regex) can define. It can also re-order columns and sync fields to those of another file, much like the join utility see below. See this link for more info.

cols

cols is a very simple, but arguably useful utility that allows you to view the data of a file aligned according to fields. Especially in conjunction with less, it’s useful if you’re manipulating a file that has 10s of columns especially if those columns are of disparate widths. cols is explained fairly well here and the cols code is available here.

paste

paste can join 2 files side by side to provide a horizontal concatenation. ie:

bash > cat file_1
aaa bbb ccc ddd
eee fff ggg hhh
iii jjj kkk lll

bash > cat file_2
111 222 333
444 555 666
777 888 999

bash > paste file_1 file2

aaa bbb ccc ddd         111 222 333
eee fff ggg hhh         444 555 666
iii jjj kkk lll         777 888 999

Note that paste inserted a TAB character between the 2 files, each of which used spaces between each field. See also the IBM DeveloperWorks tutorial

join

join is a more powerful variant of paste that acts as a simple relational join based on common fields. join needs identical field values to finish the join. See that scut described above can also do this type of operation. For much more powerful relational operations, SQLite is a fully featured relational database that can do this reasonably easily (see below). A good example of join is here

pr

pr is actually a printing utility that is mentioned here because for some tasks especially related to presentation, it can join files together in formats that are impossible to do using any other utility. For example if you want the width of the printing to expand to a nonstandard width or want to columnize the output in a particular way or modify the width of tab spacing, pr may be able to do what you need.

Complex Binary Data storage and tools

While much data is available in (or can be converted to) text format, some data is so large (typically, >1 GB) that it demands special handling. Data sets from the following domains are typically packaged in these formats:

• Global Climate Modeling
• Stock exchange transactions
• Confocal images
• Satellite and other terrestrial scans
• Microarray and other genomic data

There are a number of specialized large-data formats, but I’ll discuss a popular large-scale data format called HDF5, which has now been merged with the netCDF data format. These can be thought of as numeric databases, tho they have significant functional overlap with relational databases. One advantage is that they have no requirement for a database server, an advantage they share with SQLite, below. As the wiki pages describe in more detail, these Hierarchical Data Formats are self-describing, somewhat like XML files, which enable applications to determine the data structures without external references. HDF5 and netCDF provide sophisticated, compact, and especially hierarchical data storage, allowing an internal structure much like a modern filesystem. A single file can provide character data in various encodings (ASCII, UTF, etc), numeric data in various length integer, floating point, and complex representation, geographic coordinates, encoded data such as base+offsets for efficiency, etc. These files and the protocols for reading them, assure that they can be read and written using common functions without regard to platform or network protocols, in a number of languages.

These files are also useful for very large data as they have parallel Input/Output (I/O) interfaces. Using HDF5 or netCDF4, you can read and write these formats in parallel, increasing I/O dramatically on parallel filesystems such as Lustre, PVFS2, and GFS. Many analytical programs already have interfaces to the HDF5 and netCDF4 format, among them MATLAB, Mathematica, R, VISIT, and others.

Tools for HDF and netCDF

As noted above, some applications can open such files directly, obviating the need to use external tools to extract or extend a data set in this format. However, for those time when you have to subset, extend, or otherwise modify such a dataset, there are a couple of tools that can make that job much easier. Since this document is aimed at the beginner rather than the expert, I’ll keep this section brief, but realize that these extremely powerful tools are available (and free!).

Some valuable tools for dealing with these formats:

• nco, a suite of tools written in C/C++ mainly by UCI’s own Charlie Zender. they were originally written to manipulate netCDF 3.x files but have been updated to support netCDF4 (which uses HDF5 format). They are described in detail here, with examples. They are portable to all current Unix implementations and of course Linux. They are extremely well-debugged and their development is ongoing.
• PyTables is another utility that is used to create, inspect, modify, and/or query HDF5 tables to extract data into other HDF or text files. This project also have very good documentation and even has a couple video introductions on ShowMeDo. They can be reached from here. In addition, PyTables also has a companion graphical data browser called ViTables.

Databases

Databases (DBs) are data structures and the supporting code that allow you to store data in a particular way. Some DBs are used to store only numbers (and if this is the case, it might be quicker and compact to store that data in an HDF5 file). If you have lots of different kinds of data and you want to query that data on complex criteria (give me the names of all the people who lived in the 92655 area code and have spent more than $500 on toothpicks in the last 10 years), using a DB can be extremely useful and in fact using a DB may be the only way to extract the data you need in a timely manner.

As implied above, there are a number of different types of DBs, delineated not only by the way they store and retrieve data, but by the way the user interacts with the DB. The software can be separated into Desktop DBs (which are meant to serve mostly local queries by one person – for example, a research DB that a scientist might use to store his own data) and DB Servers, which are meant to answer queries from a variety of clients via network socket.

The latter are typically much more complex and require much more configuration to protect their contents from network mischief. Desktop DBs typically have much less security and are less concerned about answering queries from other users on other computers, tho they can typically do so. Microsoft Access and SQLite are examples of Desktop DBs.

Relational databases

Relational DBs are those that can relate data from one table (or data structure) to another. A relational DB is composed of tables which are internally related and can be joined or related to other tables with more distant information. For example, woodworker’s DB might be composed of DB tables that describe PowerTools, Handtools, Plans, Glues, Fasteners, Finishes, Woods, and Injuries, with interconnecting relationships among the tables. The Woods table definition might look like this:

TABLE Woods (
    id INTEGER PRIMARY KEY,  # the entry index
    origin VARCHAR(20),      # area of native origin
    now_grown VARCHAR (100), # areas where now grown
    local BOOLEAN,           # locally available?
    cost FLOAT,              # cost per board foot
    density FLOAT,           # density in lbs/board ft.
    hardness INT,            # relative hardness scaled 1-10
    sanding_ease INT,        # relative ease to sand to finish
    color VARCHAR(20),       # string descriptor of color range
    allergy_propensity INT,  # relative propensity to cause allergic reaction 1-10
    toxicity INT,            # relative toxicity scaled 1-10
    strength INT,            # relative strength scaled 1 (balsa) to 10 (hickory)
    appro_glue VARCHAR(20),  # string descriptor of best glue
    warnings VARCHAR(200),   # any other warnings
    notes VARCHAR (1000),    # notes about preparation, finishing, cutting
    <etc>
);

See the Wikipedia entry on Relational Databases for more.

Desktop

• SQLite (website, wikipedia) is an amazingly powerful, ACID-compliant, astonishingly tiny DB engine (could fit on a floppy disk) that can do much of what much larger DB engines can do. It is public domain, has a huge user base, has good documentation (including a book) and is well suited to using for the transition from flat files to relational database. It has support for almost every computer language, and several utilities (such as graphical browsers like sqlitebrowser and knoda to ease its use. The sqlite3 program that provides the native commandline interface to the DB system is fairly flexible, allowing generic import of TAB-delimited data into SQLite tables. Ditto the graphical sqlitebrowser programs. Here is a well-documented example of how to use SQLite in a Perl script (about 300 lines including comments).
• OpenOffice comes with it’s own DB and DB interaction tools, which are extremely powerful, tho they have not been well-documented in the past. The OpenOffice DB tools can be used not only with its own DB, but with many others including MySQL, PostgreSQL, SQLite, MS Access, and any DB that provides an ODBC interface.

Server-based

• MySQL is a hugely popular, very fast DB server that provides much of the DB needs of the WWW, both commercial and non-profit. Some of the largest, most popular web sites in the world use MySQL to keep track of the web intereactions, as well as their data. The UC Santa Cruz Genome DB uses MySQL with a schema of >300 tables to keep one of the most popular Biology web sites in the world running.
• PostgreSQL is similarly popular and has a reputation for being even more robust and full-featured. It also is formally an Object-Relational database, so its internal storage can be used in a more object oriented way. If your needs require storing Geographic Information, PostgreSQL has a parallel development called PostGIS, which is optimized for storing Geographical Information and has become a de facto standard for GIS DBs. It supports the popular Mapserver software
• Firebird is the Open Source version of a previously commercial DB called Interbase from Borland. Since its release as OSS, it has undergone a dramatic upswing in popularity and support.
• Others. There are a huge number of very good relational DBs, many of them free or OSS. Wikipedia has a page that names a number of them and briefly describes some differences, altho unless you are bound by some external constraint, you would be foolish not to choose one of SQLite, MySQL, or PostgreSQL due to the vast user-generated support and utilities.

Data visualization

You almost always want to visualize your data. It’s one thing to page thru acres of spreadsheet or numeric data, but nothing can give you a better picture of your data than … a picture. This is an area where Linux shines, tho it’s not without scrapes and scratches. I’m going to break this into 2 arbitrary sections. The first is Simple the second Complex. “Simple” alludes to that both the process and data are relatively simple; “Complex” implies that both the visualization process and data are more complex.

Simple Data Visualization

• qtiplot is “a fully fledged plotting software similar to the OriginLab Origin software”. It also is multiplatform, so it can run on Windows as well as MacOSX. This is probably what researchers coming from the Windows environment would expect to use when they want a quick look at their data. It has a fully interactive GUI and while it takes some getting used to, it is fairly intuitive and has a number of useful descriptive statistics features as well. Highly recommended.
• quickplot is a more primitive graphical plotting program but with a very large capacity if you want to plot large numbers of point (say 10s or 100s of thousands of points.) It can also read data from a pipe so you can place it at the end of a data pipeline to show the result.
• gnuplot is one of the most popular plotting applications for Linux. If you spend a few minutes with it, you’ll wonder why. If you persist and spend a few hours with it, you’ll understand. It’s not a GUI program, altho qgfe provides a primitive GUI (but if you want a GUI, try qtiplot above. gnuplot is really a scripting language for automatically plotting (and replotting) complex data sets. To see this in action, you may first have to download the demo scripts and then have gnuplot execute all of them with gnuplot /where/the/demo/data/is/all.dem. Pretty impressive (and useful, as all the demo examples ARE the gnuplot scripts) It’s an extremely powerful plotting package, but it’s not for dilettantes. Highly recommended (if you’re willing to spend the time).
• pyxplot is a graphing package sort of like gnuplot in that it relies on an input script, but instead of the fairly crude output of gnuplot, the output is truly publication quality and the greater the complexity of the plot, the more useful it is. Recommended if you need the quality or typesetting functionality.
• gretl is not a plotting package per se but a fully graphical statistical package that incorporates gnuplot and as such is worth knowing about. It is quite intuitive and the statistical functions are an added bonus for when you want to start applying them to your data. It was originally developed with its own statistics engine but can start and use R as well. Highly recommended.
• The R statistical language has some very good plotting facilities for both low and high dimensional data. Like the Gretl program immediately above, R combines impressive statistical capabilities with graphics, although R is an interpreted language that uses commands to create the graphs which are then generated in graphics windows. There are some R packages that are completely graphical and there are moves afoot to put more effort into making a completely graphical version of R, but for the most part, you’ll be typing, not clicking. that said, it’s an amazingly powerful language and one of the best statistical environments available, commercial or free. Very highly recommended.

Visualization of Multivariate Data

• ggobi is on the border of “simple” and “complex”, but since it can be started and used relatively easily and has some very compelling abilities. I have to admit to being a ggobi fan for many years – it has some features that I haven’t seen anywhere else which, if your data is multivariate, really helps you to understand it. It can be run by itself, but its real power is obvious when you use it inside of R. That interface also obviates the grotacious requirement to code all your data into XML (R does it for you automatically). The ggobi has a very good website and documentation, and also has some very compelling demos and videos showing off its abilities. For an extended example of how to use it and some output, please check out the ggobi part of An R Cheat Sheet Highly recommended.
• ncview is a viewer for netCDF files. Since such files are typically used to map variables onto a physical space, this application provides a quick visualization of the data in a netCDF file mapped to the geographical or time coordinates.
• Integrated Data Viewer (IDV) “from Unidata is a Java™-based software framework for analyzing and visualizing geoscience data. The IDV brings together the ability to display and work with satellite imagery, gridded data, surface observations, balloon soundings, NWS WSR-88D Level II and Level III radar data, and NOAA National Profiler Network data, all within a unified interface.” It supports a fantastically wide set of data and can even be run (slowly) without installing it via Java’s Web Start. It can also interactively query remote data servers to provide filtered data from much larger data sets. It is strongly tailored to mapping geophysical data onto geographic maps, so is more of a GIS tool than a strictly multivariate visualizationj tool.
• ifrit, named for “Ionization FRont Interactive Tool” for that domain, has now become a much more general data visualization tool. It is used to visualize 4 main types of data:
  • Scalar data: several scalar variables in 3D space.
  • Vector field data: a 3D field of vectors.
  • Tensor field data: a symmetric 3×3 tensor in 3D space.
  • Particle data: a set of particles (points) with several optional attributes (numbers that distinguish particles from each other) per particle. In order to supply data to ifrit, you will probably have to do some editing of the data file header to tell ifrit the layout.
• VISIT is a very sophisticated fully graphical visualization tool that can provide plotting of just about every dataset that you can supply, and do so in very high quality (including making animations). It supports the HDF and netCDF formats mentioned above as well about 40 others. To provide this visualization capability, it’s interface is fairly complex as well, but if your data visualization needs are very sophisticated, this is the application for you. The web site includes a gallery of visualizations made with VISIT.

Programmatic manipulation

Sometimes your data is of a complexity that you’ll need to reduce or filter it before it can be used as input for statistical processing or visualization. If this is the case, you may be headed into the realm of actual programming. I refer you again to the excellent Software Carpentry tutorials and related videos on ShowMeDo.

For further data manipulation, I would suggest learning a scripting (or interpreted) language, such as Python, Perl, R, or Java for the following reasons.

Python

Python, iPython & NumPy. Python is a general purpose interpreted programming language and while not the most popular, it is quite suitable for research, especially when coupled with interactive iPython shell/debugger and the numerical module Numpy. Like Java (and unlike Perl) it is strongly object-oriented, which can make it easier to understand and extend. iPython is an interactive commandline debugger (there are also many free GUI debuggers such as eric), which has a number of features that make it very easy and powerful. The Numpy module is Python’s scientific numeric package which makes it very easy to deal with multidimensional arrays and their manipulation as well as wrapping libraries from other languages to include as Python modules – as an example, see HOWTO_Pythonize_FORTRAN.html. I’ve written a simple skeleton example in Python to show how Python does some useful things.

Perl

Perl & the Perl Data Language (PDL). Perl is also a powerful general purpose language, with probably the best integration of regular expressions. If parsing files, substituting strings, and manipulating text is your bag, then Perl is your sewing machine. The PDL is an additional set of routines that provides much the same functionality as Numpy does for Python. While both Perl and Python have their own set of self-updating and library-searching routines, Perl’s CPAN is more mature and better debugged than Python’s easy_install system, altho both are very good. Perl and Python are about equally well-supported when it comes to interacting with databases and network activity.

The R Statistical Language

R as alluded to above is an interpreted language that is designed explicitly for data manipulation and analysis. It has significant graphing and database capabilities built in and if you are a biologist, the R-based BioConductor suite is a very powerful set of modules for analyses like microarray data, genomics data mining, and even sequence analysis. It is not a general purpose programming language but for a researcher (not a web programmer), it would be a very good choice.

Java

Java is yet another interpreted, general purpose, object-oriented, programming language, but one which has great support in the area of web development and cross-platform programming. Most browsers have Java plug-ins which enable them to run Java code, so your applications can run in a browser, which can sometimes be advantageous. However, the language itself is not as intuitive or concise as Python or Perl and does not have the same strengths in numerical or scientific support, altho that could be my igorance showing.

Processing.org

processing.org is a free visualization Java toolkit that makes it quite easy to write code to manipulate data in 3D. If you have a visualization need that is not met by any of the above tools, you may want to play with this toolkit to see if you can develop it yourself. The language is quite simple and intuitive and the code examples given are well-documented and are executed in the provided programming environment. There are some interesting and beautiful examples of what it can do shown here.

Version Control for Code and Documents

This is a bit beyond the original scope of the document, but since I brought up writing your own code, version control is something you should know about. This Software Carpentry page describes what it is and why it’s important. READ IT!!

Versioning tools can, unsurprisingly, help you keep track of file versions. The two such OSS systems now in widespread and growing use are git and Subversion (aka svn). git was writ by a major software developer (Linus Torvalds, he of Linux fame and name) for a major software project (tracking the development of Linux itself). As such, it is very fast, efficient, and was written to encourage branching and merging of versions by a widespread network of asynchronously online group of developers. As such it is more like a peer-to-peer repository. svn was written with the same goals (developed by the team that previously wrote CVS, one of the most successful versioning systems before Subversion), but it is harder to branch and merge versions and has some different architectural features that lend it to a server implementation.

Note that these repositories can be used not only for code but for anything based on text – notes, grants, small amounts of data, email, etc. I know a few people who use subversion as their entire backup system. One in particular who uses Linux and Open Source tools for everything (including writing papers in TeX) has very few binary format files to back up so everything goes into svn.

Both the Software Carpentry site and Showmedo have pages dedicated to version control but mostly subversion. Setting up a subversion repository is not trivial, but learning how to use it can be tremendously valuable if you need or appreciate versioned files. Google Video has a couple of tech Talks dedicated to git, one by Linus Torvalds which is very entertaining but is not much of a HOWTO and another by Randall Schwartz which is more useful and describes more of what git can be used for.

Further Reading

There is a similar, if less complete (but possibly more articulate) tutorial called Simplify data extraction using Linux text utilities from the IBM DeveloperWorks series.

For a quick introduction to more sophisticated data analysis with R that builds on this document, please refer to An R Cheatsheet and the references therein.

Appendix

List of applications noted here:

• file – what kind of file is this?
• ls, wc – how big is the file
• OpenOffice.org – native GUI MS Office work-alike Word Processor (oowriter), Spreadsheet (oocalc), Database Tools (part of oocalc), Drawing (oodraw), Presentation (ooimpress) apps
• Gnumeric – native, GUI Spreadsheet
• NeoOffice – native MacOSX fork of OpenOffice
• antiword, py_xls2csv – extract data from MS Word, Excel binary files.
• nedit, jedit, kate, xemacs – GUI editors for data files
• cat – dump a file to STDOUT
• more, less – text file pagers
• head, tail – view or slice rows from teh top or bottom of a data file
• cut, scut – how to slice columns out of a data file
• cols – columnize delimited data files
• paste – glue 2 files together side by side
• join, scut – merge 2 files based on common keys
• Complex data formats
• nco, pytables – slice/dice, extract data from netCDF, HDF files
• Relational Databases
  • SQLite
  • OpenOffice Base
  • MySQL
  • PostgreSQL
  • Firebird
• Simple Data Plots
  • qtiplot
  • quickplot
  • gnuplot, qgfe
  • pyxplot
  • gretl
  • R
• Complex Data Visualization
  • ggobi & R
  • ncview
  • IDV
  • ifrit
  • VISIT
• Programmatic Data Manipulation
  • Python, iPython, Numpy
  • Perl, Perl Data language
  • R
  • Java
  • processing.org

Copyright

Introduction and Constraints

UCI, like many UC campuses, is facing the dual squeeze of decreasing IT budgets, and increasing licensing fees for our institutional Backup Systems. We also are facing somewhat more requirements from clients as more data is being gathered or generated, analyzed, and archived. In view of these pressures, the Office of Information Technology (OIT) is evaluating what our requirements are, and what Backup solutions can be used to more economically address our needs. We are evaluating both Proprietary and Open Source Software (OSS) approaches and it may be that the optimal solution is a combination of the 2.

Any Backup approach is guided by at least 2 issues:

1. The value of the data (or the cost of replacing it).
2. The cost of backing it up.

Much of the most valuable institutional data is stored on high-cost, high-reliability, highly-secured central servers. This makes backup fairly easy and most such devices have inherent or included data redundancy or data protection, making decisions about what backup system to use much easier (in some cases, there is no choice at all since the proprietary nature of the storage allows only the vendor’s implementation).

The situation is somewhat different for a university, which brings in much of its support $ in the form of overhead on grants. Such faculty-initiated grants brought in ~$328M in external funding last year for UC Irvine alone. Those grants were being composed on and still reside substantially on personal Laptops and Desktops scattered around the campus. The vast majority of them are backed up sporadically, if at all. I’ve had personal experience in trying to rescue at least 5 grants that were lost to disk crashes or accidental deletion days before the submisison date. That is valuable data and it would be exceptionally useful to be able to provide backup services to such users, even disregarding the somewhat less critical primary data from their labs. At UCI, this includes about 400 faculty who write for external grants on a regular basis. Any Backup system should minimally cover these people and therefore the scalability and Mac/Windows client compatibility of any such system is quite important.

Below is a summary of our current backup systems, EMC Networker and EMC Retrospect.

Networker Client Load

We currently use Networker to back up ~230 clients (mostly other servers) to 3 backup servers, with storage requirements as described graphically below:

and statistically here:

Sum       109326.2 ........ total GB
Number    227 ............. # clients
Mean      481.61 .......... GB / client
Median    155.9 ...........     "
Min       0.2 ............. GB on smallest client
Max       4651.2 .......... GB on largest client
Range     4651 ............ diff between 2 above
Variance  523643.27 ....... among all clients
Std_Dev   723.63 ..........        "
SEM       48.02 ...........        "
Skew      2.43 ............        "
Std_Skew  14.99 ...........        "
Kurtosis  6.97 ............        "

We currently use about 1/4 of an FTE to administer the Networker system after setup. The IAT recharge rates for this service is listed here, but in summary, it’s $20/mo/system plus $.39/GB for storage and a $40 charge for file restores.

Retrospect Client Load

Retrospect is a disk-only based system,

We currently pay $620/yr for this license and currently have 59 clients (18 Macs, rest Windows).

We currently charge $87.50/user/year with a 50GB limit on storage, with self-service file restores.

Open Source Backup Software Evaluation

There are perhaps 50 OSS Backup systems, but most of them are too limited in their features or maturity to be considered. Only 3 seem to rise to the level of possibility, tho for different things:

• Amanda – A good, neutral description from BackupCentral. Zmanda provides commercial suport and additional code goodies.
• Bacula – A good neutral description from BackupCentral. Bacula Systems provides support.
• BackupPC – A good neutral description from BackupCentral.

Amanda, Bacula, and BackupPC share these characteristics:

• All have Commercial support available now or imminently.
• The OSS version is of course, free for unlimited number of servers and clients.
• All can backup to disk.
• All have Open Source versions (altho Amanda and Bacula have $ versions that have additional, non-OSS goodies provided.
• All can support MacOSX, Windows, *nix clients via some combination of rsync, samba, semi-proprietary protocol (Bacula protocol is OSS, but only Bacula uses it it)
• All can support 10s-100s of clients per server.
• None currently have support for the NDMP protocol (tho Bacula and Zmanda are planning it)
• None support transparent Bare Metal Restore
• None support duplicate on backup
• None natively support Snapshots altho all can be implemented on Solaris to provide a number of ZFS features such as: snapshots, filesystem compression, slightly better I/O, etc. However, while Solaris is certainly stable, there are still aspects of ZFS that seem to be causing problems. (To ease a transition from Linux to Solaris, Nexenta is a free distribution of Solaris packaged as Ubuntu).

Amanda/Zmanda

• useful for separate instances of backup servers servicing sets of clients (star config)
• store to tape or disk; can use Barcode writers, readers to generate, process labels
• Amanda uses std unix text-based config files; seems to be more easily configurable than Bacula, tho less so than BackupPC (configured by Web GUI or text files)
• very flexible backup scheduling mechanism
• uses open & common storage protocols (tar, dump, gzip, compress, etc)
• very secure by on-wire & and on-disk encryption
• CLI or GUI available.
• mature (16yr), extremely well-reviewed C++ source code. Zmanda says that they are re-writing it in Perl for easier maintenance and to encourage more external contributions.
• very large user base
• Zmanda extension for backing up MySQL.
• uses own internal datastructures, so no additional DB instance required.
• Zmanda-supplied technical PDF about current and near-future options (NDMP, others)
• Commercial support costs are as described here.

Bacula

• Commercial support available
• store to tape or disk
• allows multiple servers to service multiple clients simultaneously, allowing a much larger single instance
• uses SQLite, MySQL or PostgreSQL as DB (all well-supported & understood), but additional complexity, and DB is therefore a critical link.
• published storage code, but not as widely used as Amanda
• Commercial support costs are as described here.

BackupPC

• Zmanda will be providing commercial support for BackupPC very soon, probably ~$10/client for large academic institutions.
• store only to disk, not tapes; therefore not useful for long-term archiving, unless willing to buy appropriately large hardware.
• support for client restores
• UCI-written support for client self-registration.
• rsync support depends on Perl implementation of rsync so it lags the most recent rsync features by a few revisions.
• supports file de-dupe via hard links, but hard links limit the storage to 1 filesystem (but XFS, ext4 support up to >= 1-8 EB).
• uses file tree, not DB, as the data structure; simpler but more primitive.
• uses no proprietary or application-specific client software; therefore very simple to implement and robust for restores.
• supports encrypted data transfer for MacOSX, Lin, but not (easily) for Windows (encryption requires use of ssh public key from server and ssh remote execution of windows executables).

Some Feature Comparisons

Nice Table of Feature Comparisons among OSS Backup packages and proprietary Backup packages

Crude Measures of popularity

Via Google-linking (a very crude measure; very sensitive to key words)

• Amanda/Zmanda: 411,000 google links; 460 pages linking to zmanda.com; 252 linking to amanda.zmanda.com
• Bacula: 402,000 google links; 468 pages linking to www.bacula.org.
• BackupPC: 267,000 google links; 5,070 pages link to backuppc.sourceforge.net; 3,140 pages link to backuppc.sf.net

And for some comparison:

• Vertitas: 255,000 for veritas backup software; 5,510 linking to http://www.symantec.com (all products)
• Networker: 66,500 for EMC networker; 2,370 linking to emc.com (all products)

Google Trends indicates that the Search Volume Index is decreasing rapidly for Veritas, is holding constant for EMC Networker, Bacula and BackupPC, but Bacula is ~3x the value of BackupPC, which is itself 2x EMC Networker. Veritas has decreased to just above BackupPC. Zmanda only started in 2006, and does not have much of an index built up, but it show significant spikes in News Reference Volume. “amanda backup” has been decreasing from 2004 and remains about 1/2 of BackupPC and 1/6 of Bacula.

Future Projects

I would like to see the automatic backup of all of our faculty’s Desktops/Laptops (\~1000) to shield against catastrophic loss of recent data, but I’m not sanguine about the chances for this due to funding issues, unless we go with a pure OSS solution, which would be considerably better than nothing at all.

Details

At 10GB per faculty, this would mean a storage server of ~20TB (now a medium-sized file server), and at 1% data changing per day, that means that on the order of 100GB a day would have to be transferred for incremental backups. At 7 MB/s (a decent transfer rate over 100Mb), transmission time is only ~4 hrs, easily done in a night, in parallel sessions. Measured on an Opteron backup server (doing server-side compression & file deduping via hardlinking), it takes about 25% of a CPU to handle 1 backup session, so a 4-core machine could theoretically handle ~16 simultaneous backups, if the bandwidth can supply it with enough data. Our test backup server is currently single-homed, but has 5 interfaces, so could easily be multi-homed. If we use OSS Backup software, it will cost \~$10K for hardware to provide 1000 very valuable PCs or laptops with at least protection against catastrophic loss.

Considerations for any such decision

Please feel free to expand on (or critique) these points.

Clients

• what features of current clients are actually used? (Do you need a wiki or calendaring function in your backup software? – why pay for unused features?)
• how many clients currently served?
  • how many would you like to serve?
• OS distribution of clients
• minimum backup cycle
• how much data per client per cycle
  • max data accepted, if such a limit
• do clients need to be able to initiate their own backups?
• client email notification of problems or email just to admin?
• mechanism of backup (rsync or propr.)
• compressed on client or server (open or propr.)
• client upgrades done from server, or client involvement
• preferred client registration (self or admin registration?)
• support of mobile client IP’s or static IP only?
• local nets only or internet support (backup/restore in Europe?)
• special application backups required? MS Exchange, RDBMSs, CMSs
• does timing of backup have to be client-adjustable or just at night or doesn’t matter?
• typical client ethernet type and number of hops to server
• need to support secure backup over wireless?
• can open files be backed up? Win/Lin/Mac ⁺
• do you need to back up encrypted filesystems and if so, how many features will work with such filesystems?
• are clients cluster aware? Can they back up to a cluster or one of a set of distributed servers? ⁺
• if the software backs up Windows servers, is it SharePoint-aware? Exchange-aware? ⁺
• if the software backs up Windows servers, can it do hot reinserts into Active Directory? For example, can it restore a single security group deleted from a domain controller or a single mailbox on an Exchange server? ⁺
• can users recover files themselves? If so, are they properly restricted to only recovering files they own? ⁺

Server & Admin

• what features of the server are actually used. (why pay for unused features?)
• how many servers are required per 100 clients?
• what is the FTE setup, configuration, and support requirements for the server & each type of client?
• do servers stage-to-disk before writing to tape? ⁺
• if multiple servers, are they synchronized or independent?
• type of server required (OS, CPUs, RAM, used for other services?)
• is there an institutional inability to deal with a particular platform?
• how many net interfaces per server are typically being used?
• data format on server (open, propr.)
• storage (tape / tape robot, disk, hierarchical)
• type of server admin interface used, preferred (native GUI, Web, CLI)
• database, other support software required (OSS, propr.)
• how are server upgrades done?
• does a feature of the backup server require or obviate a specific filesystem type?
• can you have (do you need) standby servers?
• is the software database server-aware? For example, can it backup Oracle databases without having to take the databases offline? ⁺
• Does it support direct fiber/fiber-aware backups? ⁺

Backup Protocol

• what kinds of backups are required (disaster recovery, archival, incremental, snapshots)
• network protocol used, preferred
• encryption requirements
• type of data compression used, preferred – format, block-level de-duping? file-level de-duping?, with HW accel or just software?
• is dedupe technology required? Versus just buying more storage media? If needed, what level is needed? (compression of individual files, hard links to files, proprietary dedupe technology, target or source-based dedupe, etc)
• does the software deduplicate to tape? Or does it reconstitute the data, then write it to tape? ⁺
• does the de-duplication component (if the backup software has one) write both the deduplicated data and the de-duplicate table to physical tape? (This obviates the need to have the same backup server to reconstitute the data.) ⁺
• if using rsync, which version? Version 3x has significant advantages over 2x. ⁼

Cost, Cost Efficiency, & Legal Concerns

• what is the cost per user for the system?
• what is the coverage of the various systems at different price points?
• what is the tradeoff of moving up one curve and down another? (for example, the cost of using inefficient dedupe technology and compensating with more storage, vs using efficient, but proprietary dedupe technology)
• what is the institutional cost of using technology that does not cover all clients equally vs the cost of not covering them at all?
• what features in the proprietary software will actually be used? (Since we have such systems, we should be able to document this at least for Networker and Retrospect)
• what is cost and timing of scaling? If we have to provide backups for a new department suddenly, what is the $ cost and the time lag?
• if it’s possible to go out of compliance, what is the cost of doing so?
• what legal recourse is there if the system fails in some way?

Feedback and Suggestions

We are still in a preliminary mode for this evaluation, but if you have suggestions, queries, or would like to be notified of the final result and sent any documentation of the process, please let me know.

Contributed Suggestions

Resources

Books

The O’Reilly book Backup & Recovery: Inexpensive Backup Solutions for Open Systems is a good overview of some important considerations for a backup system. UC people can read it in its entirety here via O’Reilly Safari

Note that O’Reilly itself uses a combination of commercial and Open Source tools.

Web sites

Curtis Preston, the author of the above book runs a backup-related site called BackupCentral, which is a very good info clearing house / blog on all things backup.

SearchDataBackup is another backup-related site.

Whitepapers

Of variable quality, some sponsored by vendors.

Backup on a Budget (PDF) – by the ubiquitous Curtis Preston. Reiteration of many of his points, especially pointed to the fact that for many organizations, backup is not rocket science, people are the most expensive thing you pay for, and that backup to clouds may be useful (but mostly in rare occassions).

Useful(?) individual pages

fwbackups

Via TechRepublic’s 10 outstanding Linux backup utilities. Very short list, with some interesting choices. fwbackups is a very slick little program writ in Python/GTK that can work on all platforms (but GTK on Windows requires the whole GTK lib). It’s not an enterprise system, but if you have a Personal Linux box to back up it’s pretty straightforward, and can use rsync/ssh to encrypt over the wire. However, it does require your own shell login and dedicated dir space on a server.

Boxbackup is not in the running for an enterprise backup system, but is interesting for near-realtime backup for a small group of clients.

Latest version

The latest version of this document should always be here.

How to let us know what’s wrong

Since BDUC is a research cluster, it is in perpetual flux as apps, libraries, and modules are added, updated, or modified so sometimes a bug will creep in where none existed before. When you find something missing or a behavior that seems odd, please let us know. You can email the BDUC admins here.

Note that it will help considerably if you tell us more than It doesn’t work, or I can’t log in. If you want quick resolution of the problem, please send us as much relevant info as possible, including a description of what triggered the misbehavior. If the misbehavior involves an error message, doing a Google search on that error message verbatim will often produce the answer.

While many of you are not programmers, you’re dealing with programs, and if we are to have any hope of debugging the process that caused the failure, the more info the better (usually). PLEASE READ How to Report Bugs Effectively before you report a failure. At least glance at it.

If you’re going to spend a lot of time with computers, you should also read Eric Raymond’s encyclopedic How To Ask Questions The Smart Way. It will be of use thru your life.

What is a BDUC?

The Broadcom Distributed Unified Cluster (BDUC) is, as the name suggests, a distributed group of clusters unified by running under a single <a href="http://moo.nac.uci.edu/_{hjm/Sun_Grid_Engine_62_install_and_config.pdf”>Sun Grid Engine (SGE) Resource Manager. BDUC consists of subclusters of 2-48core AMD64 Opteron nodes (for a total of about 380cores) running 64bit Linux. One group of 40 nodes is in the NACS Academic Data Center and another of 40 nodes is in the ICS data center, for a total of} 125 nodes / ~380 cores. There is another smaller subcluster (see BEAR, below) running Kubuntu.

The nodes are interconnected with 1Gb ethernet and have the MPICH and MPICH2 environments for parallel jobs. All the nodes share a common /home which is on a RAID6 system but which is NOT backed up. If you generate valuable data, you should move it off ASAP.

What is a BEAR?

The Broadcom EA Replacement (BEAR) is a Broadcom-supplied subcluster consisting of 7 larger nodes administered especially for interactive use. These nodes each have 4-8 Opteron285/2.6GHz cores and 32-64GB RAM. They run the 64bit Kubuntu (10.04.1) Desktop Edition, so you can have access to the full graphical KDE desktop via VNC, as well as the individual GUI applications and shell utilities. BEAR is fully integrated with BDUC and shares its /home directories, but has a different, larger set of applications. One of the nodes (claw1) is reserved for interactive use; the others can be used for both interactive and batch runs (currently limited to 48hrs) on the claws Q.

You can compile and run jobs on all the claw nodes, but don’t saturate claw1 with multiple serial or parallel jobs.

How do I get an account?

You request an account by sending a message including your UCINetID to <bduc-request@uci.edu>. Please let us know in that message if you want to use the SGE batch system to submit long-running or multiple jobs. You should get an acknowledgement within a few hours and your account should be available then. By default, BDUC & BEAR are open to all postgrad UCI researchers, altho it will be available to undergrads with faculty sponsorship.

How do I connect to BDUC?

You must use ssh, an encrypted terminal protocol. Be sure to use the -Y or -X options, if you want to view X11 graphics (see below).

On a Mac, use the Applications → Utilities → Terminal app.
On a WinPC, use the excellent putty. See also below.
On Linux, I assume that you know how to start a Terminal session with one of the bazillion terminal apps (konsole & terminator are 2 good ones).

Telnet access is NOT available. Use your UCINetID and associated password to log into the login node (bduc-login.nacs.uci.edu) via ssh.

To connect using a Mac or Linux, open the Terminal app and type:

ssh -Y UCINetID@bduc-login.nacs.uci.edu
# the '-Y' requests that the X11 protocol is tunneled back to you inside of ssh.

As of June 15th, 2009, you can also ssh directly to the claw1 node for a 64bit interactive node from anywhere on campus.

ssh -Y UCINetID@bduc-claw1.nacs.uci.edu

How to set up passwordless ssh

ssh errors

Occasionally you may get the error below when you try to log into BDUC (or more rarely, among the BDUC nodes):

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
 Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
93:c1:d0:97:e8:a0:f5:91:13:89:7d:94:6c:aa:9b:8c.
 Please contact your system administrator.
Add correct host key in /Users/joeuser/.ssh/known_hosts to get rid of this message.
Offending key in /Users/joeuser/.ssh/known_hosts:2
RSA host key for bduc.nacs.uci.edu has changed and you have requested strict checking.
 Host key verification failed.

The reason for this error is that the computer to which you’re connecting to has changed its identification key. This might be due to the mentioned man-in-the-middle attack but is far more likely to be an administrative change that has caused the BDUC node to have changed its ID. This may be due to a change in hardware, reconfiguration of the node, a reboot, an upgrade, etc.

The fix is buried in the error message itself.

Offending key in /Users/joeuser/.ssh/known_hosts:2

Simply edit that file and delete the line referenced. When you log in again, there will be a notification that the key has been added to your known_hosts file.

Should you want to be able to log in regardless of this warning, you’ll have to edit the /etc/ssh/ssh_config file and add the 2 lines as shown below. (Macs, Linux). There are good reasons for not doing this, but it’s a convenience that many of us use. Consider it the rolling stop of ssh security.

Host *
        StrictHostKeyChecking ask

After you do that, you’ll still get the warning (which you should investigate) but you’ll be able to log in.

If you’re using putty on Windows, you won’t be able to effect this security skip-around. Read why here.

After you log in…

Logging in to bduc.nacs.uci.edu will give you access to a Linux shell (bash by default, tcsh, ksh available).

You will also have access to the resources of the BDUC via the SGE commands. The most frequently used commands for SGE will be qrsh to request an interactive node and qsub to submit a batch job. You can also check the status of various resources with the qconf command. See the SGE cheatsheet for more detail.

The login node should be considered your 1st stop in doing real work. You can copy files to and from your home directory from the login node, but you shouldn’t run any long (>10m) jobs on the login node. If you do and we notice, we’ll kill them off. To do real work, request a node from the interactive queue, like this:

# for a 64bit interactive node
hmangala@bduc-login:~ $ qrsh -q int

# wait a few seconds...

hmangala@bduc-amd64-2:~

#or you can ssh directly to one of the claw nodes:

ssh claw1

Data Storage on BDUC

No limits, but no warnings either

We have not yet imposed disk quotas on BDUC. We encourage you to use the data storage you need, up to hundreds of GB, but we also warn you that if we detect large directories that have not been used in weeks, we retain the right to clean them out. The larger the dataset, the more scrutiny it will get. IF YOU HAVE LARGE DATASETS AND ARE NOT USING THEM, THEY MAY DISAPPEAR WITHOUT WARNING. We mean it when we say that if you generate valuable data, it is up to you to back it up elsewhere ASAP.

If you have no idea of how large your data is and how it is distributed, you can find out via the du command (disk usage).

$ cd /home
$ du -sh hmangala   # you would substitute *your* home dir
5.3G    hmangala/

To see the distribution of files graphically,

$ cd; ssh -Y claw1 'kdirstat'

This will launch kdirstat which will determine the size, type and age of your files and present them in a color-coded map by size. You can then inspect and hopefully remove the ones least needed.

How do I get my files to and from BDUC?

This is covered in more detail in the document HOWTO_move_data. There are currently a few ways to get your files to and from BDUC. The most direct, most available way is via scp. Besides the commandline scp utility bundled with all Linux and Mac machines, there are GUI clients for MacOSX and Windows, and of course, Linux. If you have large collections of files or large individual files that change only partially, you might be interested in using rsync as well.

interactive=`echo $- | grep -c i `
if [ ${interactive} = 1 ] ; then
  # put all your intereractive stuff in here:
  # ie tell me what my 22 latest files are
  ls -lt | head -22
fi

Windows

The hands-down, no-question-about-it, go-to utility here is the free WinSCP, which gives you a graphical interface for SCP, SFTP and FTP.

$ dos2unix windows.file linux.file

MacOSX

There may be others but it looks like the winner here is the oddly named, but freely available Cyberduck, which provides graphical file browsing via FTP, SCP/SFTP, WebDAV, and even Amazon S3(!).

Linux

The full range of high-speed net data commandline utilities are available via the above-referenced HOWTO_move_data, however, for ease of use, it may well be easiest to use the built-in capabilities of KDE’s Swiss Army knife browser Konqueror or twin panel file manager Krusader which both support the secure file browser kio-plugin called fish. If you use a fish URL, you can connect the server via shared keys or via password:

fish://hmangala@bduc.nacs.uci.edu

archivemount

Once you’ve generated some data on BDUC, you may want to keep it handy for a short time while you’re further processing it. In order to keep it both compact and accessible, BDUC supports the archivemount utility on both the login and claw1 nodes. This allows you to mount a compressed archive (tar.gz, tar.bz2, and zip archives) on a mountpoint as a fuse filesystem. You can cd into the archive, modify files in place, copy files out of the archive, or copy files into the archive. When you unmount the archive, the changes are saved into the archive. Here’s an extended article on it from Linux Mag.

Here’s an example of how to use archivemount with a 84MB data tarball (data.tar.gz) that you want to interact with.

# how big is this thang?
$ ls -lh
total 84M
-rw-r--r-- 1 hmangala hmangala 84M Jun 15 14:55 jksrc.zip

# OK - 84MB, which is fine.  Now let's make a mount point for it.

$ mkdir jk

$ ls
jk/  jksrc.zip

# so now we have a zipfile and a mountpoint.  That's all we need to archivemount
# let's time it just to see how long it takes to unpack and mount this archive:

$ time archivemount jksrc.zip jk

real    0m0.810s  <-  less than a second wall clock time
user    0m0.682s
sys     0m0.112s

$ cd jk      # cd into the top of the file tree.

# lets see what the top of this file tree looks like.  All file utils can work on this data structure
$ tree |head -11
.
`-- kent
    |-- build
    |   |-- build.crontab
    |   |-- dosEolnCheck
    |   |-- kentBuild
    |   |-- kentGetNBuild
    |   `-- makeErrFilter
    |-- java
    |   |-- build
    |   |-- build.xml
<etc>

# and the bottom of the file tree.
$ tree |tail
            |   |-- wabaCrude.h
            |   `-- wabaCrude.sql
            |-- xaShow
            |   |-- makefile
            |   `-- xaShow.c
            `-- xenWorm
                |-- makefile
                `-- xenWorm.c

2286 directories, 12793 files <- lots of files that don't take up anymore 'real' space on the disk.

# how does it show up with 'df'?  See the last line..

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md2             373484336  11607976 342598364   4% /
/dev/md1               1019144     47180    919356   5% /boot
tmpfs                  8254876         0   8254876   0% /dev/shm
/dev/sdc             12695180544 6467766252 6227414292  51% /data
bduc-sched.nacs.uci.edu:/share/sge62
                      66946520   8335072  55155872  14% /sge62
fuse                 1048576000         0 1048576000   0% /home/hmangala/build/fs/jk

# finally, !!IMPORTANTLY!! un-mount it.

$ cd ..   # cd out of the tree

$ fusermount -u jk    # unmount it with 'fusermount -u'

sshfs

sshfs is a utility that allows you to mount remote directories in your BDUC home dir. Since it operates in user-mode, you don’t have to be root or use sudo to use it. It’s very easy to use and you don’t have to alert us to use it..

You have to be able to ssh to the machine from which you want to exchange files, typically the desktop or laptop you’re connecting to BDUC from (ergo WinPCs cannot do this without much more effort). For MacOSX and Linux, in the example below assume I’m connecting from a laptop named ringo to the BDUC claw1 node. I have a valid BDUC login (hmangala) and my login on ringo is frodo.

sshfs works on both the login and claw1 nodes.

frodo@ringo:~ $ ssh bduc-claw1  # from ringo, ssh to BDUC with passwordless ssh

 # <BDUC login stuff deleted>

# make a dir named 'ringo' for the ringo filesystem mountpoint
hmangala@bduc-claw1:~  $ mkdir ringo

# sshfs-attach the remote filesystem to BDUC on ~/ringo
# NOTE: you usually have to provide the FULL PATH to the remote dir, not '~'
# using '~' on the local side (the last arg) is OK.
# ie: this is wrong:
# hmangala@bduc-claw1:~  $ sshfs frodo@ringo.dept.uci.edu:~ ringo
#                                                         ^
# the following is right:
hmangala@bduc-claw1:~  $ sshfs frodo@ringo.dept.uci.edu:/home/frodo ~/ringo

hmangala@bduc-claw1:~  $ ls -l |head
total 4790888
drwxr-xr-x   2 hmangala hmangala          6 Dec 10 14:17 ringo/  # the new mountpoint for ringo
-rw-r--r--   1 hmangala hmangala       3388 Sep 22 16:25 9.2.zip
-rw-r--r--   1 hmangala hmangala       4636 Dec  8 10:18 acct
-rw-r--r--   1 hmangala hmangala        501 Dec  8 10:20 acct.cpu.user
-rwxr-xr-x   1 hmangala hmangala        892 Nov 11 08:55 alias*
-rw-r--r--   1 hmangala hmangala        691 Sep 30 13:21 all3.needs

 <etc>         ^^^^^^^^^^^^^^^^^ note the ownership

# now I cd into the 'ringo' dir
hmangala@bduc-claw1:~  $ cd ringo

hmangala@bduc-claw1:~/ringo  $ ls -lt |head
total 4820212
drwxr-xr-x 1 frodo frodo       20480 2009-12-10 14:43 nacs/
drwxr-xr-x 1 frodo frodo        4096 2009-12-10 14:41 Mail/
-rw------- 1 frodo frodo          61 2009-12-10 12:54 ~Untitled
-rw-r--r-- 1 frodo frodo          42 2009-12-10 12:44 testfromclaw
-rw-r--r-- 1 frodo frodo      627033 2009-12-10 11:22 sun_virtualbox_3.1.pdf

#<etc>       ^^^^^^^^^^^ note the ownership.  Even tho I'm on bduc-claw1, the original ownership is intact

# writing from BDUC to ringo filesystem
hmangala@bduc-claw1:~/ringo  $ echo "testing testing" > test_from_bduc

hmangala@bduc-claw1:~/ringo  $ cat test_from_bduc
testing testing

hmangala@bduc-claw1:~/ringo  $ ls -lt |head
total 4820216
drwxr-xr-x 1 frodo frodo       20480 2009-12-10 14:47 nacs/
-rw-r--r-- 1 frodo frodo          16 2009-12-10 14:46 test_from_bduc
drwxr-xr-x 1 frodo frodo        4096 2009-12-10 14:41 Mail/
#            ^^^^^^^^^^^  even tho I wrote it as 'hmangala' on BDUC, it's owned by 'frodo'

# and finally, unmount the sshfs mounted filesystem.
hmangala@bduc-claw1:~/ringo $ fusermount -u ringo

# get more info on sshfs with 'man sshfs'

YOU are responsible for your data

We do not have the resources to provide backups of your data. If you store valuable data on BDUC, it is ENTIRELY your responsibility to protect it by backing it up elsewhere. You can do so via the mechanisms discussed above, especially with (if using a Mac or Linux) rsync, which will copy only those bytes which have changed, making it extremely efficient. Using rsync (with examples) is described here.

How do I do stuff?

On the login node, you shouldn’t do anything too strenuous (computationally). If you run something that takes more than a minute or so to complete, you should be running on an interactive node or submit it to one of the batch queues.

qrsh given alone will start an ssh -Y session with one of the nodes in the interactive Q.

Can I compile code?

We have the full GNU toolchain available on both the CentOS interactive nodes and on all the Ubuntu/claw nodes, so normal compilation tools such as autoconf, automake, libtool, make, ant, gcc, g++, gfortran, gdb, ddd, java, python, R, perl, etc are available to you. We do not yet have any proprietary compilers or debuggers available (ie. the Intel or PGC compilers or the TotalView Debugger). Please let us know if there are other tools or libraries you need that aren’t available.

Compiling your own code

You can always compile your own (or downloaded) code. Compile it in its own subdir and when you’ve built the executables, install it rooted from your own home directory.

If the code is well-designed, it should have a configure shell script in the top-level dir. The ./configure –help command should then give you a list of all the parameters it accepts. Typically, all such scripts will accept the –prefix flag. You can use this to tell it to install everything in your $HOME dir.

ie

./configure --prefix=/home/you ...other options..

This command, when it completes successfully will generate a Makefile. At this point, you can type make (or make -j2 to compile on 2 CPUs) and the code will be compiled into whatever kind of executable is called for. Once the code has been compiled successfully (there may be a make test or make check option to run tests to check for this), you can install it in your $HOME directory tree with make install.

ie

/home/you/bin
/home/you/man
/home/you/lib
/home/you/include
/home/you/share
<etc>

Then you can run it out of your ~/bin dir without interfering with other code. In order for you to be able to run it transparently, you will have to prepend your ~/bin to the PATH environment variable, typically by editing it into the appropriate line in your ~/.bashrc.

export PATH=~/bin:${PATH}

How do I find out what’s available?

Via the module command

We use the tcl-based environment module system to wrangle non-standard software versions and subsystems into submission. To find out what modules are available, simply type:

$ module avail
------------ /apps/Modules/modulefiles ------------
R/2.10.0               gromacs_s_mpich2/4.0.7 openmpi/1.4.3
R/2.11.0               gromacs_s_ompi/4.0.7   paml/4.4
R/2.12.1               gromacs_s_ompi/4.5.1   petsc/3.1-p8
R/2.8.0                gromacs_s_ompi/4.5.2   pgc/10.6
R/dev                  gromacs_s_ompi/4.5.3   picard/1.45
abyss/1.1.2            gromacs_s_ompi/4.5.4   plink/1.07
abyss/1.2.3            hadoop/0.20.2          python/2.6.1
abyss/1.2.5            haploview/4.1          rapidminer/5.1.001
abyss/1.2.6            hdf5/1.8.4p1           ray/1.3
allpathslg/36681       hdf5/1.8.5.p1          ray/1.4
amber/11               hmmer/3.0              readline/5.2
annovar/2010Jan17      hyphy/2.0              rosetta/3.1
antlr/3.2              igv/1.5.58             samtools/0.1.13
ants/1.9               imagej/1.41            samtools/0.1.7
autodock/4.2.3         interviews/17          scilab/5.1.1
bedtools/2.6.1         java/1.6               scilab/5.3.0
bfast/0.6.3c           loni_pipeline/5.1.4    scilab/5.3.1
blat/3.4               maestro/91207          sge/6.2
boost/1.410            maq/0.7.1              simset/2.9
bowtie/0.10.1          matlab/R2008b          soap/2.20
bowtie/0.12.3          matlab/R2009b          sparsehash/1.6
breakway/0.6           matlab/R2010b          sqlite/3.6.22
bwa/0.5.7              mgltools/1.5.4         ssaha2/2.5.3
cnver/0.7.2            modeller/9v7           stampy/1.0.12
cnver/0.8.1            mosaik/1.0.1388        stampy/1.0.9
cufflinks/0.8.1        mpich/1.2.7            subversion/1.6.9
edena/2.1.1            mpich2/1.1.1p1         sva/1.02
eigensoft/3.0          mpich2/1.2.1p1         tablet/1.11.01.25
enthought_python/6.3-2 msort/20081208         tacg/4.5.1
exonerate/2.2          namd/2.6               taverna/2.2.0
freesurfer/4.5.1Dev    namd/2.7b1             tcl/8.5.5
freesurfer/5.0.0       namd/2.8b1             tcl/8.5.9
fsl/4.1                ncl/5.1.1              tinker/5.1.09
fsl/4.1.6              nco/4.0.4              tk/8.5.5
gamess/2010R1          netcdf/3.6.3           tophat/1.0.13
gapcloser/20100125     netcdf/4.1.1           tophat/1.2.0
gatk/1.0.5336          neuron/7.0             triton/4.0.0
gaussian/3.0           nmica/0.8.0            velvet/1.0.13
gnu_parallel/20101202  nwchem/6.0             velvet/1.0.19
gnuplot/4.2.4          octave/3.0.1           velvet/1.1.02
gnuplot/4.5p1          octave/3.2.0           visit/1.11.2
gpu/1.0                open64/4.2.3           vmd/1.8.7
gromacs_d/4.0.7        openmpi/1.4            zdock/3.0.1
gromacs_s/4.0.7        openmpi/1.4.2

(current as of June 8, 2011)

To load a particular module, use the module load <module/version> command:

$ module load imagej/1.41  # for example

If a module has a dependency, it should set it up for you automatically. Let us know if it doesn’t. If you note that a module has an update that we should install, tell us.

Via the shell

This is a bit tricky. there are literally thousands of applications that are available and many of them have names that are entirely unrelated to their function. In order to determine whether a well-know application is already on the system, you can simply try typing its name. If it’s NOT installed or not on your executable’s PATH, the shell will return command not found.

All the interactive nodes have TAB completion enabled at least in the bash shell. This means that if you type a few characters of the name and hit <TAB> twice, the system will try to compete the command for you. If there are multiple executables that match those characters, the shell will present all the alternatives to you. ie

$ jo<TAB><TAB>
jobs        jockey-kde  joe         join

You can then complete the command or enter enough characters to make the command unique and hit <TAB> again and the command will complete.

Via the installer Database

The 2 installer databases (one for Ubuntu’s apt-get on the claw nodes, one for CentOS’s yum on the rest) will let you search all the applications that HAVE been installed and all those that CAN be installed.

To search for the ones that CAN be installed on the BEAR (claw1-4) nodes, use the command asrch (an alias for apt-get search). This searches thru all the application names and descriptions in a case-insensitve search to find a wide variety of names that match the pattern you give it. For example:

$ asrch biology
avida-base - Auto-adaptive genetic system for Artificial Life research
biomode - [Biology] An Emacs mode to edit genetic data
bioperl - Perl tools for computational molecular biology
   <41 lines deleted>
molphy - [Biology] Program Package for MOLecular PHYlogenetics
phylip - [Biology] A package of programs for inferring phylogenies
phylip-doc - [Biology] A package of programs for inferring phylogenies
treetool - [Biology] An interactive tool for displaying trees
tacg - [Biology] a sophisticated 'grep' for nucleic acid strings

To see a more detailed descripton of the application, use ashow (an alias for apt-get show), which will provide a few lines or paragraphs of text about the application:

$ ashow phylip
Package: phylip
Priority: optional
Section: multiverse/science
Installed-Size: 5792
Maintainer: Ubuntu MOTU Developers <ubuntu-motu@lists.ubuntu.com>
Original-Maintainer: Debian-Med Packaging Team <debian-med-packaging@lists.alioth.debian.org>
Architecture: amd64
Version: 1:3.67-2
Depends: libc6 (>= 2.4), libx11-6, libxaw7, libxt6
Suggests: phylip-doc
Filename: pool/multiverse/p/phylip/phylip_3.67-2_amd64.deb
Size: 2520650
MD5sum: eacef9de8503a21b90a05bfabea9fbca
SHA1: 61a2ec92c1b0699db07ea08196848e2f41f79a6c
SHA256: 3453f9b3bc9d418bf0c4941eb722e807a96ec32ac3a041df34ee569929bd19dc
Description: [Biology] A package of programs for inferring phylogenies
 The PHYLogeny Inference Package is a package of programs for inferring
 phylogenies (evolutionary trees) from sequences.
 Methods that are available in the package include parsimony, distance
 matrix, and likelihood methods, including bootstrapping and consensus
 trees. Data types that can be handled include molecular sequences, gene
 frequencies, restriction sites, distance matrices, and 0/1 discrete
 characters.
Homepage: http://evolution.genetics.washington.edu/phylip.html
Bugs: mailto:ubuntu-users@lists.ubuntu.com
Origin: Ubuntu

HOWEVER, this only tells you that the application or library is available, not whether it’s installed. To find out whether it’s installed, you use dpkg. dpkg -S pattern will tell you whether a package containing a file that has that pattern has been installed and what package provided it: ie

$ dpkg -S ifconfig
net-tools: /sbin/ifconfig
net-tools: /usr/share/man/man8/ifconfig.8.gz

The -l flag has a different meaning, but can also be useful

dpkg -l |grep -i python |less
 <lots of output - try it>

There is a GUI application called synaptic that provides a more pointyclicky interface but asrch and dpkg are much faster via the commandline.

To search for all possible applications and libraries on the BBUC nodes using yum, it’s similar:

$ yum search lapack
Loading "downloadonly" plugin
Loading "fastestmirror" plugin
Loading mirror speeds from cached hostfile
 * epel: mirror.hmc.edu
 * dag: apt.sw.be
 * atrpms: dl.atrpms.net
 * rpmforge: ftp-stud.fht-esslingen.de
 * base: centos.cogentcloud.com
 * updates: mirrors.usc.edu
 * lscsoft: www.lsc-group.phys.uwm.edu
 * addons: mirror.stanford.edu
 * extras: centos.promopeddler.com
lapack-devel.i386 : LAPACK development libraries
blas-devel.i386 : LAPACK development libraries
lapack.i386 : The LAPACK libraries for numerical linear algebra
blas-devel.i386 : LAPACK development libraries
   <10 lines deleted>
blas.i386 : The BLAS (Basic Linear Algebra Subprograms) library.
lapack-devel.i386 : LAPACK development libraries
blas.i386 : The BLAS (Basic Linear Algebra Subprograms) library.
R-RScaLAPACK.i386 : An interface to perform parallel computation on linear algebra problems using ScaLAPACK

To find out if a package has been installed:

$ yum list lapack-devel.i386
Loading "downloadonly" plugin
Loading "fastestmirror" plugin
Loading mirror speeds from cached hostfile
 * epel: mirror.hmc.edu
 * dag: apt.sw.be
 * atrpms: dl.atrpms.net
 * rpmforge: ftp-stud.fht-esslingen.de
 * base: mirrors.xmission.com
 * updates: mirrors.usc.edu
 * lscsoft: www.lsc-group.phys.uwm.edu
 * addons: centos.cogentcloud.com
 * extras: mirror.hmc.edu
Installed Packages
lapack-devel.i386                        3.1.1-1.el5.rf         installed

Via the Internet

Obviously, a much wider ocean to search. My first approach is to use a Google search constructed of the platform, application name, and/or function of the software. Something like

linux image photography hdr 'high dynamic range'  # '' enforce exact phrase

which yields this page of results.

Also, don’t be afraid to try Google’s Advanced Search or even Google’s Linux Search.

After evaluating the results, you’ll come to a package that seems to be what you’re after, pfstools, for example. If you didn’t find this in the previous searches of the application databases, you can look again, searching explicitly:

$ashow pfstools
 ...
Description: command line HDR manipulation programs
 pfstools is a set of command line (and two GUI) programs for reading,
 writing, manipulating and viewing high-dynamic range (HDR) images and video
 frames. All programs in this package exchange data using a simple generic
 file format (pfs) for HDR data. It is an attempt to integrate existing file
 formats by providing a simple data format that can be used to exchange data
 between applications.
 ...

and then you can ask an admin to install it for you. Typically the apps found in the application repositories lag the latest releases by a few point versions, so if you really need the latest version, you’ll have to download the source code or binary package and install it from that package. You can compile your own version as a private package, but to install it as a system binary, you’ll have to ask one of the admins.

Interactive Use

Logging on to an interactive node may be all that you need. If you want to slice & dice data interactively, either with a graphical app like MATLAB, VISIT, JMP, or clustalx, or a commandline app like nco or scut or even hybrids like gnuplot or R, you can run them from any of the interactive nodes, read, analyze and save data to your /home directory. As long as you satisfy the graphics requirements, you can view the output of the X11 graphics programs as well.

byobu and screen: keeping a session alive between logins

In most cases, when you log out of an interactive session, the processes associated with that login will also be killed off, even if you’ve put them in the background (by appending the starting command with &). If you regularly need a process to continue after you’ve logged out, you should submit it to the SGE scheduler with qsub (see immediately below).

However, sometimes it is convenient to continue a long-running process when you have to log out (as when you have to shut down your network connection to take your laptop home). In this case, you can use the underappreciated screen program, which establishes a long-running proxy connection on the remote machine that you can detach from and then re-attach to without losing the connection. As far as the remote machine is concerned, you’ve never logged off, so your running processes aren’t killed off. When you re-establish the connection by logging in again, you can re-attach to the screen proxy and take up as if you’ve never been away.

You can also use screen as a terminal multiplexer, allowing multiple terminal sessions to be used from one login, especially useful if you’re using Windows with PuTTY that doesn’t have a multiple terminal function built into it.

For these reasons, screen by itself is a very powerful and useful utility, but it is admittedly hard to use. To the rescue comes a screen wrapper called byobu which provides a much easier-to-use interface to the screen utility. byobu has been installed on all the interactive nodes on BDUC and can be started by typing:

$ byobu

There will a momentary screen flash as it refreshes and re-displays the login, and then the screen will look similar, except for 2 lines along the bottom that show the screen status. In the images below, the one at left is without byobu; at right is with byobu. The byobu screen shows 3 active sessions: login, claw_1, and bowtie. The graphical tabs at the bottom are part of the KDE application konsole which also supports multiplexed sessions (allowing you to multi-multiplex sessions (polyplex?))

The help screen, shown below, can always be gotten to by hitting the <F9> key, followed by the <Enter> key.

Byobu 2.57 is an enhancement to GNU Screen, a command line
tool providing live system status, dynamic window management,
and some convenient keybindings:

F2    Create a new window    |  F6    Detach from the session
F3    Go to the prev window  |  F7    Enter scrollback mode
F4    Go to the next window  |  F8    Re-title a window
F5    Reload profile         |  F9    Configuration
                             |  F12   Lock this terminal
'screen -r'  - reattach      |  <ctrl-A> Escape sequence
'man screen' - screen's help | 'man byobu'  - byobu's help

Most usefully, you can create new sessions with the F2 key, switch between them with F3/F4 and detach from the screen session with F6.

Note that you must have started a screen session before you can detach, so to make sure you’re always in a screen session, you can have it start automatically on login by changing the state of the Byobu currently launches at login flag (at bottom of screen after the 1st F9.

When you log back in after having detached, type byobu again to re-attach to all your running processes. If you set byobu to start automatically on login, there will be no need of this, of course, as it will have started.

SGE Batch Submission & Queues

If you have jobs that are very long or require multiple nodes to run, you’ll have to submit jobs to an SGE Queue (aka Q).

qsub job_name.sh will submit the job described by job_name.sh to SGE, which will look for an appropriate Q and then start the job running via that Q. For example, if you need a long running Q, you can request it explicitly: qsub -q long job_name.sh , which will try to run it on the least loaded machine.

Once you log into the login node (via ssh -Y <your_UCINetID>@bduc-login.nacs.uci.edu), you can get an idea of the hosts that are currently up by issuing the qhost command. You can find out the status of your jobs with qstat alone, which will tell you the status of your jobs or

qstat -u '*'

will tell you the status of all jobs currently queued or running. A very useful PDF cheatsheet for the SGE q commands is here.

To get an overall idea of the status of the entire cluster, type bduc_status, which will dump a listing of:

• who’s logged into the node
• the top 100 jobs currently running
• nodes/Qs in error state
• overall cluster node usage by Q.

qsub scripts

The shell script that you submit (job_name.sh above) should be written in bash and should completely describe the job, including where the inputs and outputs are to be written (if not specified, the default is your home directory. The following is a simple shell script that defines bash as the job environment, calls date, waits 20s and then calls it again.

#!/bin/bash
# (c) 2008 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms.
# This is a simple example of a SGE batch script

# request Bourne shell as shell for job
#$ -S /bin/bash

# print date and time
date
# Sleep for 20 seconds
sleep 20
# print date and time again
date

Note that your script has to include (usually at the end) at least one line that executes something – generally a compiled program but it could also be a Perl or Python script (which could also invoke a number of other programs). Otherwise your SGE job won’t do anything.

Using qsub scripts to keep data local

BDUC depends on a network-shared /home filesystem. The actual disks are in the bduc-login node so users are local to the data when they log in. However, when you submit an SGE job, unless otherwise specified, the nodes have to read the data over the network and write it back across the network. This is fine when the total data involved is a few MB, such as is often the case with molecular dynamics runs – small data in, lots of computation, small data out. However, if your jobs involve 100s or 1000s of MB, the network traffic can grind the entire cluster to a halt.

To prevent this network armaggedon, there is a /scratch directory on each node which is writable by all users, but is sticky – the files written can only be deleted by the user who wrote them.

$ ls -ld /scratch
drwxrwxrwt 6 root root 4096 Oct 29 18:20 /scratch/
         ^
         + the 't' indicates 'stickiness'

If there is a chance that your job will consume or emit lots of data, please use the local /scratch dir to stage your input data, and especially write your output.

This is dirt simple to do. Since your qsub script executes on each node, your script should copy the data from your $HOME dir to /scratch/$USER/input to stage the data, then specify /scratch/$USER/input as input, with your application writing to /scratch/$USER/output_node#. When the application has finished, copy the output files back to your $HOME dir again, and finally cleaning up the /scratch/$USER/whatever afterwards.

Here’s another page of information on using scratch space.

An example script that does this data copying.

Note

Staging data – some important caveats Staging data to the remote node makes sense when you have large input data and it has to be repeatedly parsed. It makes less sense when a lot of data has to be read once and is then ignored. (If the data is only read once, why copy it? Just read it in the script.) If you stage it to /scratch, it is still traversing the network once so there is little advantage. (If you have significant data to be re-read on an ongoing basis, contact me and depending on circumstances, we may be able to let you leave it on the /scratch system of a set of nodes for an extended period of time. Otherwise, we expect that all /scratch data will be cleaned up post-job. If it does make sense to stage your data, please try to follow the guidelines below. If the cluster locks up, offending jobs will be deleted without warning so ask me if you have questions. Limit your staging bandwidth If your job(s) are going to require a mass copy (for example, if you submit 20 jobs that each have to copy 1GB), then throttle your job appropriately by using a bandwidth-limiting protocol like scp -C -l 2000 instead of cp. This scp command compresses the data and also limits the bandwidth to ~250KB/s in the above case (2000 refers to KiloBITS, not KiloBYTES). scp will work without requiring passwords, just like ssh within the cluster. The syntax is slightly different tho. # use scp to copy from bduc-login to a local node dir as would be required in a qsub script scp -C -l 2000 bduc-login:~/my_file /scratch/hmangala This prevents a few bandwidth-unlimited jobs from causing the available cluster bandwidth to drop to zero, locking up all users. If you have a single job that will copy a single 100MB file, then don’t worry about it; just copy it directly. Assume the aggregate bandwidth of the cluster is about 30 MB/s. No set of jobs should exceed half of that, so if you’re submitting 50 jobs, the total bandwidth should be set to no more than 25MB/s or 0.5 MB/s per job or in scp terms -l 4000. Check the network before you submit a job While there’s no way to prodict the cluster environment after you submit a job, there’s no reason to make an existing BAD situation worse. If the cluster is exhibiting network congestion, don’t add to it by submitting 100 staging jobs. (and if it does appear to be lagging, please let me know) How to check for cluster congestion On the login node, you can use a number of tools to see what the status is. bduc_status will dump a long description detailing who’s logged in, what the SGE Q status, including the 1st 100 jobs, any Qs in error state, the Queue Summary, the hosts that are down, and the overall cluster load. top give you an updating summary of the top CPU-using processes on the node. If the top processes include nfsd, and the load average is above 4 with no user processes exceeding 100%, then the cluster can be considered congested. For those that don’t have the fancy prompt, you can add it by inserting the following line into your /.profile or ~/.bashrc. PS1="\n\[33[01;34m\]\d \t \[33[00;33m\][\$(cat /proc/loadavg | cut -f1,2,3 -d' ')] \ \[33[01;32m\]\u@\[33[01;31m\]\h:\[33[01;33m\]\w\n\! \$ \[33[00m\]" ifstat will produce a continuous, instantaneous chart of network interface output. dstat will produce a similar readout of many system parameters including CPU, memory usage, network, and storage activity. htop produces a colored, top-like output that is multiply sortable to debug what’s happening with the system. atop produces yet another top-like output but highlights saturated systems. It provides more info to the root user, but is also useful for regular users. iftop produces a very useful (but only available to root) text-based, updating diagram of network bandwidth by endpoints. Mentioned as it might be useful to users on their own machines.

Debugging why your job isn’t running

You can (at least partially) diagnose your own SGE problems. It may well be that the Qs are set up sub-optimally (and if so, we’ll try to work with you to optimize them), but you can see very quickly if that’s the case or if it’s due to a more mundane problem

$ qstat                  # will give you a list of your SGE jobs
$ qstat -j <job number>  # will give you an exhaustive list of reasons
                         #  that your job is not executing

More example qsub scripts

• sleeper1.sh is a slightly more elaborate one.
• fsl_sub is a longer, much more elaborate one that uses a variety of parameters and tests to set up the run.
• array_job.sh is a qsub script that implements an array job – it uses SGE’s internal counter to vary the parameters to a command. This example also uses some primitive bash arithmetic to calculate the parameters.
• qsub_generate.py is a Python script for generating serial qsubs, in a manner similar to the SGE array jobs. However, if you need more control over your inputs & outputs and /or are more familiar with Python, it may be useful.

Current Queue Organization

The batch queues have been to reorganized for clarity. They now are organized as follows:

Queue        time*      total CPUs   Type
===============================================

long-ics     ( 78 batch cores)
long-adc     ( 64 batch cores)
long         (191 batch cores)

int           2hr          4         interactive (*)

long-ics     240hr         78        batch

long-adc     240hr         64        batch

long-quad    240hr        124        batch (all 4core motherboards)

long         240hr        191        batch

* for the 'int' Q, you have 2 hr of aggregate CPU time (not
    wallclock time).

To submit short jobs (<12hr), you can most easily not specify a Q – it will go on any batch Q. To run on a longxxx Q, either specify the estimated runtime in the submission script by including the -l h_rt parameter

#$ -l h_rt=00:30:00 #30 min run

(also see below)

or submit specifically to one of the long Qs.

ie:

$ qsub -q long-ics yourshellname.sh

# or include the Q spec in the script:

#$ -q long-ics

Fixing qsub errors

Occasionally, a script will hiccup and put your job into an error state. This can be seen by the qstat state output:

$ qstat -u '*'

job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   6868 0.62500 simple.sh  hmangala     E     06/08/2009 11:29:02 claws@claw3.bduc                   1
                                       ^^^

the E (^{^}) means that the job is in an ERROR state. You can either delete the job with qdel:

qdel <Job ID> # deletes the job

or often change it’s status with the qmod command.

qmod -cj <Job ID> # clears the error state of the job

Some useful SGE script parameters

When you submit an SGE script, it is processed by both bash and SGE. In order to protect the SGE directives from being misinterpreted by bash, they are prefixed by #$ This prefix causes bash to ignore the rest of the line (considers it a comment), but allows SGE to process the directive correctly.

So, the rules are:

• If it’s a bash command, don’t prefix it at all.
• If it’s an SGE directive, prefix it with both characters (#$).
• If it’s a comment, prefix it only with a #.

Here are some of the most frequently used

#$ -N job_name     # this name shows in qstat
#$ -S /bin/bash    # run with this shell
#$ -q long-ics     # run in this Q
#$ -l h_rt=50:00:00  # need 50 hour runtime
#$ -l mem_free=2G  # need 2GB free RAM
#$ -l scr_free=XG  # need X GB scratch space
#$ -pe mpich 4     # define parallel env
#$ -cwd            # run the job out of the current directory
                   # (the one from which you ran the script)
#$ -o job_name.out # the name of the output file
#$ -e job_name.err # the name of the error file
#  or
#$ -o job_name.out -j y            # '-j y' merges stdout and stderr

#$ -t 0-10:2       # task index range (for looping); generates 0 2 4..10
#                 Uses $SGE_TASK_ID to find out whether they are task
                  0, 2, 4, 6, 8 or 10

#$ -notify
#$ -M <email> - send mail about this job to the given email address.
#$ -m beas          # send a mail to owner when the job
#                       begins (b), ends (e), aborted (a),
#                       and suspended(s).

When a job starts, a number of SGE environment variables are set and are available to the job script. Read about all of them here (bottom of page).

For more on SGE shell scripts, see here.

For a sample SGE script that uses mpich2, see below

Where do I get more info on SGE?

Oracles purchase of Sun has resulted in a major disorganization of SGE (now OGE) documentation. If a link doesn’t work, it may be because of this kerfuffle. tell me if a link doesn’t work anymore and I’ll try to fix it.

• The ROCKS group has a very good SGE Introduction from the User’s perspective. Ignore the ROCKS-specific bits.
• Google Sun Grid Engine is a good, easy start. Maybe you’ll be lucky..
• Chris Dagdigian’s SGE site is very good and has an excellent wiki
• The official Sun (now Oracle) Grid Engine site has a lot of good links.
• The SGE docs are the final word, but there are a lot of pages to cover.

If you need to run an MPI parallel job, you can request the needed resources by Q as well by specifying the resources inside the shell script (more on this later) or externally via the -q and -pe flags (type man sge_pe on one of the BDUC nodes).

Special cases

Editing Huge Files

In a word, don’t. Many research domains generate or use multi-GB text files. Prime offenders are log files and High-Thruput Sequencing files such as those from Illumina. These are meant to be processed programmatically, not with an interactive editor. When you use such an editor, it typically tried to load the entire thing into memory and generates various cache files. (If you know of a text editor that handles such files without doing this, please let me know.)

Otherwise, use the utilities head, tail, grep, split, less, sed, and tr, possibly in combinations with Perl/Python to peek into such files and or change them.

grep especially is one of the most useful tools for text processing you’ll ever use.

For example, the following command starts at 2,000,000 lines into a file and stops at 2,500,000 lines and shows that range in the less pager.

$ perl -n -e 'print if ( 2000000 .. 2500000)' humongo.txt | less

In addition, please use the commandline utilities gzip/gunzip, bzip2, zip, zcat, etc instead of the ark graphical utility on such files. ark apparently tries to store everything in RAM before dumping it.

NAMD scripts

namd is a molecular dynamics application that interfaces well with VMD. Both of these are available on BDUC – see the output of the module avail command. The scripts to submit a namd run to the SGE Q’ing system are a bit tricky due to the way namd is compiled and run – it uses an MPI-like parallelization approach without explicitly requiring an external MPI library – those functions are linked into the charmrun executable supplied with the namd package. However, SGE requires an MPI parallel environment to reserve the cores necessary for such a run.

namd_sge_submit.sh is a SGE submission script that runs successfully on BDUC if given a valid namd input file. It must be submitted to SGE as follows:

qsub -pe openmpi <#cores> <name_of_script>
# or explicitly, for an 8core job
qsub -pe openmpi 8 namd_sge_submit.sh

(thanks to Chad Cantwell for the hints and pointer to the original page)

SATe

SATe is a Python wrapper around a number of Phylogenetic tools. It, along with its requisite tools (ClustalW2, MAFFT, MUSCLE, OPAL, PRANK, RAxML) are installed in the shared /usr/local/bin directory of the Claw nodes.

The test cases work with the default settings, but if you want to change any parameters, you have to edit the configuration file and feed it to run_sate.py with -c as shown below.

export SH=/usr/local/bin # <- shortens the following lines considerably

run_sate.py -c sate.cfg  -i $SH/sate_data/small.fasta -t $SH/sate_data/small.tree -j test
            ^^^^^^^^^^^

You can name the configuration file anything you like, but it has a specific format. Especially, do NOT try to start comments anywhere except the 1st character of a line, and then only beginning with #.

Here is a good SATe configuration file to start from.

Here is the same SATE configuration file with some bad comments (marked as such with BAD COMMENT in the offending line.)

Here is an example qsub submission script for SATe. Submit to SGE as:

qsub SATE.sh

R on BDUC

R is an object-oriented language for statistical computing, like SAS (see below). It is becoming increasingly popular among both academic and commercial users to the extent that it was noted in the New York Times in early 2009. For a very simple overview with links to other, better resources, see this link

There are multiple versions of R on BDUC, and they do not all behave identically. Since we have a split cluster (most nodes (_80; 160 cores) run CentOS (RedHat-based) ; the 4 claw nodes (16 cores) run a version of Ubuntu, (Debian-based). Because of slightly different library structures and versions, some R add-ons don’t work across the subclusters, so in those situations, we concentrate on getting the standard approach working on the CentOS nodes, and provide work-arounds on the claw nodes.

The module system provides R versions 2.10.0 and 2.8.0 for all nodes. Additionally, the claw nodes provide version 2.9.0 because it is the default version. Finally, I’ve added the R development version which is automatically downloaded, compiled, and re-installed every night from the R archives. This is the VERY LATEST version, so new that it (infrequently) fails. Howeer, if you need the latest and greatest version, it’s available. To load any of these versions, inquire what the available versions are with module avail and then use the appropriate module load R/<version> to set up the paths.

For most things, everything works identically. The things that don’t usually have to do with parallel processing in R and the underlying Message Passing Interface (MPI) technology:

• Rmpi should work on all CentOS nodes with version 2.10.0. The claw nodes will not work with the 2.10.0 version as it has a complicated lib dependency that leads into some very bushy areas. Rmpi DOES work on the claw nodes, but only under R 2.9.0 (the default).
• rsprng (R’s wrapping of SPRNG) is available on all the CentOS nodes for R/2.10.0 and on the default 2.9.0 version on the claw nodes.
• snow and snowfall are available on the CentOS nodes with version 2.10.0 and on the claw nodes with the default 2.9.0 version.

SAS 9.2 for Linux

We have a single node-locked license for SAS 9.2 on claw1, a 4core Opteron node with 32GB RAM. While the license is for that node only, as many instances of SAS can be run as there is RAM for it.

To start SAS on claw1, type:

# cd to the directory where your data is
cd /dir/holding/data

# and start SAS
sas

This will start an X11 SAS session, opening several windows on your monitor (as long as you have an active X11 server running). If you’re connecting from Mac or Windows, please see this link.

You can use the SAS program editor (one of the windows that opens automatically), or use any other editor you want and paste or import that code into SAS. The combination of emacs and ESS (Emacs Speaks Statistics) is a very powerful combination. It’s mostly targeted to the R language, but it also supports SAS and Stata.

Nedit also has a template file for SAS.

# convenience shortcut
export SASPATH=/home/apps/SAS-x86_64/9.2

# following is required to allow 32bit java to find its libs
export LD_LIBRARY_PATH=${SASPATH}/jre1.5.0_21/lib/i386:\
${SASPATH}/jre1.5.0_21/lib/i386/server:${LD_LIBRARY_PATH}

# Need to set the CLASSPATH to the JRE root so when SAS calls java, the right executable is called.
export JAVAHOME=${SASPATH}/jre1.5.0_21/

Parallel jobs

BDUC supports several MPI variants.

MPICH2

BDUC is running MPICH2 version 2-1.1.1p1. Using it is not hard, but requires a few things:

• To compile MPI binaries, you’ll have to module load the MPICH2 environment:

module load mpich2

• You need to set up passwordless ssh so that you can ssh to any BDUC node without entering a password, including editing your ~/.ssh/config file to prevent 1st-time connection warnings from interrupting your jobs
• you need to create the file ~/.mpd.conf, as below:

cd
# replace 'thisismysecretpassword' with something random.
# You won't have to remember it.
echo "MPD_SECRETWORD=thisismysecretpassword" >.mpd.conf
chmod og-rw .mpd.conf

• your mpich2 qsub scripts have to include the 2 following lines in order to allow SGE to find the PATHS to executables and libraries

module load mpich2
export MPD_CON_EXT="sge_$JOB_ID.$SGE_TASK_ID"

A full MPICH2 script is shown below. Note the #$ -pe mpich2 8 line which sets up the MPICH2 parallel environment for SGE and requests 8 slots (CPUs). (see above for more SGE script parameters)

#!/bin/bash
# good idea to be explicit about using /bin/bash (NOT /bin/sh).
# Some Linux distros symlink bash -> dash for a lighter weight
# shell, which works 99% of the time but causes unimaginable pain
# in those 1% occassions.

# Note that SGE directives are prefixed by '#$' and plain comments are prefixed by '#'.
# Text after the '<-' should be removed before executing.

#$ -q long    <- the name of the Q you want to submit to
#$ -pe mpich2 8    <- load the mpich2 parallel env and ask for 8 slots
#$ -S /bin/bash    <- run the job under bash
#$ -M harry.mangalam@uci.edu <- mail this guy ..
#$ -m bea          <- .. when the script (b)egins, (e)nds, or (a)borts
#$ -N cells500     <- name of the job in the qstat output
#$ -o cells500.out <- name of the output file.
#
module load mpich2              <- load the mpich2 environment
export MPD_CON_EXT="sge_$JOB_ID.$SGE_TASK_ID" <- this is REQUIRED for SGE to set it up.
module load neuron              <- load another env (specific for 'neuron')
export NRNHOME=/apps/neuron/7.0 <- ditto
cd /home/hmangala/newmodel      <- cd to this dir before executing
echo "calling mpiexec now"      <- some deugging text
mpiexec -np 8 nrniv -mpi -nobanner -nogui /home/hmangala/newmodel/model-2.1.hoc
# above, start the job with 'mpiexec -np 8', followed by the executable command.

MATLAB

MATLAB can be started from the login node by typing matlab. This will log you into a 64bit interactive node and start the MATLAB Desktop. matlabbig will start an interactive session on one of the claw nodes (32GB RAM).

We have 3 licenses for interactive MATLAB on the BDUC cluster. Those licenses are decremented from the campus MATLAB license pool. They are meant for running interactive, relatively short-term MATLAB jobs, typically less than a couple hours. If they go longer than that, or I see that you’ve launched several MATLAB jobs, they are liable to be killed off.

If you want to run long jobs using MATLAB code, the accepted practice is to compile your MATLAB .m code to a native executable using the MATLAB compiler mcc and then submitting that code, along with your data to a batch Q (see above for submitting batch jobs). This approach does not require a MATLAB license, so you can run as many instances of this compiled code for as long as you want without impacting the campus licenses.

The official mechanics of doing this is described here.

Some additional notes from someone who has done this is in the Appendix.

MATLAB Alternatives

There are a number of MATLAB alternatives, the most popular of which are available on BDUC. Since these are Open Source, they aren’t limited in the number of simultaneous uses, altho you should always try to run batch jobs in the SGE queue if possible. See this doc for an overview of them and further links.

Hadoop

Hadoop is a Java-based framework for running large-grained, parallel jobs on clusters. It now encompasses a large number of subprojects, but it is usually used with the MapReduce approach. It scales very well, but it is complex to run on BDUC because it requires its own filesystem and scheduler. Since on BDUC (and other general-purpose clusters which are not dedicated to hadoop full-time) the job scheduling is more general-purpose, we have to run it as a meta-job. That is, you submit a request to SGE to allocate a number of nodes on which to run hadoop; SGE allocates them to hadoop; hadoop sets up the logical structures it needs on those allocated nodes and everyone’s happy. We run hadoop under myHadoop a small bit of middleware designed to handle the interactions between SGE and Hadoop.

Note that Hadoop initializes its own filesystem on the existing /scratch directory for each node. Because Hadoop can end up storing many GBs of data, we have set up a dedicated hadoop SGE Q named (surprise) hadoop. All the nodes in this Q should have >300GB available and you can test this by running the following command on the login node:

cf --config=/usr/local/bin/cfrc --target=HADOOP 'df -h |grep sda3 | scut --c1=3'

This will start an interative script that will create a subdir in the current directory which will contain files named for all the hadoop nodes which lists the free diskspace on /scratch. You can see the results by doing this:

cd REMOTE_CMD-df--h--grep-sda3---s-<timestamp> # timestamp changes obviously
grep G *
a64-141:334G
a64-142:337G
..
..

An SGE submit script for Hadoop is here.

The usual way to exploit Hadoop is to write your application in Java, typically wrapping it into a jarfile. This Java requirement is not absolute tho. You can also write your application in Jython (Python written in Java) and therefore essentially write Java using Python. You can also write your hadoop app in pure Python (or even in C++). Here is a page describing that approach.

The hadoop installation can be found at $HADOOP_HOME. By referring to this variable, you can, for example, add standard libraries to the class path of your java application (e.g. Class-Path: $HADOOP_HOME/lib/hadoop-0.20.2-tools.jar).

(Thanks to Fabian Lindenberg for helping to set up and debug Hadoop on BDUC.)

GPUs

Thanks to Dr. Steve Jenks, the claw8 node contains 2 Nvidia C1060 Graphics Processing Units (GPUs), each with 240 cores. These cores are specialized to do SIMD tasks very fast. For example, if your code supports that kind of processing, you can get 10-100X speedup for those parts of the code. You can learn more about programming these GPUs with CUDA at the local docs or via the more up-to-date docs at NVIDIA.

In order to use the GPUs, you will have to be registered to use the GPU SGE Queue (contact harry.mangalam@uci.edu) and of course will have to provide your own code, altho the entire Nvidia SDK examples are compiled and available locally: source code at /apps/gpu/1.0/NVIDIA_GPU_Computing_SDK/C/src/, compiled biaries at /apps/gpu/1.0/NVIDIA_GPU_Computing_SDK/C/bin/. Of course, since only claw8 has the GPUs installed, you’ll only be able to run them there.

In order to run the compiled GPU code for long runs, you’ll have to submit them thru the SGE scheduler using the gpu queue which is initialized by using the module load gpu directive in your qsub script. We expect that you’ll do your debugging on your own machine altho you can do it on claw8 after you have registered and been added to the gpu group.

We currently have the CUDA Toolkit 4.0 installed and will try to remain current with NVIDIA upgrades.

Graphics

All the interactive nodes will have the full set of X11 graphical tools and libraries. However, since you’ll be running remotely, any application that requires OpenGL, while it will probably run, will run so slowly that you won’t want to run it for long. If you have an application that requires OpenGL, you’ll be much better off downloading the processed data to your own desktop and running the application locally.

If you connect using Linux

In order to have access to these X11 tools via Linux, your local Linux must have the X11 libraries available. Unless you have explicitly excluded them, all modern Linux distros include X11 runtime libraries. Don’t forget to use the the -Y flag when you connect using ssh to tunnel the X11 display back to your machine:

ssh -Y your_UCINetID@bduc.nacs.uci.edu

If you connect using MacOSX

The MacOSX installation DVDs come with a free, Apple-certified X11 installation. On Leopard it’s in Optional Installs → Optional Installs.mpkg All you have to do is install it and start it running in the background to accept the X11 windows (Applications → Utilities → X11) Ditto the -Y ssh flag as above.

If you connect using Windows

There are quite a few ways to use a Linux system besides logging into it directly from the console.

• remote shell access, using PuTTY, a free ssh client, which even allows X11 forwarding so that you can use it with Xming (below) to view Graphical apps from BDUC. Putty is a straight ssh terminal connection that allows you to securely connect to the Linux server and interact with it in a purely text-based basis. For a shell/terminal cognoscenti, it’s considerably less capable than any of the terminal apps (konsole, eterm, gnome-terminal, etc) that come with Linux, but it’s fine for establishing the 1st connection to the Linux server. If you’re going to run anything that requires an X11 GUI, you’ll need to set PuTTY to do X11 forwarding. To enable this, double-click the PuTTY icon to bring up the PuTTY configuration window. On the left Pane, follow the clickpath: Connection → SSH → X11 → set the Enable X11 Forwarding. After setting this, click on Session at top of the pane, and set a name in Saved Sessions on lower right pane, click the [Save] button to save the connection information so that the next time you need to connect, the correct setting will already be set.
• Xming, a lightweight and free X11 server (client, in normal terminology). Xming provides only the X server, as opposed to Cygwin/X below. Xming provides the X server that displays the X11 GUI information that comes from the Linux machine. When started, it looks like it has done nothing, but it has started a hidden X11 window (note the Xming icon in the toolbar). When you start an X application on the Linux server (after logging in with PuTTY as described above), it will accept a connection from the Linux machine and display the X11 app as a single window that looks very much like a normal MS WinXP window. You’ll be able to move it around, minimize it, maximize it and close it by clicking on the appropriate button in the title bar. There may be a slight lag in response in that window, but over the University network, it should be be acceptable.
• if you have trouble setting up Putty and Xming, please see this page which describes it in more detail, with screenshots
• Cygwin/X, another free, but much larger and capable X server (combined with an entire Linux-on-Windows implementation). Provides much more power and requires much more user configuration than Xming. Cygwin/X provides not only a free Xserver but nearly the entire Linux experience to Windows. This is more than what most normal users want (both in diskspace and configuration), especially if you have a real Linux server to use. The X11 server is very good tho, as you might expect.
• VNC server and client. Run the server on the Linux machine and connect to it with the client running on your Windows Desktop. Can provide the entire Linux Desktop experience on your Windows machine, altho with less graphics performance (it’s fine to connect to a machine on the university network, but slow across the Internet). VNC is mechanism that can present the entire Linux Desktop to the user, including not only the application windows, but the Desktop itself, with all the bells and whistles that that metaphor provides. The RealVNC package for Windows provides both the Viewer and the Server, so you can provide remote access to your Windows Desktop as well. This can be especially useful if you’re trying to demo a Desktop application to others – you can configure the VNC server to allow multiple read-only clients (they can’t take control of your desktop) to watch you run the app. Combined with the multiplaform VOIP application Skype which can run on the same machine, you have a very cheap tele-screensharing setup good for demo’ing applications. The Windows VNC server is efficient enough to support at least 10 viewing clients and the refresh rate is good enough for a mostly 2D demo across UC.
• NoMachine Server and Clients, a system much like the VNC system but much more efficient and therefore has better performance. It is also more complicated to set up. Please read this review for an overview of what is required and how to install it. For personal use, there is a free server and client. For more connections, you’ll need a commercial license. There are also 2 free NX servers: FreeNX and Google’s recently released NeatX, both of which are fairly easy to install and allow unlimited connections. BDUC uses both the free versions and it is described in more detail immediately below.

NoMachine NX connections

We’ve added GPL’ed NXservers to both the login and the claw1 nodes. Both have direct external connections. With the nxclient software installed on your machine, you can run the entire Linux Desktop (Gnome on login, KDE on claw1) with remarkable speed. Here’s a screenshot of my laptop screen with the nxclient Desktop from claw1 running SAS, matlab, and tablet (a genomics assembly viewer):

Get the appropriate client software for your platform here.

Configuring the nxclient

The configuration is fairly simple. The initial pane allows you to set your Login (your UCINetID and Password (your UCINetID password) and name the session anything you like.

Clicking the Configure… button takes you to a set of tabbed configuration pages. The only one that needs to be modified is the 1st one General:

The screenshot above shows the setup for logging into the claw1 node (which supports the KDE Desktop). If you want to use it on the login node which supports the Gnome Desktop, see below:

DO NOT change the default Key unless you have problems logging into the login node.

-----BEGIN DSA PRIVATE KEY-----
MIIBuwIBAAKBgQCXv9AzQXjxvXWC1qu3CdEqskX9YomTfyG865gb4D02ZwWuRU/9
C3I9/bEWLdaWgJYXIcFJsMCIkmWjjeSZyTmeoypI1iLifTHUxn3b7WNWi8AzKcVF
aBsBGiljsop9NiD1mEpA0G+nHHrhvTXz7pUvYrsrXcdMyM6rxqn77nbbnwIVALCi
xFdHZADw5KAVZI7r6QatEkqLAoGBAI4L1TQGFkq5xQ/nIIciW8setAAIyrcWdK/z
5/ZPeELdq70KDJxoLf81NL/8uIc4PoNyTRJjtT3R4f8Az1TsZWeh2+ReCEJxDWgG
fbk2YhRqoQTtXPFsI4qvzBWct42WonWqyyb1bPBHk+JmXFscJu5yFQ+JUVNsENpY
+Gkz3HqTAoGANlgcCuA4wrC+3Cic9CFkqiwO/Rn1vk8dvGuEQqFJ6f6LVfPfRTfa
QU7TGVLk2CzY4dasrwxJ1f6FsT8DHTNGnxELPKRuLstGrFY/PR7KeafeFZDf+fJ3
mbX5nxrld3wi5titTnX+8s4IKv29HJguPvOK/SI7cjzA+SqNfD7qEo8CFDIm1xRf
8xAPsSKs6yZ6j1FNklfu
-----END DSA PRIVATE KEY-----

Unlike the commercial NoMachine NXserver, these servers allow any number of connections. There may be a fairly long wait (up to a minute) before the session is initially validated and the screen comes up (the Desktop is loading on the server), but after that, the interaction is very fast.

Terminating the session

Note that when you close the session, you have 2 options – to Disconnect (closes the client but leaves the session running so you can reconnect to the same session you left) or Terminate (closes the client and kills the session, so you’ll start from a new Desktop instance).

Unless there is good reason to keep it running, please Terminate the session to free up resources.

I have run into a situation whereby if the session is ended oddly it will leave a ghost session (ie, you kill the nxclient by killing the shell from which it was started). When you next start up the nxclient, it will offer to let you re-connect to an existing session, but then be unable to reconnect. If this happens, you should still be able to start a new session, but please call me to address that situation – I have to manually remove the ghost session credentials in /usr/local/var/lib/neatx/sessions/.

export XKEYSYMDB=/usr/share/X11/XKeysymDB

How to Manipulate Data on Linux

This is a topic for a whole ‘nother document named Manipulating Data on Linux and the documents and sites referred to therein.

Frequently Asked Questions

OK, maybe not frequently, but cogently, and CAQ just doesn’t have the same ring. If you have other questions, please ask them. If they address a frequent theme, I’ll add them here. In any case, I’ll try to answer them.

Q&A

1. What’s a node? Is it the same as a processor? A node refers to a self-contained chassis that has its own power supply, motherboard (containing RAM, CPU, controllers, IO slots and devices (like ethernet ports), various wires and unidentifiable electrogrunge). It usually contains a disk, altho this is not necessary with boot-over-the-network. It’s not the same as a processor. Typical BDUC nodes (from the Jurassic period) have 2-4 CPU cores per node. Modern nodes have 8 to >100 cores.
2. When I submit a .sh script with qsub, does the following line refer to 10 processors or 10 nodes? #$ -pe openmpi 10 10 processor cores. Most modern physical CPUs (the thing that plugs into the motherboard socket) have multiple processor cores internally these days.
3. What about the call to mpiexec? mpiexec -np 10 nrniv -mpi -nobanner -nogui modelbal.hoc Same thing. That’s why they should be the same number.
4. Is it possible for the processors on one node to be working on different jobs? Yes, altho the scheduler can be told to try to keep the jobs on 1 node (better for sharing memory objects like libs, but worse if there’s significant contention for other resources like disk & network IO). Most of the MPI environments on BDUC are currently set to spread out the jobs rather than bunch them together on as few nodes as possible.
5. If processor 1 (working on Job A) fails, does it bring down processor 2 (working on Job B) as well? No, and in fact it doesn’t typically work that way. A job does not run on a particular CPU; on a multi-core node, different threads of the same job can hop among CPU cores. The kernel allocates threads and processes to whatever resources it has to optimize the job.
6. Is the performance of processor 1 dependent on whether processor 2 is engaged in the same or different job? It depends. The computational bits of a thread, when they are being executed on a CPU, don’t interfere much with the other processor. They do share memory, interrupts, and IO so if they’re doing roughly the same thing at roughly the same time, they’ll typically want to read and write at the same time and thus compete for those resources. That was the rationale for spreading out the MPI jobs rather than filling up nodes.
7. Is it possible for one processor to use more than its “share” of the memory available to the node, i.e., is it wrong for me to count on having a certain amount of memory just because I’ve specified a certain number of processors (nodes?) for my job? The CPU running prog1 will request the RAM that it needs independent of other CPUs running prog1 or prog2, prog3, etc. If the node gets close to running out of real RAM, it will start to swap idle (haven’t-been- accessed-recently) pages of RAM to the disk, freeing up more RAM for active programs. If the computer runs out of both RAM and swap, it will hopefully kill off the offending programs until it regains enough RAM to function and then it will continue until it happens again. This is why you should try to estimate the amount of RAM your prog will use and indicate that to the scheduler with the -l mem_free directive. See the section above.
8. I can ssh to BDUC but I can’t scp files to it. Why? Probably because you edited your .bashrc (or .zrc or .tcshrc) to emit something useful when you log in. (Both scp and ssh have a useful option -v that puts it into verbose mode that tells you much more about what the process is doing and why it fails). You need to mask this output from non-interactive logins like scp and remote ssh execution by placing such commands inside a test for an interactive shell. When using bash, you would typically do something like this:

interactive=`echo $- | grep -c i `
if [ ${interactive} = 1 ] ; then
  # tell me what my 22 latest files are
  ls -lt | head -22
fi

Appendix

HOWTO: Passwordless ssh

Passwordless ssh will allow you to ssh/scp to frequently used hosts without entering a passphrase each time. The process below works on Linux and Mac only. Windows clients can do it as well, but it’s a different procedure. However, regardless of your desktop machine, you can use passwordless ssh to log in to all the nodes of the BDUC cluster once you’ve logged into the login node.

In a terminal on your Mac or Linux machine, type:

# for no passphrase, use
ssh-keygen -b 1024 -N ""

# if you want to use a passphrase:
ssh-keygen -b 1024 -N "your passphrase"
# but you probably /don't/ want a passphrase - else why would you be going thru this?

save to the default places.

For the BDUC cluster case: Since all cluster nodes share a common /home, all you have to do is rename the public key file (normally id_rsa.pub in your ~/.ssh dir) to authorized_keys.

For unrelated (non-cluster) hosts: Linux users, use the ssh-copy-id command, included as part of your ssh distribution. (Mac users will have to do it manually, described just below.) ssh-copy-id does all the copying one shot, using your ~/.ssh/id_rsa.pub key (by default; use the -i option to specify another identity file, say ~/.ssh/id_dsa.pub if you’re using DSA keys)

ssh-copy-id  your_bduc_login@bduc.nacs.uci.edu
# you'll have to enter your password one last time to get it there.

What this does is to scp id_rsa.pub to the remote host (the ssh server your’re trying to log into) and append that key to the remote file ~/.ssh/authorized_keys. If things don’t work, check that the id_rsa.pub file has been appended correctly.

Then verify that it’s worked by ssh’ing to BDUC. You shouldn’t have to enter a password anymore.

For Mac users, scp the same keys to the remote host and append your public key to the remote ~/.ssh/authorized_keys. Here are the commands below. Just modify the UCINETID value and mouse them into the Terminal window on your local Mac.

bash  # starts the bash shell just to make sure the rest of the commands work
cd    # makes sure you're in your local home dir
export UCINETID=""  # fill in the empty quotes with *your UCINETID*

# you'll need to enter the password manually for the next 2 commands)

scp ~/.ssh/id_rsa.pub ${UCINETID}@bduc-login.nacs.uci.edu:~/.ssh/id_rsa.pub
ssh ${UCINETID}@bduc-login.nacs.uci.edu 'cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys'

# and now you should be able to ssh in without a password
ssh ${UCINETID}@bduc-login.nacs.uci.edu

The authenticity of host 'bduc-login.nacs.uci.edu (128.200.15.20)' can't be established.
RSA key fingerprint is 57:70:23:8e:e1:15:8c:51:b0:52:ca:c7:a8:e9:26:9b.
Are you sure you want to continue connecting (yes/no)?

Host *
   StrictHostKeyChecking no

chmod go-rw ~/.ssh/config

Notes on using the MATLAB comiler on the BDUC cluster

(Thanks to Michael Vershinin and Fan Wang for their help and patience in debugging this procedure).

As noted, the official docs for compiling your MATLAB code is is described here. Before you start hurling your .m code at the compiler, please read the following for some hints.

The following is a simple case where all the MATLAB code is in a single file, say test.m. Note that for the easiest path, you should write your MATLAB code to compile as a function. This means that keyword function has to be used to define the MATLAB code (see example below). If you want to pass parameters to the function, you have include a function parameter indicating this.

# Before you use any MATLAB utilities, you will have to load the
# MATLAB environment via the 'module' command

module load matlab/R2009b

# for a C file dependency, you compile it with 'mex'.  Note that mex doesn't like
# C++ style comments (//), so you'll have to change them to the C style /* comment */

mex some_C_code.c    # -> produces 'some_C_code.mexa64'

# then compile the MATLAB code for a standalone application.
# (type mcc -? for all mcc options)

# If the m-code has a C file dependency which has already been mex-compiled,
# mcc will detect the requirement and link the '.mexa64' file automatically.

mcc -m test.m  # -> 'test'  (can take a minute or more)

# !! if you have additional files that are dependencies, you may have to define
# !! them via the '-I /path/to/dir' flags to describe the dirs where your
# !! additional m code resides.

# for a _C_ shared lib (named libmymatlib.so) with multiple input .m files

mcc -B csharedlib:libmymatlib file1.m file2.m file3.m

# for a _C++_ shared lib (named libmymatlib.so) with multiple input .m files

mcc -B cpplib:libmymatlib file1.m file2.m file3.m

In the standalone case which will probably be the most popular approach on BDUC, the mcc compilation will generate a number of files:

readme.txt  ...............  autogen'd description of the process
test   ....................  the 'semi-executable'
test.m  ...................  original 'm code'
test_main.c  ..............  C code wrapper for the converted m code
test_mcc_component_data.c .  m code translated into C code
run_test.sh  ..............  the script that wraps and runs the executable
test.prj  .................  XML description of the entire compilation
                               dependencies (Project file)

In order to now run the executable, you often can’t submit the auto-generated run_test.sh directly in the SGE Q. You have to submit it wrapped in a SGE script which finally calls the run_test.sh script which sets up all the necessary environment variables and paths to run the executable.

So while you can test it for a few minutes like this:

./run_test.sh [matlab_root] ./test

# where the [matlab_root] would be '/apps/matlab/r2009b' for the
# matlab version that supports the compiler

Note that if you have already loaded the MATLAB module, you can usually run the compiled executable alone from the commandline.

However, for long/production runs, you will have to create a bash script (call it runmycode.sh) like this:

#!/bin/bash

#$ -S /bin/bash          # run with this shell

#$ -N comp_matlab_run    # this name shows in qstat
#$ -q long               # run in this Q
#$ -l h_rt=50:00:00      # need 50 hour runtime
#$ -l mem_free=2G        # need 2GB free RAM
#$ -l scr_free=1G        # need 1 GB scratch space
#$ -cwd            # run the job out of the current directory
                   # (the one from which you ran the script)

#$ -notify
#$ -M <email> - send mail about this job to the given email address.
#$ -m beas          # send a mail to owner when the job
#                       begins (b), ends (e), aborted (a),
#                       and suspended(s).

./run_test.sh  /apps/matlab/r2009b ./test

and qsub it to SGE:

qsub runmycode.sh

MATLAB Compilation Example

Below is a very simple example showing how to compile and execute some MATLAB code. Save the following code to a file named average.m.

function y = average(x)
% AVERAGE Mean of vector elements.
% AVERAGE(X) is the mean of vector, where X is a vector of
% elements. Nonvector input results in an error.
[m,n] = size(x);
if (~((m == 1) | (n == 1)) | (m == 1 & n == 1))
    error('Input must be a vector')
end
y = sum(x)/length(x);      % Actual computation
y

Once the code is saved as average.m, compile by copying and pasting into a terminal window.

module load matlab/R2009b   # load the MATLAB environment
mcc -m average.m;           # compile the code (takes many seconds)
z=1:99                      # assign the input vector to a shell variable
./average $z                # call the executable with the range.

Note also that if you’re going to run this under SGE as multiple instances, each instance will have to run with the appropriate MATLAB environment so you will have to preface each exec with the module load matlab/R2009b directive.

Release information & Latest version

The latest version of this document should always be available here. The asciidoc source is available here.

This document is released under the GNU Free Documentation License.