April 29, 1993

PAW version 2.02

A new version of PAW (2.02) has been installed in the PRO area on June 8. This version is available on all CERN supported systems and all the sources and tar files are also available from the ASIS server. Some features and enhancements in the new version are described below.

PIAF - The Parallel Interactive Analysis Facility

PIAF is a cluster of five HP-755 workstations each rated at 27 CERN units and equipped with 128 MBytes memory and 8 GBytes disk space. The machines are interconnected by an FDDI ring and accessible from the CERN wide Ethernet.

Piaf provides a service for processing large Ntuples. Users store their data files on Piaf and then use them from within a PAW session running on their local system. PAW version 2.02 incorporates the facilities to connect to a server process on one of the Piaf machines. The only condition is the availability of TCP/IP.

The Piaf server handles CPU and I/O intensive tasks and sends the results back to the client PAW session. Except for the speed the user will see no difference between a local and a Piaf file. In the first phase there is only one server process per client. A further speed-up can be reached in the second phase where the server talking to the client will launch slave servers on all available machines to process Ntuple requests in parallel.

The Piaf server requires password authentication. Piaf accounts will be allocated on a per-group basis with all group members sharing the same account. Each user has his own working directory corresponding to the login name on the client system.

A connection to the Piaf server is established by the PAW command CONNECT. The first time the group account name and password will be prompted for and is then kept in a local file ".piafrc" for future references. Once the server is started the usual Unix file manipulation commands "ls", "rm", etc. can be used. The ftp-like commands PUT and GET allow to transfer files between the client system and Piaf. A STAGE command for accessing tapes and remote data files is in preparation.

The only difference between local and Piaf files is in the HISTO/FILE command. Piaf files have to be indicated by the prefix "//piaf/". The remaining part of the path is relative to the user's working directory. "~" references the group directory. For example,

 
    HISTO/FILE 1 //piaf/file                 |   ~group/user/file
    HISTO/FILE 2 //piaf/~/file               |   ~group/file
    HISTO/FILE 3 //piaf/~/pub/file           |   ~group/pub/file

Afterwards the files are used like local files, e.g.

 
    CD //LUN1
    HISTO/LIST
    NT/PRINT 10
    NT/PLOT 10.x

Initially we want to restrict the access to Piaf to a small user community in order to gain experience with the operation and management of the service. After a consolidation phase Piaf will become a public service for PAW users inside CERN and at collaborating outside institutes.

- Ntuple Chains -- Logical Ntuples of Unlimited Size.

Using the command NTUPLE/CHAIN one can build logical Ntuples of unlimited size (assuming an unlimited amount of diskspace). To create a chain give as arguments to the chain command a chain name followed by a list of files, already open files (e.g. //LUN3) or other chain names (these chains may be defined later). Because a chain can contain other chains we can easily build a chain tree in this way. For example:

 
 CHAIN   Year93  Jan Feb March April May ...
 CHAIN   Jan     Week1 Week2 Week3 Week4
 CHAIN   Week1   file1.hbook file2.hbook ...
 CHAIN   Week2   file3.hbook file4.hbook ...
 ...
 ...

After a chain has been defined it can be traversed, by all Ntuple commands (NT/PLOT, NT/PROJ, NT/LOOP), by changing the current working directory to the chain name, e.g., CD //Year93 or CD //Week1.

Now to plot the distribution of, for example, the Pt for all events in Ntuple 10 in all files referenced in the chain we simply type: NT/PLOT 10.pt. For this to work, however, Ntuple 10 must be in all files in the same directory and Ntuple 10 must of course be Ntuple 10 in all files.

The path of all files referenced in a chain can be changed by the -P option of the chain command. If all the files of the above example, file1.hbook, file2.hbook, etc. are moved to another directory, or actually reside in a directory which is not the current working directory one can tell the chain command to prepend a path to the file names. For example:

 
 CHAIN Year93 -P /delphi/stage

will tell Paw to look for the files referenced in chain Year93 in directory /delphi/stage (e.g. /delphi/stage/file1.hbook).

On how to list chains, delete chains, delete files from chains, etc., see the online help of the chain command in PAW.

Recovery procedure for ntuples when the HBOOK job crashes

With the new version of HBOOK, the ntuple header containing the essential definitions for the ntuple are now written to the file when the first buffer is written. If the job producing the ntuple does not terminate in a clean way (job crash or missing CALL HROUT), it is now possible to rebuild the ntuple header from the information available in the file. The events corresponding to the last ntuple buffer in memory are lost. The HBOOK routine HRECOV can be used (CALL HRECOV(ID,' ')) and a new command NTUPLE/RECOVER has been implemented in PAW (See HELP).

Note on memory usage of Ntuples in PAW

Ntuple record length.

Currently the PAWC common in Paw is dimensioned to 500000 words. So, if you create Ntuples with very large buffer sizes, BSIZE, (set via HBSET) you might run out of memory in Paw when analysing many columns at a time. Paw needs to keep one buffer, of BSIZE words, in memory for each column being analysed. The error message will be coming from HNBUFR, telling you that there is not enough memory. To prevent this problem tune the BSIZE to the number of columns in your Ntuple while keeping the upper limit of 500000 words in mind. E.g. with BSIZE=32000 you could access at most 15 columns in a Comis function or via a cut tree (probably less because other parts of Paw also use the PAWC common).

Note 1:: BSIZE is the Ntuple buffer size specified via the HBSET routine (by default it is 1024). It is not the physical record length specified in the HROPEN routine.
Note 2:: On Unix and Apollo machines the PAWC common is dimensioned to 2000000 words.

Number of columns an Ntuple can contain.

Hbook limits the number of columns to 9999, where an array counts only for one column. E.g. 'NTRACK,HITS(100),NRUN' counts for three columns. PAW limits the combined number of columns of type int*4, real*4 and logical to 1000 and the number of real*8 and char*32 columns each to 1000. In this case each array element counts for one column. E.g. 'NTRACK,HITS(100),NRUN' counts for 102 columns.

Graphical cuts are operational and more powerful.

   PAW > Ntuple/plot 10.var2%sqrt(var1)
   PAW > Cut 1 g    | select an area with the mouse
   PAW > Ntuple/plot 10.var3  $1  | Note that the cut $1 can be referenced
                                  | when plotting a variable not referenced
                                  | in the graphical cut

In our previous implementation, graphical cuts were restricted to the original variables of the ntuple. With the new version, a graphical cut can reference operations on the original variables.

Commands MINIM and MAXIM are operational for 2-D plots

In the previous version, MINIM, MAXIM and SCALE were mutually exclusive. This has required a change in the HBOOK data structure. The words previously used to store the inverse of the bin width in X and Y are now used to store the minimum and maximum scale (HMINIM/HMAXIM).

W A R N I N G !!!!!!

As a consequence, histogram files generated with the new version of HBOOK and containing calls to HMAXIM/HMINIM for 2-D histograms cannot be saved and processed by previous versions of HBOOK.

Histogram subranges

Histogram subranges can be specified in 2 different ways:

        1- h/pl id(ic1:ic2) with ic1 and ic2 integers means plot
                            from channel ic1 to channel ic2
        2- h/pl id(x1:x2) with x1 and x2 reals (with a .) means plot
                            from channel corresponding to x1

Note that the mixed mode h/pl id(x1:ic2) is also accepted This subrange works also for 2-DIM cases.

Histogram subranges can be used in all graphics and fit commands.

Changes in FIT commands and routines

In Histo/fit and Vector/Fit commands, one can now reference an already compiled COMIS routine (see example below). The previous version accepted only COMIS files.

Fit commands work for a subrange of a 2-D plot.

IMPORTANT NOTE

Bug fixed in HBOOK routine HMINUT(thanks to D.Lellouch). In case of log-likelihood fits, the MINUIT ERRDEF was incorrectly set to 0.5. As a consequence, errors on parameters were underestimated by sqrt(2).

The HBOOK fit routines HFITH and HFITHN use DOUBLE PRECISION for all computations in case of predefined functions (G,E,Pn). a new option 'U' is available with these 2 routines (and in PAW with the commands HIST/FIT and VECTOR/FIT) when DOUBLE PRECISION is necessary for users functions. When the option 'U' is given, The user must reference the common block /HCFITD/ containing the current parameters in DOUBLE PRECISION and store the result of the function into the double precision variable FITFUN. Example of use in a COMIS function in a macro:

 
      macro test
      application COMIS QUIT
            FUNCTION UFIT(X)
            DOUBLE PRECISION DPAR(24),FITFUN !dimension of DPAR MUST be 24
            COMMON/HCFITD/DPAR,FITFUN
            FITFUN = DPAR(1)+DPAR(2)*SQRT(X) +DPAR(3)/X
            UFIT=FITFUN
            END
      QUIT
      vector/create par(3) r 0.5 1.2 -89.6
      Histo/Fit  10  ufit  U  3 par
      return

Changes and New options in HPLOT/HIGZ.

In addition to the Cartesian coordinate systems, Polar, cylindrical, spherical, pseudo-rapidity/phi coordinates are available for 1-D and 2-D plots, including stacked lego plots. Try for example:

      PAW > Histo/plot 10+20+30  LEGO1CYL  | stacked cylindrical lego plot
      PAW > Histo/plot 10+20+30  LEGO1POL  |         polar
      PAW > Histo/plot 10+20+30  LEGO1SPH  |         spherical
      PAW > Histo/plot 10+20+30  LEGO1PSD  |         pseudo-rapidity/phi

Note that the viewing angles may be changed via the command ANGLES.

When plotting the error bars,the symbols are drawn only if they are not exactly on the minimum or on the maximum of the current normalisation transformation window.

HPLAX (Atitle) restore the text alignment.

LOG scales in X and Y are now available for LEGO and SURFace plots.

Improvements for drawing the cursor and cross-hair (X11 mode)

Mods in various HIGZ routines for the DECGKS version (Lynn Garren)

Special characters [,],$ correctly drawn for file names (OPT FILE) and selection expressions (CSEL)

Many improvements in COMIS:

several problems reported by Walter Mueller have been fixed. (Many thanks to Walter for submitting his long list of comments)
Spurious syntax errors in continuation lines: fixed
TABS again possible.
No restrictions for list-directed I/O statements.
Inline comments with ! now possible
The following intrinsic functions have been added to the list of callable functions:IOR IEOR IAND ISHFT BTEST IBSET IBCLR and NINT
Passing arguments to the COMIS routine from the command line or in ntuple commands.
Passing character type parameters to COMIS on VAX now fixed.
The COMIS_SCRATCH file does not grow after each COMIS compilation. We hope that the next version of COMIS will not require this file anymore.
KUIP vectors are correctly managed in case of garbage collection. This was the case when the COMIS routine called HBOOK routines (HFN) or graphics routines.
Vectors can be defined in the main program
You may have several APPLication COMIS inside the same macro
The size of the COMIS stack has been increased from 10000 words to 20000 words. This allows bigger routines to be compiled. On Unix systems, the COMIS working space is dynamically allocated.
IMPLICIT NONE is now working. A Warning is issued when local or global variables are not explicitly typed.
INCLUDE files now supported. The INCLUDE statement syntax is:
```
         1. INCLUDE  name
         2. INCLUDE 'name'
         3. INCLUDE "name"
```
In 1st and 3th case the file name is converted to uppercase. In 2nd case the file case is kept.
An included file can not contain other INCLUDE statements.
The INCLUDE statement can appear anywhere in a COMIS source file. Multiple INCLUDE statements can appear in the source file.
Vladimir Berezhnoi is currently making several other improvements to COMIS (reliability and efficiency). Users experiencing problems or willing to make suggestions should contact Vladimir directly: (berezhno@hpcnas.cern.ch)
The new COMIS documentation should be available at the time of the release.

KUIP enhancements

"$name" will be replaced by value of environment variable if it is defined and "name" is not a KUIP system function.

Allow switching to STYLE G from logon kumac.

STYLE G: high-light last choice in menus; if panel is defined display it along with menus (reported by Robert Franchisseur). STYLE G could not be used in PAWLOGON.

Increased limit for number of macro arguments from 9 to 99.

IBM: disabled trapping of floating point underflows (CALL XUFLOW). Remove call to KUALFA from error handler. This caused recursive Fortran I/O in the GKS version.

DECstation : make it compile without "-common" option.

Fixes for MSDOS version using f2c and gcc (Valery Fine).

Set IQUEST(1)=0 before calling action routine to fix problem of VECTOR/READ with good filename being ignored after a VECTOR/READ with an invalid filename (reported by IVERSEN@VSFYS1.FI.UIB.NO)

KUGETx returns last value for optional parameters without default value. Interactive Geant relies on this e.g. for the DRAW command.

KUGETE did not work when prompting for missing mandatory argument (reported by Steve Fisher).

KUARGS: mismatch between code and manual. Option BATCH should NOT imply NOLOG, i.e. LOGON.KUMAC should still be executed.

Version 2.01/02 introduced that all floating point numbers were printed with a decimal point. This broke the usage of $SIGMA for index calculations. The old behaviour that e.g. $SIGMA(2*3+1) returns "7" is restored. To solve the problem which originally triggered the change a new system function $RSIGMA is available now. This function has be to used in COMIS calls expecting a REAL argument, e.g.

CALL file.f($RSIGMA(sqrt(x(1)))

Otherwise the value may be passed as an INTEGER if the SIGMA result turns out to be a whole number.

New system function $FORMAT(number,format) to format a number according to a Fortran-like FORMAT string, e.g. $FORMAT([x],F9.3). Supports F,E,G,I, and Z (hexadecimal).

New routine KUCOMV(SUB) to register a subroutine SUB(CHNAME) which is called whenever a vector is deleted. This is mainly needed for COMIS which keeps a table of defined vectors.

Filecase conversion for EXEC statements inside KUMACS.

The invisible vector ? was not correctly treated for subranges (reported by Nicole Lhermitte).

A continuation line start in the first column is concatenated as-is, e.g.

     VEC/READ x,y,_
     z,t file.dat

is equivalent to "VEC/READ x,y,z,t file.dat". Before there was put in at least one blanks between the concatenated lines.

getline(): implemented ^J/LF (delete word), ^R (refresh) and ^X for style DCL and do output buffering. Prefer Posix over BSD termio to solve problem with type-ahead on Sun.

KUIP/SHELL for Apollo: use pgm_$invoke() instead of system(). Otherwise /com/sh is suspected to overwrite memory because of an unassigned I/O stream (reported by Robert Franchisseur).

New supported Platforms

The PAW system is now available on ALPHA platforms (VMS and OSF)

The MSDOS version (NDP and GNU versions) have been consolidated by Valery Fine.

The LINUX version has been introduced (V.Balashov and A.Lomov)