October 22, 1993

PAW version 2.03/22

A new version of PAW (2.03/22) has been released on October 22th together with CERNLIB 93D. The source files are already available on the asis system. The availability of the binary libraries, executables and tar files on asis will be announced in the coming days by the Program Library office.

New features and enhancements in this version are described below.


PIAF --- The Parallel Interactive Analysis Facility

PIAF is a cluster of five HP-755 workstations each rated at 27 CERN units and equipped with 128 MBytes memory and 8 GBytes disk space. The machines are interconnected by an FDDI ring and accessible from the CERN wide Ethernet.

PIAF provides a service for processing large Ntuples. Users store their data files on PIAF and then use them from within a PAW session running on their local system. PAW version 2.02 incorporated the facilities to connect to a server process on one of the PIAF machines, only one server process per client. With the new version 2.03, a further speed-up can be reached where the server talking to the client is now able to launch slave servers on all available machines to process Ntuple requests in parallel. PIAF can process in a few seconds queries that would take several minutes on a workstation.

The PIAF server is available to all PAW clients running on their own workstation, on CERNVM or on the central VAXes. It can also be accessed by PAW clients outside of CERN if the connection speed is at least 64 kbits/s.

The PIAF server requires password authentication. PIAF accounts are allocated on a per-group basis with all group members sharing the same account. Each user has his own working directory corresponding to the login name on the client system.

A connection to the PIAF server is established by the PAW command CONNECT. The first time the group account name and password will be prompted for and is then kept in a local file ".piafrc" for future references. Once the server is started the usual Unix file manipulation commands "ls", "rm", "mkdir", etc. can be used. The ftp-like commands PUT and GET allow to transfer files between the client system and PIAF. A STAGE command for accessing tapes and remote data files is in preparation.

By default, PIAF is in parallel mode (up to 5 slave servers per client). For special cases (mentioned below) PIAF can be switched to sequential mode by sending the command "MODE SEQ". The only difference between local and PIAF files is in the HISTO/FILE command. PIAF files have to be indicated by the prefix "//piaf/". The remaining part of the path is relative to the user's working directory. "~" references the group directory. For example,

 
    HISTO/FILE 1 //piaf/file                 |   ~group/user/file
    HISTO/FILE 2 //piaf/~/file               |   ~group/file
    HISTO/FILE 3 //piaf/~/pub/file           |   ~group/pub/file

Afterwards the files are used like local files, e.g.

 
    CD //LUN1
    HISTO/LIST
    NT/PRINT 10
    NT/PLOT 10.x

Initially PIAF has been restricted to a small user community in order to gain experience with the operation and management of the service. We are now opening a public service for PAW users inside CERN and at collaborating outside institutes. PIAF can be accessed from a PAW session running in a remote site if the line speed is at least 64 Kbits/s.

The PIAF software is available in the PAW master file (PATCH PIAFS). It can be ported to any Unix platform and does not require special additional software. However, for the time being, we want to concentrate our efforts on the existing CERN platform. Cloning PIAF requires fast machines with a lot of memory (128 Mbytes per machine), fast disks (more than 4 Mbytes/s each) and a fast interconnect (more than 6 Mbytes/s).


Processing Ntuple Commands in Parallel

Although we tried to make the usage of PIAF as transparent as possible there are still a few points one has to think about when running in parallel mode:

  1. Check the initialization part of COMIS functions (where you typically create histograms). Don't use the value of IDNEVT to trigger initialization. The following won't work:
  2. *
    *-- begin of user's code
    *
          IF (IDNEVT .EQ. 1) THEN
             CALL HBOOK1(10, ...)
             CALL HBOOK2(20, ...)
             ...
          ENDIF
          ...
    
        because the start value of IDNEVT will be different on
        each slave server. Only one slave server will correctly initialize
        the histograms 10 and 20, while the others will fail
        since they can not retrieve the definition of the histograms.
        The correct way of doing things is:
    
          LOGICAL INIT
          DATA    INIT /.TRUE./
    *
    *-- begin of user's code
    *
          IF (INIT) THEN
             CALL HBOOK1(10, ...)
             CALL HBOOK2(20, ...)
             ...
             INIT = .FALSE.
          ENDIF
          ...
    
  3. Writing to vectors in COMIS functions is not supported in parallel mode since the semantics of this operation is not clear (should the vectors coming from each of the slaves be added or is the contents a true/false value, etc.). To use writable vectors switch to sequential mode. Currently, however, the sequential mode does not yet support writable vectors.
  4. Currently, histogramming Ntuple character variables does not produce correct results in parallel mode (problem due to adding this kind of histograms). A new version of the server will fix this soon. Use the sequential mode for the time being when histogramming character variables.
  5. Masks are not (yet) supported on PIAF (in either mode).
  6. Files opened in COMIS functions are not shared between slave servers and therefore cannot be used to store data for cross-reference between events. Use the sequential mode in case your COMIS function depends on this feature. Files created on PIAF can be retrieved using the GET command.
  7. In general, if any assumption is made on the processing order of events in a COMIS function use the sequential mode.

Having said all this, it is our experience that most of the COMIS functions will work in parallel mode and the user will enjoy substantial performance increases using the parallel mode compared to the sequential mode.


New Ntuple facilities

The old routine PAWFCU has been replaced by a new set of routines to compile and interpret the ntuple selection expressions. These new routines are automatically generated by the YACC and LEX systems from the grammar definition stored in new PATChes YACC and LEX. We hope this new system will be more reliable and open to new expansions. The new system supports all the mechanisms already described in CNL210. Some new features relative to array processing have also been introduced:

  1. you can access an array by its name and a description of its index
  2. nt/plot 10.V(2:3) will histogram columns 2 and 3 in one go in the same histogram (1000000).

    nt/pl 10.V <=> nt/pl 10.V(min:max) nt/pl 10.V(6:) <=> nt/pl 10.V(6:max) nt/pl 10.V2(3:9,2) nt/pl 10.px%py 4 are also allowed.

  3. The vsum function
  4. nt/pl 10.V vsum(V(1:3))<10 is nt/pl 10.V (V(1)+V(2)+V(3))<10

  5. Restriction(s)
  6. Examples with CWN ntuples
    1. Example to generate a CWN ntuple with variable length array
    2.       Program hcwn
            parameter (nwpaw=1000000)
            common/pawc/paw(nwpaw)
            parameter (MAXTRACK=31,MAXCLUST=1000,nevent=10000)
            common/event/idrun,vx,vy,vz,pbeam1,pbeam2,ntrack,itype(MAXTRACK),
           +px(MAXTRACK),py(MAXTRACK),pz(MAXTRACK),etot(MAXTRACK),
           +iclust(MAXTRACK),nclust,eclust(MAXCLUST),itrack(MAXCLUST)
      
      *
            call hlimit(nwpaw)
            call hropen(1,'TEST','hcwn.hbook','N',1024,istat)
            call hbset('BSIZE',4096,istat)
            call hbnt(1,'HCWN',' ')
      *
            call hbname(1,'Block1',idrun,'idrun,vx,vy,vz,pbeam1,pbeam2')
            call hbname(1,'Block2',ntrack,
           +'ntrack[0,31],itype(ntrack)[-20,20],px(ntrack),py(ntrack),'//
           +'pz(ntrack),etot(ntrack),iclust(ntrack)[0,1000]')
            call hbname(1,'Block3',nclust,'Nclust[0,1000],'//
           +'eclust(nclust)[0,250],itrack(nclust)[0,31]')
      *
            idrun=1
            do 100 ievent=1,nevent
               call rannor(vx,vy)
               call rannor(zz,uu)
               vz=zz*10.
               call rannor(dp1,dp2)
               pbeam1=45.+0.1*dp1
               pbeam2=45.+0.1*dp2
               ntrack=rndm(ievent)*31
               call rannor(xc,yc)
               nclust=50.*sqrt(xc*xc+yc*yc)
               if(nclust.gt.maxclust)nclust=maxclust
               ttot=0.
               ebeam=pbeam1+pbeam2
               if(ntrack.eq.0)go to 60
               do 10 it=1,ntrack
                  itype(it)=20*rndm(it)-10
                  etot(it)=rndm(ip)*50.
                  ttot=ttot+etot(it)
                  phi=6.28*rndm(it)
         5        call rannor(theta,dum)
                  if(abs(theta).gt.3.14/2.)go to 5
                  px(it)=etot(it)*sin(theta)*sin(phi)
                  py(it)=etot(it)*sin(theta)*cos(phi)
                  pz(it)=etot(it)*cos(theta)
        10     continue
               ratio=ebeam/ttot
               do 50 it=1,ntrack
                  etot(it)=ratio*etot(it)
                  px(it)=ratio*px(it)
                  py(it)=ratio*py(it)
                  pz(it)=ratio*pz(it)
                  iclust(it)=rndm(it)*nclust
        50     continue
        60     continue
               do 70 ic=1,nclust
                  eclust(ic)=ebeam*rndm(ic)/nclust
                  itrack(ic)=rndm(ic)*ntrack
        70     continue
      *
        90     call hfnt(1)
       100  continue
      *
            call hprint(1)
            call hrout(1,icycle,' ')
            call hrend('TEST')
            close(1)
            end
      
    3. Result of the NT/PRINT 1 command
    4.      ******************************************************************
           * Ntuple ID = 1      Entries = 10000     HCWN
           ******************************************************************
           * Var numb * Type * Packing *    Range     *  Block   *  Name    *
           ******************************************************************
           *      1   * I*4  *         *              * BLOCK1   * idrun
           *      2   * R*4  *         *              * BLOCK1   * vx
           *      3   * R*4  *         *              * BLOCK1   * vy
           *      4   * R*4  *         *              * BLOCK1   * vz
           *      5   * R*4  *         *              * BLOCK1   * pbeam1
           *      6   * R*4  *         *              * BLOCK1   * pbeam2
           *      1   * I*4  *         * [0,31]       * BLOCK2   * ntrack
           *      2   * I*4  *    6    * [-20,20]     * BLOCK2   * itype(ntrack)
           *      3   * R*4  *         *              * BLOCK2   * px(ntrack)
           *      4   * R*4  *         *              * BLOCK2   * py(ntrack)
           *      5   * R*4  *         *              * BLOCK2   * pz(ntrack)
           *      6   * R*4  *         *              * BLOCK2   * etot(ntrack)
           *      7   * I*4  *    11   * [0,1000]     * BLOCK2   * iclust(ntrack)
           *      1   * I*4  *         * [0,1000]     * BLOCK3   * Nclust
           *      2   * R*4  *         * [.0000,250.0 * BLOCK3   * eclust(Nclust)
           *      3   * I*4  *    6    * [0,31]       * BLOCK3   * itrack(Nclust)
           ******************************************************************
           *  Block   *  Entries  * Unpacked * Packed *   Packing Factor    *
           ******************************************************************
           * BLOCK1   *  10000    * 24       * 24     *       1.000         *
           * BLOCK2   *  10000    * 748      * Var.   *    Variable         *
           * BLOCK3   *  10000    * 8004     * Var.   *    Variable         *
           * Total    *    ---    * 8776     * Var.   *    Variable         *
           ******************************************************************
           * Blocks = 3            Variables = 16      Max. Columns = 2194  *
           ******************************************************************
      
    5. Examples of selections on the previously build ntuple hcwn.hbook


New Function facilities

You can now use the cut and selection mechanism described above with the function commands fun/plot, fun/fun1 and fun/fun2. These commands now use

the new set of routines to compile and interpret selection expressions.

Example:

        PAW > CUTS $1 ABS(x)>2
        PAW > FUN/PLOT $1.AND.X**2 -5 5
        PAW > SET FCOL 1005
        PAW > CUTS $2 X**2>15
        PAW > FUN/PLOT $1.AND.x**2.AND.$2 -5 5


APPLIcation HMINUIT

Thanks to a reconfiguration of the MINUIT package to input interactive commands and the corresponding implementation in HBOOK, it is now possible to provide MINUIT interactive commands in a macro, via the application HMINUIT. Note that Application HMINUIT must precede the Fit commands.

Example of a macro:

      Macro test
      Application HMINUIT EXIT
         set eps 1.e-12
         migrad
         minos
         contour 1 2
      EXIT
      Histo/Fit 10 G M


Changes and New options in HBOOK.

The HBOOK Histogram directory table is now created and expanded by chunks of 500 words and links. This speeds up considerably programs using a large number of histograms.

New routine HRSORT(CHOPT) to sort IDs in a RZ directory. HRSORT is automatically called by HLDIR if option 'S' is given. The routine HLNEXT automatically sorts IDs in increasing order. HLNEXT is the routine called by the PAW++ browser. In command mode to list the current directory in increasing IDs, do: Histo/List S To sort permanently directories in HBOOK files, attach the file in Update mode:

Example:

     Histo/File 1 myfile.hbook ! U
     Histo/list S
     Close 1

The maximum number of integer*4, real*4 and logical columns, in CWN Ntuples, has been increased from 1000 to 5000.

The maximum number of files open simultaneously has been extended to 50.

Changes in Unix mapping memory routines. SGI moved from BSD to SYS5 class. New routines HRESETM,HRESETM1,HREZ0M to reset histograms in shared memory (from W.Bruckner). New routines HPLISM,HPLISM1 to list histograms in shared memory. Mods in HLDIR to call HPLISM. The PAW command Histo/List is now able to list the histogram directory in a VMS global section or Unix shared-memory section.

The fitting routines have been interfaced to the new Interactive MINUIT.

Protection in HRESET to not reset ntuples and to correctly reset the HFIT data structure.

New routine HPROF2 to make profile histograms from 2-D histograms

Several minor bugs fixed in CWN handling routines (PAW and HBOOK). Deck HFNT2: when value is out of range substitute max or min value, also check range when only number of bits is specified. Bug fixed in HGNTBF for indexed variables in ntuples.

There was an unpleasant feature in HBOOK on computers with IEEE 754 floating point arithmetic (eg. on DECstations under Ultrix). Calculations on these machines can give as result Infinity or Not-a-Number (NaN). If one tries to fill these numbers into histograms (and this happens in real-world applications, especially during the development phase of a program), some problems may arise:

  1. Filling of +/- infinity as value. This case is handled properly by the checks for over/underflow and the value is counted as such.
  2. Filling of NaN as value. The over/underflow checks fail because relations like .LT. or .GE. are always false if one of the operands is NaN. Then the value is converted to a bin address, which is an undefined integer and finally results in either a segmentation fault or an overwriting of memory somewhere. However, this could easily be fixed, for example in HF1 (the other routines are analogous): By changing the overflow check from .GE. to .NOT. .LT. the NaN value will be counted as overflow. This change, proposed by Ulrich Mueller has been made in all HBOOK filling routines
  3. Filling of Infinity or NaN as a weight. This is maybe the worst case. The value is added to the bin, and this resulted in a crash when the ZEBRA structure was written to a file, or in a crash of PAW if one tries to plot the bad histogram. Ulrich Mueller has suggested a change specific to some Unix machines. The suggested change has not been incorporated. However, thanks to a modification in ZEBRA/RZ, you should not see anymore a crash when writing the histogram or ntuple into a file.


Changes in PAW++

The PIAF file system can now be browsed via the PAW++ browser.

PAW++ is now available on VMS systems.

Several cosmetic changes.


Changes and New options in HPLOT/HIGZ.

HIGZ

FALCO driver (7878, 7879):

Text drawing:

3d plots:

PostScript:

HPLOT

New command GRAPHICS/HPLOT/AERRORS:

 
 * GRAPHICS/HPLOT/AERRORS X Y EXL EXU EYL EYU N [ ISYMB SSIZE CHOPT ]
 
 
   X          C 'Vector of X coordinates'
   Y          C 'Vector of Y coordinates'
   EXL        C 'Vector of X error bars (Low)'
   EXU        C 'Vector of X error bars (Up)'
   EYL        C 'Vector of Y error bars (Low)'
   EYU        C 'Vector of Y error bars (Up)'
   N          I 'Number of points' D=1
   ISYMB      I 'Symbol number' D=24
   SSIZE      R 'Symbol size' D=0.28
   CHOPT      C 'Options' D=' '
 
   Possible CHOPT values are:
 
   ' ' Coordinates are expressed in histogram coordinates (of the last drawn
       histogram). Error bars are drawn.
    C  Coordinates are expressed in centimeters.
    W  A new window is defined and axis are drawn.
    0  Draw the error bars (default).
    1  Draw small lines at the end of the error bars.
    2  Draw error rectangles.
    3  Draw a filled area through the end points of the vertical error bars.
    4  Draw a smoothed filled area through the end points of the vertical
       error bars.
 
   Draw (according to the CHOPT value) a series of points using a symbol and
   asymmetric error bars in horizontal and vertical direction in the current
   normalization transformation.  If ISYMB = 0 or SSIZE = 0. no symbol is
   drawn.  Note that the options can be cumulated.

New options in GRAPHICS/HPLOT/ERRORS

    W  A new window is defined and axis are drawn.
    0  Draw the error bars (default).
    1  Draw small lines at the end of the error bars.
    2  Draw error rectangles.
    3  Draw a filled area through the end points of the vertical error bars.
    4  Draw a smoothed filled area through the end points of the vertical
       error bars.

New options in HISTOGRAM/PLOT

    E      Draw error bars and current marker. E0 is also allowed.
    E1     Draw small lines at the end of the error bars.
    E2     Draw error rectangles.
    E3     Draw a filled area through the end points of the vertical error
           bars.
    E4     Draw a smoothed filled area through the end points of the vertical
           error bars.


Improvements in COMIS:

The COMIS interpreter has been optimized for:

  1. character expressions and assignments;
  2. calls to F77-routines when arguments are:
  3. calls to CS-routines;
  4. access to array elements with constant subscript values;
  5. relational operators like "exp.eq.0", "exp.ne.0".

All the code to support PageMemory has been removed. In particular

        ******************************************************
        *  the file "comis_scratch.dat" is not used anymore  *
        ******************************************************

Length of character buffer for the FORMAT statement changed from 256 to 512 characters.

Maximum number of vectors changed from 100 to 500.

READONLY and SHARED options added for OPEN statement in VAX case, routine CSFILE.

Bugs fixed:

in case of unformatted READ : routine MHLOC;

in case conversion type complex to type double precision;

in CSINTX when calculating the address of name KUIP vector. ( in case "call subr(vec_name)" );

in CSKCAL for VAX version: any call to f77-routine caused an 'access violation';

in parsing expressions like (DPVAR.GT.30.);

New routines callable from COMIS

       MATHLIB routines: DENLAN,DSTLAN,DIFLAN,XM1LAN,XM2LAN,RANLAN.
       MINUIT routine MNERRS
       HBOOK routines HGNTB and HXI
       ZEBRA routine RZQUOT
       HPLOT routine HPLFRA


Improvements in SIGMA:

SIGMA is undergoing a serious revision, including cutting out of unused and dead code, reduction of COMMON areas, and a certain speed-up.

At the same time, the following functions are now available:

 
 
          DAWSON       Dawson's Integral
          DILOG        Dilogarithm Function
          ADIGAM       Digamma or Psi Function
          ASINH        Hyperbolic Arcsine
          ATANI        Arctangent integral
          FRSIN        Fresnel Integrals
          FRCOS        Fresnel Integrals
          GAMMF        Gamma Function for Real Argument
          GAUSIN       Inverse of Gaussian Distribution
          RANGAM       Random Numbers in Gamma or Chisquare Distribution
          STRH0        Struve Functions of Order Zero
          STRH1        Struve Functions of Order One

Hooks for matrices operations are now there, although these will be available only later.


KUIP enhancements

The new KUIP version, which is currently being released, contains a substantial extension to the handling of expressions in assignments, DO loops, IF statements, etc. The previous, Fortran-based parser could only handle expressions envolving one operator. For example,

 
    a = 2
    b = 3
    c = [a]*[b]                 ==> "6"

would give an arithmetic result while in

 
    d = [a]*[b]+1               ==> "2*3+1"

only variable substitution was performed to give a string result. In order to get an arithmetic result one had to use

 
    d = $SIGMA([a]*[b]+1)       ==> "7"

The new version contains now a Yacc-generated parser which can handle complex expressions, i.e.

 
    d = [a]*[b]+1               ==> "7"

The system function $EVAL allows expressions to be used as command argu- ments:

 
    message $EVAL(2*([a]*[b]+1)-[b])

Besides numerical constants the expressions can also contain references to vector elements:

 
    vec/create v(4) r 1 2 3 4
    f = v(1)*v(3) - v(2)*v(4)   ==> "-5"

Expressions can also be used as vector indizes, e.g.

 
    f = v([i]+1)

In addition to the operators "+", "-", "*", "/" the intrinsic functions ABS(x), INT(x), and MOD(x,y) are provided:

 
    f = abs(-1.5)               ==> "1.5"
    f = int(5/2)                ==> "2"
    f = mod(7.5, 6.1)           ==> "1.4"

Note that all numbers are treated as floating point values, i.e. "5/2" is equivalent to "5.0/2.0". The right-hand side of an assignment statement is only evaluated if the entire line is a valid expression. Partial arithmetic expressions are not evaluated, i.e.

 
    f = [a]*[b]+x               ==> "2*3+x"

and NOT "6+x". (However, if "x" is a vector with a single element the result is a numeric value again.) The operator "//" performs string concatenation:

 
    g = abc//def                ==> "abcdef"

The quotes around string constants can only be omitted if they do not contain any characters used as arithmetic operators:

 
    g = abc//'(*)'              ==> "abc(*)"

Otherwise the r.h.s is classified as invalid expression. Note also that numeric constants are converted into a canonical string representation with leading and trailing zeroes stripped off and using an exponent for absolute values less than 0.0001 or greater/equal than 1000000:

 
    h = 007                     ==> "7"
    h = 7.00                    ==> "7"
    h = 0.00007                 ==> "7E-5"
    h = 7000000                 ==> "7E+6"

If the original form should be preserved the value has to be quoted:

 
    h = '007'                   ==> "007"

The new expression parser also allows to combine several comparisons in a single IF statement using the ".AND." and ".OR" operators. E.g. one can now use

 
    IF [a]<[b] .and. [c]>0 THEN
       ...
    ENDIF

instead of nesting

 
    IF [a]<[b] THEN
       IF [c]>0 THEN
          ...
       ENDIF
    ENDIF

and

 
    IF [a]<[b] .or. [c]>0 THEN
       ...
    ENDIF

instead of the clumsy

 
    IF [a]<[b] THEN
       doit = 1
    ELSEIF [c]>0 THEN
       doit = 1
    ELSE
       doit = 0
    ENDIF
    IF [doit]=1 THEN
       ...
    ENDIF

INCOMPATIBILITIES:

The new parser removes some very inconvinient restrictions of the prev- ious version which clearly results in incompatibilities for macros which depend on obscure features of the old logic. I want to give examples of cases where the behaviour is known to be different and how these cases will have to be reformulated. Obviously,

 
    a = 2
    b = 3
    c = [a]*[b]+1
    IF [c]='2*3+1' THEN
       ...

does not work anymore because the value of "[c]" is "7". It has be written as

 
    c = [a]//'*'//[b]//'+1'

String constants containing special characters in string comparisons have to be quoted:

 
    s = -
    IF [s]=- THEN
       ...

has to be written as

 
    IF [s]='-' THEN

Note that

 
    IF [s]=abc THEN

is still accepted. Special characters which have to be quoted are those appearing as operator:

 
    + - * / ( ) < = >

A variable cannot be used anymore to in the place of an operator, i.e.

 
    op = <
    IF [a][op][b] THEN

... is illegal and will have to replaced by

 
    IF [a]$INLINE([op])[b] THEN

$INLINE is a special function to inline the variable content into the parser's input string. There are also a number of other new system functions:

Due to the new parser it is now possible to nest system functions and to use expressions in their arguments, e.g.

 
   foo = $lower($substring($OS//$MACHINE,[k]+1,2*[n]))

The expression parser is also implicitly called for all numeric command parameters. This makes the use of $EVAL unnecessary in most places. For example, you can now write

 
   NULL -[x]/2 [x]/2 [y] [y]+[dy]

Before numeric arguments could only be constants or vector elements.

We would hope that the increased power of the new expression parser is ample compensation for any inconveniencies caused by the above mentioned incompatibilities. The cases known not to be working anymore seem rather pathological anyway.

We want to urge all users to try the new version in order to locate other possible incompatibilities before it becomes the default production version.

Special thanks to all users who have already tried the new version and helped to overcome the teething problems in the initial implementation.


New supported Platforms

The PAW system is now available on ALPHA/PC under WINDOWS/NT.

The LINUX version has been introduced on asis.


Release NotesKnown bugsFAQsContributionsTutorialReference manualDown loadMiscellaneous

Paw.Support@cern.ch