April 4, 1996

PAW version 2.07/22

The PAW version 2.07/22 is part of the 96A release of CERNLIB. The source files, binaries and libraries will be available, via anonymous ftp, on the asis system:

   ftp asisftp.cern.ch

The PAW team is:

Contributions from:

In the file the following subjects can be found:


Release Highlights

The 96A CERNLIB release will be the last major PAW release, although PAW will naturally have to be supported throughout the lifetime of the LEP experiments. An essential step will be the gathering of USER REQUIREMENTs for a future data analysis tool, including requirements for the components e.g. histograming, minimization, data input.

In this release some importants points are:


New version of the Ntuple Query Processor

Interactive Analysis of Ntuples in PAW.

The interactive analysis of high energy physics event data using HBOOK Ntuples has always been one of the major strengths of PAW. With the design and implementation of the PIAF system and the advent of Column Wise Ntuples (CWNs) the amount of event data which can be analyzed interactively has been significantly increased. Other improvements due to the use of CWNs include the support for more data types and new possibilities for structuring the data. CWNs can store boolean, bit field, integer, real, double precision and character string data types. Data can be structured in arrays of one or more dimensions, where the last dimension of each array can be of variable length. This allows for efficient storage of lists of values, lists of vectors etc. The actual length of the array in a given event is defined by an integer variable in the Ntuple.

In the following table the main commands for accessing Ntuples in PAW are listed. For all commands the arguments IDN, UWFUNC, NEVENT and IFIRST can be specified.

 +------------------+--------------------------------------------------+
 |     Command      |    Arguments [ Optional Arguments ]              |
 +------------------+--------------------------------------------------+
 |     LOOP         |    IDN UWFUNC [ NEVENT IFIRST ]                  |
 +------------------+--------------------------------------------------+
 |     PROJECT      |    IDH IDN [ UWFUNC NEVENT IFIRST ]              |
 +------------------+--------------------------------------------------+
 |     PLOT         |    IDN [ UWFUNC NEVENT IFIRST NUPD OPTION IDH ]  |
 +------------------+--------------------------------------------------+
 |     SCAN         |    IDN [ UWFUNC NEVENT IFIRST OPTION VARLIS ]    |
 +------------------+--------------------------------------------------+
 |     DUMP         |    IDN [ UWFUNC NEVENT IFIRST FILENAME OPTION ]  |
 +------------------+--------------------------------------------------+

The LOOP command is somewhat special because it just calls a user function (COMIS) to process each event.

The part of PAW which evaluates the expressions and performs the appropriate actions on the values is called the Query Processor. An important property of the Query Processor is the language in which the expressions are expressed.

Motivation for a new Query Processor

The current version of the system was developed a few years ago to accommodate the features that became available with the Column Wise Ntuples as described above. It also significantly extended the language which is used to formulate the queries.

Sadly enough the software version is unstable and difficult to modify. It suffers from arbitrary limits on the size of expressions and internal tables. It only handles arrays correctly in the simplest cases. Finally it understands only two types, float and string, which makes it impossible to call COMIS functions which take integer or double arguments.

The aim of the new query processor is to have a more powerful syntax, to fully support all types available in the Column Wise Ntuples and to remove most limitations.

New features

In this section the most important features of the new Query Processor are presented. It is maybe useful to remark that all of the old functionality is also reimplemented and that many bugs that caused erroneous input to be accepted are no longer present. It is hoped that implementation of the new processor will result in easier maintenance, and easier extensibility, and a better system for the user.

Regular syntax and full type support

The new query processor supports all types available in column wise Ntuples: boolean, unsigned integer, signed integer, real and string. The integer and real types are available in multiple sizes, 32 and 64 bit, if the hardware support is available. This is normally only the case for type real. The numerical types can be combined in expressions, in which case automatic type conversions are applied. The new syntax is fully recursive and therefore allows expressions within expressions. It is also allowed to have indices (re)calculated for each event. The rigorous checking of the syntax and types gives predictable results. In the new system most tables are allocated dynamically and grow when needed. This will alleviate the most common problem with the old system which is the overflowing of fixed length internal tables.

Shape mapping of (dynamic) arrays

A uniform principle has been adopted for the use of arrays in expressions. The operators are applied element by element, e.g. when adding two arrays they have to have the same shape, and the result is an array of the same shape where each element is the sum of the corresponding elements in the two operands. If one of the operands is a scalar, e.g. multiplying a scalar and an array then the result is again an array of the same shape as the input array with each element the product of the scalar and the corresponding element. It is important to realize how this principle works out in the case of such operators as `less-than' and boolean `and' and `or'. Following the same principle the plot expressions and the weight are processed. This allows for one weight for all plotted values but also for a specific weight for each element of an array. Typical use of these features will be shown in the examples below. The query processor checks the proper matching of shapes statically as far as possible. When variable length arrays are used or slices of arrays the proper matching is checked at runtime.

New functions to manipulate array variables

KUIP vectors

The new query processor allows KUIP vectors to be used in expressions just like Ntuple variables.

One of the benefits is the possibility of parameterized expressions and selections.

The second benefit is an added level of indirection, an Ntuple variable can be used as an index of a KUIP array. The use of this will be illustrated in the examples.

Enhanced access to COMIS

Access to COMIS is improved in several ways:

Improved Cuts

The old query processor used cuts as a macro mechanism, a textual substitution was carried out before interpreting the expression.

In the new system cuts are treated as function calls. The cut expression is evaluated once for each event and the result is cached.

A complex sub-expression, e.g. a call to COMIS or array slice, can now be used in both the selection and the plot expressions without being evaluated more than once.

Examples

The following examples illustrate new features of the new query processor. The first example is fairly simple:

               PAW > ntuple/plot 10.a%log(b) c+d>42

If we assume a, b, c and d to be scalar variables this will create a plot where for each event the point (a,log(b)) is plotted if the value of c plus d for that event is larger than 42. If a, b and c are matrices of dimension (3,4), d still scalar then effectively the following happens:

                  For i = 1 To 3 Do
                     For j = 1 To 4 Do
                        Ntuple/Plot 10.a(i,j)%log(b(i,j)) c(i,j)+d>42
                     End
                  End

internally this is of course executed much more efficiently. This is the mechanism by which all non scalar values are combined in expressions. It should be noted that not all built-in functions like "log" will be expanded this way. For the moment COMIS functions never expand like this. Next the use of a KUIP vector is shown:

               PAW > ntuple/plot 10.tdcVal-pedestal(istrip+100*iplane)

in this example tdcVal, iStrip and iPlane are scalar Ntuple variables and pedestal is a KUIP vector. For each event the index is calculated and the right element of pedestal is subtracted from tdcVal.

A future extension mentioned below will allow this expression to work if all variables are vectors. In the following example cuts and COMIS functions are illustrated:

               PAW > ntuple/cut 1 myfun.f(2*ix,sin(y))

               PAW > ntuple/plot 10.x%$1 0..lt.$1.and.0.lt.x.lt.1

the first command defines a cut in which a user supplied fortran function myfun() is to be interpreted using COMIS, it is called with an integer and a real argument. The arguments are expressions using the ix and y variables. It is checked if myfun() is supposed to be called with these types of argument and what the return type is. When the second command is executed, the value of cut $1 is calculated and used in the selection expression. If the selection evaluates to non-zero a point is plotted using the cached value of $1. The type of the value of a cut need not be a scalar (as in this example) but can be any type acceptable to the query processor.

Performances

As explain in the previous paragraph, the new Ntuple queries processor is now complete (all kind of Ntuples variables are handle), has more functionnalities and do more checking on the validity of the query.

Because of that the time to complete a query can be longer than with the old system. Depending on the complexity of query, the size of the ntuple, the type of ntuple (RWN or CWN), the machine used a factor between 1 and 3 may be observed.

Note a more detailed performance comparison between the old and the new query processor has been done. It shows that used in the same conditions the new query processor is not slower.

Compatibility with the previous version

The new query processor is largely compatible with the old one. There is some incompatibilities listed here. The incompatabilities have been introduced to improve coherence of the query processor. There is nothing which can be done before and is impossible now.

  1. The most important difference is the evaluation order of the selection and plot expressions, which has been reversed. The new system evaluates the selection expression first and only evaluates the plot expressions when necessary.
  2. The internal variables VIDN1, VIDN2 and VIDN3 can no longer be supported in COMIS selection functions. The use of these (scalar) variables was anyway not well defined in the case of non scalar expressions. In all cases these variables can be replaced by cuts or arguments to user functions.
  3. With the old query processor expression like:
  4.         NT/PLOT 10
    

    was allowed. In a such case the first variable of the ntuple was ploted. With the new query processor this kind of construct is not allowed ie: a variable name or an expression of variable must be specified .

  5. With the old query processor it was possible to access the ntuple variables by number. In expression like:
  6.         NT/PLOT 10.1
    

    It was not clear if 1 means ``the variable number 1'' or ``the value 1''. Now ntuple variables can be accessed only by name.

  7. In the command NT/LOOP, the ntuple identifier is now an integer only. No variables name or expression of variables is allowed. In the previous version it was simply ignore. In the previous version one can typed:
  8.        NT/LOOP 10.X FUNC.F
    

    X was ignored but accessible in the VIDNx variables inside FUNC.F. Now if one want to have access to a ntuple variable in FUNC.F just do:

           NT/LOOP 10 FUNC.F(X)
    

    No expression is allowed after the dot of the ntuple identifier. Note that the number of parameter given to the function is not limited.

  9. With the old query processor it was possible to use a COMIS function with parameters without specifying the parameters. Now all the required parameters must be specified. If one or several parameters are ommited the following error message is printed:
  10.        /NTUPLE/PLOT: Wrong number of arguments for 'SELECT.FOR' (expect 1)
    

  11. In the old version of the query processor, file names (for COMIS functions) with all kind of non alphanumeric characters where allowed. But sometimes, a file name was ambigous with other construct. For example on VMS characters like [,] can be used or / in UNIX file names which can be interpreted as a divide operation. To escape from those kind of ambiguties, file names can be enclose with double quotes "...". Note that for simple file names like myfile.f the double quotes can be ommited.
  12. The command MASK has been replaced by a set of commands. The old masks generated with the old command MASK are incompatible with the new commands. The masks should be regenerated once.
  13. The command GCUT replace the option 'G' of the command CUT.
  14. The cuts can be written in ASCII file with the option 'W' of the command CUT. The format of these file has changed and is more coherent that the previous one. The conversion from the old format to the new format is not difficult.
  15. With the old version of the query processor, it was possible to end a loop on a ntuple by returning a negative value. In:
           NT/LOOP 10 FUNC.F(X)
    
    The loop stoped when FUNC.F was negative. This has been suppressed because it forbid negative weights. To limit the number of loop use the NEVENT parameter.

For more details see the complete description of the new query processor features.


PAW

New command for graphical cuts manipulation

New set of commands for MASKs manipulation

Other new commands

New features, improvements and bug fixes

Help improvements

PIAF

System Functions

PAW++

Obsolete functionnalities


HBOOK

New routines

New features, improvements and bug fixes


HPLOT

New features

Improvements

Bug fixes


HIGZ

New features

Improvements

Bug Fixes


KUIP

New HELP Mechanism

The new ``HELP mechanism'' as descibed in CNL 219 (section 5.3) will be operational with this release. The major change is the introduction of a new directive ''>Keyword'' in the KUIP Command Definition File (CDF). From a user point of view this means that, for any KUIP-based application, instead of getting help only on valid command names, users can get help associated with keywords. For instance, in PAW, when typing :

   PAW > help editor    

one gets:

   Keyword "EDITOR" found in commands:

    1: * /KUIP/EDIT    Invoke the editor on the file.
    2: * /KUIP/SET_SHOW/HOST_EDITOR  
                       Set the host command to invoke the editor.
   ______
    0:                 Top level menu
   Enter a number ('Q'=command mode): q

N.B. The introduction of ``Keywords'' for the PAW CDF files is not yet finalized and is under way. If you think a specific keyword should be added you can send a mail to paw.support@cern.ch.

Although users might see little differences in the output given by this new HELP compared to what they were used, we think that the funtionality (or information provided) has been kept.

The new HELP for the command /KUIP/HELP is:

* KUIP/HELP [ ITEM OPTION ]

   ITEM       C 'Command or menu name or keyword(s)' D=' '
   OPTION     C 'Option' D='N'

Possible OPTION values are:

    EDIT     The help text is written to a file and the editor is invoked,
    NOEDIT   The help text is output on the terminal output.
    KEYWORD  give access to all commands associated to that keyword(s).
    E        Same as 'EDIT'.
    N        Same as 'NOEDIT'
    K        Same as 'KEYWORD'

Find help information by command name, menu name or keywords.

If ITEM is a valid command name (and there is only one such command) then full explanation on that command is provided: syntax (as given by the command USAGE), functionality, list of parameters with their attributes. If ITEM also corresponds to other commands associated to it with a "keyword" then a "See also" message, followed by the names of these commands is given.

If ITEM is a menu (or a submenu) a dialogue is guiding the user in traversing the tree command structure for getting full explanation on a specific command from that tree.

If HELP is entered without parameters, the search start from the top level menu and the user is guided in traversing the complete tree command structure.

'HELP -KEYWORD' (or 'HELP -K') followed by one or more keywords causes HELP to give access to all commands associated to that (list of) keyword(s). If the keyword corresponds to a valid command or (sub)menu name all corresponding commands are accessible. This option is especially useful when you do not know the exact name of a valid command or menu and you can only describe it by its functionality (e.g. 'HELP -KEYWORD POSTSCRIPT').

N.B. If ITEM does not correspond to any valid command or menu name then the option `-KEYWORD' is automatically invoked.

'HELP -EDIT' (or just 'CHELP -E') switches to edit mode: instead of writing the help text to the terminal output, it is written into a temporary file and the pager or editor defined by the command HOST_PAGER is invoked. (On Unix workstations the pager can be defined to display the help text asynchrously in a separated window.) 'CHELP -NOEDIT' (or just 'CHELP -N') switches back to standard mode. The startup value is system dependent.

See also:

    /KUIP/USAGE, /KUIP/MANUAL

Important note to application programmers:

This modification affecting the KUIP CDF, it means that application programmers who based their application on KUIP MUST re-compile all the CDF files related to the application (otherwise the executable module will fail at the very beginning with an error message).

Various bugs correction

Motif interface


COMIS

Shared libraries

The Shared libraries mechanism is implemented on two new platforms

Various improvments


Release NotesKnown bugsFAQsContributionsTutorialReference manualDown loadMiscellaneous

Paw.Support@cern.ch