October 22, 1993
A new version of PAW (2.03/22) has been released on October 22th together with CERNLIB 93D. The source files are already available on the asis system. The availability of the binary libraries, executables and tar files on asis will be announced in the coming days by the Program Library office.
New features and enhancements in this version are described below.
PIAF is a cluster of five HP-755 workstations each rated at 27 CERN units and equipped with 128 MBytes memory and 8 GBytes disk space. The machines are interconnected by an FDDI ring and accessible from the CERN wide Ethernet.
PIAF provides a service for processing large Ntuples. Users store their data files on PIAF and then use them from within a PAW session running on their local system. PAW version 2.02 incorporated the facilities to connect to a server process on one of the PIAF machines, only one server process per client. With the new version 2.03, a further speed-up can be reached where the server talking to the client is now able to launch slave servers on all available machines to process Ntuple requests in parallel. PIAF can process in a few seconds queries that would take several minutes on a workstation.
The PIAF server is available to all PAW clients running on their own workstation, on CERNVM or on the central VAXes. It can also be accessed by PAW clients outside of CERN if the connection speed is at least 64 kbits/s.
The PIAF server requires password authentication. PIAF accounts are allocated on a per-group basis with all group members sharing the same account. Each user has his own working directory corresponding to the login name on the client system.
A connection to the PIAF server is established by the PAW command CONNECT. The first time the group account name and password will be prompted for and is then kept in a local file ".piafrc" for future references. Once the server is started the usual Unix file manipulation commands "ls", "rm", "mkdir", etc. can be used. The ftp-like commands PUT and GET allow to transfer files between the client system and PIAF. A STAGE command for accessing tapes and remote data files is in preparation.
By default, PIAF is in parallel mode (up to 5 slave servers per client). For special cases (mentioned below) PIAF can be switched to sequential mode by sending the command "MODE SEQ". The only difference between local and PIAF files is in the HISTO/FILE command. PIAF files have to be indicated by the prefix "//piaf/". The remaining part of the path is relative to the user's working directory. "~" references the group directory. For example,
HISTO/FILE 1 //piaf/file | ~group/user/file
HISTO/FILE 2 //piaf/~/file | ~group/file
HISTO/FILE 3 //piaf/~/pub/file | ~group/pub/file
Afterwards the files are used like local files, e.g.
CD //LUN1
HISTO/LIST
NT/PRINT 10
NT/PLOT 10.x
Initially PIAF has been restricted to a small user community in order to gain experience with the operation and management of the service. We are now opening a public service for PAW users inside CERN and at collaborating outside institutes. PIAF can be accessed from a PAW session running in a remote site if the line speed is at least 64 Kbits/s.
The PIAF software is available in the PAW master file (PATCH PIAFS). It can be ported to any Unix platform and does not require special additional software. However, for the time being, we want to concentrate our efforts on the existing CERN platform. Cloning PIAF requires fast machines with a lot of memory (128 Mbytes per machine), fast disks (more than 4 Mbytes/s each) and a fast interconnect (more than 6 Mbytes/s).
Although we tried to make the usage of PIAF as transparent as possible there are still a few points one has to think about when running in parallel mode:
*
*-- begin of user's code
*
IF (IDNEVT .EQ. 1) THEN
CALL HBOOK1(10, ...)
CALL HBOOK2(20, ...)
...
ENDIF
...
because the start value of IDNEVT will be different on
each slave server. Only one slave server will correctly initialize
the histograms 10 and 20, while the others will fail
since they can not retrieve the definition of the histograms.
The correct way of doing things is:
LOGICAL INIT
DATA INIT /.TRUE./
*
*-- begin of user's code
*
IF (INIT) THEN
CALL HBOOK1(10, ...)
CALL HBOOK2(20, ...)
...
INIT = .FALSE.
ENDIF
...
Having said all this, it is our experience that most of the COMIS functions will work in parallel mode and the user will enjoy substantial performance increases using the parallel mode compared to the sequential mode.
The old routine PAWFCU has been replaced by a new set of routines to compile and interpret the ntuple selection expressions. These new routines are automatically generated by the YACC and LEX systems from the grammar definition stored in new PATChes YACC and LEX. We hope this new system will be more reliable and open to new expansions. The new system supports all the mechanisms already described in CNL210. Some new features relative to array processing have also been introduced:
nt/plot 10.V(2:3) will histogram columns 2 and 3 in one go in the same histogram (1000000).
nt/pl 10.V <=> nt/pl 10.V(min:max) nt/pl 10.V(6:) <=> nt/pl
10.V(6:max) nt/pl 10.V2(3:9,2) nt/pl 10.px%py 4
nt/pl 10.V vsum(V(1:3))<10 is nt/pl 10.V (V(1)+V(2)+V(3))<10
NT/PL 10.array array>>mask(1)}
but
NT/PL 10.array(1) array(1)>>mask(1)
is possible
NT/PL 10.p but NT/PL 10.p(1:3,2) is OK
It is recommended to switch to COMIS for queries involving multidimensional arrays.
Program hcwn
parameter (nwpaw=1000000)
common/pawc/paw(nwpaw)
parameter (MAXTRACK=31,MAXCLUST=1000,nevent=10000)
common/event/idrun,vx,vy,vz,pbeam1,pbeam2,ntrack,itype(MAXTRACK),
+px(MAXTRACK),py(MAXTRACK),pz(MAXTRACK),etot(MAXTRACK),
+iclust(MAXTRACK),nclust,eclust(MAXCLUST),itrack(MAXCLUST)
*
call hlimit(nwpaw)
call hropen(1,'TEST','hcwn.hbook','N',1024,istat)
call hbset('BSIZE',4096,istat)
call hbnt(1,'HCWN',' ')
*
call hbname(1,'Block1',idrun,'idrun,vx,vy,vz,pbeam1,pbeam2')
call hbname(1,'Block2',ntrack,
+'ntrack[0,31],itype(ntrack)[-20,20],px(ntrack),py(ntrack),'//
+'pz(ntrack),etot(ntrack),iclust(ntrack)[0,1000]')
call hbname(1,'Block3',nclust,'Nclust[0,1000],'//
+'eclust(nclust)[0,250],itrack(nclust)[0,31]')
*
idrun=1
do 100 ievent=1,nevent
call rannor(vx,vy)
call rannor(zz,uu)
vz=zz*10.
call rannor(dp1,dp2)
pbeam1=45.+0.1*dp1
pbeam2=45.+0.1*dp2
ntrack=rndm(ievent)*31
call rannor(xc,yc)
nclust=50.*sqrt(xc*xc+yc*yc)
if(nclust.gt.maxclust)nclust=maxclust
ttot=0.
ebeam=pbeam1+pbeam2
if(ntrack.eq.0)go to 60
do 10 it=1,ntrack
itype(it)=20*rndm(it)-10
etot(it)=rndm(ip)*50.
ttot=ttot+etot(it)
phi=6.28*rndm(it)
5 call rannor(theta,dum)
if(abs(theta).gt.3.14/2.)go to 5
px(it)=etot(it)*sin(theta)*sin(phi)
py(it)=etot(it)*sin(theta)*cos(phi)
pz(it)=etot(it)*cos(theta)
10 continue
ratio=ebeam/ttot
do 50 it=1,ntrack
etot(it)=ratio*etot(it)
px(it)=ratio*px(it)
py(it)=ratio*py(it)
pz(it)=ratio*pz(it)
iclust(it)=rndm(it)*nclust
50 continue
60 continue
do 70 ic=1,nclust
eclust(ic)=ebeam*rndm(ic)/nclust
itrack(ic)=rndm(ic)*ntrack
70 continue
*
90 call hfnt(1)
100 continue
*
call hprint(1)
call hrout(1,icycle,' ')
call hrend('TEST')
close(1)
end
******************************************************************
* Ntuple ID = 1 Entries = 10000 HCWN
******************************************************************
* Var numb * Type * Packing * Range * Block * Name *
******************************************************************
* 1 * I*4 * * * BLOCK1 * idrun
* 2 * R*4 * * * BLOCK1 * vx
* 3 * R*4 * * * BLOCK1 * vy
* 4 * R*4 * * * BLOCK1 * vz
* 5 * R*4 * * * BLOCK1 * pbeam1
* 6 * R*4 * * * BLOCK1 * pbeam2
* 1 * I*4 * * [0,31] * BLOCK2 * ntrack
* 2 * I*4 * 6 * [-20,20] * BLOCK2 * itype(ntrack)
* 3 * R*4 * * * BLOCK2 * px(ntrack)
* 4 * R*4 * * * BLOCK2 * py(ntrack)
* 5 * R*4 * * * BLOCK2 * pz(ntrack)
* 6 * R*4 * * * BLOCK2 * etot(ntrack)
* 7 * I*4 * 11 * [0,1000] * BLOCK2 * iclust(ntrack)
* 1 * I*4 * * [0,1000] * BLOCK3 * Nclust
* 2 * R*4 * * [.0000,250.0 * BLOCK3 * eclust(Nclust)
* 3 * I*4 * 6 * [0,31] * BLOCK3 * itrack(Nclust)
******************************************************************
* Block * Entries * Unpacked * Packed * Packing Factor *
******************************************************************
* BLOCK1 * 10000 * 24 * 24 * 1.000 *
* BLOCK2 * 10000 * 748 * Var. * Variable *
* BLOCK3 * 10000 * 8004 * Var. * Variable *
* Total * --- * 8776 * Var. * Variable *
******************************************************************
* Blocks = 3 Variables = 16 Max. Columns = 2194 *
******************************************************************
PAW > nt/pl 1.vx | 10000 entries
PAW > nt/pl 1.px | 150163 entries
PAW > nt/pl 1.px(1:2) | 18968 entries
PAW > nt/pl 1.px px>0 | 75013 entries
PAW > nt/pl 1.px vsum(px)<2 | 87330 entries
PAW > nt/pl 1.px abs(px)>10 | 3326 entries
PAW > nt/pl 1.px%py 4
You can now use the cut and selection mechanism described above with the function commands fun/plot, fun/fun1 and fun/fun2. These commands now use
the new set of routines to compile and interpret selection expressions.
Example:
PAW > CUTS $1 ABS(x)>2
PAW > FUN/PLOT $1.AND.X**2 -5 5
PAW > SET FCOL 1005
PAW > CUTS $2 X**2>15
PAW > FUN/PLOT $1.AND.x**2.AND.$2 -5 5
Thanks to a reconfiguration of the MINUIT package to input interactive commands and the corresponding implementation in HBOOK, it is now possible to provide MINUIT interactive commands in a macro, via the application HMINUIT. Note that Application HMINUIT must precede the Fit commands.
Example of a macro:
Macro test
Application HMINUIT EXIT
set eps 1.e-12
migrad
minos
contour 1 2
EXIT
Histo/Fit 10 G M
The HBOOK Histogram directory table is now created and expanded by chunks of 500 words and links. This speeds up considerably programs using a large number of histograms.
New routine HRSORT(CHOPT) to sort IDs in a RZ directory. HRSORT is automatically called by HLDIR if option 'S' is given. The routine HLNEXT automatically sorts IDs in increasing order. HLNEXT is the routine called by the PAW++ browser. In command mode to list the current directory in increasing IDs, do: Histo/List S To sort permanently directories in HBOOK files, attach the file in Update mode:
Example:
Histo/File 1 myfile.hbook ! U
Histo/list S
Close 1
The maximum number of integer*4, real*4 and logical columns, in CWN Ntuples, has been increased from 1000 to 5000.
The maximum number of files open simultaneously has been extended to 50.
Changes in Unix mapping memory routines. SGI moved from BSD to SYS5 class. New routines HRESETM,HRESETM1,HREZ0M to reset histograms in shared memory (from W.Bruckner). New routines HPLISM,HPLISM1 to list histograms in shared memory. Mods in HLDIR to call HPLISM. The PAW command Histo/List is now able to list the histogram directory in a VMS global section or Unix shared-memory section.
The fitting routines have been interfaced to the new Interactive MINUIT.
Protection in HRESET to not reset ntuples and to correctly reset the HFIT data structure.
New routine HPROF2 to make profile histograms from 2-D histograms
Several minor bugs fixed in CWN handling routines (PAW and HBOOK). Deck HFNT2: when value is out of range substitute max or min value, also check range when only number of bits is specified. Bug fixed in HGNTBF for indexed variables in ntuples.
There was an unpleasant feature in HBOOK on computers with IEEE 754 floating point arithmetic (eg. on DECstations under Ultrix). Calculations on these machines can give as result Infinity or Not-a-Number (NaN). If one tries to fill these numbers into histograms (and this happens in real-world applications, especially during the development phase of a program), some problems may arise:
The PIAF file system can now be browsed via the PAW++ browser.
PAW++ is now available on VMS systems.
Several cosmetic changes.
- The EPS (-113) files contains now a showpage. - Better centering of the A4 landscape PS files.
* GRAPHICS/HPLOT/AERRORS X Y EXL EXU EYL EYU N [ ISYMB SSIZE CHOPT ]
X C 'Vector of X coordinates'
Y C 'Vector of Y coordinates'
EXL C 'Vector of X error bars (Low)'
EXU C 'Vector of X error bars (Up)'
EYL C 'Vector of Y error bars (Low)'
EYU C 'Vector of Y error bars (Up)'
N I 'Number of points' D=1
ISYMB I 'Symbol number' D=24
SSIZE R 'Symbol size' D=0.28
CHOPT C 'Options' D=' '
Possible CHOPT values are:
' ' Coordinates are expressed in histogram coordinates (of the last drawn
histogram). Error bars are drawn.
C Coordinates are expressed in centimeters.
W A new window is defined and axis are drawn.
0 Draw the error bars (default).
1 Draw small lines at the end of the error bars.
2 Draw error rectangles.
3 Draw a filled area through the end points of the vertical error bars.
4 Draw a smoothed filled area through the end points of the vertical
error bars.
Draw (according to the CHOPT value) a series of points using a symbol and
asymmetric error bars in horizontal and vertical direction in the current
normalization transformation. If ISYMB = 0 or SSIZE = 0. no symbol is
drawn. Note that the options can be cumulated.
W A new window is defined and axis are drawn.
0 Draw the error bars (default).
1 Draw small lines at the end of the error bars.
2 Draw error rectangles.
3 Draw a filled area through the end points of the vertical error bars.
4 Draw a smoothed filled area through the end points of the vertical
error bars.
E Draw error bars and current marker. E0 is also allowed.
E1 Draw small lines at the end of the error bars.
E2 Draw error rectangles.
E3 Draw a filled area through the end points of the vertical error
bars.
E4 Draw a smoothed filled area through the end points of the vertical
error bars.
The COMIS interpreter has been optimized for:
All the code to support PageMemory has been removed. In particular
******************************************************
* the file "comis_scratch.dat" is not used anymore *
******************************************************
Length of character buffer for the FORMAT statement changed from 256 to 512 characters.
Maximum number of vectors changed from 100 to 500.
READONLY and SHARED options added for OPEN statement in VAX case, routine CSFILE.
Bugs fixed:
in case of unformatted READ : routine MHLOC;
in case conversion type complex to type double precision;
in CSINTX when calculating the address of name KUIP vector. ( in case "call subr(vec_name)" );
in CSKCAL for VAX version: any call to f77-routine caused an 'access violation';
in parsing expressions like (DPVAR.GT.30.);
New routines callable from COMIS
MATHLIB routines: DENLAN,DSTLAN,DIFLAN,XM1LAN,XM2LAN,RANLAN.
MINUIT routine MNERRS
HBOOK routines HGNTB and HXI
ZEBRA routine RZQUOT
HPLOT routine HPLFRA
SIGMA is undergoing a serious revision, including cutting out of unused and dead code, reduction of COMMON areas, and a certain speed-up.
At the same time, the following functions are now available:
DAWSON Dawson's Integral
DILOG Dilogarithm Function
ADIGAM Digamma or Psi Function
ASINH Hyperbolic Arcsine
ATANI Arctangent integral
FRSIN Fresnel Integrals
FRCOS Fresnel Integrals
GAMMF Gamma Function for Real Argument
GAUSIN Inverse of Gaussian Distribution
RANGAM Random Numbers in Gamma or Chisquare Distribution
STRH0 Struve Functions of Order Zero
STRH1 Struve Functions of Order One
Hooks for matrices operations are now there, although these will be available only later.
The new KUIP version, which is currently being released, contains a substantial extension to the handling of expressions in assignments, DO loops, IF statements, etc. The previous, Fortran-based parser could only handle expressions envolving one operator. For example,
a = 2
b = 3
c = [a]*[b] ==> "6"
would give an arithmetic result while in
d = [a]*[b]+1 ==> "2*3+1"
only variable substitution was performed to give a string result. In order to get an arithmetic result one had to use
d = $SIGMA([a]*[b]+1) ==> "7"
The new version contains now a Yacc-generated parser which can handle complex expressions, i.e.
d = [a]*[b]+1 ==> "7"
The system function $EVAL allows expressions to be used as command argu- ments:
message $EVAL(2*([a]*[b]+1)-[b])
Besides numerical constants the expressions can also contain references to vector elements:
vec/create v(4) r 1 2 3 4
f = v(1)*v(3) - v(2)*v(4) ==> "-5"
Expressions can also be used as vector indizes, e.g.
f = v([i]+1)
In addition to the operators "+", "-", "*", "/" the intrinsic functions ABS(x), INT(x), and MOD(x,y) are provided:
f = abs(-1.5) ==> "1.5"
f = int(5/2) ==> "2"
f = mod(7.5, 6.1) ==> "1.4"
Note that all numbers are treated as floating point values, i.e. "5/2" is equivalent to "5.0/2.0". The right-hand side of an assignment statement is only evaluated if the entire line is a valid expression. Partial arithmetic expressions are not evaluated, i.e.
f = [a]*[b]+x ==> "2*3+x"
and NOT "6+x". (However, if "x" is a vector with a single element the result is a numeric value again.) The operator "//" performs string concatenation:
g = abc//def ==> "abcdef"
The quotes around string constants can only be omitted if they do not contain any characters used as arithmetic operators:
g = abc//'(*)' ==> "abc(*)"
Otherwise the r.h.s is classified as invalid expression. Note also that numeric constants are converted into a canonical string representation with leading and trailing zeroes stripped off and using an exponent for absolute values less than 0.0001 or greater/equal than 1000000:
h = 007 ==> "7"
h = 7.00 ==> "7"
h = 0.00007 ==> "7E-5"
h = 7000000 ==> "7E+6"
If the original form should be preserved the value has to be quoted:
h = '007' ==> "007"
The new expression parser also allows to combine several comparisons in a single IF statement using the ".AND." and ".OR" operators. E.g. one can now use
IF [a]<[b] .and. [c]>0 THEN
...
ENDIF
instead of nesting
IF [a]<[b] THEN
IF [c]>0 THEN
...
ENDIF
ENDIF
and
IF [a]<[b] .or. [c]>0 THEN
...
ENDIF
instead of the clumsy
IF [a]<[b] THEN
doit = 1
ELSEIF [c]>0 THEN
doit = 1
ELSE
doit = 0
ENDIF
IF [doit]=1 THEN
...
ENDIF
The new parser removes some very inconvinient restrictions of the prev- ious version which clearly results in incompatibilities for macros which depend on obscure features of the old logic. I want to give examples of cases where the behaviour is known to be different and how these cases will have to be reformulated. Obviously,
a = 2
b = 3
c = [a]*[b]+1
IF [c]='2*3+1' THEN
...
does not work anymore because the value of "[c]" is "7". It has be written as
c = [a]//'*'//[b]//'+1'
String constants containing special characters in string comparisons have to be quoted:
s = -
IF [s]=- THEN
...
has to be written as
IF [s]='-' THEN
Note that
IF [s]=abc THEN
is still accepted. Special characters which have to be quoted are those appearing as operator:
+ - * / ( ) < = >
A variable cannot be used anymore to in the place of an operator, i.e.
op = <
IF [a][op][b] THEN
... is illegal and will have to replaced by
IF [a]$INLINE([op])[b] THEN
$INLINE is a special function to inline the variable content into the parser's input string. There are also a number of other new system functions:
args = $WORD([*],3,$WORDS([*]))
DO i=1,$WORDS([args])
arg = $WORD([args],[i])
...
ENDDO
CALL foo($QUOTE([bar]))
It can also be used if an alias containing blanks should be treated as a single token, e.g. ALIAS/CREATE htitle 'Histogram Title' 1d 10 htitle 100 0 1
htitle = 'Histogram title'
1d 10 [htitle] 100 0 1
works but
limits = 100 0 1
1d 10 'Title' [limits]
does not. With
1d 10 'Title' $UNQUOTE([limits])
the variable content will be interpreted as several tokens.
Due to the new parser it is now possible to nest system functions and to use expressions in their arguments, e.g.
foo = $lower($substring($OS//$MACHINE,[k]+1,2*[n]))
The expression parser is also implicitly called for all numeric command parameters. This makes the use of $EVAL unnecessary in most places. For example, you can now write
NULL -[x]/2 [x]/2 [y] [y]+[dy]
Before numeric arguments could only be constants or vector elements.
We would hope that the increased power of the new expression parser is ample compensation for any inconveniencies caused by the above mentioned incompatibilities. The cases known not to be working anymore seem rather pathological anyway.
We want to urge all users to try the new version in order to locate other possible incompatibilities before it becomes the default production version.
Special thanks to all users who have already tried the new version and helped to overcome the teething problems in the initial implementation.
The PAW system is now available on ALPHA/PC under WINDOWS/NT.
The LINUX version has been introduced on asis.