create new tag
view all tags
Hello,     I upgraded trend's soft/sim/python code at CC. 
Let me recall that CC IN2P3 has upgraded from SL5 to SL6. 
Also, the new default python is 2.7. Python 2.6 does not seem to be supported anymore. 
For example, numpy, scipy and pylab modules are only installed for python 2.7. 
The good news is that 2.7 is backward compatible with 2.6 such that, except a few unrelated bugs that I found meanwhile, I only had to change a bit the makefiles and recompile the shared objects. 
Below are step by step instructions for running the production tools at CCIN2P3. Note that you could as well svn checkout soft/sim/python locally, compile *.so and run them from your local PC, if more convenient for you. 
Especially if you want to show up plots ect ... that would be much more convenient for you actually ...     
I) Context     
- In order to simulate our current data taking setup, or a future GIANT one, we need to compute the radio signal emission from ultra high energy particle showers (10^16+ eV, radio source in ultra relativistic motion) developing in the atmosphere. 
The good news is that, that for, we have a toolsuit: EVA/conex+EVA/radio developed by experts in the field, which allows precise computation of the radio signals taking non trivial relativistic effects into account. 
The bad news is that a precise computation is long. Therefore, we need to deploy this over batch at CCIN2P3 or the grid.    
 - The computation can be separated in 2 two main tasks: the shower simulation (~1-10 h/per shower depending on the energy) and the radio signal computation (~20-180 min per antenna, depending on the frequency content). 
A full simulation resumes then to (1) simulate a set of showers, for example EAS from cosmic rays with energy >= 10^16 eV. (2) given a list of antennas compute the electric field from shower s at antenna location a. 
Hence from the simulated shower set {s in S} and the antenna set {a in A} we form the set or pairs {(s,a) in (S,A)}. Note that air shower simulations (EVA/conex) are performed in free sky, with no topography information. 
Therefore, over the current prototype setup there is an almost exact invariance by space translation of the simulated showers. The incoming direction matters a lot however. Consequently it is enough to simulate a reduce set of showers and to 
`reuse` them at various positions over the experimental setup (randomized uniformly in (x, y) the impact coordinates in an horizontal plan at height z=2650.m for example).    
 II) Simulation software    
 - The simulation soft is written in C(++) and python 2.6. The neutrino simulation part is currently fully in C (linking to Pythia which is written in FORTRAN). It is located in $SVNTREND/soft/sim/cxx. 
The neutrino simulation chains end at the level of the tau decays, whose products are recorded. The shower simulation and radio computation parts are centralized in $SVNTREND/soft/sim/python.     
- The subfolder trasi (Trend Radio SIgnal) contains a set of interface tools to EVA (trasi/Interface.py). EVA is a set of FORTRAN codes glued together with bash scripts. There are various archives in $THRONG_DIR/soft/sim/arx. 
The default one comes with a static compiled version of EVA/conex (eva.tar.gz) for usage on the grid. It is replicated in Olivier's area in grid storage, which is used for grid's shower simulation jobs. There is also a lighter version of EVA 
(eva.~conex.tar.gz) which does not contain EVA/conex. It is used in order to generate a stand-alone executable for radio signal computation, that can run on various backends (batch, grid, locally, ...). 
If you do a local installation of the soft, you will need to download at least this lighter archive for radio signal computations.    - Production tools are centralized in python/production folder. There are two submodules: dirac: 
an interface to DIRAC, and bookkeeping: DB tools for the production. The folders scripts contain management scripts, detailed later.         
*** IF you work at CCIN2P3 you don't need to do the step below. The soft is already installed there.         
- In order to install the simulation software locally you can try `make` from soft/sim/python. It works at CCIN2P3 and my own SL6. For production tools you will need to install dirac also, but not any version. 
Best is to copy install-franasia-dirac.(c)sh from $THRONG_DIR/soft/dirac and source it in your DIRAC local install directory. It will request your grid certificate 2 times. First will likely fail, Don't worry. 
Once dirac installed and dirac/bashrc or dirac/cshrc sourced go to python/production and call `make install`. BEWARE: dirac uses its own version of python, a 2.6 basis. Sourcing dirac/bashrc will change your current python to dirac's one. 
Production tools run with dirac's python BUT other simulation modules don't. Because, dirac python's is not fully compatible with modules the like pylab. At least on my SL6 it was crashing.   
 - In order to have an easy access to some visualisation tools (plotting showers, electric fields, ect ...) you can also go to python/trasi and call `make install`. Note that before you shold restart a session/term 
if you sourced dirac bashrc. In order to use your default python.     
III) Production strategy and tools.    
 - Since we do not have a massive volume of data, but mainly need lot of CPU, we decided to store simulation results locally or on /sps/trend for example as compressed tarballs. The data are organized as a tree of directories. 
The name of the root directory is the name of the simulation chain. For example you can have a look at /sps/trend/trend-50 for the TREND-50 simulation chain of the current prototype experimental setup. 
It is structured as:      
... coord_antennas_all.txt     .
.. fields/     
... ... 6588527/     
... ... ... out.tar.gz     
... showers/     
... ... 3190264/    
 ... ... ... out.tar.gz    
 The file `coord_antennas_all.txt` contains the antennas coordinates. The name can be changed but must contain `antennas`. The subfolders fields and shower store the result of electric field and shower computation. 
The name of the subfolder below are tags from the production backend. Here it correspond to the grid job id. The real data are stored in out.tar.gz. Other files are logs from the production backend.      
- In order to keep tracks of all simulated data, tasks left to do, and pending jobs, we use a local DB. Note that it is only a helper tool, no simulated data are stored there. Only meta-data (high level info). 
This production DB can be rebuilt at any time, by scanning the data location. It takes some time though ... Following, a simulation chain consist in (1) registering antennas and tasks to do to the DB (showers simulation,
 field computations for couples (shower, antenna)), (2) submit the tasks to a backend (grid or locally, or on a batch system. Only grid backend is automated for now) and (3) collect finished tasks results by storing them in a 
centralized place and updating the related meta-data.     
IV) Step by step illustration of existing tools with the TREND-50 current prototype setup.     
(0) I assumed here that you managed to install the simulation software locally, or that you are running the one installed at CC. For a local install you will also need to copy already simulated data located in /sps/trend/trend-50.     
(1) Set up a work directory, from which you will centralize the production:            
mkdir -p ~/production/trend-50        
 cp $THRONG_DIR/soft/sim/python/production/scripts/setup.(c)sh .        
mv setup.(c)sh setup.production.(c)sh        
cp $THRONG_DIR/soft/sim/python/production/scripts/*tr50* .        
cp $THRHONG_DIR/soft/sim/python/trasi/setup.(c)sh .        
mv setup.(c)sh setup.trasi.(c)sh     
Adapt path and extensions to trend's soft install and your shell. This will provide you with a set of example monitoring and management scripts.     
(2) Build the local production DB:         
source  setup.production.sh or source setup.production.csh        
import-db /sps/trend/trend-50     
(3) Register some new shower and/or field computations:         
python register-tr50-(fields|showers).py                
You can have a look at the corresponding script with your favorite text editor. Global tunning settings are centralized at the beginning of the script. For the showers we loop on all existing shower data, simulated at 5e17 eV and register showers 
with same zenith, azimuth, ect ... but different energy, taken in the list of values at the beginning of the script. For the fields we loop on the pairs (simulated showers, antenna) and register an electric field. 
The `generate_shower_core_position` function at the beginning of the field would allow to randomize the shower's position.     
(4) Submit some new task to the grid:          
dirac-proxy-init (if you dind't already do it ...)        
 python submit-tr50-jobs.py          
Again, have a look at the corresponding python script. This scripts builds a self extracting python zip executable from the DB info on registered jobs to perform. Then this executable is submitted to the backend, the grid here. 
There is a flag `submit = True` which when set to false would disable the job's submission. You can for example try to run the generated executable locally (it might take some time though ...).     
(5) Check job's status and get finished jobs:          
python get-tr50-jobs.py          
This will look for pending jobs in the local DB, check its status and retrieve and store the result if finished. Note that a running job not registered in the DB will be lost. You would have to retrieve an properly store it manually and update the 
DB with import-db command. For that can also use the trend/dirac interface functions. They work like batch commands but are prefixed with g*. For example:          gstat          will list your `current` grid jobs. 
Current means today. See `gstat -h` for help on other selection criteria. To retrieve the job there is the command gget. To kill it: gkill, to submit a new one: gsub. All those scripts are located in soft/sim/python/production/dirac/scripts if you need.        
(6) Plot a job result (shower histos, electric field):          
*** FROM A NEW SESSION ***         
. setup.trasi.sh / source setup.trasi.csh         
plot-conex /sps/trend/trend-50/showers/3223505/out.tar.gz         
plot-eva /sps/fields/trend-50/fields/6711246/out.tar.gz          
change the folder name to your needs. You can call `plot-* -h` for help on options. Note that currently one needs a separate session for dirac related tasks and for pylab/ploting ones         
I could not test all with the last updates. There will be bugs left ...  But if you manage to go through these steps it would be a good start already! And since you will have read/write access to svn you can correct them :P 
After that it will be time to make precise plans on what we want to run as simulations, taking CPU limitations into account. Currently the shower simulation strategy seems too crude in my opinion. 
All showers have energy 5e17 eV. A realistic scheme would simulate showers from an energy spectrum and mass composition in some energy range: 1e17 -> 1e21 eV for example.       
Cheers,      Valentin

-- OlivierMartineauHuynh - 2013-12-17

Topic revision: r1 - 2013-12-16 - OlivierMartineauHuynh
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback