create new tag
view all tags

TREND data processing using grid computing


In this area we document how to use the grid for processing TREND data.

Setting up your working environment

As an individual, you need to get the necessary credentials to be authenticated as a grid user. In this section we present how to get such credentials and install them on your personal computer. This is a one-time process, or at least once-per-year process.

Individuals affiliated to CAS (including NAOC and IHEP), need to apply to get a certificate from the IHEP certification authority. The procedure is as follows. Go to the IHEP certification authority https://cagrid.ihep.ac.cn.Click on "Request a Certificate" and follow the guidelines to submit your application. After a few hours, you will be contacted by e-mail with the instructions on how to dowload your personal certificate. In order to keep your browser happy, you are encouraged to download and install the certificate of the IHEP certification authority in your personal computer.

<!--- Point your browser to the page https://cagrid.ihep.ac.cn/pki/pub/cacert/cacert.crt and click on the DER link. Your browser will guide you on where and how to install this certificate.--->

Individuals affiliated to any of the CNRS institutes (including IN2P3) need to apply for their certificate to the CNRS certification authority. Go to the page https://igc.services.cnrs.fr/GRID2-FR and click on the left-side menu "Certificat Personnel" --> "Demander", fill the required information and submit your application. You will be instructed by e-mail when your certificate is available and you can download it following the provided instructions.

Once you obtain your personnal certificate you can use it as your identity card to indentify yourself with some web sites and also as your credentials for using the grid. This certificate is very personal and identifies you as an individual, so treat it accordingly.

Once your certificate is installed in your browser, you need to export it so to be able to use it also for commands interacting with grid services, such as the ones for submitting jobs or querying job status. The exact procedure depends on the browser you use and on the operating system your computer runs. Here we assume that you are using a flavor of UNIX (Linux or MacOS X).

The first step is to export your personal certificate from your browser to a disk file on your computer. If you are using a recent version of Firefox on Linux go to "Edit" --> "Preferences" --> "Advanced" --> "Encryption" tab and "Your Certificates" tab. If you are using Safari on MacOS X, the first step is "Firefox" --> "Preferences". If you are using any browser on MacOS X, use Keychain for this process.

Once you select your new certificate, export (or backup) it and save the file in a safe location on your computer. The next step is to convert your personal certificate and save it in the location expected by the grid commands you are going to use later.

The conversion process needs a software package called OpenSSL that usually comes installed with any flavor of Linux and MacOS X. If it is not installed on your machine, you have to make this conversion on a secure Linux machine available to you.

If you saved your certificate in the file $HOME/myPersonalCertificate, you can use the following commands to convert it:

mkdir $HOME/.globus 
chmod 700 $HOME/.globus 
openssl pkcs12 -in  $HOME/myPersonalCertificate -clcerts -nokeys -out $HOME/.globus/usercert.pem 
openssl pkcs12 -in  $HOME/myPersonalCertificate -nocerts -out $HOME/.globus/userkey.pem
chmod 600 $HOME/.globus/usercert.pem $HOME/.globus/userkey.pem 

After successfully typing these commands, your user certificate and associated private key will be stored in the appropriate format in the directory $HOME/.globus where the commands expect to find them.

Get enrolled in the France-Asia virtual organization and registered in DIRAC

Once you have got your personal certificate, the next steps are to get registered as a member of the France-Asia virtual organization and as a user of the DIRAC instance operated by CC-IN2P3 for France-Grilles.

For asking registration to the France-Asia VO, please go to the following link:


Your request will arrive to one of the managers of the virtual organization and eventually get approved. This step is important because it grants you access to the grid sites which support the France-Asia VO.

In addition, you will also need to be registered as a user of DIRAC, the middleware for managing your jobs on the grid. For this purpose, the subject of your personnal certificate is needed. To retrieve it please use the following command:

openssl x509 -in $HOME/.globus/usercert.pem -subject -noout

Copy the result of this command and send it by e-mail to Fabio, who will interact with the relevant people at CC-IN2P3 for getting you registered as a user of DIRAC.

Install DIRAC client software on your personal computer

Next step is to install the DIRAC client on your personal computer. The DIRAC client contains all the necessary commands you will need to submit jobs, query their status and retrieve their output.

To install DIRAC client follow the procedure below:

  • Download the installer
    cd $HOME/DIRAC #or whatever directory you want to install the software in
    curl -k -o dirac-install.py https://raw.github.com/DIRACGrid/DIRAC/master/Core/scripts/dirac-install.py
  • In order to execute the installer, you need a working Python interpreter in your machine. This is usually the case in Linux and MacOS X. If not, please consider installing one following the instructions in http://www.python.org
    chmod u+x ./dirac-install.py; ./dirac-install.py -V gridfr
  • Create a proxy (using your previously installed personal certificate) and setup the DIRAC configuration
    source bashrc
    dirac-proxy-init --upload --group franasia_user
    NOTE: after executing this command for the first time, you may get a "normal error message" similar to '... Joe Smith not registered', but please continue with the process.
    dirac-configure defaults-gridfr.cfg
    dirac-proxy-init --upload --group franasia_user
  • Detailed instructions for installing the DIRAC client are also provided by the DIRAC project. You may want to look a them at https://github.com/DIRACGrid/DIRAC/wiki/ClientInstallation

Submit grid jobs through DIRAC

A comprehensive walkthrough DIRAC is available as a set of slides presented in a DIRAC end-user tutorial in Bordeaux (France). You can look at them here: https://indico.in2p3.fr/conferenceOtherViews.py?view=standard&confId=6573

Monitor your DIRAC jobs

The web portal of the DIRAC instance operated by CC-IN2P3 and used by TREND is here: https://dirac.france-grilles.fr/DIRAC

There you can see the activity of the server, monitor your jobs (go to "Jobs" --> "Jobs Monitor") and even submit simple jobs to the system. You are encouraged to get familiar with this tool

-- FabioHernandez - 2012-10-26

Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r4 - 2016-01-18 - OlivierMartineauHuynh
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback