Platform based on DIRAC

What is DIRAC?

• DIRAC(Distributed Infrastructure with Remote Agent Control) is a complete solution for using distributed computing resources.
• DIRAC forms a layer between a particular community and various computing resources to allow optimized, transparent and reliable usage, including lcg resources, clusters, clouds, etc.

BesDirac

• BesDirac is built as a uniform framework or platform for the BESIII experiment and users to manage and use distributed resources for production and analysis acitivities, which also allows an efficient use of non-reliable resources.
• It would integrate:
  • Workload management: Data-driven workload scheduling system, various compute resources
  • Data management system: File and metadata catalog, data transfer, various storage resources
  • Job and storage monitoring system: local ang global monitoring system
• Status:
  • 2011.4 the besdirac server is set up
  • 2011.6 four LCG and PBS sites(IHEP-PBS,IHEP-Grid, GUCAS, Dubna) are joined to accept jobs
  • 2011.7 two storage elements(dcache, dpm) are joined to accept data
  • 2011.8 working with ganga to accept mass production jobs
  • 2011.9 enable inputdata functions for analysis jobs
• Link:
  • https://cmsdbs.ihep.ac.cn/DIRAC/

Mass job management with Ganga

What is Ganga?

• Ganga is an easy-to-use frontend for mass job definition and management, implemented in Python
• Ganga Home page: http://ganga.web.cern.ch/ganga/

GangaBoss

• GangaBoss is the package which we developed on ganga for BOSS users with all the necessary BES-specific plug-ins
• It includes the following plug-ins:
  • Dirac -- use DIRAC API to talk to DIRAC server
  • Boss -- support BOSS applications, including generating run script,etc
  • Dataset -- support dataset-based operations Interface with BES metadata catalog, resolving and registering BOSS dataset Interface with BES data transfer system, subscribing dataset transfers
  • Splitter and Merger -- define a set of splitters and mergers to meet requirements of BES data processing

• Status
  • Can well support both PBS and DIRAC backend
  • Able to split and submit BESIII production jobs to PBS and DIRAC
  • Ready to support BESIII reconstruction and analysis
• Link

How to start as a user

get your private certificate ready and join BES VO

• apply your certificates with https://gridca.ihep.ac.cn
• download it to your home directory (~/.globus/), and generate userkey.pem and usercert.pem
• join Bes VO with https://voms.ihep.ac.cn:8443/voms/bes/

register in dirac CS

• your DN are automatically registered in dirac CS after you registered as a BES VO memeber
• test your proxy with dirac command
  • source /afs/ihep.ac.cn/bes3/bvrs/ganga/diracenv (for bash)
    source /afs/ihep.ac.cn/bes3/bvrs/ganga/diracenv.csh (for tcsh)
  • proxy-init -g bes_user

set up your environment to use ganga

• source boss env
• source ganga env
  • source /afs/ihep.ac.cn/bes3/bvrs/ganga/gangaEnv (for bash)
    source /afs/ihep.ac.cn/bes3/bvrs/ganga/gangaEnv.csh (for tcsh)
  • cp /afs/ihep.ac.cn/users/z/zhangxm/.gangarc .gangarc
  • define your private parameters in .gangarc
    • gangadir = /home/cms/zhangxm/gangadir # log directory
    • DataOutput = /home/cms/zhangxm/gangadir # data output dir

run ganga

• run and submit a job to pbs
  • /afs/ihep.ac.cn/bes3/bvrs/ganga/Ganga/install/5.6.2/bin/ganga /afs/ihep.ac.cn/bes3/bvrs/ganga/tutorial/sim_pbs.py

How to start as a site

• PBS site
  • register your site in DIRAC

            (1) the name of the torque server
            (2) the queue to be used for BES
            (3) a user account which can do the submission to PBS resources.
            (4) the name of UI where the user can login to do the submission to PBS resources.
         

  • network requirements

           (1) UI need to be accessible from the outside
           (2) work nodes need to access abroad
         

  • install CVMFS client to get BOSS software deployed

            (1) The installation and configuration of CVMFS client, please refer to the CVMFS technical reports:
            http://cernvm.cern.ch/portal/filesystem
            (2) Add support for BES in one of configuration files(/etc/cvmfs/default.local)
            CVMFS_REPOSITORIES=boss.cern.ch
        

-- ZhangXiaomei - 2011-10-16: