Introduction to DBS

• Data Bookkeeping Service
• Catalog all CMS event data from MC and online
  • Store dataset and block information, including files and SE info
  • LFN info of files in datasets and blocks
• Global DBS vs. Local DBS
  • Global DBS, for official data information
    • for public use
    • https://cmsweb.cern.ch/dbs_discovery/
  • Local DBS, for private data information
    • for local groups, particular processing purposes
    • http://cmsdbs.ihep.ac.cn:8003

Goals of CMS local DBS

• Share datasets among local group, PAG ,etc
• Able to deal with private datasets in crab
  • Register your private dataset into local DBS
  • Read your private dataset from local DBS
• Manage private datasets conveniently
  • Create, delete, etc

Functions of CMS local DBS

• Publish your private dataset with CRAB
• Use your private dataset with CRAB
• Delete your private dataset with CRAB
• Query your private dataset over web (http://cmsdbs.ihep.ac.cn:8003)
• Create and Register your private dataset with CMS DBS API

Publish your private dataset with CRAB

• Specify these jobs published results to local DBS in crab.cfg

       publish_data=1
       #<primarydataset>/<publish_data_name>/USER, ###/Zee/xmzhang-DBS_test01-83ec*/USER
       publish_data_name = DBS_test01
       dbs_url_for_publication = http://cmsdbs.ihep.ac.cn:8080/DBS/servlet/DBSServlet
       

• Do the submission

       crab -create -submit 1
       

• Get output of jobs when jobs are in "Done" state

       crab --status
       crab --getoutput
       

• Publish the dataset into the IHEP local DBS

       crab --publish http://cmsdbs.ihep.ac.cn:8080/DBS/servlet/DBSServlet
       

Use your private dataset with CRAB

• Fill the DBS url in crab.cfg

       dbs_url = http://cmsdbs.ihep.ac.cn:8080/DBS/servlet/DBSServlet
       

• Specify your private dataset name

       datasetpath=/Zee/xmzhang-DBS_test-83ec64b9af82f6c4b27a36630c9d7d4e/USER
       

• Do the submission

   
       Crab –create –submit 1
       

Delete your private dataset with CRAB

• Use the tool DBSDeleteData.py from CRAB in the directory /python

       /home/cms/zhangxm/test_crab/CRAB_2_5_1/python
       

• Issue the command to do the deletion

         python DBSDeleteData.py 
          --DBSURL=http://cmsdbs.ihep.ac.cn:8080/DBS/servlet/DBSServlet 
          --datasetPath=/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO
       

Query your private dataset over web

• Open the web page http://cmsdbs.ihep.ac.cn:8003
• Query your datasets or other datasets with DBS QUERY LANGUAGE (DBS-QL)
  • Refer to the DBS-QL rules(http://cmsdbs.ihep.ac.cn:8003/_help?userMode=user)

Create and Register your private dataset with CMS DBS API

• Set up env for DBS client in cmsui01 /home/cms/zhangxm/DBS/slc4_ia32_gcc345/cms/dbs-client/DBS_2_0_7-cmp/lib/setup.(c)sh
• Following the steps to create and register your private dataset
  1. Create a Primary Dataset (If one doesn't exist already)
  2. Create an Algorithm (Combinition of Application and ParametersetID)
  3. Insert Run(s)
  4. Create a Processed Dataset for the Primary dataset, providing data tier etc)
  5. Insert Lumi Section (s)
  6. Create a Block
  7. Insert files into Processed dataset
• Use the following scripts in the directory /home/cms/zhangxm/DBS/slc4_ia32_gcc345/cms/dbs-client/DBS_2_0_7-cmp/lib/RegisteFile for each step
  • dbsInsertPrimary.py
  • dbsInsertAlgorithm.py
  • dbsInsertRun.py
  • dbsInsertProcessed.py
  • dbsInsertLumiSection.py
  • dbsInsertBlock.py
  • dbsInsertFiles.py