The GRID introduces the concept of a GRID certificate to authenticate users. A GRID certificate is issued by a Certificate Authority (CA) which checks the identity of the user and guarantees that the holder of this certificate is existing and his certificate is valid.
The certificate is used for authentication instead of the user's account to avoid the replication of the user's account to all GRID sites. When authenticating to a site, the user's certificate is mapped to a local account under which all commands are executed.
The certificate itself is not used during the actual GRID usage. All GRID jobs use a proxy of the certificate with a limited lifetime. This enhances security because the user has to re-establish the validty of his certificate after the lifetime of the proxy has ended.
In the following, the application procedure for a GRID certificate and the generation of a proxy is described. The application has to be done once followed by the installation of the certificate in the home directory of the user's account. The proxy generation has to be repeated everytime no valid proxy exists on the user's submission machine.
Getting Grid Certificate
To work in Grid Environment the user should have a valid grid certificate signed by recognized grid computing certification authority, as for our country user can apply for certificate to "Indian Grid Certificate Authority (IGCA)"
Make sure you executed all steps carefully and after recieving the certificate please load it into your internet browser
Please register yourself with the virtual organization (VO membership) e.g. CMS VO membership
Getting an account at India-CMS Tier-III at TIFR, Mumbai and T2_IN_TIFR
After getting certificate and VO membership, Please send an email with your lxplus username and certificate DN, to all members in this list and your guide or supervisor.
You will recieve your account information and machine name in the reply to you email, you will also recieve temporary onetime password which you have change when you login into your account.
Configure your account with your grid certificate ( As described in these steps GRIDCredentials)
Installation of the certificate
After the successful application, the certificate has to be installed in the user's home directory following these instructions:
Export or 'backup' the certificate from the browser used for the application. The interface for this varies from browser to browser. The exported file will probably have the extension .p12 or .pfx. Guard this file carefully. Store it off your computer, or remove it once you are finished with this process.
Copy the file to the user's home directory.
Create a directory in the user's home directory:
Extract the certificate creating a public and a private key file replacing _YourCert.p12_ with the filename chosen during step 1:
The user will be asked to define a *passphrase* during this step. This passphrase has to be entered every time a proxy is created from the certificate. For security reasons, an empty passphrase is not adviseable.
Set the access mode on your userkey.pem and usercert.pem files:
Further protection of the *$HOME/.globus* directory is necessary to prevent everyone except the user to enter this directory:
chmod go-rx $HOME/.globus
The user's GRID certificate (usercert.pem and userkey.pem) can be copied to every other machine to access the GRID by transporting the $HOME/.globus directory. The security measures described above have to be repeated.
---++ Proxy generation
Note: the execution of commands indicates below requires a installed and setup GRID user interface (UI) on the user's machine. The installation and setup of an UI is described in the following section: User interface.
After installation of the certificate in the user's home directory, the following command creates a user proxy:
voms-proxy-init -voms cms
using the passphrase defined during installation.
To check how long the user's proxy is valid, use the following command:
A valid proxy should produce a similar output like:
---+ How to use Tier-II account at T2_IN_TIFR
At T2_IN_TIFR we are using Disk Pool Manager (DPM) as storage solution. As per WLCG recommendations we have deployed multiple access technologies for accessing the storage area. Details for utilising T2 resources.
Following are the access protocols currently enabled.
The user space at T2_IN_TIFR can be used in following ways,
1 Storing your analysis data
1 To stage out the output of your jobs running on any WLCG site (including T2_IN_TIFR)
---++ Stageout and publication
CRAB allows you to copy your analysis outputs directly to a Tier2 or Tier3 Storage Element. You can decide to store them in a Storage Element of an "Official CMS site" or in some SE of your choosing. The difference between the two is technical, not political. "Official CMS site" simply means a site fully described in CMS data transfer tool (PhEDEx). If the target site has a working PhEDEx node connected to the stageout location, then the site is listed in [[https://twiki.cern.ch/twiki/bin/view/CMS/SiteDB][SiteDB]] and crab can discover all the details needed for the stageout from the site name in the CMS format Tx_CC_SiteName, where x is the tier number (usually 2 or 3) and CC is the country code, e.g. T2_IT_Pisa, T2_US_UCSD, T3_IT_Trieste etc. If the target Storage Element is not known to PhEDEx, you will have to indicate full contact details in
---++++ 1. Stage out to an "official CMS site" without publication in DBS
you have to configure the crab.cfg with:
copy_data = 1
storage_element = T2_IN_TIFR
user_remote_dir = subdirectory where your output will be stored
1 To stage out the output of the jobs (batch jobs or CRAB Jobs) running at Tier-III at TIFR
Note* - It is advised that If you are using more then 5 TB of storage space in your user area, Please inform any of the system administrator via email
---+ Setup and usage instructions for Tier-III account.
At present tier III account has been set up with condor as the batch system, system has been designed in such a way that as per the load on the system and user requirement no of job slots can be increased or decreased. We have dedicated NFS based tier-III storage area, which is separate from the user space at tier II storage. Directory structure and it's details are following
With your credentials you can login on User Interface machine at TIFR. Currently ew have two user interface machines.
Hostname for SLC6 - ui.indiacms.res.in
Hostname for SLC5 - ui2.indiacms.res.in
CMS software is mounted using CERN Virtual Machine File System (CVMFS) , where all supported versions of CMSSW software are available. Directory location is /cvmfs/cms.cern.ch/
Tier III storage space is available under your user home directory as. /home/<username>/t3store You are requested to use this location to store your user data.
---+++ Test your grid certificate
1 Is your personal certificate able to generate Grid proxies? To find out, after having setup your environment run this command:
grid-proxy-init -debug -verify
In case of failure, the possible causes are:
* the certificate/key pair is not installed under $HOME/.globus/
* the certificate has expired
* the certificate and the private key do not match
In the first case, you either do not have a certificate at all or have to install it on the UI; in the second case, you should get a new certificate; in the third case you probably have incorrectly installed your certificate.
1 Are you a member of the CMS VO? To see if this is the case, you can execute this command:
---++ Submissions of CMSSW jobs locally - ui.indiacms.res.in
Before submitting jobs to the Grid, it is a good practice to run the CMSSW code locally over a few events to discard problems not related with CRAB.
To run an analysis CMSSW code, one needs to specify an input dataset file directly in the CMSSW parameter-set configuration file (specifying as well how to access/open the file). One could either copy one file of the remote input dataset to a local machine, or, more conveniently, open the file remotely. For both, the recommended tool is the Xrootd service (please refer to the https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookXrootdService to learn the basics about how to use Xrootd). We will choose to open a file remotely. In any case, one firt shas to find out the LFN of such a file. We used DAS to find the files in the dataset.
The datasets are in status "VALID" (otherwise we wouldn't be able to run on them).
The /GenericTTbar/HC-CMSSW_5_3_1_START53_V5-v1/GEN-SIM-RECO MC dataset has 1 block, 177 files and 300K events.
---++++++ CMSSW configuration file to process an existing dataset
This section shows the CMSSW parameter-set configuration file that we will use in this tutorial when running over an existing dataset (either MC or Data, either CMS official or user-produced). We call it pset_tutorial_analysis.py.
This analysis code will produce an output ROOT file called output.root containing only the recoTracks___* branches for a maximum of 10 events in the input dataset.
Notice that we added the string root://se01.indiacms.res.in/ before the LFN. The first part, root:, specifies that the file should be opened with ROOT.
Note: When running CMSSW code locally with cmsRun, we suggest to do it in a separate (fresh) shell, where the user sets up the Grid and CMS environments, but skips the CRAB setup. This is to avoid the CRAB environment to interfere with the CMSSW environment.
For the list of supported and production releases of CMSSW software please refer to this link
#edmConfigHash your_python_file.py ( _To test your .py file whether it complies with CMSSW_ )
Now we run the analysis locally:
Note* After setting up cmsenv. rfio set of commands wil stop working because of environment variable conflict. therefor you have to execute following command for using rfdir and setting up proxy etc etc.*
Following are the commands used to submit condor jobs
#condor_submit myjob.submit ( to submit the jobs)
#condor_q job_id (toknow the status of job)
#condor_status (to know the status of available job slots)
when_to_transfer_output = ON_EXIT
echo "I'm process id $$ on" hostname
echo "This is sent to standard error" 1>&2
echo "Running as binary $0" "$@"
echo "My name (argument 1) is $1"
echo "My sleep duration (argument 2) is $2"
echo "Sleep of $2 seconds finished. Exiting"
#to stageout at site T2 storage (return_data = 0 copy_data = 1)
return_data = 0
copy_data = 1
storage_element = T2_IN_TIFR
user_remote_dir = Test
email = email@example.com
jobtype = cmssw
with the above file your crab otput will be staged out in
following are the additonal steps for submitting standard CRAB jobs.
#source /cvmfs/cms.cern.ch/crab3/crab.sh (for bash shell)
#source /cvmfs/cms.cern.ch/crab3/crab.csh (for tcsh shell)
#crab checkwrite ----site=T2_IN_TIFR ( To check permissions to write on the site storage )
#crab -create -submit -cfg YOUR_CRAB.cfg
---++ How to open root file
In addtion to the above steps, In order to open root file in your T3 account at ui.indiacms.res.in. Xrootd is now the standard way of openining files using root locally or anywhere ( Any of the CMS T1s or T2s )