public class BalancedCluster extends Basic
Modifier and Type | Class and Description |
---|---|
protected class |
BalancedCluster.BundleValue |
protected class |
BalancedCluster.PoolTransfer
A container to store the transfers that need to be done per staging site
The transfers are stored over a collection of Transfer Containers with
each transfer container responsible for one transfer job.
|
protected class |
BalancedCluster.TransferContainer
A container class for storing the name of the transfer job, the list of
file transfers that the job is responsible for.
|
Modifier and Type | Field and Description |
---|---|
static String |
DEFAULT_LOCAL_STAGE_IN_CLUSTER_FACTOR
The default bundling factor that identifies the number of transfer jobs
that are being created per execution pool for stageing in data for
the workflow.
|
static String |
DEFAULT_LOCAL_STAGE_OUT_CLUSTER_FACTOR
The default bundling factor that identifies the number of transfer jobs
that are being created per execution pool for stageing out data for
the workflow.
|
static String |
DEFAULT_REMOTE_STAGE_IN_CLUSTER_FACTOR
The default bundling factor that identifies the number of transfer jobs
that are being created per execution pool for stageing in data for
the workflow.
|
static String |
DEFAULT_REMOTE_STAGE_OUT_CLUSTER_FACTOR
The default bundling factor that identifies the number of transfer jobs
that are being created per execution pool for stageing out data for
the workflow.
|
static String |
DESCRIPTION
A short description of the transfer refinement.
|
protected boolean |
mAddNodesForSettingXBit
A boolean indicating whether chmod jobs should be created that set the
xbit in case of executable staging.
|
private int |
mCurrentSILevel
The current level of the jobs being traversed.
|
private int |
mCurrentSOLevel
The current level of the jobs being traversed.
|
protected String |
mJobPrefix
The job prefix that needs to be applied to the job file basenames.
|
protected Pegasus |
mPegasusProfilesInProperties
Pegasus Profiles specified in the properties.
|
private ReplicaCatalogBridge |
mRCB
The handle to the replica catalog bridge.
|
private Map |
mRelationsParentMap
The map indexed by compute jobnames that contains the list of stagin job
names that are being added during the traversal of the workflow.
|
protected Map |
mSetupMap
The map indexed by staged executable logical name.
|
protected SiteStore |
mSiteStore
Handle to the SiteStore
|
protected BalancedCluster.BundleValue |
mStageinLocalBundleValue
The BundleValue that evaluates for local stage in jobs.
|
private Map<String,BalancedCluster.PoolTransfer> |
mStageInLocalMapPerLevel
The map containing the list of stage in transfer jobs that are being
created for the workflow indexed by the execution poolname.
|
protected BalancedCluster.BundleValue |
mStageInRemoteBundleValue
The BundleValue that evaluates for remote stage-in jobs.
|
private Map<String,BalancedCluster.PoolTransfer> |
mStageInRemoteMapPerLevel
The map containing the list of stage in transfer jobs that are being
created for the workflow indexed by the execution poolname.
|
protected BalancedCluster.BundleValue |
mStageOutLocalBundleValue
The BundleValue that evaluates for local stage out jobs.
|
private Map<String,BalancedCluster.PoolTransfer> |
mStageOutLocalMapPerLevel
A map indexed by site name, that contains the pointer to the local stage out
PoolTransfer objects for that site.
|
protected BalancedCluster.BundleValue |
mStageOutRemoteBundleValue
The BundleValue that evaluates for remote stage out jobs.
|
private Map<String,BalancedCluster.PoolTransfer> |
mStageOutRemoteMapPerLevel
A map indexed by site name, that contains the pointer to the remote stage out
PoolTransfer objects for that site.
|
mCreateRegistrationJobs, mFileTable, mLogMsg, mPPS, mRelationsMap
mDAG, mLogger, mPOptions, mProps, mRemoteTransfers, mTPT, mTXInterImplementation, mTXStageInImplementation, mTXStageOutImplementation, mTXSymbolicLinkImplementation, mXMLStore
INTER_POOL_PREFIX, LOCAL_PREFIX, REGISTER_PREFIX, REMOTE_PREFIX, STAGE_IN_PREFIX, STAGE_OUT_PREFIX
Constructor and Description |
---|
BalancedCluster(ADag dag,
PegasusBag bag)
The overloaded constructor.
|
Modifier and Type | Method and Description |
---|---|
void |
addStageInXFERNodes(Job job,
boolean localTransfer,
Collection files,
int type,
Map<String,BalancedCluster.PoolTransfer> stageInMap,
BalancedCluster.BundleValue bundleValue,
Implementation implementation)
Adds the stage in transfer nodes which transfer the input files for a job,
from the location returned from the replica catalog to the job's execution
pool.
|
void |
addStageInXFERNodes(Job job,
Collection<FileTransfer> files,
Collection<FileTransfer> symlinkFiles)
Adds the stage in transfer nodes which transfer the input files for a job,
from the location returned from the replica catalog to the job's execution
pool.
|
void |
addStageOutXFERNodes(Job job,
Collection files,
ReplicaCatalogBridge rcb,
boolean localTransfer,
boolean deletedLeaf)
Adds the stageout transfer nodes, that stage data to an output site
specified by the user.
|
void |
done()
Signals that the traversal of the workflow is done.
|
protected String |
getComputeJobBundleValue(Job job)
Returns the bundle value associated with a compute job as a String.
|
protected String |
getDefaultClusterValueFromProperties(String key,
String defaultKey,
String defaultValue)
Returns the default value for the clustering/bundling of jobs to be used.
|
String |
getDescription()
Returns a textual description of the transfer mode.
|
BalancedCluster.PoolTransfer |
getStageOutPoolTransfer(String site,
boolean localTransfer,
int num)
Returns the appropriate pool transfer for a particular site.
|
protected void |
initializeClusterValues()
Initializes the bundle value variables, that are responsible determining
the bundle values.
|
Map<String,BalancedCluster.PoolTransfer> |
resetStageInMap(Map<String,BalancedCluster.PoolTransfer> stageInMap,
Implementation implementation,
int stageInJobType,
boolean localTransfer)
Signals that the traversal of the workflow is done.
|
protected void |
resetStageInMaps()
Resets the local and remote stage out maps.
|
protected Map |
resetStageOutMap(Map<String,BalancedCluster.PoolTransfer> map,
boolean localTransfer)
Resets a single map
|
protected void |
resetStageOutMaps()
Resets the local and remote stage out maps.
|
addInterSiteTXNodes, addJob, addRelation, addRelation, addStageInXFERNodes, addStageOutXFERNodes, appendAttribute, constructFileKey, createRegistrationJob, getJobPriority, logRefinerAction
loadImplementations
addStageInXFERNodes, getWorkflow, getXMLProducer, isSiteThirdParty, logConfigMessages, refinerPreferenceForLocalTransferJobs, refinerPreferenceForTransferJobLocation, runTPTOnRemoteSite, runTransferRemotely
public static final String DESCRIPTION
public static final String DEFAULT_LOCAL_STAGE_IN_CLUSTER_FACTOR
public static final String DEFAULT_REMOTE_STAGE_IN_CLUSTER_FACTOR
public static final String DEFAULT_LOCAL_STAGE_OUT_CLUSTER_FACTOR
public static final String DEFAULT_REMOTE_STAGE_OUT_CLUSTER_FACTOR
private Map<String,BalancedCluster.PoolTransfer> mStageInLocalMapPerLevel
private Map<String,BalancedCluster.PoolTransfer> mStageInRemoteMapPerLevel
private Map mRelationsParentMap
protected BalancedCluster.BundleValue mStageinLocalBundleValue
protected BalancedCluster.BundleValue mStageInRemoteBundleValue
protected BalancedCluster.BundleValue mStageOutLocalBundleValue
protected BalancedCluster.BundleValue mStageOutRemoteBundleValue
protected Map mSetupMap
private Map<String,BalancedCluster.PoolTransfer> mStageOutLocalMapPerLevel
private Map<String,BalancedCluster.PoolTransfer> mStageOutRemoteMapPerLevel
private int mCurrentSOLevel
private ReplicaCatalogBridge mRCB
protected String mJobPrefix
protected Pegasus mPegasusProfilesInProperties
protected SiteStore mSiteStore
protected boolean mAddNodesForSettingXBit
private int mCurrentSILevel
public BalancedCluster(ADag dag, PegasusBag bag)
dag
- the workflow to which transfer nodes need to be added.bag
- the bag of initialization objectsprotected void initializeClusterValues()
protected String getDefaultClusterValueFromProperties(String key, String defaultKey, String defaultValue)
return value of pegasus profile key if it exists, else return value of pegasus profile defaultKey if it exists, else the defaultValue
key
- the pegasus profile keydefaultKey
- the default pegasus profile keydefaultValue
- the default value.public void addStageInXFERNodes(Job job, Collection<FileTransfer> files, Collection<FileTransfer> symlinkFiles)
addStageInXFERNodes
in class Basic
job
- Job
object corresponding to the node to
which the files are to be transferred to.files
- Collection of FileTransfer
objects containing the
information about source and destURL's.symlinkFiles
- Collection of FileTransfer
objects containing
source and destination file url's for symbolic linking
on compute site.public void addStageInXFERNodes(Job job, boolean localTransfer, Collection files, int type, Map<String,BalancedCluster.PoolTransfer> stageInMap, BalancedCluster.BundleValue bundleValue, Implementation implementation)
job
- Job
object corresponding to the node to
which the files are to be transferred to.localTransfer
- boolean indicating whether transfer has to happen on local site.files
- Collection of FileTransfer
objects containing the
information about source and destURL's.type
- the type of transfer job being createdstageInMap
- Map indexed by site name that gives all the transfers for
that site.bundleValue
- used to determine the bundling factor to employ for a job.implementation
- the transfer implementation to use.public void addStageOutXFERNodes(Job job, Collection files, ReplicaCatalogBridge rcb, boolean localTransfer, boolean deletedLeaf)
addStageOutXFERNodes
in class Basic
job
- Job
object corresponding to the node to
which the files are to be transferred to.files
- Collection of FileTransfer
objects containing the
information about source and destURL's.rcb
- bridge to the Replica Catalog. Used for creating registration
nodes in the workflow.localTransfer
- whether the transfer should be on local site or not.deletedLeaf
- to specify whether the node is being added for
a deleted node by the reduction engine or not.
default: falseprotected String getComputeJobBundleValue(Job job)
job
- public void done()
protected void resetStageInMaps()
public Map<String,BalancedCluster.PoolTransfer> resetStageInMap(Map<String,BalancedCluster.PoolTransfer> stageInMap, Implementation implementation, int stageInJobType, boolean localTransfer)
stageInMap
- maps site names to PoolTransferimplementation
- the transfer implementation to usestageInJobType
- whether a stagein or symlink stagein joblocalTransfer
- indicates whether transfer job needs to run on
local site or not.public String getDescription()
getDescription
in class Basic
public BalancedCluster.PoolTransfer getStageOutPoolTransfer(String site, boolean localTransfer, int num)
site
- the site for which the PT is reqd.localTransfer
- whethe the associated transfer job runs on local site or remote.num
- the number of Stageout jobs required for that Pool.protected void resetStageOutMaps()
protected Map resetStageOutMap(Map<String,BalancedCluster.PoolTransfer> map, boolean localTransfer)
map
- the map to be resetlocalTransfer
- whether the transfer jobs need to run on local site or not