public class CPlanner extends Executable
Modifier and Type | Field and Description |
---|---|
static String |
CLEANUP_DIR
The basename of the directory that contains the submit files for the
cleanup DAG that for the concrete dag generated for the workflow.
|
static String |
DEFAULT_MEGADAG_MODE
The default megadag mode that is used for generation of megadags in
deferred planning.
|
private static String |
EMPTY_FINAL_WORKFLOW_MESSAGE
The final successful message that is to be logged.
|
static String |
JAVA_COMMAND_LINE_PROPERTY_REGEX
The regex used to match against a java property that is set using
-Dpropertyname=value in the argument string
|
private PegasusBag |
mBag
The object containing the bag of pegasus objects
|
private NumberFormat |
mNumFormatter
The number formatter to format the run submit dir entries.
|
private PlannerMetrics |
mPMetrics
The PlannerMetrics object storing the metrics about this planning instance.
|
private PlannerOptions |
mPOptions
The object containing all the options passed to the Concrete Planner.
|
private String |
mUser
The user name of the user running Pegasus.
|
static String |
NOOP_PREFIX
The prefix for the NoOP jobs that are created.
|
static String |
PEGASUS_MONITORD_LAUNCH_PROPERTY_KEY
The name of the property key that determines whether pegasus-run
should monitord or not.
|
private static String |
SUCCESS_MESSAGE
The message to be logged in case of empty executable workflow.
|
mLogger, mLogMsg, mProps, mVersion
Constructor and Description |
---|
CPlanner()
Default constructor.
|
CPlanner(LogManager logger)
The overload constructor.
|
Modifier and Type | Method and Description |
---|---|
protected void |
construct(Job job,
String key,
String value)
Constructs a condor variable in the condor profile namespace
associated with the job.
|
protected Job |
createNoOPJob(String name)
It creates a NoOP job that runs on the submit host.
|
protected boolean |
createSymbolicLink(String source,
String destination,
File directory,
boolean logErrorToDebug)
This method generates a symlink between two files
|
protected String |
determineRelativeSubmitDirectory(ADag dag,
String dir,
String user,
String vogroup,
boolean timestampBased)
Creates the submit directory for the workflow.
|
protected String |
determineRelativeSubmitDirectory(String label,
String dir,
String user,
String vogroup,
boolean timestampBased)
Creates the submit directory for the workflow.
|
private String |
determineRootWorkflowUUID(ADag dag,
PlannerOptions options,
PegasusProperties properties)
Determines the workflow uuid for a workflow
|
protected String |
doBackupAndCreateSymbolicLinkForSubmitDirectory(String baseDir,
String relativeSubmitDir) |
private Collection<File> |
doDeferredPlanning()
This ends up invoking the deferred planning code, that generates
the MegaDAG that is used to submit the partitioned daxes in layers.
|
protected void |
doPartitionAndPlan(PegasusProperties properties,
PlannerOptions options)
Partitions and plans the workflow.
|
void |
executeCommand()
Executes the command on the basis of the options specified.
|
Collection<File> |
executeCommand(PlannerOptions options)
Executes the command on the basis of the options specified.
|
private void |
generateStampedeEventsForAbstractWorkflow(ADag workflow,
PegasusBag bag)
Generates events for the abstract workflow.
|
gnu.getopt.LongOpt[] |
generateValidOptions()
Tt generates the LongOpt which contain the valid options that the command
will accept.
|
protected String |
getDAGFilename(ADag dag,
PlannerOptions options)
Returns the basename of the dag file
|
String |
getNOOPJobName(ADag dag)
Returns the name of the noop job.
|
private String |
getPegasusRunInvocation()
Returns the pegasus-run invocation on the workflow planned.
|
protected String |
getRandomDirectory(ADag dag)
Sets the basename of the random directory that is created on the remote
sites per workflow.
|
protected boolean |
handleRescueDAG(ADag dag,
PlannerOptions options)
Checks for rescue dags, and determines whether to plan or not.
|
protected boolean |
handleRescueDAG(String dag,
String dir,
int numOfRescues)
Checks for rescue dags, and determines whether to submit a rescue dag
or not.
|
void |
initialize(String[] opts,
char confChar)
Initialize the executable object
|
void |
loadProperties()
Loads all the properties that are needed by this class.
|
private SiteStore |
loadSiteStore(Set<String> sites)
Loads the sites from the site catalog into the site store
|
private void |
logIDHierarchyMessage(ADag dag,
String parentType,
String parentID)
Logs a message that connects the jobs with DAX/DAG
|
private void |
logSuccessfulCompletion(boolean emptyWorkflow)
Logs the successful completion message.
|
static void |
main(String[] args)
The main program for the CPlanner.
|
PlannerOptions |
parseCommandLineArguments(String[] args)
Parses the command line arguments using GetOpt and returns a
PlannerOptions contains all the options passed by the
user at the command line. |
PlannerOptions |
parseCommandLineArguments(String[] args,
boolean sanitizePath)
Parses the command line arguments using GetOpt and returns a
PlannerOptions contains all the options passed by the
user at the command line. |
void |
printLongVersion()
Prints the long description, displaying in detail what the various options
to the command stand for.
|
void |
printShortVersion()
Prints out a short description of what the command does.
|
protected static void |
sanityCheck(File dir)
Checks the destination location for existence, if it can
be created, if it is writable etc.
|
boolean |
submitWorkflow(String invocation)
Submits the workflow for execution using pegasus-run, a wrapper around
pegasus-submit-dag.
|
protected boolean |
writeOutMetrics(PlannerMetrics pm)
Writes out the planner metrics to the global log.
|
convertException, convertException, getCommandLineOptions, getEnvValue, getGVDSVersion, initialize, log, lookupConfProperty, sanityCheckOnProperties, setupLogging
public static final String DEFAULT_MEGADAG_MODE
public static final String CLEANUP_DIR
public static final String NOOP_PREFIX
public static final String PEGASUS_MONITORD_LAUNCH_PROPERTY_KEY
public static final String JAVA_COMMAND_LINE_PROPERTY_REGEX
private static final String EMPTY_FINAL_WORKFLOW_MESSAGE
private static final String SUCCESS_MESSAGE
private PlannerOptions mPOptions
private PegasusBag mBag
private PlannerMetrics mPMetrics
private NumberFormat mNumFormatter
private String mUser
public CPlanner()
public CPlanner(LogManager logger)
logger
- the logger object to use. can be null.public void initialize(String[] opts, char confChar)
Executable
initialize
in class Executable
opts
- the command line argument passed by the userconfChar
- the short option corresponding the conf property.public static void main(String[] args)
args
- the main arguments passed to the planner.public void loadProperties()
loadProperties
in class Executable
public void executeCommand()
args
- the command line options.public Collection<File> executeCommand(PlannerOptions options)
options
- the command line options.File
objects for the files written
out.public String getNOOPJobName(ADag dag)
dag
- the workflowprotected Job createNoOPJob(String name)
name
- the name to be assigned to the noop jobprotected void construct(Job job, String key, String value)
job
- contains the job description.key
- the key of the profile.value
- the associated value.public PlannerOptions parseCommandLineArguments(String[] args)
PlannerOptions
contains all the options passed by the
user at the command line.args
- the arguments passed by the user at command line.public PlannerOptions parseCommandLineArguments(String[] args, boolean sanitizePath)
PlannerOptions
contains all the options passed by the
user at the command line.args
- the arguments passed by the user at command line.sanitizePath
- whether to sanitize path during construction of optionspublic boolean submitWorkflow(String invocation)
invocation
- the pegasus run invocationprotected void doPartitionAndPlan(PegasusProperties properties, PlannerOptions options)
properties
- the properties passed to the planner.options
- the options passed to the planner.protected String getRandomDirectory(ADag dag)
dag
- the DAG containing the abstract workflow.public gnu.getopt.LongOpt[] generateValidOptions()
generateValidOptions
in class Executable
LongOpt
objects , corresponding to the valid
optionspublic void printShortVersion()
printShortVersion
in class Executable
public void printLongVersion()
printLongVersion
in class Executable
private String determineRootWorkflowUUID(ADag dag, PlannerOptions options, PegasusProperties properties)
dag
- the workflowoptions
- the options passed to the plannerproperties
- the properties passed to the plannerprivate Collection<File> doDeferredPlanning()
File
objects for the files written
out.protected String determineRelativeSubmitDirectory(ADag dag, String dir, String user, String vogroup, boolean timestampBased) throws IOException
dag
- the workflow being worked upon.dir
- the base directory specified by the user.user
- the username of the user.vogroup
- the vogroup to which the user belongs to.timestampBased
- boolean indicating whether to have a timestamp based dir or notIOException
- in case of unable to create submit directory.protected String determineRelativeSubmitDirectory(String label, String dir, String user, String vogroup, boolean timestampBased) throws IOException
label
- the label of the workflowdir
- the base directory specified by the user.user
- the username of the user.vogroup
- the vogroup to which the user belongs to.timestampBased
- boolean indicating whether to have a timestamp based dir or notIOException
- in case of unable to create submit directory.protected static void sanityCheck(File dir) throws IOException
dir
- is the new base directory to optionally create.IOException
- in case of error while writing out files.protected boolean writeOutMetrics(PlannerMetrics pm)
pm
- the metrics to be written out.protected String getDAGFilename(ADag dag, PlannerOptions options)
dag
- the dag that was parsed.options
- the planner optionsprotected boolean handleRescueDAG(ADag dag, PlannerOptions options)
dag
- the dag that was parsed.options
- the planner optionsprotected boolean handleRescueDAG(String dag, String dir, int numOfRescues)
dag
- the dag file for the daxdir
- the submit directory.numOfRescues
- the number of rescues to handle.protected boolean createSymbolicLink(String source, String destination, File directory, boolean logErrorToDebug)
source
- the file that has to be symlinkeddestination
- the destination of the symlinkdirectory
- the directory in which to execute the commandlogErrorToDebug
- whether to log messeage to debug or notprivate void generateStampedeEventsForAbstractWorkflow(ADag workflow, PegasusBag bag)
workflow
- the parsed daxbag
- the initialized object bagprivate SiteStore loadSiteStore(Set<String> sites)
sites
- private void logIDHierarchyMessage(ADag dag, String parentType, String parentID)
dag
- the DAG objectparentType
- the parent typeparentID
- the parent idprivate void logSuccessfulCompletion(boolean emptyWorkflow)
emptyWorkflow
- indicates whether the workflow created was empty or not.private String getPegasusRunInvocation()
protected String doBackupAndCreateSymbolicLinkForSubmitDirectory(String baseDir, String relativeSubmitDir) throws IOException
IOException
Copyright © 2011 The University of Southern California. All Rights Reserved.