nl.esciencecenter.xenon.adaptors.schedulers

Class JobQueueScheduler

java.lang.Object

nl.esciencecenter.xenon.schedulers.Scheduler

nl.esciencecenter.xenon.adaptors.schedulers.JobQueueScheduler

All Implemented Interfaces:

java.lang.AutoCloseable

public class JobQueueScheduler
extends Scheduler

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private class	JobQueueScheduler.DaemonThreadFactory Simple thread factory which returns daemon threads instead of normal threads

Field Summary

Fields

Modifier and Type	Field and Description
private java.lang.String	adaptorName
private InteractiveProcessFactory	factory
private FileSystem	filesystem
private java.util.concurrent.atomic.AtomicLong	jobID
private static org.slf4j.Logger	LOGGER
static int	MAX_POLLING_DELAY The maximum allowed value for the polling delay
static int	MIN_POLLING_DELAY The minimal allowed value for the polling delay
private static java.lang.String	MULTI_QUEUE_NAME
private java.util.concurrent.ExecutorService	multiExecutor
private java.util.List<JobExecutor>	multiQ
private long	pollingDelay
private java.util.ArrayList<java.util.List<JobExecutor>>	queues
private static java.lang.String	SINGLE_QUEUE_NAME
private java.util.concurrent.ExecutorService	singleExecutor
private java.util.List<JobExecutor>	singleQ
private long	startupTimeout
private static java.lang.String	UNLIMITED_QUEUE_NAME
private java.util.concurrent.ExecutorService	unlimitedExecutor
private java.util.List<JobExecutor>	unlimitedQ
private Path	workingDirectory

Fields inherited from class nl.esciencecenter.xenon.schedulers.Scheduler

properties

Constructor Summary

Constructors

Constructor and Description
JobQueueScheduler(java.lang.String uniqueID, java.lang.String adaptorName, java.lang.String location, Credential credential, InteractiveProcessFactory factory, FileSystem filesystem, Path workingDirectory, int multiQThreads, long pollingDelay, long startupTimeout, XenonProperties properties)

Method Summary

Modifier and Type	Method and Description
JobStatus	cancelJob(java.lang.String jobIdentifier) Cancel a job.
private boolean	cleanupJob(java.util.List<JobExecutor> queue, java.lang.String jobIdentifier)
private void	cleanupJob(java.lang.String jobIdentifier)
void	close() Close this Scheduler.
void	end()
private JobExecutor	findJob(java.util.List<JobExecutor> queue, java.lang.String jobIdentifier)
private JobExecutor	findJob(java.lang.String jobIdentifier)
long	getCurrentJobID()
java.lang.String	getDefaultQueueName() Get the name of the default queue.
FileSystem	getFileSystem() Retrieve the FileSystem used internally by this Scheduler.
private void	getJobs(java.util.List<JobExecutor> list, java.util.List<java.lang.String> out)
java.lang.String[]	getJobs(java.lang.String... queueNames) Get all job identifier of jobs currently in (one ore more) queues.
JobStatus	getJobStatus(java.lang.String jobIdentifier) Get the status of a Job.
java.lang.String[]	getQueueNames() Get the queue names supported by this Scheduler.
QueueStatus	getQueueStatus(java.lang.String queueName) Get the status of the queue.
QueueStatus[]	getQueueStatuses(java.lang.String... queueNames) Get the status of all queues.
boolean	isOpen() Test if the connection of this Scheduler is open.
private JobExecutor	submit(JobDescription description, boolean interactive)
java.lang.String	submitBatchJob(JobDescription description) Submit a batch job.
Streams	submitInteractiveJob(JobDescription description) Submit an interactive job (optional operation).
private void	verifyJobDescription(JobDescription description, boolean interactive)
JobStatus	waitUntilDone(java.lang.String jobIdentifier, long timeout) Wait until a job is done or until a timeout expires.
JobStatus	waitUntilRunning(java.lang.String jobIdentifier, long timeout) Wait until a job starts running, or until a timeout expires.

Methods inherited from class nl.esciencecenter.xenon.schedulers.Scheduler

assertNonNullOrEmpty, assertPositive, create, create, create, create, equals, getAdaptorDescription, getAdaptorDescriptions, getAdaptorName, getAdaptorNames, getCredential, getJobStatuses, getLocation, getProperties, hashCode

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

SINGLE_QUEUE_NAME

private static final java.lang.String SINGLE_QUEUE_NAME

See Also:

Constant Field Values

MULTI_QUEUE_NAME

private static final java.lang.String MULTI_QUEUE_NAME

See Also:

Constant Field Values

UNLIMITED_QUEUE_NAME

private static final java.lang.String UNLIMITED_QUEUE_NAME

See Also:

Constant Field Values

MIN_POLLING_DELAY

public static final int MIN_POLLING_DELAY

The minimal allowed value for the polling delay

See Also:

Constant Field Values

MAX_POLLING_DELAY

public static final int MAX_POLLING_DELAY

The maximum allowed value for the polling delay

See Also:

Constant Field Values

adaptorName

private final java.lang.String adaptorName

filesystem

private final FileSystem filesystem

workingDirectory

private final Path workingDirectory

singleQ

private final java.util.List<JobExecutor> singleQ

multiQ

private final java.util.List<JobExecutor> multiQ

unlimitedQ

private final java.util.List<JobExecutor> unlimitedQ

singleExecutor

private final java.util.concurrent.ExecutorService singleExecutor

multiExecutor

private final java.util.concurrent.ExecutorService multiExecutor

unlimitedExecutor

private final java.util.concurrent.ExecutorService unlimitedExecutor

pollingDelay

private final long pollingDelay

startupTimeout

private final long startupTimeout

factory

private final InteractiveProcessFactory factory

jobID

private final java.util.concurrent.atomic.AtomicLong jobID

queues

private final java.util.ArrayList<java.util.List<JobExecutor>> queues

Constructor Detail

JobQueueScheduler

public JobQueueScheduler(java.lang.String uniqueID,
                         java.lang.String adaptorName,
                         java.lang.String location,
                         Credential credential,
                         InteractiveProcessFactory factory,
                         FileSystem filesystem,
                         Path workingDirectory,
                         int multiQThreads,
                         long pollingDelay,
                         long startupTimeout,
                         XenonProperties properties)
                  throws BadParameterException

Throws:

BadParameterException

Method Detail

getCurrentJobID

public long getCurrentJobID()

getJobs

private void getJobs(java.util.List<JobExecutor> list,
                     java.util.List<java.lang.String> out)

getDefaultQueueName

public java.lang.String getDefaultQueueName()
                                     throws XenonException

Description copied from class: Scheduler

Get the name of the default queue.

Specified by:

getDefaultQueueName in class Scheduler

Returns:

the name of the default queue for this scheduler, or null if no default queue is available.

Throws:

NotConnectedException - If scheduler is closed.

XenonException - If an I/O error occurred.

getJobs

public java.lang.String[] getJobs(java.lang.String... queueNames)
                           throws NoSuchQueueException

Description copied from class: Scheduler

Get all job identifier of jobs currently in (one ore more) queues. If no queue names are specified, the job identifiers for all queues are returned. Note that job identifiers of jobs submitted by other users or other schedulers may also be returned.

Specified by:

getJobs in class Scheduler

Parameters:

queueNames - the names of the queues.

Returns:

an array containing the resulting job identifiers .

Throws:

NoSuchQueueException - If the queue does not exist in the scheduler.

findJob

private JobExecutor findJob(java.util.List<JobExecutor> queue,
                            java.lang.String jobIdentifier)
                     throws XenonException

Throws:

XenonException

findJob

private JobExecutor findJob(java.lang.String jobIdentifier)
                     throws XenonException

Throws:

XenonException

cleanupJob

private boolean cleanupJob(java.util.List<JobExecutor> queue,
                           java.lang.String jobIdentifier)

cleanupJob

private void cleanupJob(java.lang.String jobIdentifier)

getJobStatus

public JobStatus getJobStatus(java.lang.String jobIdentifier)
                       throws XenonException

Description copied from class: Scheduler

Get the status of a Job.

Specified by:

getJobStatus in class Scheduler

Parameters:

jobIdentifier - the job identifier of the job to get the status for.

Returns:

the status of the Job.

Throws:

NoSuchJobException - If the job is not known.

XenonException - If the status of the job could not be retrieved.

waitUntilDone

public JobStatus waitUntilDone(java.lang.String jobIdentifier,
                               long timeout)
                        throws XenonException

Description copied from class: Scheduler

Wait until a job is done or until a timeout expires.

This method will wait until a job is done (either gracefully or by being killed or producing an error), or until the timeout expires, whichever comes first. If the timeout expires, the job will continue to run.

The timeout is in milliseconds and must be >= 0. When timeout is 0, it will be ignored and this method will wait until the jobs is done.

A JobStatus is returned that can be used to determine why the call returned.

Specified by:

waitUntilDone in class Scheduler

Parameters:

jobIdentifier - the identifier of the to wait for.

timeout - the maximum time to wait for the job in milliseconds.

Returns:

the status of the Job.

Throws:

NoSuchJobException - If the job is not known.

XenonException - If the status of the job could not be retrieved.

waitUntilRunning

public JobStatus waitUntilRunning(java.lang.String jobIdentifier,
                                  long timeout)
                           throws XenonException

Description copied from class: Scheduler

Wait until a job starts running, or until a timeout expires.

This method will return as soon as the job is no longer waiting in the queue, or when the timeout expires, whichever comes first. If the job is no longer waiting in the queue, it may be running, but it may also be killed, finished or have produced an error. If the timeout expires, the job will continue to be queued normally.

The timeout is in milliseconds and must be >= 0. When timeout is 0, it will be ignored and this method will wait until the job is no longer queued.

A JobStatus is returned that can be used to determine why the call returned.

Specified by:

waitUntilRunning in class Scheduler

Parameters:

jobIdentifier - the identifier of the to wait for.

timeout - the maximum time to wait in milliseconds.

Returns:

the status of the Job.

Throws:

NoSuchJobException - If the job is not known.

XenonException - If the status of the job could not be retrieved.

verifyJobDescription

private void verifyJobDescription(JobDescription description,
                                  boolean interactive)
                           throws XenonException

Throws:

XenonException

submit

private JobExecutor submit(JobDescription description,
                           boolean interactive)
                    throws XenonException

Throws:

XenonException

submitBatchJob

public java.lang.String submitBatchJob(JobDescription description)
                                throws XenonException

Description copied from class: Scheduler

Submit a batch job.

Specified by:

submitBatchJob in class Scheduler

Parameters:

description - the description of the batch job to submit.

Returns:

the job identifier representing the running job.

Throws:

IncompleteJobDescriptionException - If the description did not contain the required information.

InvalidJobDescriptionException - If the description contains illegal or conflicting values.

UnsupportedJobDescriptionException - If the description is not legal for this scheduler.

XenonException - If the Scheduler failed to get submit the job.

submitInteractiveJob

public Streams submitInteractiveJob(JobDescription description)
                             throws XenonException

Description copied from class: Scheduler

Submit an interactive job (optional operation).

Specified by:

submitInteractiveJob in class Scheduler

Parameters:

description - the description of the interactive job to submit.

Returns:

a Streams object containing the job identifier and the standard streams of a job.

Throws:

IncompleteJobDescriptionException - If the description did not contain the required information.

InvalidJobDescriptionException - If the description contains illegal or conflicting values.

UnsupportedJobDescriptionException - If the description is not legal for this scheduler.

XenonException - If the Scheduler failed to get submit the job.

cancelJob

public JobStatus cancelJob(java.lang.String jobIdentifier)
                    throws XenonException

Description copied from class: Scheduler

Cancel a job.

A status is returned that indicates the state of the job after the cancel. If the job was already done it cannot be cancelled.

A JobStatus is returned that can be used to determine the state of the job after cancelJob returns. Note that it may take some time before the job has actually terminated. The waitUntilDone method can be used to wait until the job is terminated.

Specified by:

cancelJob in class Scheduler

Parameters:

jobIdentifier - the identifier of job to kill.

Returns:

the status of the Job.

Throws:

NoSuchJobException - If the job is not known.

XenonException - If the status of the job could not be retrieved.

getQueueStatus

public QueueStatus getQueueStatus(java.lang.String queueName)
                           throws XenonException

Description copied from class: Scheduler

Get the status of the queue.

Specified by:

getQueueStatus in class Scheduler

Parameters:

queueName - the name of the queue.

Returns:

the resulting QueueStatus.

Throws:

NoSuchQueueException - If the queue does not exist in the scheduler.

XenonException - If the Scheduler failed to get its status.

getQueueNames

public java.lang.String[] getQueueNames()

Description copied from class: Scheduler

Get the queue names supported by this Scheduler.

Specified by:

getQueueNames in class Scheduler

Returns:

the queue names supported by this Scheduler.

getQueueStatuses

public QueueStatus[] getQueueStatuses(java.lang.String... queueNames)
                               throws XenonException

Description copied from class: Scheduler

Get the status of all queues. Note that this method will only throw an exception when this exception will influence all status requests. For example, if the scheduler is no longer connected. Exceptions that only refer to a single queue are returned in the QueueStatus returned for that queue.

Specified by:

getQueueStatuses in class Scheduler

Parameters:

queueNames - the names of the queues.

Returns:

an array containing the resulting QueueStatus.

Throws:

XenonException - If the Scheduler failed to get the statuses.

end

public void end()

getFileSystem

public FileSystem getFileSystem()
                         throws XenonException

Description copied from class: Scheduler

Retrieve the FileSystem used internally by this Scheduler.

Often, a Scheduler needs to access files or directories on the machine it will schedule jobs. For example, to ensure a working directory exists, or to redirect the stdin, stdout or stderr streams used by a job.

This method returns this FileSystem so it can also be used by the application to prepare input files for the jobs, or retrieve the output files produced by the jobs.

Specified by:

getFileSystem in class Scheduler

Returns:

the FileSystem used by this Scheduler.

Throws:

XenonException - if this Scheduler does not use a FileSystem internally.

close

public void close()
           throws XenonException

Description copied from class: Scheduler

Close this Scheduler.

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in class Scheduler

Throws:

XenonException - If the Scheduler failed to close.

isOpen

public boolean isOpen()
               throws XenonException

Description copied from class: Scheduler

Test if the connection of this Scheduler is open.

Specified by:

isOpen in class Scheduler

Returns:

true if the connection of this Scheduler is still open, false otherwise.

Throws:

XenonException - If an I/O error occurred.