org.apache.mesos

Class MesosSchedulerDriver

java.lang.Object

org.apache.mesos.MesosSchedulerDriver

All Implemented Interfaces:

SchedulerDriver

public class MesosSchedulerDriver
extends Object
implements SchedulerDriver

Concrete implementation of a SchedulerDriver that connects a Scheduler with a Mesos master. The MesosSchedulerDriver is thread-safe.

Note that scheduler failover is supported in Mesos. After a scheduler is registered with Mesos it may failover (to a new process on the same machine or across multiple machines) by creating a new driver with the ID given to it in Scheduler.registered(org.apache.mesos.SchedulerDriver, org.apache.mesos.Protos.FrameworkID, org.apache.mesos.Protos.MasterInfo).

The driver is responsible for invoking the Scheduler callbacks as it communicates with the Mesos master.

Note that blocking on the MesosSchedulerDriver (e.g., via join()) doesn't affect the scheduler callbacks in anyway because they are handled by a different thread.

Note that the driver uses GLOG to do its own logging. GLOG flags can be set via environment variables, prefixing the flag name with "GLOG_", e.g., "GLOG_v=1". For Mesos specific logging flags see src/logging/flags.hpp. Mesos flags can also be set via environment variables, prefixing the flag name with "MESOS_", e.g., "MESOS_QUIET=1".

See src/examples/java/TestFramework.java for an example of using the MesosSchedulerDriver.

Constructor Summary

Constructors

Constructor and Description
MesosSchedulerDriver(Scheduler scheduler, org.apache.mesos.Protos.FrameworkInfo framework, String master) Creates a new driver for the specified scheduler.
MesosSchedulerDriver(Scheduler scheduler, org.apache.mesos.Protos.FrameworkInfo framework, String master, org.apache.mesos.Protos.Credential credential) Same as the above constructor, except that it accepts 'credential' as a parameter.

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.mesos.Protos.Status	abort() Aborts the driver so that no more callbacks can be made to the scheduler.
org.apache.mesos.Protos.Status	declineOffer(org.apache.mesos.Protos.OfferID offerId) Declines an offer in its entirety.
org.apache.mesos.Protos.Status	declineOffer(org.apache.mesos.Protos.OfferID offerId, org.apache.mesos.Protos.Filters filters) Declines an offer in its entirety and applies the specified filters on the resources (see mesos.proto for a description of Filters).
protected void	finalize()
protected void	initialize()
org.apache.mesos.Protos.Status	join() Waits for the driver to be stopped or aborted, possibly blocking the current thread indefinitely.
org.apache.mesos.Protos.Status	killTask(org.apache.mesos.Protos.TaskID taskId) Kills the specified task.
org.apache.mesos.Protos.Status	launchTasks(Collection<org.apache.mesos.Protos.OfferID> offerIds, Collection<org.apache.mesos.Protos.TaskInfo> tasks) Launches the given set of tasks.
org.apache.mesos.Protos.Status	launchTasks(Collection<org.apache.mesos.Protos.OfferID> offerIds, Collection<org.apache.mesos.Protos.TaskInfo> tasks, org.apache.mesos.Protos.Filters filters) Launches the given set of tasks on a set of offers.
org.apache.mesos.Protos.Status	launchTasks(org.apache.mesos.Protos.OfferID offerId, Collection<org.apache.mesos.Protos.TaskInfo> tasks)
org.apache.mesos.Protos.Status	launchTasks(org.apache.mesos.Protos.OfferID offerId, Collection<org.apache.mesos.Protos.TaskInfo> tasks, org.apache.mesos.Protos.Filters filters)
org.apache.mesos.Protos.Status	reconcileTasks(Collection<org.apache.mesos.Protos.TaskStatus> statuses) Allows the framework to query the status for non-terminal tasks.
org.apache.mesos.Protos.Status	requestResources(Collection<org.apache.mesos.Protos.Request> requests) Requests resources from Mesos (see mesos.proto for a description of Request and how, for example, to request resources from specific slaves).
org.apache.mesos.Protos.Status	reviveOffers() Removes all filters, previously set by the framework (via SchedulerDriver.launchTasks(java.util.Collection<org.apache.mesos.Protos.OfferID>, java.util.Collection<org.apache.mesos.Protos.TaskInfo>, org.apache.mesos.Protos.Filters)).
org.apache.mesos.Protos.Status	run() Starts and immediately joins (i.e., blocks on) the driver.
org.apache.mesos.Protos.Status	sendFrameworkMessage(org.apache.mesos.Protos.ExecutorID executorId, org.apache.mesos.Protos.SlaveID slaveId, byte[] data) Sends a message from the framework to one of its executors.
org.apache.mesos.Protos.Status	start() Starts the scheduler driver.
org.apache.mesos.Protos.Status	stop() Stops the scheduler driver assuming no failover.
org.apache.mesos.Protos.Status	stop(boolean failover) Stops the scheduler driver.

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MesosSchedulerDriver

public MesosSchedulerDriver(Scheduler scheduler,
                    org.apache.mesos.Protos.FrameworkInfo framework,
                    String master)

Creates a new driver for the specified scheduler. The master should be one of:

 host:port
     zk://host1:port1,host2:port2,.../path
     zk://username:password@host1:port1,host2:port2,.../path
     file:///path/to/file (where file contains one of the above)
 
 

The driver will attempt to "failover" if the specified FrameworkInfo includes a valid FrameworkID.

Any Mesos configuration options are read from environment variables, as well as any configuration files found through the environment variables.

Parameters:

scheduler - The scheduler implementation which callbacks are invoked upon scheduler events.

framework - The frameworkInfo describing the current framework.

master - The address to the currently active Mesos master.

MesosSchedulerDriver

public MesosSchedulerDriver(Scheduler scheduler,
                    org.apache.mesos.Protos.FrameworkInfo framework,
                    String master,
                    org.apache.mesos.Protos.Credential credential)

Same as the above constructor, except that it accepts 'credential' as a parameter.

Parameters:

scheduler - The scheduler implementation which callbacks are invoked upon scheduler events.

framework - The frameworkInfo describing the current framework.

master - The address to the currently active Mesos master.

credential - The credentials that will be used used to authenticate calls from this scheduler.

Method Detail

start

public org.apache.mesos.Protos.Status start()

Description copied from interface: SchedulerDriver

Starts the scheduler driver. This needs to be called before any other driver calls are made.

Specified by:

start in interface SchedulerDriver

Returns:

The state of the driver after the call.

See Also:

Protos.Status

stop

public org.apache.mesos.Protos.Status stop(boolean failover)

Description copied from interface: SchedulerDriver

Stops the scheduler driver. If the 'failover' flag is set to false then it is expected that this framework will never reconnect to Mesos. So Mesos will unregister the framework and shutdown all its tasks and executors. If 'failover' is true, all executors and tasks will remain running (for some framework specific failover timeout) allowing the scheduler to reconnect (possibly in the same process, or from a different process, for example, on a different machine).

Specified by:

stop in interface SchedulerDriver

Parameters:

failover - Whether framework failover is expected.

Returns:

The state of the driver after the call.

See Also:

Protos.Status

stop

public org.apache.mesos.Protos.Status stop()

Description copied from interface: SchedulerDriver

Stops the scheduler driver assuming no failover. This will cause Mesos to unregister the framework and shutdown all its tasks and executors. Please see SchedulerDriver.stop(boolean) for more details.

Specified by:

stop in interface SchedulerDriver

Returns:

The state of the driver after the call.

abort

public org.apache.mesos.Protos.Status abort()

Description copied from interface: SchedulerDriver

Aborts the driver so that no more callbacks can be made to the scheduler. The semantics of abort and stop have deliberately been separated so that code can detect an aborted driver (i.e., via the return status of SchedulerDriver.join(), see below), and instantiate and start another driver if desired (from within the same process).

Specified by:

abort in interface SchedulerDriver

Returns:

The state of the driver after the call.

join

public org.apache.mesos.Protos.Status join()

Description copied from interface: SchedulerDriver

Waits for the driver to be stopped or aborted, possibly blocking the current thread indefinitely. The return status of this function can be used to determine if the driver was aborted (see mesos.proto for a description of Status).

Specified by:

join in interface SchedulerDriver

Returns:

The state of the driver after the call.

run

public org.apache.mesos.Protos.Status run()

Description copied from interface: SchedulerDriver

Starts and immediately joins (i.e., blocks on) the driver.

Specified by:

run in interface SchedulerDriver

Returns:

The state of the driver after the call.

requestResources

public org.apache.mesos.Protos.Status requestResources(Collection<org.apache.mesos.Protos.Request> requests)

Description copied from interface: SchedulerDriver

Requests resources from Mesos (see mesos.proto for a description of Request and how, for example, to request resources from specific slaves). Any resources available are offered to the framework via Scheduler.resourceOffers(org.apache.mesos.SchedulerDriver, java.util.List<org.apache.mesos.Protos.Offer>) callback, asynchronously.

Specified by:

requestResources in interface SchedulerDriver

Parameters:

requests - The resource requests.

Returns:

The state of the driver after the call.

See Also:

Protos.Request, Protos.Status

launchTasks

public org.apache.mesos.Protos.Status launchTasks(org.apache.mesos.Protos.OfferID offerId,
                                         Collection<org.apache.mesos.Protos.TaskInfo> tasks)

Specified by:

launchTasks in interface SchedulerDriver

Parameters:

offerId - The offer ID.

tasks - The collection of tasks to be launched.

Returns:

The state of the driver after the call.

launchTasks

public org.apache.mesos.Protos.Status launchTasks(org.apache.mesos.Protos.OfferID offerId,
                                         Collection<org.apache.mesos.Protos.TaskInfo> tasks,
                                         org.apache.mesos.Protos.Filters filters)

Specified by:

launchTasks in interface SchedulerDriver

Parameters:

offerId - The offer ID.

tasks - The collection of tasks to be launched.

filters - The filters to set for any remaining resources.

Returns:

The state of the driver after the call.

launchTasks

public org.apache.mesos.Protos.Status launchTasks(Collection<org.apache.mesos.Protos.OfferID> offerIds,
                                         Collection<org.apache.mesos.Protos.TaskInfo> tasks)

Description copied from interface: SchedulerDriver

Launches the given set of tasks. See above for details.

Specified by:

launchTasks in interface SchedulerDriver

Parameters:

offerIds - The collection of offer IDs.

tasks - The collection of tasks to be launched.

Returns:

The state of the driver after the call.

launchTasks

public org.apache.mesos.Protos.Status launchTasks(Collection<org.apache.mesos.Protos.OfferID> offerIds,
                                         Collection<org.apache.mesos.Protos.TaskInfo> tasks,
                                         org.apache.mesos.Protos.Filters filters)

Description copied from interface: SchedulerDriver

Launches the given set of tasks on a set of offers. Resources from offers are aggregated when more then one is provided. Note that all offers must belong to same slave. Any resources remaining (i.e., not used by the tasks or their executors) will be considered declined. The specified filters are applied on all unused resources (see mesos.proto for a description of Filters). Invoking this function with an empty collection of tasks declines offers in their entirety (see SchedulerDriver.declineOffer(org.apache.mesos.Protos.OfferID, org.apache.mesos.Protos.Filters)).

Specified by:

launchTasks in interface SchedulerDriver

Parameters:

offerIds - The collection of offer IDs.

tasks - The collection of tasks to be launched.

filters - The filters to set for any remaining resources.

Returns:

The state of the driver after the call.

See Also:

Protos.OfferID, Protos.TaskInfo, Protos.Filters, Protos.Status

killTask

public org.apache.mesos.Protos.Status killTask(org.apache.mesos.Protos.TaskID taskId)

Description copied from interface: SchedulerDriver

Kills the specified task. Note that attempting to kill a task is currently not reliable. If, for example, a scheduler fails over while it was attempting to kill a task it will need to retry in the future Likewise, if unregistered / disconnected, the request will be dropped (these semantics may be changed in the future).

Specified by:

killTask in interface SchedulerDriver

Parameters:

taskId - The ID of the task to be killed.

Returns:

The state of the driver after the call.

declineOffer

public org.apache.mesos.Protos.Status declineOffer(org.apache.mesos.Protos.OfferID offerId)

Description copied from interface: SchedulerDriver

Declines an offer in its entirety. See above for details.

Specified by:

declineOffer in interface SchedulerDriver

Parameters:

offerId - The ID of the offer to be declined.

Returns:

The state of the driver after the call.

See Also:

Protos.OfferID, Protos.Status

declineOffer

public org.apache.mesos.Protos.Status declineOffer(org.apache.mesos.Protos.OfferID offerId,
                                          org.apache.mesos.Protos.Filters filters)

Description copied from interface: SchedulerDriver

Declines an offer in its entirety and applies the specified filters on the resources (see mesos.proto for a description of Filters). Note that this can be done at any time, it is not necessary to do this within the Scheduler.resourceOffers(org.apache.mesos.SchedulerDriver, java.util.List<org.apache.mesos.Protos.Offer>) callback.

Specified by:

declineOffer in interface SchedulerDriver

Parameters:

offerId - The ID of the offer to be declined.

filters - The filters to set for any remaining resources.

Returns:

The state of the driver after the call.

See Also:

Protos.OfferID, Protos.Filters, Protos.Status

reviveOffers

public org.apache.mesos.Protos.Status reviveOffers()

Description copied from interface: SchedulerDriver

Removes all filters, previously set by the framework (via SchedulerDriver.launchTasks(java.util.Collection<org.apache.mesos.Protos.OfferID>, java.util.Collection<org.apache.mesos.Protos.TaskInfo>, org.apache.mesos.Protos.Filters)). This enables the framework to receive offers from those filtered slaves.

Specified by:

reviveOffers in interface SchedulerDriver

Returns:

The state of the driver after the call.

See Also:

Protos.Status

sendFrameworkMessage

public org.apache.mesos.Protos.Status sendFrameworkMessage(org.apache.mesos.Protos.ExecutorID executorId,
                                                  org.apache.mesos.Protos.SlaveID slaveId,
                                                  byte[] data)

Description copied from interface: SchedulerDriver

Sends a message from the framework to one of its executors. These messages are best effort; do not expect a framework message to be retransmitted in any reliable fashion.

Specified by:

sendFrameworkMessage in interface SchedulerDriver

Parameters:

executorId - The ID of the executor to send the message to.

slaveId - The ID of the slave that is running the executor.

data - The message.

Returns:

The state of the driver after the call.

See Also:

Protos.ExecutorID, Protos.SlaveID

reconcileTasks

public org.apache.mesos.Protos.Status reconcileTasks(Collection<org.apache.mesos.Protos.TaskStatus> statuses)

Description copied from interface: SchedulerDriver

Allows the framework to query the status for non-terminal tasks. This causes the master to send back the latest task status for each task in 'statuses', if possible. Tasks that are no longer known will result in a TASK_LOST update. If statuses is empty, then the master will send the latest status for each task currently known.

Specified by:

reconcileTasks in interface SchedulerDriver

Parameters:

statuses - The collection of non-terminal TaskStatuses to reconcile.

Returns:

The state of the driver after the call.

See Also:

Protos.TaskStatus, Protos.SlaveID

initialize

protected void initialize()

finalize

protected void finalize()

Overrides:

finalize in class Object