org.apache.flink.runtime.dispatcher

Class Dispatcher

java.lang.Object

org.apache.flink.runtime.rpc.RpcEndpoint

org.apache.flink.runtime.rpc.FencedRpcEndpoint<DispatcherId>

org.apache.flink.runtime.dispatcher.Dispatcher

All Implemented Interfaces:

DispatcherGateway, SubmittedJobGraphStore.SubmittedJobGraphListener, LeaderContender, FencedRpcGateway<DispatcherId>, RpcGateway, RestfulGateway

Direct Known Subclasses:

MiniDispatcher, StandaloneDispatcher

public abstract class Dispatcher
extends FencedRpcEndpoint<DispatcherId>
implements DispatcherGateway, LeaderContender, SubmittedJobGraphStore.SubmittedJobGraphListener

Base class for the Dispatcher component. The Dispatcher component is responsible for receiving job submissions, persisting them, spawning JobManagers to execute the jobs and to recover them in case of a master failure. Furthermore, it knows about the state of the Flink session cluster.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Dispatcher.DefaultJobManagerRunnerFactory Singleton default factory for JobManagerRunner.
static interface	Dispatcher.JobManagerRunnerFactory Factory for a JobManagerRunner.

Nested classes/interfaces inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

RpcEndpoint.MainThreadExecutor

Field Summary

Fields

Modifier and Type	Field and Description
static String	DISPATCHER_NAME
protected String	restAddress

Fields inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

log, rpcServer

Constructor Summary

Constructors

Constructor and Description

Dispatcher(RpcService rpcService, String endpointId, org.apache.flink.configuration.Configuration configuration, HighAvailabilityServices highAvailabilityServices, SubmittedJobGraphStore submittedJobGraphStore, ResourceManagerGateway resourceManagerGateway, BlobServer blobServer, HeartbeatServices heartbeatServices, JobManagerMetricGroup jobManagerMetricGroup, String metricServiceQueryPath, ArchivedExecutionGraphStore archivedExecutionGraphStore, Dispatcher.JobManagerRunnerFactory jobManagerRunnerFactory, FatalErrorHandler fatalErrorHandler, String restAddress, HistoryServerArchivist historyServerArchivist)

Method Summary

Modifier and Type	Method and Description
CompletableFuture<Acknowledge>	cancelJob(org.apache.flink.api.common.JobID jobId, org.apache.flink.api.common.time.Time timeout) Cancel the given job.
CompletableFuture<Acknowledge>	disposeSavepoint(String savepointPath, org.apache.flink.api.common.time.Time timeout) Dispose the given savepoint.
CompletableFuture<Integer>	getBlobServerPort(org.apache.flink.api.common.time.Time timeout) Returns the port of the blob server.
void	grantLeadership(UUID newLeaderSessionID) Callback method when current resourceManager is granted leadership.
void	handleError(Exception exception) Handles error occurring in the leader election service.
protected void	jobNotFinished(org.apache.flink.api.common.JobID jobId)
protected void	jobReachedGloballyTerminalState(ArchivedExecutionGraph archivedExecutionGraph)
CompletableFuture<Collection<org.apache.flink.api.common.JobID>>	listJobs(org.apache.flink.api.common.time.Time timeout) List the current set of submitted jobs.
void	onAddedJobGraph(org.apache.flink.api.common.JobID jobId) Callback for SubmittedJobGraph instances added by a different SubmittedJobGraphStore instance.
protected void	onFatalError(Throwable throwable)
void	onRemovedJobGraph(org.apache.flink.api.common.JobID jobId) Callback for SubmittedJobGraph instances removed by a different SubmittedJobGraphStore instance.
CompletableFuture<Void>	postStop() User overridable callback.
CompletableFuture<ClusterOverview>	requestClusterOverview(org.apache.flink.api.common.time.Time timeout) Requests the cluster status overview.
CompletableFuture<ArchivedExecutionGraph>	requestJob(org.apache.flink.api.common.JobID jobId, org.apache.flink.api.common.time.Time timeout) Requests the ArchivedExecutionGraph for the given jobId.
CompletableFuture<JobResult>	requestJobResult(org.apache.flink.api.common.JobID jobId, org.apache.flink.api.common.time.Time timeout) Requests the JobResult of a job specified by the given jobId.
CompletableFuture<JobStatus>	requestJobStatus(org.apache.flink.api.common.JobID jobId, org.apache.flink.api.common.time.Time timeout) Request the JobStatus of the given job.
CompletableFuture<Collection<String>>	requestMetricQueryServicePaths(org.apache.flink.api.common.time.Time timeout) Requests the paths for the MetricQueryService to query.
CompletableFuture<MultipleJobsDetails>	requestMultipleJobDetails(org.apache.flink.api.common.time.Time timeout) Requests job details currently being executed on the Flink cluster.
CompletableFuture<OperatorBackPressureStatsResponse>	requestOperatorBackPressureStats(org.apache.flink.api.common.JobID jobId, JobVertexID jobVertexId) Requests the statistics on operator back pressure.
CompletableFuture<String>	requestRestAddress(org.apache.flink.api.common.time.Time timeout) Requests the REST address of this RpcEndpoint.
CompletableFuture<Collection<org.apache.flink.api.java.tuple.Tuple2<ResourceID,String>>>	requestTaskManagerMetricQueryServicePaths(org.apache.flink.api.common.time.Time timeout) Requests the paths for the TaskManager's MetricQueryService to query.
CompletableFuture<Acknowledge>	rescaleJob(org.apache.flink.api.common.JobID jobId, int newParallelism, RescalingBehaviour rescalingBehaviour, org.apache.flink.api.common.time.Time timeout) Trigger rescaling of the given job.
void	revokeLeadership() Callback method when current resourceManager loses leadership.
CompletableFuture<Acknowledge>	shutDownCluster()
void	start() Starts the rpc endpoint.
CompletableFuture<Acknowledge>	stopJob(org.apache.flink.api.common.JobID jobId, org.apache.flink.api.common.time.Time timeout) Stop the given job.
CompletableFuture<Acknowledge>	submitJob(JobGraph jobGraph, org.apache.flink.api.common.time.Time timeout) Submit a job to the dispatcher.
CompletableFuture<String>	triggerSavepoint(org.apache.flink.api.common.JobID jobId, String targetDirectory, boolean cancelJob, org.apache.flink.api.common.time.Time timeout) Triggers a savepoint with the given savepoint directory as a target.

Methods inherited from class org.apache.flink.runtime.rpc.FencedRpcEndpoint

callAsyncWithoutFencing, getFencingToken, getMainThreadExecutor, getUnfencedMainThreadExecutor, runAsyncWithoutFencing, setFencingToken

Methods inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

callAsync, getAddress, getEndpointId, getHostname, getRpcService, getSelfGateway, getTerminationFuture, runAsync, scheduleRunAsync, scheduleRunAsync, shutDown, stop, validateRunsInMainThread

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.rpc.FencedRpcGateway

getFencingToken

Methods inherited from interface org.apache.flink.runtime.rpc.RpcGateway

getAddress, getHostname

Methods inherited from interface org.apache.flink.runtime.leaderelection.LeaderContender

getAddress

Field Detail

DISPATCHER_NAME

public static final String DISPATCHER_NAME

See Also:

Constant Field Values

restAddress

@Nullable
protected final String restAddress

Constructor Detail

Dispatcher

public Dispatcher(RpcService rpcService,
                  String endpointId,
                  org.apache.flink.configuration.Configuration configuration,
                  HighAvailabilityServices highAvailabilityServices,
                  SubmittedJobGraphStore submittedJobGraphStore,
                  ResourceManagerGateway resourceManagerGateway,
                  BlobServer blobServer,
                  HeartbeatServices heartbeatServices,
                  JobManagerMetricGroup jobManagerMetricGroup,
                  @Nullable
                  String metricServiceQueryPath,
                  ArchivedExecutionGraphStore archivedExecutionGraphStore,
                  Dispatcher.JobManagerRunnerFactory jobManagerRunnerFactory,
                  FatalErrorHandler fatalErrorHandler,
                  @Nullable
                  String restAddress,
                  HistoryServerArchivist historyServerArchivist)
           throws Exception

Throws:

Exception

Method Detail

postStop

public CompletableFuture<Void> postStop()

Description copied from class: RpcEndpoint

User overridable callback.

This method is called when the RpcEndpoint is being shut down. The method is guaranteed to be executed in the main thread context and can be used to clean up internal state.

IMPORTANT: This method should never be called directly by the user.

Specified by:

postStop in class RpcEndpoint

Returns:

Future which is completed once all post stop actions are completed. If an error occurs this future is completed exceptionally

start

public void start()
           throws Exception

Description copied from class: RpcEndpoint

Starts the rpc endpoint. This tells the underlying rpc server that the rpc endpoint is ready to process remote procedure calls.

IMPORTANT: Whenever you override this method, call the parent implementation to enable rpc processing. It is advised to make the parent call last.

Overrides:

start in class RpcEndpoint

Throws:

Exception - indicating that something went wrong while starting the RPC endpoint

submitJob

public CompletableFuture<Acknowledge> submitJob(JobGraph jobGraph,
                                                org.apache.flink.api.common.time.Time timeout)

Description copied from interface: DispatcherGateway

Submit a job to the dispatcher.

Specified by:

submitJob in interface DispatcherGateway

Parameters:

jobGraph - JobGraph to submit

timeout - RPC timeout

Returns:

A future acknowledge if the submission succeeded

listJobs

public CompletableFuture<Collection<org.apache.flink.api.common.JobID>> listJobs(org.apache.flink.api.common.time.Time timeout)

Description copied from interface: DispatcherGateway

List the current set of submitted jobs.

Specified by:

listJobs in interface DispatcherGateway

Parameters:

timeout - RPC timeout

Returns:

A future collection of currently submitted jobs

disposeSavepoint

public CompletableFuture<Acknowledge> disposeSavepoint(String savepointPath,
                                                       org.apache.flink.api.common.time.Time timeout)

Description copied from interface: RestfulGateway

Dispose the given savepoint.

Specified by:

disposeSavepoint in interface RestfulGateway

Parameters:

savepointPath - identifying the savepoint to dispose

timeout - RPC timeout

Returns:

A future acknowledge if the disposal succeeded

cancelJob

public CompletableFuture<Acknowledge> cancelJob(org.apache.flink.api.common.JobID jobId,
                                                org.apache.flink.api.common.time.Time timeout)

Description copied from interface: RestfulGateway

Cancel the given job.

Specified by:

cancelJob in interface RestfulGateway

Parameters:

jobId - identifying the job to cancel

timeout - of the operation

Returns:

A future acknowledge if the cancellation succeeded

stopJob

public CompletableFuture<Acknowledge> stopJob(org.apache.flink.api.common.JobID jobId,
                                              org.apache.flink.api.common.time.Time timeout)

Description copied from interface: RestfulGateway

Stop the given job.

Specified by:

stopJob in interface RestfulGateway

Parameters:

jobId - identifying the job to stop

timeout - of the operation

Returns:

A future acknowledge if the stopping succeeded

rescaleJob

public CompletableFuture<Acknowledge> rescaleJob(org.apache.flink.api.common.JobID jobId,
                                                 int newParallelism,
                                                 RescalingBehaviour rescalingBehaviour,
                                                 org.apache.flink.api.common.time.Time timeout)

Description copied from interface: RestfulGateway

Trigger rescaling of the given job.

Specified by:

rescaleJob in interface RestfulGateway

Parameters:

jobId - specifying the job to rescale

newParallelism - new parallelism of the job

rescalingBehaviour - defining how strict the rescaling has to be executed

timeout - of this operation

Returns:

Future which is completed with Acknowledge once the rescaling was successful

requestRestAddress

public CompletableFuture<String> requestRestAddress(org.apache.flink.api.common.time.Time timeout)

Description copied from interface: RestfulGateway

Requests the REST address of this RpcEndpoint.

Specified by:

requestRestAddress in interface RestfulGateway

Parameters:

timeout - for this operation

Returns:

Future REST endpoint address

requestClusterOverview

public CompletableFuture<ClusterOverview> requestClusterOverview(org.apache.flink.api.common.time.Time timeout)

Description copied from interface: RestfulGateway

Requests the cluster status overview.

Specified by:

requestClusterOverview in interface RestfulGateway

Parameters:

timeout - for the asynchronous operation

Returns:

Future containing the status overview

requestMultipleJobDetails

public CompletableFuture<MultipleJobsDetails> requestMultipleJobDetails(org.apache.flink.api.common.time.Time timeout)

Description copied from interface: RestfulGateway

Requests job details currently being executed on the Flink cluster.

Specified by:

requestMultipleJobDetails in interface RestfulGateway

Parameters:

timeout - for the asynchronous operation

Returns:

Future containing the job details

requestJobStatus

public CompletableFuture<JobStatus> requestJobStatus(org.apache.flink.api.common.JobID jobId,
                                                     org.apache.flink.api.common.time.Time timeout)

Description copied from interface: RestfulGateway

Request the JobStatus of the given job.

Specified by:

requestJobStatus in interface RestfulGateway

Parameters:

jobId - identifying the job for which to retrieve the JobStatus

timeout - for the asynchronous operation

Returns:

A future to the JobStatus of the given job

requestOperatorBackPressureStats

public CompletableFuture<OperatorBackPressureStatsResponse> requestOperatorBackPressureStats(org.apache.flink.api.common.JobID jobId,
                                                                                             JobVertexID jobVertexId)

Description copied from interface: RestfulGateway

Requests the statistics on operator back pressure.

Specified by:

requestOperatorBackPressureStats in interface RestfulGateway

Parameters:

jobId - Job for which the stats are requested.

jobVertexId - JobVertex for which the stats are requested.

Returns:

A Future to the OperatorBackPressureStatsResponse or null if the stats are not available (yet).

requestJob

public CompletableFuture<ArchivedExecutionGraph> requestJob(org.apache.flink.api.common.JobID jobId,
                                                            org.apache.flink.api.common.time.Time timeout)

Description copied from interface: DispatcherGateway

Requests the ArchivedExecutionGraph for the given jobId. If there is no such graph, then the future is completed with a FlinkJobNotFoundException.

Note: We enforce that the returned future contains a ArchivedExecutionGraph unlike the super interface.

Specified by:

requestJob in interface DispatcherGateway

Specified by:

requestJob in interface RestfulGateway

Parameters:

jobId - identifying the job whose AccessExecutionGraph is requested

timeout - for the asynchronous operation

Returns:

Future containing the AccessExecutionGraph for the given jobId, otherwise FlinkJobNotFoundException

requestJobResult

public CompletableFuture<JobResult> requestJobResult(org.apache.flink.api.common.JobID jobId,
                                                     org.apache.flink.api.common.time.Time timeout)

Description copied from interface: RestfulGateway

Requests the JobResult of a job specified by the given jobId.

Specified by:

requestJobResult in interface RestfulGateway

Parameters:

jobId - identifying the job for which to retrieve the JobResult.

timeout - for the asynchronous operation

Returns:

Future which is completed with the job's JobResult once the job has finished

requestMetricQueryServicePaths

public CompletableFuture<Collection<String>> requestMetricQueryServicePaths(org.apache.flink.api.common.time.Time timeout)

Description copied from interface: RestfulGateway

Requests the paths for the MetricQueryService to query.

Specified by:

requestMetricQueryServicePaths in interface RestfulGateway

Parameters:

timeout - for the asynchronous operation

Returns:

Future containing the collection of metric query service paths to query

requestTaskManagerMetricQueryServicePaths

public CompletableFuture<Collection<org.apache.flink.api.java.tuple.Tuple2<ResourceID,String>>> requestTaskManagerMetricQueryServicePaths(org.apache.flink.api.common.time.Time timeout)

Description copied from interface: RestfulGateway

Requests the paths for the TaskManager's MetricQueryService to query.

Specified by:

requestTaskManagerMetricQueryServicePaths in interface RestfulGateway

Parameters:

timeout - for the asynchronous operation

Returns:

Future containing the collection of instance ids and the corresponding metric query service path

getBlobServerPort

public CompletableFuture<Integer> getBlobServerPort(org.apache.flink.api.common.time.Time timeout)

Description copied from interface: DispatcherGateway

Returns the port of the blob server.

Specified by:

getBlobServerPort in interface DispatcherGateway

Parameters:

timeout - of the operation

Returns:

A future integer of the blob server port

triggerSavepoint

public CompletableFuture<String> triggerSavepoint(org.apache.flink.api.common.JobID jobId,
                                                  String targetDirectory,
                                                  boolean cancelJob,
                                                  org.apache.flink.api.common.time.Time timeout)

Description copied from interface: RestfulGateway

Triggers a savepoint with the given savepoint directory as a target.

Specified by:

triggerSavepoint in interface RestfulGateway

Parameters:

jobId - ID of the job for which the savepoint should be triggered.

targetDirectory - Target directory for the savepoint.

timeout - Timeout for the asynchronous operation

Returns:

A future to the external pointer of the savepoint.

shutDownCluster

public CompletableFuture<Acknowledge> shutDownCluster()

Specified by:

shutDownCluster in interface RestfulGateway

onFatalError

protected void onFatalError(Throwable throwable)

jobReachedGloballyTerminalState

protected void jobReachedGloballyTerminalState(ArchivedExecutionGraph archivedExecutionGraph)

jobNotFinished

protected void jobNotFinished(org.apache.flink.api.common.JobID jobId)

grantLeadership

public void grantLeadership(UUID newLeaderSessionID)

Callback method when current resourceManager is granted leadership.

Specified by:

grantLeadership in interface LeaderContender

Parameters:

newLeaderSessionID - unique leadershipID

revokeLeadership

public void revokeLeadership()

Callback method when current resourceManager loses leadership.

Specified by:

revokeLeadership in interface LeaderContender

handleError

public void handleError(Exception exception)

Handles error occurring in the leader election service.

Specified by:

handleError in interface LeaderContender

Parameters:

exception - Exception being thrown in the leader election service

onAddedJobGraph

public void onAddedJobGraph(org.apache.flink.api.common.JobID jobId)

Description copied from interface: SubmittedJobGraphStore.SubmittedJobGraphListener

Callback for SubmittedJobGraph instances added by a different SubmittedJobGraphStore instance.

Important: It is possible to get false positives and be notified about a job graph, which was added by this instance.

Specified by:

onAddedJobGraph in interface SubmittedJobGraphStore.SubmittedJobGraphListener

Parameters:

jobId - The JobID of the added job graph

onRemovedJobGraph

public void onRemovedJobGraph(org.apache.flink.api.common.JobID jobId)

Description copied from interface: SubmittedJobGraphStore.SubmittedJobGraphListener

Callback for SubmittedJobGraph instances removed by a different SubmittedJobGraphStore instance.

Specified by:

onRemovedJobGraph in interface SubmittedJobGraphStore.SubmittedJobGraphListener

Parameters:

jobId - The JobID of the removed job graph