org.apache.flink.runtime.clusterframework

Class BootstrapTools

java.lang.Object

org.apache.flink.runtime.clusterframework.BootstrapTools

public class BootstrapTools
extends Object

Tools for starting JobManager and TaskManager processes, including the Actor Systems used to run the JobManager and TaskManager actors.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BootstrapTools.FixedThreadPoolExecutorConfiguration Configuration for a fixed thread pool executor.
static class	BootstrapTools.ForkJoinExecutorConfiguration Configuration for a fork join executor.

Method Summary

Modifier and Type	Method and Description
static org.apache.flink.configuration.Configuration	cloneConfiguration(org.apache.flink.configuration.Configuration configuration) Clones the given configuration and resets instance specific config options.
static org.apache.flink.configuration.Configuration	generateTaskManagerConfiguration(org.apache.flink.configuration.Configuration baseConfig, String jobManagerHostname, int jobManagerPort, int numSlots, scala.concurrent.duration.FiniteDuration registrationTimeout) Generate a task manager configuration.
static String	getStartCommand(String template, Map<String,String> startCommandValues) Replaces placeholders in the template start command with values from startCommandValues.
static String	getTaskManagerShellCommand(org.apache.flink.configuration.Configuration flinkConfig, ContaineredTaskManagerParameters tmParams, String configDirectory, String logDirectory, boolean hasLogback, boolean hasLog4j, boolean hasKrb5, Class<?> mainClass) Generates the shell command to start a task manager.
static org.apache.commons.cli.Option	newDynamicPropertiesOption() Get an instance of the dynamic properties option.
static org.apache.flink.configuration.Configuration	parseDynamicProperties(org.apache.commons.cli.CommandLine cmd) Parse the dynamic properties (passed on the command line).
static akka.actor.ActorSystem	startActorSystem(org.apache.flink.configuration.Configuration configuration, String listeningAddress, int listeningPort, org.slf4j.Logger logger) Starts an Actor System at a specific port.
static akka.actor.ActorSystem	startActorSystem(org.apache.flink.configuration.Configuration configuration, String listeningAddress, int listeningPort, org.slf4j.Logger logger, org.apache.flink.runtime.clusterframework.BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration) Starts an Actor System at a specific port.
static akka.actor.ActorSystem	startActorSystem(org.apache.flink.configuration.Configuration configuration, String actorSystemName, String listeningAddress, int listeningPort, org.slf4j.Logger logger, org.apache.flink.runtime.clusterframework.BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration) Starts an Actor System at a specific port.
static akka.actor.ActorSystem	startActorSystem(org.apache.flink.configuration.Configuration configuration, String listeningAddress, String portRangeDefinition, org.slf4j.Logger logger) Starts an ActorSystem with the given configuration listening at the address/ports.
static akka.actor.ActorSystem	startActorSystem(org.apache.flink.configuration.Configuration configuration, String listeningAddress, String portRangeDefinition, org.slf4j.Logger logger, org.apache.flink.runtime.clusterframework.BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration) Starts an ActorSystem with the given configuration listening at the address/ports.
static akka.actor.ActorSystem	startActorSystem(org.apache.flink.configuration.Configuration configuration, String actorSystemName, String listeningAddress, String portRangeDefinition, org.slf4j.Logger logger, org.apache.flink.runtime.clusterframework.BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration) Starts an ActorSystem with the given configuration listening at the address/ports.
static WebMonitor	startWebMonitorIfConfigured(org.apache.flink.configuration.Configuration config, HighAvailabilityServices highAvailabilityServices, LeaderGatewayRetriever<JobManagerGateway> jobManagerRetriever, MetricQueryServiceRetriever queryServiceRetriever, org.apache.flink.api.common.time.Time timeout, ScheduledExecutor scheduledExecutor, org.slf4j.Logger logger) Starts the web frontend.
static void	substituteDeprecatedConfigKey(org.apache.flink.configuration.Configuration config, String deprecated, String designated) Sets the value of a new config key to the value of a deprecated config key.
static void	substituteDeprecatedConfigPrefix(org.apache.flink.configuration.Configuration config, String deprecatedPrefix, String designatedPrefix) Sets the value of a new config key to the value of a deprecated config key.
static void	updateTmpDirectoriesInConfiguration(org.apache.flink.configuration.Configuration configuration, String defaultDirs) Set temporary configuration directories if necessary.
static void	writeConfiguration(org.apache.flink.configuration.Configuration cfg, File file) Writes a Flink YAML config file from a Flink Configuration object.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

startActorSystem

public static akka.actor.ActorSystem startActorSystem(org.apache.flink.configuration.Configuration configuration,
                                                      String listeningAddress,
                                                      String portRangeDefinition,
                                                      org.slf4j.Logger logger)
                                               throws Exception

Starts an ActorSystem with the given configuration listening at the address/ports.

Parameters:

configuration - The Flink configuration

listeningAddress - The address to listen at.

portRangeDefinition - The port range to choose a port from.

logger - The logger to output log information.

Returns:

The ActorSystem which has been started

Throws:

Exception - Thrown when actor system cannot be started in specified port range

startActorSystem

public static akka.actor.ActorSystem startActorSystem(org.apache.flink.configuration.Configuration configuration,
                                                      String listeningAddress,
                                                      String portRangeDefinition,
                                                      org.slf4j.Logger logger,
                                                      @Nonnull
                                                      org.apache.flink.runtime.clusterframework.BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration)
                                               throws Exception

Starts an ActorSystem with the given configuration listening at the address/ports.

Parameters:

configuration - The Flink configuration

listeningAddress - The address to listen at.

portRangeDefinition - The port range to choose a port from.

logger - The logger to output log information.

actorSystemExecutorConfiguration - configuration for the ActorSystem's underlying executor

Returns:

The ActorSystem which has been started

Throws:

Exception - Thrown when actor system cannot be started in specified port range

startActorSystem

public static akka.actor.ActorSystem startActorSystem(org.apache.flink.configuration.Configuration configuration,
                                                      String actorSystemName,
                                                      String listeningAddress,
                                                      String portRangeDefinition,
                                                      org.slf4j.Logger logger,
                                                      @Nonnull
                                                      org.apache.flink.runtime.clusterframework.BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration)
                                               throws Exception

Starts an ActorSystem with the given configuration listening at the address/ports.

Parameters:

configuration - The Flink configuration

actorSystemName - Name of the started ActorSystem

listeningAddress - The address to listen at.

portRangeDefinition - The port range to choose a port from.

logger - The logger to output log information.

actorSystemExecutorConfiguration - configuration for the ActorSystem's underlying executor

Returns:

The ActorSystem which has been started

Throws:

Exception - Thrown when actor system cannot be started in specified port range

startActorSystem

public static akka.actor.ActorSystem startActorSystem(org.apache.flink.configuration.Configuration configuration,
                                                      String listeningAddress,
                                                      int listeningPort,
                                                      org.slf4j.Logger logger)
                                               throws Exception

Starts an Actor System at a specific port.

Parameters:

configuration - The Flink configuration.

listeningAddress - The address to listen at.

listeningPort - The port to listen at.

logger - the logger to output log information.

Returns:

The ActorSystem which has been started.

Throws:

Exception

startActorSystem

public static akka.actor.ActorSystem startActorSystem(org.apache.flink.configuration.Configuration configuration,
                                                      String listeningAddress,
                                                      int listeningPort,
                                                      org.slf4j.Logger logger,
                                                      org.apache.flink.runtime.clusterframework.BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration)
                                               throws Exception

Starts an Actor System at a specific port.

Parameters:

configuration - The Flink configuration.

listeningAddress - The address to listen at.

listeningPort - The port to listen at.

logger - the logger to output log information.

actorSystemExecutorConfiguration - configuration for the ActorSystem's underlying executor

Returns:

The ActorSystem which has been started.

Throws:

Exception

startActorSystem

public static akka.actor.ActorSystem startActorSystem(org.apache.flink.configuration.Configuration configuration,
                                                      String actorSystemName,
                                                      String listeningAddress,
                                                      int listeningPort,
                                                      org.slf4j.Logger logger,
                                                      org.apache.flink.runtime.clusterframework.BootstrapTools.ActorSystemExecutorConfiguration actorSystemExecutorConfiguration)
                                               throws Exception

Starts an Actor System at a specific port.

Parameters:

configuration - The Flink configuration.

actorSystemName - Name of the started ActorSystem

listeningAddress - The address to listen at.

listeningPort - The port to listen at.

logger - the logger to output log information.

actorSystemExecutorConfiguration - configuration for the ActorSystem's underlying executor

Returns:

The ActorSystem which has been started.

Throws:

Exception

startWebMonitorIfConfigured

public static WebMonitor startWebMonitorIfConfigured(org.apache.flink.configuration.Configuration config,
                                                     HighAvailabilityServices highAvailabilityServices,
                                                     LeaderGatewayRetriever<JobManagerGateway> jobManagerRetriever,
                                                     MetricQueryServiceRetriever queryServiceRetriever,
                                                     org.apache.flink.api.common.time.Time timeout,
                                                     ScheduledExecutor scheduledExecutor,
                                                     org.slf4j.Logger logger)
                                              throws Exception

Starts the web frontend.

Parameters:

config - The Flink config.

highAvailabilityServices - Service factory for high availability services

jobManagerRetriever - to retrieve the leading JobManagerGateway

queryServiceRetriever - to resolve a query service

timeout - for asynchronous operations

scheduledExecutor - to run asynchronous operations

logger - Logger for log output

Returns:

WebMonitor instance.

Throws:

Exception

generateTaskManagerConfiguration

public static org.apache.flink.configuration.Configuration generateTaskManagerConfiguration(org.apache.flink.configuration.Configuration baseConfig,
                                                                                            String jobManagerHostname,
                                                                                            int jobManagerPort,
                                                                                            int numSlots,
                                                                                            scala.concurrent.duration.FiniteDuration registrationTimeout)

Generate a task manager configuration.

Parameters:

baseConfig - Config to start from.

jobManagerHostname - Job manager host name.

jobManagerPort - Port of the job manager.

numSlots - Number of slots to configure.

registrationTimeout - Timeout for registration

Returns:

TaskManager configuration

writeConfiguration

public static void writeConfiguration(org.apache.flink.configuration.Configuration cfg,
                                      File file)
                               throws IOException

Writes a Flink YAML config file from a Flink Configuration object.

Parameters:

cfg - The Flink config

file - The File to write to

Throws:

IOException

substituteDeprecatedConfigKey

public static void substituteDeprecatedConfigKey(org.apache.flink.configuration.Configuration config,
                                                 String deprecated,
                                                 String designated)

Sets the value of a new config key to the value of a deprecated config key.

Parameters:

config - Config to write

deprecated - The old config key

designated - The new config key

substituteDeprecatedConfigPrefix

public static void substituteDeprecatedConfigPrefix(org.apache.flink.configuration.Configuration config,
                                                    String deprecatedPrefix,
                                                    String designatedPrefix)

Sets the value of a new config key to the value of a deprecated config key. Taking into account the changed prefix.

Parameters:

config - Config to write

deprecatedPrefix - Old prefix of key

designatedPrefix - New prefix of key

newDynamicPropertiesOption

public static org.apache.commons.cli.Option newDynamicPropertiesOption()

Get an instance of the dynamic properties option.

Dynamic properties allow the user to specify additional configuration values with -D, such as -Dfs.overwrite-files=true -Dtaskmanager.network.memory.min=536346624

parseDynamicProperties

public static org.apache.flink.configuration.Configuration parseDynamicProperties(org.apache.commons.cli.CommandLine cmd)

Parse the dynamic properties (passed on the command line).

getTaskManagerShellCommand

public static String getTaskManagerShellCommand(org.apache.flink.configuration.Configuration flinkConfig,
                                                ContaineredTaskManagerParameters tmParams,
                                                String configDirectory,
                                                String logDirectory,
                                                boolean hasLogback,
                                                boolean hasLog4j,
                                                boolean hasKrb5,
                                                Class<?> mainClass)

Generates the shell command to start a task manager.

Parameters:

flinkConfig - The Flink configuration.

tmParams - Parameters for the task manager.

configDirectory - The configuration directory for the flink-conf.yaml

logDirectory - The log directory.

hasLogback - Uses logback?

hasLog4j - Uses log4j?

mainClass - The main class to start with.

Returns:

A String containing the task manager startup command.

getStartCommand

public static String getStartCommand(String template,
                                     Map<String,String> startCommandValues)

Replaces placeholders in the template start command with values from startCommandValues.

If the default template ConfigConstants.DEFAULT_YARN_CONTAINER_START_COMMAND_TEMPLATE is used, the following keys must be present in the map or the resulting command will still contain placeholders:

• java = path to the Java executable
• jvmmem = JVM memory limits and tweaks
• jvmopts = misc options for the Java VM
• logging = logging-related configuration settings
• class = main class to execute
• args = arguments for the main class
• redirects = output redirects

Parameters:

template - a template start command with placeholders

startCommandValues - a replacement map placeholder -> value

Returns:

the start command with placeholders filled in

updateTmpDirectoriesInConfiguration

public static void updateTmpDirectoriesInConfiguration(org.apache.flink.configuration.Configuration configuration,
                                                       @Nullable
                                                       String defaultDirs)

Set temporary configuration directories if necessary.

Parameters:

configuration - flink config to patch

defaultDirs - in case no tmp directories is set, next directories will be applied

cloneConfiguration

public static org.apache.flink.configuration.Configuration cloneConfiguration(org.apache.flink.configuration.Configuration configuration)

Clones the given configuration and resets instance specific config options.

Parameters:

configuration - to clone

Returns:

Cloned configuration with reset instance specific config options