Implementing a Custom Scheduler

To run a Heron cluster, you’ll need to set up a scheduler that is responsible for cluster management. Heron currently supports the following schedulers out of the box:

If you’d like to run Heron on a not-yet-supported system, such as Amazon ECS, you can create your own scheduler using Heron’s spi, as detailed in the sections below.

Java is currently the only supported language for custom schedulers. This may change in the future.

Java Setup

In order to create a custom scheduler, you need to import the heron-spi library into your project.

Maven

com.twitter.heron heron-spi 0.14.0

Gradle

dependencies { compile group: "com.twitter.heron", name: "heron-spi", version: "0.14.0" }

Interfaces

Creating a custom scheduler involves implementing each of the following Java interfaces:

Interface Role Examples
IPacking Defines the algorithm used to generate physical plan for a topology. RoundRobin
ILauncher Defines how the scheduler is launched Aurora, local
IScheduler Defines the scheduler object used to construct topologies local
IUploader Uploads the topology to a shared location accessible to the runtime environment of the topology local hdfs s3

Heron provides a number of built-in implementations out of box.

Running the Scheduler

To run the a custom scheduler, the implementation of the interfaces above must be specified in the config. By default, the heron-cli looks for configurations under ${HERON_HOME}/conf/. The location can be overridden using option --config-path. Below is an example showing the command for topology submission:

$ heron submit [cluster-name-storing-your-new-config]/[role]/[env] \ --config-path [config-folder-path-storing-your-new-config] \ /path/to/topology/my-topology.jar \ biz.acme.topologies.MyTopology

The implementation for each of the interfaces listed above must be on Heron’s classpath.