Why Another Workflow Engine?

Production Ready: Cylc is not just a research tool. It has been used 24/7 since 2010 for production weather forecasting - which is notorious for the size and complexity of its application workflows.

Efficient Cycling: On deployment to NIWA production systems Cylc's automatic cycle interleaving cut ~24 hours of catch-up from delays down to ~30 minutes. (This is the original reason for Cylc. It also predates some current non-cycling systems).

• ISO 8601 cycling supports any date-time sequence for weather and climate workflows
• integer cycling supports arbitrary repetitive workflows (e.g. satellite data processing).

Open Source: Cylc is available on the GPLv3 license and is actively and openly developed on GitHub.

Distributed Architecture: There are no central workflow or database servers to manage, so Cylc has low admin overhead and a small security footprint. Each workflow gets its own ad hoc server program, which runs as the user, and only relies on the filesystem.

Cross-workflow Triggering: Via the upstream workflow database, so the upstream workflow doesn't even have to be running at the time.

Distributed Workflows: Cylc easily manages task jobs on multiple remote hosts.

Workflow Configuration: A human-readable config file, so modest workflows are easy to write, even for non-programmers. This is not "just a static config file" however: efficient programmatic generation of large complex workflows is supported by task inheritance and parameterization, and Python-like general template processing. (A Python API is also planned for Cylc 9).

Workflow Scaling: Single workflows scale to thousands of tasks per cycle (however, Cylc is a distributed system so several smaller workflows may be preferred).

System Scaling: Cylc is built to transparently use a pool of host VMs, with load balancing at start-up, and it scales arbitrarily well with the number of host VMs. Running workflows can self-migrate to another host at maintenance time.

Powerful UIs: GUIs for workflow visualization, and runtime monitoring and control (GTK for Cylc 7; a web UI for Cylc 8, coming next year). The CLI can flexibly operate on many tasks at once, e.g. to re-trigger all failed tasks matching a certain name pattern in a particular cycle point.

Flexibility and Generality: Cylc can be used equally for every kind of workflow in climate, weather, and weather-driven forecasting (for example), in research, test, and production. Portable workflows are feasible that run "out of the box" in different environments; and even at different sites. This can lead to huge savings: transitioning complex systems from research to operations, for instance, is notoriously difficult if the workflows are different; and much worse again if conversion to a different workflow engine is required.

HPC: Cylc has great support for common HPC batch systems (workload managers) like PBS and Slurm. It can also manage plain background jobs on any Linux host.

Remote Job Hosts: If you need to use ssh to reach a remote job host, Cylc supports that as well. This includes remote job query and kill, filesystem configuration and file installation, and job log retrieval. (Some of this is via Rose but soon to be built in to Cylc).

General External Triggering: Trigger tasks off of anything in the external world - just supply a Python function that checks your external condition.

Resilience: Cylc has many features to help deal with system problems, e.g.:

• configurable retries for both job submission and job execution
• state checkpointing, with safe restarts even in the event of a power outage (Cylc even determines what happened to jobs orphaned during downtime)
• polling to confirm that jobs are still queued or running as expected (if not, job status is updated accordingly)
• manual job control: poll, kill, or retrigger jobs
• failure (and other state) triggering - e.g. define tasks that only run if other tasks fail
• flexible event handling - connect handler scripts to many events (such as task failure); and built-in email alerting

Other Capabilities in Brief:

• Powerful web-based viewer for thousands of job logs
• Validation - catch many errors prior to run time.
• Edit Run - edit job scripts on the fly just before submission.
• Broadcast task runtime settings (including environment variables) to tasks or groups of tasks, at runtime.
• Simulation modes with proportional job run length.
• Internal Queues to limit the number of concurrent active tasks.
• Conditional triggering.
• Clock-triggers - trigger off the clock as well as off other tasks.
• External triggers - trigger off external events as well as other tasks - via user-defined Python plug-in functions.
• Expiring tasks - optionally skip tasks that are too far behind the clock.