Eventline handbook

Preface

What is Eventline?

Writing scripts to automate recurring tasks is the bread and butter of every developer, SRE or data scientist out there. But running these scripts is always annoying: hosting, deployment, secret management, monitoring, everything has to be handled correctly.

Eventline is a platform that stores, schedules, executes and monitors all your jobs. You define what the job does, where to run it and when to trigger it; Eventline takes care of everything else.

Features

Eventline is battery included by design, and provides multiple features designed to make your life easier:

Declarative job definitions with inline or external source code.
Support for multiple runners: local execution, Docker, Kubernetes…
Events used to trigger jobs when certain conditions are met.
Manual execution with custom parameters and auto-generated forms.
Identities to store credentials, with full OAuth2 support.
Observability with access to all past runs of your jobs, success rate and performance monitoring, output logs…
Full runtime control, letting you abort jobs, restart them and replay events.
HTTP API and command line tool.

Digging in

Interested? Good! In the next section, we will install Eventline and start writing our first job.

Curious about the way Eventline works? Jump directly to the source code on GitHub.

Feel free to contact us at any time if you have any question. Or ask directly on GitHub discussions.

1. Getting started

In this chapter, we will install Eventline and get it ready to use. While there are multiple ways to deploy Eventline depending on your needs, we will use Docker Compose to run everything locally with minimum effort.

You will need a UNIX system supporting Docker Compose, for example Linux or Mac OS, with Docker and Docker Compose installed.

1.1. Eventline

Installation

First download the Docker Compose definition file, docker-compose.yaml, provided with each Eventline release. You can either get it on GitHub release page or download it directly from https://github.com/exograd/eventline/releases/latest/download/docker-compose.yaml.

Run Docker Compose in the directory you downloaded the file to:

docker-compose up -d

You now have Eventline running along its PostgreSQL database. It also includes MailCatcher so that you can send email notifications without having to configure access to an email server.

Start your favourite web browser and open http://localhost:8087. You should see the login page of Eventline.

First connection

The first time Eventline starts, it creates an account with username admin and password admin. Use these credentials to connect to the web interface.

Start by going to the "Account" section and click the "Change password" button. Choose a strong password to protect the admin account.

Caution

When installing Eventline, always change the default password immediately after starting the platform to prevent unauthorized access.

1.2. Evcli

Installation

Evcli is the command line tool used to interact with Eventline. You can also get it from the GitHub release page, where it is available as a binary for all supported platforms, or simply use our installation script:

curl -sSLf https://github.com/exograd/eventline/releases/latest/download/install-evcli | sh -

At this point, you should be able to run evcli version and see the version number of the last release.

Note	`install-evcli` put the `evcli` executable file in `/usr/local/bin`. Therefore you should make sure that the `PATH` environment variable contains this location.

Authentication

Eventline uses keys to authenticate clients on the API. While you could create an API key manually and add it to the Evcli configuration file, it is much simpler to use the login command.

Run evcli login; the command asks for the endpoint (prefilled as "http://localhost:8085", which will work for the current setup), username and password, then takes care of everything.

To make sure everything works as intended, run evcli list-projects. You should see the default project, main.

1.3. Writing your first job

Now that everything is running, let us write a job. Eventline jobs are small YAML documents describing when, where and how to run the job. Open a file named hello-world.yaml with your favourite text editor and copy the following content:

name: "hello-world"
trigger:
  event: "time/tick"
  parameters:
    periodic: 30
steps:
  - code: |
      echo "Hello world!"

We define a job and give it a name. We then describe when to run the job using a trigger: the job will react to the tick events from the time connector, and will be executed periodically, every 30 seconds. Finally we provide the code to run as a list of steps.

Save the file, and deploy the job using Evcli:

evcli deploy-job hello-world.yaml

This is it! The job is now saved in Eventline and will be executed every 30 seconds as requested. Open your web browser; on the "Jobs" page, you can see your new hello world job:

If you click on it, you will see the job execution page, which lists all executions of this job.

After a few seconds, Eventline should execute the job for the first time. You can see its status changes in real time. You can also click on this particular execution and see the status and output of each step:

You can now delete job: go to the "Jobs" page, click on the "…" button to the right of the "hello-world" list entry, and select "Delete". This will delete the job and all its executions.

Alternatively, you can use Evcli:

evcli delete-job hello-world

In the next chapters, we will explore more advanced features of Eventline.

2. Writing jobs

2.1. Definition files

Jobs are defined in YAML documents. Each file contains a single object, the job specification. When you deploy one of these files, Eventline uses the name field to either create a new job or update an existing one.

Note	While job filenames have no particular meaning to Eventline, using the name of the job as filename is helpful when organizing multiple job files.

2.2. Deployment

Jobs are deployed using Evcli. The deploy-job command deploys a single job file, while deploy-jobs can be used to deploy multiple jobs at the same time.

Both commands accept the --dry-run (or -n) option which validate job files without actually deploying them.

Tip	While you can write and deploy jobs manually, you will probably reach a point where you want to organize all your jobs in a central place with version control. In that case, you could use a job which calls evcli to deploy all jobs in the repository, and trigger it on new commits.

2.3. Export

Jobs can be exported out of Eventline at any time using Evcli and the export-job command. The job is exported as a job specification file which can then be edited and re-deployed.

2.4. Writing code

Jobs contain a list of steps. Each steps is executed as a small independent program. There are several ways to define each step, each one with its pros and cons.

Code blocks

Code blocks let you write code inline.

Example

name: "code-blocks"
steps:
  - code: |
      echo "Hello world!"
  - code: |
      #!/usr/bin/env ruby

      puts "Use the language of your choice."

The first code block does not start with a shebang line, so Eventline automatically inserts the default project header which contains the #!/bin/sh shebang line.

In the second block, we provide the shebang line ourselves, and write the code in Ruby.

Commands

Commands are a simple way to execute a program with or without arguments.

Example

name: "command"
steps:
  - command:
      name: "ls"
      arguments: ["-l", "/tmp"]

Here we simply execute the ls program with two arguments.

Scripts

Scripts give you the ability to use files written out of the job definition file. This is particularly nice when you have to write more than a couple lines of code, and want to edit the code properly instead of writing inline YAML content.

Example

name: "script"
steps:
  - script:
      path: "utils/generate-report.rb"
      arguments: ["--output", "/tmp/report.pdf"]

In this example, the actual code is written in the utils/generate-report.rb file. We instruct Eventline to use this code during execution with the provided arguments.

During deployment, Eventline reads script files and embeds their content in the job specification structure as if it had been written inline.

Even better, if the job is exported later, Evcli will recreate the original script file as you would expect.

2.5. Reference

Job specification

A job is an object containing the following fields:

name (string): The name of the job.
description (optional string): A textual description of the job.
trigger (optional object): The specification of a trigger indicating when to execute the job.
parameters (optional object array): A list of parameter specifications used to execute the job manually.
runner (optional object): The specification of the runner used to execute the job.
concurrent (optional boolean, default to false): Whether to allow concurrent executions for this job or not.
retention (optional integer): The number of days after which past executions of this job will be deleted. This value override the global job_retention setting.
identities (optional string array): The names of the identities to inject during job execution.
environment (optional object): A set of environment variables mapping names to values to be defined during job execution.
steps (object array): A list of steps which will be executed sequentially.

Trigger specification

A trigger is an object containing the following fields:

event (string): The event to react to formatted as <connector>/<event>.
parameters (optional object): The set of parameters associated to the event. Refer to the connector documentation to know which parameters are available for each event.
identity (optional string): The name of an identity to use for events which require authentication. For example the github/push event needs an identity to create the GitHub hook used to listen to push events.
filters (optional object array): A list of filters used to control whether an event matches the trigger or not.

Parameter specification

A parameter is an object containing the following fields:

name (string)

The name of the parameter.

description (optional string)

A textual description of the parameter.

type (string)

The type of the parameter. The following types are supported:

number: Either an integer or an IEEE 754 double precision floating point value.
integer: An integer.
string: A character string.
boolean: A boolean.

values (optional string array)

For parameters of type string, the list of valid values.

default (optional value)

The default value of the parameter. The type of the field must be compatible with the type of the parameter.

environment (optional string)

The name of an environment variable to be used to inject the value of this parameter during execution.

Filter specification

Each filter is an object made of a path and zero or more predicates. The path is a JSON pointer (see RFC 6901) applied to the data of the event.

Predicates are additional members which are applied to the value referenced by the path. An event matches a filter if all predicates are true.

The following predicates are supported:

is_equal_to (optional value): Matches if the value referenced by the path is equal to the value associated with the predicate.
is_not_equal_to (optional value): Matches if the value referenced by the path is different from the value associated with the predicate.
matches (optional value): The associated value is a regular expression; the predicate matches if the value referenced by the path is a string which matches this regular expression. Eventline supports the RE2 syntax.
does_not_match (optional value): The associated value is a regular expression; the predicate matches if the value referenced by the path is a string which does not match this regular expression.

Example

filters:
  - path: "/branch"
    matches: "^feature-"
  - path: "/repository"
    is_not_equal_to: "tests"

When applied to a github/push event, this filters will match push events on branches whose name starts with feature- but not if the repository is named tests.

Runner specification

A runner is an object containing the following fields:

name (string): The name of the runner.
parameters (optional object): The set of parameters associated to the runner. Refer to the runner documentation to know which parameters are available for each runner.
identity (optional string): The name of an identity to use for authentication purposes associated with the runner. Refer to the runner documentation to know which identities can be used for each runner.

Step specification

A step is an object containing the following fields:

label (optional string)

A short description of the step which will be displayed on the web interface.

code (optional string)

The fragment of code to execute for this step.

command (optional object)

The command to execute for this step. Contains the following members:

name (string): The name of the command.
arguments (optional string array): The list of arguments to pass to the command.

script (optional object)

An external script to execute for this step. Contains the following members:

path (string): The path of the script file relative to the job file.
arguments (optional string array): The list of arguments to pass to the script.

Each step must contain a single field among code, command and script indicating what will be executed.

3. Using identities

Identities provide a way to store credentials of various types. Each identity contains a set of typed values, making it possible to store secrets more complex than single passwords or keys

Connectors provide identities; for example, the github connector contains identities for the authentication mechanisms of the GitHub platform. Refer to the connector documentation for a list of all available identities.

Eventline encrypts identity data in the database using AES-256-CBC with the global encryption key.

3.1. Lifecycle

Identities have one of the following statuses:

pending: The identity has been created, but some parts of the initialization process have not completed.
ready: The identity has been created and initialized, and is ready to be used.
error: An error occurred during the initialization process of the identity.

Identities with statuses pending or error cannot be used in jobs.

Most identities are self-sufficient and will be ready as soon as they are created. Others such as OAuth2 identities require extra initialization steps to get ready. If these steps are not completed, for example if the web browser is closed during a redirection, the identity will stay with the pending status. If one of these steps fails, the identity will have the error status, and an error message will be displayed on the identity view page.

Editing an identity will always reset their state, and the platform will restart all initialization steps. If your identity is stuck with the pending or error state, simply edit it to restart the initialization process.

3.2. Refresh

Some identities must be refreshed on a regular basis. This is the case for OAuth2 identities which contain a refresh token: it must be used to regularly fetch an access token. Without the refresh process, the access token will expire.

When an OAuth2 provider supports refresh tokens, for example Slack, Eventline will regularly use them to obtain new access tokens and store them in the identity.

If the refresh process fails, Eventline will send an email notification. See the notification settings documentation for more information.

Users can also force a refresh at any moment using the "Refresh" button on the page of the identity.

4. Runners

Eventline has the ability to run jobs in different environments using runners. Each job can select the runner to use for execution and its parameters.

Caution

Running job in an environment means that sensible data can be stored and accessed during execution. Make sure to take into account users and programs who can access system when they are executing Eventline jobs.

The default runner is the most basic one, the local runner.

4.1. `local`

The local runner executes jobs directly on the machine where Eventline is hosted.

Its bigger advantage is how fast it is: since there is no intermediary execution layer, it lets you execute jobs in a couple milliseconds.

Using the local runner is also useful for jobs containing lower level tasks that could be difficult to run in a virtualized environment.

Example

runner:
  name: "local"

Note that the local runner being the default runner, jobs using it do not need to include a runner field.

Configuration

The local runner supports the following settings:

root_directory (optional string, default to /tmp/eventline/execution): The directory used to store temporary data during the execution of each job. The path must be absolute.

Parameters

There are no parameters for jobs using the local runner.

Identity

The local runner does not use any identity.

4.2. `docker`

The docker runner executes jobs in containers executed by the Docker daemon. All the steps in a job are executed sequentially in the same container.

Example

runner:
  name: "docker"
  parameters:
    image: "alpine:3.16"
    cpu_limit: 2.0
    memory_limit: 1000

Configuration

The docker runner supports the following settings:

uri (optional string)

The URI of the Docker HTTP endpoint. The scheme must be either unix or tcp. If it is not set, Eventline will let the Docker server pick the default URI (usually unix:///var/run/docker.sock).

ca_certificate_path (optional string)

The path of the CA certificate file to use to connect to the Docker daemon.

certificate_path (optional string)

The path of the certificate file to use to connect to the Docker daemon.

private_key_path (optional string)

The path of the private key file to use to connect to the Docker daemon.

mount_points (optional array)

A list of mount points; each mount point is an object containing the following fields:

source (string): The path of the file or directory on the host.
target (string): The path of the file or directory in the container.
read_only (boolean): Whether to mount the source in read-only mode or not.

Parameters

Jobs using the docker runner support the following parameters:

image (string): The reference of the image to use for the container.
cpu_limit (optional number): Set a limit to the number of virtual CPUs that can be used during execution.
memory_limit (optional integer): Set a limit to the amount of memory in megabytes that can be used during execution.

Identity

The following identities can be used with the runner:

dockerhub/password: A DockerHub username and password.
dockerhub/token: A DockerHub username and token.
github/oauth2: A GitHub username and OAuth2 access token for the ghcr.io image registry.
github/token: A GitHub username and personal access token for the ghcr.io image registry.

4.3. `ssh`

The ssh runner uses the Secure Shell Protocol to execute the job on a remote server.

Example

runner:
  name: "ssh"
  parameters:
    host: "my-server.example.com"
    user: "my-user"
  identity: "my-ssh-key"

OpenSSH server configuration

The OpenSSH server does not let clients set arbitrary environment variables. In order to be able to run Eventline jobs on a server, edit the SSH daemon configuration file at /etc/ssh/sshd_config and add the following setting:

AcceptEnv EVENTLINE EVENTLINE_*

If you want to use other environment variables in your jobs, you will have to add them to the AcceptEnv setting as well.

Do not forget to restart the SSH daemon.

Warning

If this setting is not added, jobs using the ssh runner will fail with the ssh: setenv failed error message.

Configuration

The ssh runner supports the following settings:

root_directory (optional string, default to /tmp/eventline/execution): The directory used to store temporary data during the execution of each job on the remote server. The path must be absolute.

Parameters

Jobs using the ssh runner support the following parameter:

host (string): The hostname or IP address of the server to connect to.
port (optional integer, default to 22): The port number to use.
user (optional string, default to root): The user to connect as.
host_key (optional string): The expected host key of the server as a public key encoded using Base64.
host_key_algorithm (optional string): The algorithm of the host key if host_key is provided. Must be one of ssh-dss, ssh-rsa, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, ecdsa-sha2-nistp521 and ssh-ed25519

Tip	You can obtain the host key of a remote server using `ssh-keyscan`.

Identity

The following identities can be used with the runner:

generic/password: Authenticate using the password in the identity. The login field is ignored.
generic/ssh_key: Authenticate using the private key in the identity.

4.4. `kubernetes`

The kubernetes runner executes jobs in a Kubernetes cluster. All the steps in a job are executed sequentially in the same container of the same pod.

Example

runner:
  name: "kubernetes"
  parameters:
    image: "alpine:3.16"
    namespace: "eventline"

Note	The `kubernetes` runner is only available in Eventline Pro.

Execution

Each job is executed in a new pod and container. Eventline sets the following pod labels:

eventline.net/project-id: The identifier of the project of the job.
eventline.net/job-name: The name of the job.
eventline.net/job-execution-id: The identifier of the job execution.

Execution data, including identities, are injected using a secret.

All Kubernetes resources are created with the eventline field manager. See the Kubernetes documentation for more information.

Configuration

The kubernetes runner supports the following settings:

config_path (optional string): The path of the kubeconfig file to use to connect to the cluster. If not set, Eventline will either use the value of the KUBECONFIG environment variable if it set or $HOME/.kube/config otherwise.
namespace (optional string, default to default): The namespace to create pods into.

Parameters

Jobs using the kubernetes runner support the following parameters:

image (string): The reference of the image to use for the container.
namespace (optional string): The namespace to create the pod into. If not set, the runner uses the namespace defined in the configuration.
labels (optional object): A set of name and values to be added to each created pod as labels. Values are strings.
cpu_request (optional number): Set the number of virtual CPUs requested for execution.
cpu_limit (optional number): Set a limit to the number of virtual CPUs that can be used during execution.
memory_request (optional integer): Set the amount of memory in megabytes requested for execution.
memory_limit (optional integer): Set a limit to the amount of memory in megabytes that can be used during execution.

See the Kubernetes documentation for more information regarding resource requests and limits.

Identity

The following identities can be used with the runner:

dockerhub/password: A DockerHub username and password.
dockerhub/token: A DockerHub username and token.
github/oauth2: A GitHub username and OAuth2 access token for the ghcr.io image registry.
github/token: A GitHub username and personal access token for the ghcr.io image registry.

5. Executing jobs

Jobs act as simple definitions. For their steps to be executed, they need to be instantiated. The instantiation process creates a job execution containing everything required for one run. The job execution is then stored, waiting for scheduling.

5.1. Triggers

Each job can contain a trigger which specifies when to execute it. The trigger defines an event, a set of parameters and optional additional filters. When the event occurs and filters match, Eventline instantiates the job.

If a job does not contain a trigger, it will never be instantiated automatically, but can still be executed manually.

Example

name: "stable-branch-commits"
trigger:
  event: "github/push"
  parameters:
    organization: "my-organization"
    repository: "my-product"
  filters:
    - path: "/branch"
      is_equal_to: "stable"
  identity: "github-oauth2"

See the trigger specification for a list of all trigger fields.

5.2. Events

Events represent something that happened and that was detected by Eventline. When an event is created, Eventline looks for matching triggers and instantiates associated jobs.

See the connector documentation for a list of all events in each connector.

Replay

Event replay is an advanced feature designed to help people re-execute jobs as if the original event was received again.

When an event is replayed, a new event is created with the same connector, event type and data. Eventline will look for matching triggers and execute any associated job.

This feature is particularly useful when writing complex jobs. Event replay makes it easy work on new definitions of your job. Update your job, deploy the project, and simply replay the original event to see what happens.

5.3. Manual execution

All jobs can be executed manually, either on the web interface or using the execute-job Evcli command.

On the web interface, Eventline dynamically generates a form to provide the parameters defined by the job.

A job execution instantiated manually does not have any associated event.

Caution

If you define a job to be executable with either a trigger or manual execution, you need to make sure that the code executed by the job handle the fact that there may or may not be an available event.

5.4. Lifecycle

Job executions have one of the following status:

created: The job execution has just been created and is waiting for scheduling.
started: The job execution is currently running.
aborted: The job execution was stopped before completion, usually by an action of a user.
successful: The job execution has completed successfully.
failed: The job execution has failed, either because one of the steps failed or due to an internal error.

Each step in the job execution has a status code with the same possible values, allowing users to track the execution process.

Users can affect this lifecycle by aborting or restarting jobs. Both actions can be done on the web interface, with Evcli or with the HTTP API.

5.5. Timeouts

Job executions are refreshed on a regular basis once they have been started. A separate worker monitors jobs and stops those which have not been refreshed for some time. This mechanism is useful in two situations:

If Eventline is killed, for example due to SIGKILL or a server failure, jobs which were running will ultimately be detected and stopped instead of staying in a started status forever.
If a runner is stuck due to an external system or because of an unexpected error, the job execution associated with it will ultimately be stopped.

This system is controlled by controlled by two settings:

job_execution_refresh_interval: the interval between two refresh in seconds.
job_execution_timeout: the time after which a non-refreshed job execution is considered dead in seconds.

See the configuration documentation for more information.

Abortion

Created or started job executions can be aborted. If execution has not started yet, it will be cancelled. If the job is running, Eventline will try to stop it. Steps which have not been executed yet will have status aborted.

Restart

Finished job executions can be restarted. When that happens, the job execution is reinitialized, ready for scheduling. Job executions are restarted using the same job specification, even if the definition of the job was changed in the mean time.

Note that the content of identities is not part of job definitions: if an identity is changed, execution after a restart will use the new set of identity data.

Caution

Eventline is a scheduling platform: jobs execute code which usually affect external systems. Restarting a job could have unexpected consequences regarding these systems. In general, writing jobs in an idempotent way will help a lot in keeping your technical processes robust.

5.6. Runtime environment

Filesystem

All runners create and populate a directory containing various files useful during execution.

context.json: The execution context.
parameters/: A directory containing a file for each parameter of the job if there are any. For example, if the job has a username parameter, its value is stored in parameters/username.
event/: A directory containing a file for each field in the event if there is one. For example, in a github/raw event, the delivery id is stored in event/delivery_id.
identities/<name>/: A set of directories, one for each identity listed in the job, containing a file for each field of the identity. For example, if a job uses a github/oauth2 identity named gh, the access token will be available in identities/gh/access_token.

Caution

The location of this directory depends on the runner due to technical limitations; always use the EVENTLINE_DIR to build paths.

Having event and identity fields available as simple text files makes it really simple to write jobs written as shell scripts. Using the previous example, reading the access token of the identity in shell in trivial:

access_token=$(cat $EVENTLINE_DIR/identities/gh/access_token)

Alternatively, in higher level languages, simply load context.json and access all data directly.

Execution context

The execution context is available in $EVENTLINE_DIR/context.json; the top-level object contains the following fields:

event (object): The event object if the job execution was instantiated in reaction to an event.
identities (object): The set of all identities listed in the job.
parameters (object): The set of job parameters.

Environment variables

Eventline injects several environment variables during the execution of each job:

EVENTLINE: Always set to true, indicating that the task is being executed in the Eventline platform.
EVENTLINE_PROJECT_ID: The identifier of the current project.
EVENTLINE_PROJECT_NAME: The name of the current project.
EVENTLINE_JOB_ID: The identifier of the current job.
EVENTLINE_JOB_NAME: The name of the current job.
EVENTLINE_DIR: The absolute path of the directory containing Eventline data, including the context file.

6. Connectors

6.1. `aws`

The aws connector provides identities for Amazon Web Services. These identities can also be used for AWS compatible services, for example all services using the S3 protocol.

Note	The `aws` connector is only available in Eventline Pro.

Identities

`access_key`

The aws/access_key identity is used to connect to a service using the AWS authentication system.

Data fields

access_key_id (string): The access key identifier.
secret_access_key (string): The secret access key.

Environment variables

AWS_ACCESS_KEY_ID: The access key identifier.
AWS_SECRET_ACCESS_KEY: The secret access key.

6.2. `dockerhub`

The dockerhub connector is used to provide identities related to the DockerHub platform.

Identities

`password`

The dockerhub/password identity is used for password authentication. It can be used for authentication in runners docker and kubernetes.

It contains the following fields:

username (string): The username of the DockerHub account.
password (string): The password of the DockerHub account.

`token`

The dockerhub/token identity is used for access token authentication. It can be used for authentication in runners docker and kubernetes.

Data fields

username (string): The username of the DockerHub account.
token (string): The access token.

6.3. `eventline`

The eventline connector is used to provide identities related to the Eventline platform itself.

It is currently only used for Eventline API keys.

Identities

`api_key`

The eventline/api_key identity is used to connect to the Eventline platform with Evcli.

Data fields

key (string): The API key.

Environment variables

EVENTLINE_API_KEY: The API key.

This environment variable is used by the Evcli command line tool.

6.4. `generic`

The generic connector is used to provide generic identities to store credentials for whom there are no dedicated connectors.

Identities

`api_key`

A generic API key.

Data fields

key (string): The API key.

`password`

A generic login and password pair, the login being optional.

Data fields

login (optional string): The login.
password (string): The password.

`gpg_key`

A GPG key.

Note	While GPG keys are technically PGP keys, the name "GPG" has become so widely used that using "PGP" would be confusing for most users.

Data fields

private_key (string): The private key in ASCII armor format.
public_key (optional string): The public key in ASCII armor format.
password (optional string): The password protecting the private key.

`ssh_key`

A SSH key.

Data fields

private_key (string): The private key in PEM format.
public_key (optional string): The public key in PEM format.
certificate (optional string): The certificate in PEM format.

Note that OpenSSH will fail to load a private key, public key or certificate file which does not end with a new line character (\n). Eventline will automatically add one at the end of each field of this identity if there is not already one present.

`oauth2`

The generic/oauth2 identity can be used for any platform which complies with OAuth2 specifications. If discovery is true, Eventline will use OAuth2 discovery (RFC 8414) to find other endpoints.

Data fields

issuer (string): The issuer URI.
discovery (optional boolean, default to false): Whether to enable OAuth2 discovery.
discovery_endpoint (optional string): The URI of the discovery endpoint. Note that this field will be ignored if discovery is false. The default value is the endpoint mandated by OAuth2 specifications.
authorization_endpoint (optional string): The URI of the authorization endpoint. The default value is the endpoint mandated by OAuth2 specifications or the endpoint found during discovery if discovery is true.
token_endpoint (optional string): The URI of the token endpoint. The default value is the endpoint mandated by OAuth2 specifications or the endpoint found during discovery if discovery is true.
client_id (string): The client identifier.
client_secret (string): The client secret.
scopes (string array): A comma-separated list of scopes to request.
access_token (optional string): The OAuth2 access token. Automatically handled by Eventline.
refresh_token (optional string): The OAuth2 refresh token. Automatically handled by Eventline.
expiration_time (optional string): The expiration date for the access token. Automatically handled by Eventline.

6.5. `github`

The github connector provides identities and events for the GitHub platform.

Identities

`oauth2`

The github/oauth2 identity contains a username and an OAuth2 access token.

During the creation of the identity, you will be redirected to the GitHub website to authorize the creation of a new access token.

This identity can be used for authentication in runners docker and kubernetes.

Data fields

username (string): The name of the GitHub account.
scopes (string array): The list of OAuth2 scopes.
access_token (optional string): The OAuth2 access token. Automatically handled by Eventline.
refresh_token (optional string): The OAuth2 refresh token. Automatically handled by Eventline.
expiration_time (optional string): The expiration date for the access token. Automatically handled by Eventline.

Environment variables

GITHUB_USER: The name of the GitHub account.
GITHUB_TOKEN: The GitHub access token.

These environment variables are used by the official GitHub command line tool among others.

`token`

The github/token identity is used to store GitHub personal access tokens.

Note	`github/token` identities used in triggers require the `admin:repo_hook` and `admin:org_hook` scopes. Using them as runner identities (e.g. for the `docker` runner) will also require the `read:packages` scope.

You may need to add additional permissions if you intend to use this identity in jobs for other purposes, for example to interact with the GitHub API.

This identity can be used for authentication in runners docker and kubernetes.

Data fields

username (string): The name of the GitHub account.
token (string): The GitHub private access token.

Environment variables

GITHUB_USER: The name of the GitHub account.
GITHUB_TOKEN: The GitHub private access token.

These environment variables are used by the official GitHub command line tool among others.

Subscription parameters

organization (string): The name of the GitHub organization.
repository (optional string): The name of the repository.

Setting only organization will subscribe to events for all repositories in the origanization, while settings both fields will subscribe to events for a single repository.

Events

`raw`

The github/raw event is used to access raw data from GitHub events and is emitted for every single organization or repository event. It can be used when other events do not contain the required information.

Data fields

delivery_id (string): The delivery id of the event. See the GitHub documentation for more information.
event_type (string): The type of the event.
event (object): The raw event payload delivered by GitHub.

Warning

Subscribing to github/raw events will potentially result in lots of events created and lots of jobs executed. Make sure you actually need this kind of low level access to event data.

`repository_creation`

The github/repository_creation event is emitted when a repository is created in an organization.

Data fields

organization (string): The name of the GitHub organization.
repository (string): The name of the repository.

`repository_deletion`

The github/repository_deletion event is emitted when a repository is deleted in an organization.

Data fields

organization (string): The name of the GitHub organization.
repository (string): The name of the repository.

`tag_creation`

The github/tag_creation event is emitted when a tag is created in a repository.

Note	Due to a GitHub limitation, this event will not be emitted if more than three tags are pushed at the same time.

Data fields

organization (string): The name of the GitHub organization.
repository (string): The name of the repository.
tag (string): The name of the tag.
revision (string): The hash of the revision associated with the tag.

`tag_deletion`

The github/tag_deletion event is emitted when a tag is deleted in a repository.

Data fields

organization (string): The name of the GitHub organization.
repository (string): The name of the repository.
tag (string): The name of the tag.
revision (string): The hash of the revision associated with the tag.

`branch_creation`

The github/tag_created event is emitted when a branch is created in a repository.

Data fields

organization (string): The name of the GitHub organization.
repository (string): The name of the repository.
branch (string): The name of the branch.
revision (string): The hash of the revision the branch starts from.

`branch_deletion`

The github/branch_deleted event is emitted when a branch is deleted in a repository.

Data fields

organization (string): The name of the GitHub organization.
repository (string): The name of the repository.
branch (string): The name of the branch.
revision (string): The hash of the revision the branch pointed to when the deletion occurred.

`push`

The github/push event is emitted when one or more commits are pushed in a repository.

Data fields

organization (string): The name of the GitHub organization.
repository (string): The name of the repository.
branch (string): The branch where commits were pushed.
old_revision (optional string): The hash of the revision the branch pointed to before the push. This field is not set for the first push in a repository.
new_revision (string): The hash of the revision the branch pointed to after the push.

Examples

Commits on the stable branch

name: "stable-branch-commits"
trigger:
  event: "github/push"
  parameters:
    organization: "my-organization"
    repository: "my-product"
  filters:
    - path: "/branch"
      is_equal_to: "stable"
  identity: "github-oauth2"

New demo branches in the organization

name: "new-demo-branches"
trigger:
  event: "github/branch_creation"
  parameters:
    organization: "my-organization"
  filters:
    - path: "/branch"
      matches: "^demo-"
  identity: "github-oauth2"

6.6. `postgresql`

The postgresql connector provides identities for the PostgreSQL database.

Identities

`password`

The postgresql/password identity is used to connect to a PostgreSQL database.

Data fields

user (string): The PostgreSQL user.
password (string): The PostgreSQL password.

Environment variables

PGUSER: The PostgreSQL user.
PGPASSWORD: The PostgreSQL password.

6.7. `slack`

The slack connector provides identities for the Slack platform.

Note	The `slack` connector is only available in Eventline Pro.

Identities

`oauth2_bot`

The slack/oauth2_bot identity is used to store Slack bot tokens. It is used to interact with the Slack API independently of any user account.

Data fields

scopes (string array): The list of OAuth2 scopes.
access_token (optional string): The OAuth2 access token. Automatically handled by Eventline.
refresh_token (optional string): The OAuth2 refresh token. Automatically handled by Eventline.
expiration_time (optional string): The expiration date for the access token. Automatically handled by Eventline.
incoming_webhook_channel (string): The name of the channel associated with the incoming webhook if the incoming-webhook scope was selected.
incoming_webhook_uri (string): The URI of the incoming webhook if the incoming-webhook scope was selected.

`oauth2_user`

The slack/oauth2_user identity is used to store Slack user tokens. It is used to interact with the Slack API on behalf of a specific user account.

scopes (string array): The list of OAuth2 scopes.
access_token (optional string): The OAuth2 access token. Automatically handled by Eventline.
refresh_token (optional string): The OAuth2 refresh token. Automatically handled by Eventline.
expiration_time (optional string): The expiration date for the access token. Automatically handled by Eventline.
incoming_webhook_channel (string): The name of the channel associated with the incoming webhook if the incoming-webhook scope was selected.
incoming_webhook_uri (string): The URI of the incoming webhook if the incoming-webhook scope was selected.

6.8. `time`

The time connector provides a way to execute jobs using various kind of timers.

Subscription parameters

Parameters must include one of the following fields:

oneshot (string): The trigger will be activated one single time. The value is a datetime string.
periodic (integer): The trigger will be activated at a fixed time interval. The value is the number of seconds between two activations.
hourly (object): The trigger will be activated every hour. The value is an object containing two optional fields minute and second indicating the precise activation time.
daily (object): The trigger will be activated every day. The value is an object containing three optional fields hour, minute, and second indicating the precise activation time.
weekly (object): The trigger will be activated every week. The value is an object containing four fields day, hour, minute, and second indicating the precise activation time. The mandatory day field is a string indicating the day of the week between monday and sunday. Other fields are optional.

Events

`tick`

The time/tick event is emitted when a timer goes off.

Job triggered by time/tick events are instantiated when the timer goes off.

For non-periodic triggers, Eventline will instantiate job executions for past, missed datetimes. For example, if Eventline is down for a whole day and a job is supposed to be run every hour, it will instantiate a job execution for every hour as soon as it goes up. These executions will be queued for scheduling and executed sequentially or concurrently depending on the job definition.

Tip	In `time/tick` events, the `event_time` field is the date the timer was supposed to go off, while the `creation_time` is the date the event was actually created. You may need both values depending on the work you are doing.

Examples

Periodic timer

name: "every-5-minutes"
trigger:
  event: "time/tick"
  parameters:
    periodic: 300

Hourly timer

name: "every-hour"
trigger:
  event: "time/tick"

Daily timer

name: "every-day-at-7am"
trigger:
  event: "time/tick"
  parameters:
    daily:
      hour: 7

Weekly timer

name: "every-friday-at-18h30"
trigger:
  event: "time/tick"
  parameters:
    weekly:
      day: "friday"
      hour: 18
      minute: 30

7. Organization with projects

It is easy to create lots of jobs and identities and end up having trouble finding what you need. To help with organization, Eventline provides a namespace mechanism, projects.

A project contains jobs and identities and keep them separate from other projects. When you are using the web interface, Evcli or the HTTP API, you only interact with a single project.

7.1. Project selection

Web interface

On the web interface, you can change the current project by clicking on the project name on the top right part of the page to access to project selector:

Evcli

With Evcli, the -p, --project and --project-id options can be used to select the project you want to interact with. Refer to the Evcli documentation for more information.

7.2. Configuration

You can configure a project by clicking on the gear icon on the top right of the web interface.

The following settings are available:

Code header

A fragment of code inserted at the beginning of each code block of each step in all jobs in the project unless the block already contains a shebang line. The default value is:

#!/bin/sh

set -eu

7.3. Notifications settings

Each project can specify when notifications are sent.

Eventline send notifications in the following situations:

When jobs succeed.
When jobs succeed for the first time after one or more failures or abortions.
When jobs fail.
When jobs are aborted.
When OAuth2 identities cannot be refreshed.

You can also provide a list of email addresses to send notifications to.

Tip	If you want to send notifications to several people, e.g. an entire team, it is advised to create a user group in the software managing emails in your organization. You can then use the group address as recipient for notifications.

8. Managing your account

8.1. Configuration

The account configuration page lets you update several settings:

Date format: The format used to display dates on the web interface, either absolute (e.g. "2022-07-15T17:11:50Z") or relative (e.g. "3 hours ago").
Default page size: The number of elements displayed on a page listing multiple elements, for example the event list page.

8.2. API keys

Each account can define multiple API keys. Keys are used to access the HTTP API, either using Evcli or with a client of your choice.

Caution

API key give access to your account and must be treated as sensitive data.

Eventline only stores a cryptographic hash (SHA256) of each API key. When you create a new API key, you must save its value: you will not be able to recover it later.

API keys can be deleted at any moment to revoke access to Eventline.

9. Deployment and configuration

9.1. Requirements

Eventline is written in Go and is distributed as a statically linked executable for various platforms.

The following platforms are officially supported for production:

Linux x86_64.
FreeBSD x86_64.

Binaries are also released on a best effort basis for the following platforms:

Mac OS x86_64.
Mac OS arm64 (M1).

Eventline uses a PostgreSQL database version 14 or higher; the pgcrypto extension must be installed. It does not require local filesystem storage.

Eventline can also send metrics to an InfluxDB server.

9.2. Installation

Binary tarballs

Eventline is released primarily as a set of binary tarballs. Each tarball contains everything required to run the program:

Both the eventline and the evcli executables.
The Eventline license.
The data directory containing web assets, database migrations and template files.
This handbook as a HTML file with associated images and javascript code.

While you can run Eventline directly from the tarball directory, you will want something more practical for production. As such, binary tarballs are generally only used by people building packages or images themselves.

Stable and experimental builds

Stable builds are derived from Git revisions tagged as v.<major>.<minor>.<patch>. Experimental builds are derived from any other Git revisions.

Experimental packages are made available for various platforms on a regular basis. Their goal is to allow users to try new features before they are released in a stable build.

Caution

Experimental builds are inherently less reliable than stable builds and must not be used in production.

FreeBSD package

FreeBSD packages are available in the Exograd pkg repository. In order to install them, you first need to add the repository as a pkg source:

curl -sSfL -o /usr/local/etc/pkg/repos/exograd-public.conf \
     https://pkg.exograd.com/public/freebsd/exograd.conf

Then you can update the package index and install Eventline:

pkg update
pkg install eventline

The repository provides both eventline and eventline-experimental.

Ubuntu package

Ubuntu packages are available in the Exograd APT repository. In order to install them, you first need to add the repository as a source for APT:

curl -sSfL -o /etc/apt/sources.list.d/exograd-public.list \
     https://pkg.exograd.com/public/ubuntu/exograd.list

Note	The package repository is only available over HTTPS for security. You will need to install the `ca-certificates` packages.

Then you can update the package index and install Eventline:

apt-get update
apt-get install eventline

The repository provides both eventline and eventline-experimental.

Docker

If you intend to use the docker runner, you will need to install Docker. We recommend following the official Docker documentation.

You will also have to add the eventline user to the docker group to give it access to the Docker socket:

usermod -a -G docker eventline

Archlinux package

Coming soon.

Docker image

The exograd/eventline image is available on DockerHub. Versioning is the same as Eventline: for example, Eventline 0.9.0 is available with exograd/eventline:0.9.0.

While the latest tag usually points to the latest available release, it is strongly recommended not to use it and to always use a specific version tag to avoid unexpected updates.

License management

Eventline Pro requires a license allowing a client to use the product.

Each client is attributed credentials provided as two values, name and secret. These values must be stored in the configuration file. See the configuration specification for more information.

Eventline Pro automatically uses these credentials to authenticate the Exograd license server and download a license file.

The license is renewed automatically at startup, and on a regular basis afterward.

Configuration

When running in this Docker image, Eventline can be configured two ways.

One way is to mount a configuration file of your choice in the container at /etc/eventline/eventline.yaml. This gives you maximum flexibility.

An over way is to rely on the embedded configuration file which uses environment variables for configuration. The following variables are available:

EVENTLINE_WEB_HTTP_SERVER_URI (optional, default to localhost:8087): The value to use for the web_http_server_uri setting.
EVENTLINE_PG_URI (optional): The URI of the PostgreSQL server. The default value is postgres://eventline:eventline@localhost:5432/eventline.
EVENTLINE_ENCRYPTION_KEY: The value to use for the encryption_key setting.
EVENTLINE_CONNECTORS_GITHUB_WEBHOOK_SECRET: The value of the webhook_secret setting for the github connector. Setting this variable automatically enable the connector.
EVENTLINE_MAX_PARALLEL_JOB_EXECUTIONS: The value to use for the max_parallel_job_executions setting.
EVENTLINE_JOB_EXECUTION_RETENTION: The value to use for the job_execution_retention setting.
EVENTLINE_SESSION_RETENTION: The value to use for the session_retention setting.
EVENTLINE_NOTIFICATIONS_SMTP_SERVER_ADDRESS (optional, default to localhost:25): The address of the SMTP server to use for notifications.
EVENTLINE_NOTIFICATIONS_SMPT_SERVER_USERNAME (optional): The username to use to authenticate to the SMTP server.
EVENTLINE_NOTIFICATIONS_SMPT_SERVER_PASSWORD (optional): The password to use to authenticate to the SMTP server.
EVENTLINE_NOTIFICATIONS_FROM_ADDRESS (optional): The email address to use in the From header field.
EVENTLINE_NOTIFICATIONS_SUBJECT_PREFIX (optional): A character string to use as prefix for the Subject header field.
EVENTLINE_NOTIFICATIONS_SIGNATURE: A character string to insert as signature at the end of all emails.

See the configuration specification for more information about settings.

Building your own image

The exograd/eventline is provided for convenience. For production use, it is advised to build your own image. This allows you to follow the conventions used by your organization and gives you total control on the base system, environment and configuration.

Feel free to start from the default Dockerfile or to write your own from scratch.

Helm chart

Coming soon.

9.3. Configuration

Configuration file

Eventline uses a configuration file whose path is provided with the -c command line option.

Caution

The configuration file contains the global encryption key used to secure storage of sensitive information in the database. You must make sure that the UNIX user executing Eventline is the only user able to read the configuration file. Alternatively, you can use templating and environment variables to provide sensitive settings.

Templating

The configuration file is treated as a template using the Go template format. Templating currently supports the following functions:

env <name>: Return the value of the <name> environment variable.

Example:

data_directory: "/usr/share/eventline"

encryption_key: {{ env "EVENTLINE_ENCRYPTION_KEY" }}

pg:
  uri: {{ env "EVENTLINE_PG_URI" }}

At startup, Eventline loads the configuration file, renders it and then parses it as a YAML document.

Specification

A configuration file is an object containing the following fields:

logger (optional object)

The configuration of the logger used to print information and errors. The default value is:

backend_type: "terminal"
terminal_backend:
  color: true
  domain_width: 32

data_directory (optional string, default to data)

The path of the directory containing Eventline data files.

api_http_server (optional object)

the HTTP server configuration of the API interface. The default value is:

address: "localhost:8085"

web_http_server (optional object)

the HTTP server configuration of the web interface. The default value is:

address: "localhost:8087"

pg (optional object)

The configuration of the PostgreSQL server.

encryption_key (string)

The global encryption key used to encrypt sensitive information in the database. The key must be a 32 byte AES key encoded using Base64. You can generate a key using OpenSSL:

openssl rand -base64 32

web_http_server_uri (optional string, default to http://localhost:8087)

The URI which can be used to access the Eventline web interface from outside of the server. This URI will be used to generate webhook URIs among other thing.

insecure_http_cookies (optional boolean, default to false)

If true, do not set the secure attribute for HTTP cookies sent by the web HTTP server. This allows Eventline to be used over HTTP.

Caution

Using Eventline without HTTPS is fundamentally insecure: anyone able to inspect network traffic can obtain critical information transferred between the web browser and Eventline. Do not do it.

connectors (optional object)

The configuration of each connector. Refer to the connector documentation for the settings available for each connector.

max_parallel_job_executions (optional integer)

If set, the maximum number of jobs which can run in parallel for the entire platform.

job_execution_retention (optional integer)

If set, a number of days after which old job executions will be deleted. Note that changing this setting will not affect job executions which have already been terminated.

job_execution_refresh_interval (optional integer, default: 10)

The number of seconds between two job execution refresh. See execution documentation for more information on the refresh process.

job_execution_timeout (optional integer, default: 120)

The number of seconds without refresh after which a job is considered abandonned. See execution documentation for more information on the refresh process.

session_retention (optional integer)

If set, a number of days after which sessions will be deleted.

allowed_runners (optional string array)

If set, a list of the runners which can be used in submitted jobs. Jobs using other runners will be rejected during deployment.

runners (optional object)

The configuration of each runner. Refer to the runner documentation for the settings available for each runner.

notifications (optional object)

The configuration of the email notification system. The default value is:

smtp_server:
  address: "localhost:25"
from_address: "no-reply@localhost"
subject_prefix: "[eventline] "
signature: "This email is a notification sent by the Eventline job scheduling software."

pro (optional object)

Configuration specific to Eventline Pro. Ignored for the open source version.

HTTP server specification

The configuration of a HTTP server is an object containing the following fields:

address (optional string, default to localhost:8080)

The address to listen on as a <host>:<port> string.

tls (optional object)

If set, use TLS for the connection. The object contains the following fields:

certificate (string): The path of the TLS certificate file.
private_key (string): The path of the TLS private key.

PostgreSQL specification

The configuration of the PostgreSQL server is an object containing the following fields:

uri (optional string): The URI of the PostgreSQL server. The default value is postgres://eventline:eventline@localhost:5432/eventline.

Notifications specification

The configuration for the notification system is an object containing the following fields:

smtp_server (optional object)

The configuration of the SMTP server to use when sending emails.

from_address (optional string)

The email address to use in the From header field.

subject_prefix (optional string)

A character string to use as prefix for the Subject header field.

signature (optional string)

A character string to insert as signature at the end of all emails.

allowed_domains (optional string array)

A list of domains which can used in notification email addresses. If the array is empty, all domains are allowed.

Tip	the `allowed_domains` setting is useful to make sure you that all notifications are sent to email addresses you control: this way you can limit the chances of losing notifications due to a mistake, and you guarantee that you will not send emails to someone out of your organization.

SMTP server specification

The configuration of the SMTP server is an object containing the following fields:

address (optional string, default to localhost:25: The address of the server using the <host>:<port> format.
username (optional string): The username to use for authentication.
password (optional string): The password to use for authentication.

Eventline Pro specification

The Eventline Pro configuration is an object containing the following fields:

license_path (string): The path of the license file.
license_name (string): The name of the license, used to authenticate against the license server.
license_secret (string): The secret value used to authenticate against the license server.

See documentation on license management for more information.

10. Evcli

10.1. Installation

Evcli is distributed both in binary tarballs along with the Eventline binary, and as a stand-alone executable file. Each release on GitHub contains binaries for multiple platforms which can be downloaded and used without any specific installation procedure.

Alternatively, GitHub releases contain an install-evcli executable script which detects the operating system and architecture of the machine, downloads the right Evcli executable and installs it.

While we try to keep Evcli both backward and forward compatible with different versions of Eventline, it is advised to use the same version for both Evcli and Eventline.

10.2. Configuration

The configuration file used by Evcli is stored in $HOME/.evcli/config.json. You can either write it yourself, or let Evcli manage it.

Caution

The configuration file contains the API key used to connect to the Eventline API. You must therefore make sure that other users on the same machine cannot read it by settings file permissions to 0600.

10.3. Usage

Evcli supports multiple commands and options:

evcli GLOBAL-OPTIONS <command> COMMAND-OPTIONS

You can print a list of all commands with evcli -h.

10.4. Global options

The following options can be used for all commands:

--debug <level>: Print all debug messages whose level is higher or equal to <level>.
-h, --help: Print help about Evcli.
--no-color: Disable the use of colors when printing data.
-p <name>, --project <name>: Select the current project by name.
--project-id <id>: Select the current project by identifier.
-q, --quiet: Do not print status or information messages.
-y, --yes: Skip all confirmation and automatically approve all questions.

10.5. Commands

`abort-job-execution`

Abort a specific job execution. Execution is cancelled if it has not started, and interrupted if it has.

`create-project`

Create a new project.

`create-identity`

Create a new identity. The command cannot be used to create identities which rely on a web browser for initialization, for example OAuth2 identities.

`delete-job`

Delete a job. All past job executions will also be deleted.

`delete-identity`

Delete an identity.

`delete-project`

Delete a project and everything it contains.

`deploy-job`

Deploy a single job file. The --dry-run command option can be used to validate the job instead of deploying it.

`deploy-jobs`

Deploy one or more job files or directories. The --dry-run command option can be used to validate the jobs instead of deploying them.

For each argument passed to the command:

If it is a file, it is assumed to be a job specification file and loaded directly.
If it is a directory, Evcli loads all files whose extension is .yml or .yaml stored inside. If the --recursive option is set, Evcli will recurse into nested directories.

The operation is atomic: either all jobs are deployed, or none are.

`describe-job`

Print information about a job.

`execute-job`

Execute a job. The name of the job is passed as first arguments. Additional arguments are used to set parameter values. Each parameter value is passed as a <name>=<value> argument.

Example

evcli execute-job create-env branch=experimental public=true

If the --wait option is passed, Evcli will monitor execution and wait for it to finish before exiting.

If both the --wait and --fail options are passed, Evcli with exit with status 1 if execution fails.

`export-job`

Export a job to a file. The file is written to the current directory by default. The --directory command option can be used to write to another path.

`get-config`

Obtain the value from the configuration file and print it.

Example

evcli get-config api.endpoint

`help`

When called without argument, print help about Evcli. When called with the name of a command as argument, print help about this command.

`list-jobs`

Print a list of all jobs in the current project.

`list-identities`

List all identities in the current project.

`list-projects`

Print a list of all projects.

`login`

Prompt for an endpoint, login and password, connects to Eventline and create an API key. The key is then stored in the Evcli configuration file.

This command is the fastest way to start using Evcli.

`rename-job`

Rename a job.

If the --description command option is used, also update its description.

Note	Renaming a job will affect its specification: if the job is deployed from a job specification file, you will have to update it manually.

`replay-event`

Replay an event as if it has just been created for the first time. Any job whose trigger matches the event will be instantiated.

`restart-job-execution`

Restart a specific job execution.

`set-config`

Set the value of an entry in the configuration file.

Example

evcli set-config interface.color false

`show-config`

Print the current configuration file as a JSON object.

If the --entries command option is used, print the list of configuration entries as a table instead.

`update`

Update Evcli by downloading a pre-built binary from the last available GitHub release.

If the --build-id command option is used, download a specific version instead.

Note	If Evcli is installed in a location which is not writable by the user such as the path used by `install-evcli` (`/usr/local/bin`), the `update` command must be executed with the appropriate permissions, for example using `sudo`.

`update-identity`

Update an existing identity.

`version`

Print the version of the Evcli program.

11. HTTP API

The Eventline HTTP API lets users access the various features of the platform in a programmatic way.

11.1. Interface

Endpoint

The API is available on a separate endpoint. The HTTP interface is configured by the api_http_server setting. See the configuration documentation for more information. The default port is 8085.

Authentication

Access to the API requires a valid API key. Authentication is based on the HTTP Authorization header field with the Bearer scheme.

For example, for the fictional API key 082ef11a-908f-4fae-819c-eb98b7fd98f6, requests must include the following header field:

Authorization: Bearer 082ef11a-908f-4fae-819c-eb98b7fd98f6

Project selection

Most API routes operate on a specific project. This project must be identified in each request by sending its identifier using the X-Eventline-Project-Id header field.

For example, for a fictional project whose identifier is 23B1NoaoigGQfmiP9PB9X2nJh4W, requests must include the following header field:

X-Eventline-Project-Id: 23B1NoaoigGQfmiP9PB9X2nJh4W

When an API route does not depend on a project, the X-Eventline-Project-Id can be omitted from the request.

Tip	In order to obtain the identifier of a project using its name, you can use the `GET /projects/name/{name}` route to fetch the project by name and read the `id` field.

Error handling

The Eventline API uses conventional HTTP status codes to indicate success or failure. In general, 2xx status codes indicate success, 4xx status codes indicate an error caused by the request, and 5xx status codes indicate an error in the Eventline platform.

Error responses sent by Eventline servers will contain a body representing an error object.

Caution

It is possible to receive error responses with a body which is not encoded in JSON, for example for errors coming from a load balancer or reverse proxy. Clients should use the Content-Type header field to determine the format of the body. Errors originating from Eventline API servers will always have the application/json content type.

Pagination

Various API routes return collections of elements. Most of these routes use pagination to group elements.

Paginated routes return a single object representing the page, i.e. the required subset of the collection of elements.

Cursors

Pagination is controlled by cursors. A cursor contains the parameters controlling the selection of elements to be returned and their order.

Each cursor contains the following parameters:

before (optional string): An opaque key; return elements positioned before the element designated by the key in the defined order.
after (optional string): An opaque key; return elements positioned after the element designated by the key in the defined order.
size (optional integer, default to 20): The number of elements to return, between 0 and 100.
sort (optional string): The sort to apply to elements. Different types of elements support different sorts; all elements support the id sort. The default sort depends on the type of the element.
order (optional string, default to asc): The order to use for elements, either asc for ascending order or desc for descending order.

Cursors must include one and only one of the before and after parameters.

Requests

When sending requests to fetch elements, the cursor is passed using HTTP query parameters. The before and after parameters must be Base64-encoded (see RFC 4648).

For example, sending a request to /jobs?after=TWpOQ2VFTlZOVVJvT1RCQ00xaFdSalJWY210WGNtNXFaRU5Q&size=5&order=id&order=desc will result in a response containing up to 5 jobs at position MjNCeENVNURoOTBCM1hWRjRVcmtXcm5qZENP in the whole list of pipelines ordered by id.

All paginated requests may result in pages containing less elements that the number required using the size parameter.

Responses

When cursors are sent in responses, for example to indicate the previous or next page, they are represented as JSON objects.

For example:

{
  "after": "MjNCeENVNURoOTBCM1hWRjRVcmtXcm5qZENP",
  "size": 5,
  "order": "id",
  "order": "desc"
}

The response to a paginated query is a single page object.

11.2. Data

Data format

Unless mentioned otherwise, requests and responses contain data formatted using JSON as described by RFC 8259.

Note that as mandated by RFC 8259, JSON documents must be encoded used UTF-8 and must not start with a BOM (Byte Order Mark).

Errors

Error objects contain the following fields:

error (string): A human-readable description of the error.
code (string): An error code identifying the precise reason which caused the error.
data (optional value): A value, usually an object, containing additional data related to the error.

Example

{
  "error": "route not found",
  "code": "route_not_found",
  "data": {
    "target": "\/foo\/bar"
  }
}

Identifiers

Most objects in Eventline are referenced by their unique identifier. Identifiers are KSUID. They are represented in JSON by strings containing the textual representation of the KSUID, e.g. "23BUGaDgMUxucZnXMzyUEGRcujl".

Names

Names are used as human-readable identifiers for various elements.

Names must only contain lower case alphanumeric characters, dash characters ('-') or underscore characters ('_'); they must also start an alphanumeric character. For example, "build-archive" and "run\_http\_test\_42" are valid names; "-register" and "Send Notifications" are not. Additionally, names must contain at least one character, and must not contain more than 100 characters.

Dates

Dates are represented as JSON strings containing the RFC 3339 representation of the date and time.

For example, "2021-10-15T15:07:39Z" is a valid date.

Eventline always represents dates using Coordinated Universal Time (UTC).

Pages

The response to a paginated query is a page, represented by a JSON object containing the following fields:

elements (array): The list of elements contained in the page.
previous (optional object): The cursor corresponding to the previous page if there is one.
next (optional object): The cursor corresponding to the next page if there is one.

Example

{
  "elements": [
    {
      "id": "22bQgVViNfTUAaToCrk9fMXPGsj",
      "name": "project-1",
    },
    {
      "id": "23E0dGLAnH943qkNMvMEZeFsWya",
      "name": "project-2",
    }
  ],
  "next": {
    "after": "MjNFMGRHTEFuSDk0M3FrTk12TUVaZUZzV3lh",
    "size": 2,
    "sort": "name",
    "order": "asc"
  }
}

Accounts

Accounts are represented as JSON objects containing the following fields:

id (identifier): The identifier of the account.
creation_time (date): The date the account was created.
username (string): The name of the user owning the account. The username must contain at least 3 characters and no more than 100 characters.
role (string): The role of the account, either user or admin.
last_login_time (optional date): The date of the last time someone used this account to access Eventline.
last_project_id (optional identifier): The identifier of the last project selected as current project for this account.
settings (object): An object containing settings used by the account.

Projects

Projets are represented as JSON objects containing the following fields:

id (identifier): The identifier of the project.
name (name): The name of the project.

Jobs

Jobs are represented as JSON objects containing the following fields:

id (identifier): The identifier of the job.
project_id (identifier): The identifier of the project the job is part of.
creation_time (date): The date the job was created.
update_time (date): The date the job was last modified.
disabled (optional boolean, default to false): Whether the job is disabled or not.
spec (object): The specification of the job. See the job documentation for more information.

Job executions

Job executions are represented as JSON objects containing the following fields:

id (identifier): The identifier of the execution.
project_id (identifier): The identifier of the project the execution is part of.
job_id (identifier): The identifier of the job.
job_spec (object): A copy of the specification of the job.
event_id (optional identifier): If the job execution was triggered by an event, the identifier of the event.
parameters (optional object): If the job was executed without any event, the set of parameters used.
creation_time (date): The date the execution was created.
update_time (date): The date the execution was last modified.
scheduled_time (date): The date execution is supposed to start.
status (string): The current status of the execution, either created, started, aborted, successful or failed. See the job execution documentation for more information.
start_time (optional date): The date execution started.
end_time (optional date): The date execution ended.
refresh_time (optional date): The date execution was last refreshed. See the job execution documentation for more information.
expiration_time (optional date): The date the job execution will be deleted for being too old.
failure_message (optional string): If execution failed, the last error message encountered.

Events

Events are represented as JSON objects containing the following fields:

id (identifier): The identifier of the event.
project_id (identifier): The identifier of the project the event is part of.
job_id (identifier): The identifier of the job at the origin of the event.
creation_time (date): The date the event was created.
event_time (date): The date the event actually happened.
connector (string): The name of the connector.
name (string): The name of the event.
data (object): The set of data contained by the event. Depends on the connector and name. See the documentation of the event in the connector section for more information.
processed (optional boolean, default to false): Whether the event was processed for job instantiation or not.
original_event_id (optional identifier): If the event is associated with a replayed event, the identifier of the original event.

Example

{
  "id": "22gBNze4y3o57HpES4WC8MwKvwo",
  "project_id": "1zY1y6offsPNwvhFxgpteVO0GvM",
  "job_id": "22g5sGMNkKvAtoH1BBMP2aDtUGb",
  "creation_time": "2021-12-23T08:48:33Z",
  "event_time": "2021-12-23T08:48:32Z",
  "connector": "github",
  "name": "push",
  "data": {
    "branch": "test",
    "new_revision": "f1d2d2f924e986ac86fdf7b36c94bcdf32beec15",
    "old_revision": "62cdb7020ff920e5aa642c3d4066950dd1f01f4d",
    "organization": "example",
    "repository": "my-project"
  }
}

Identities

Identities are represented as JSON objects containing the following fields:

id (identifier): The identifier of the identity.
project_id (identifier): The identifier of the project the identity is part of.
name (name): The name of the identity.
status (string): The current status of the identity, either pending, ready or error.
error_message (optinoal string): If the identity has status error, a description of the error.
creation_time (date): The date the identity was created.
update_time (date): The date the identity was created.
last_use_time (optional date): The last time the identity was used.
refresh_time (optional date): The last time the identity was refreshed for identities which must be refreshed regularly.
connector (string): The name of the connector.
type (string): The type of the identity.
data (object): The set of data contained by the identity. Depends on the connector and type. See the documentation of the identity in the connector section for more information.

Example

{
  "id": "2X7hQLbWL5iQa6z17pMnXFlbqtP",
  "project_id": "1zY1y6offsPNwvhFxgpteVO0GvM",
  "name": "bob-account",
  "status": "ready",
  "creation_time": "2023-10-22T16:47:04Z",
  "connector": "generic",
  "type": "password",
  "data": {
    "login": "bob",
    "password": "67d4155eb1840d15f2bb67a481d88ee5d11770081ac2a810"
  }
}

11.3. Routes

Accounts

`GET /account`

Fetch the account of the API key used to send the request. The response is an account object.

Projects

`GET /projects`

Fetch a paginated list of projects.

The response is a page of project objects.

`POST /projects`

Create a new project.

The request must be a JSON object containing the following field:

name (name): The name of the project.

The response is the project object which was created.

`GET /projects/id/{id}`

Fetch a project by identifier.

The response is a project object.

`GET /projects/name/{name}`

Fetch a project by name.

The response is a project object.

`PUT /projects/id/{id}`

Update an existing project.

The request must be a JSON object containing the following field:

name (name): The name of the project.

The response is the modified project object.

`DELETE /projects/id/{id}`

Delete a project by identifier.

Jobs

`GET /jobs`

Fetch a paginated list of jobs.

The response is a page of job objects.

`PUT /jobs`

Deploy a set of jobs in the current project. If jobs already exist with the same names, they are updated. If not, new jobs are created. The operation is atomic: if one of the job cannot be deployed, none will be.

The request is an array of job specification objects.

The response is an array of job objects representing the jobs created or updated from the specifications.

If the dry-run query parameter is set, Eventline validates job specifications but does not deploy them.

`GET /jobs/id/{id}`

Fetch a job by identifier.

The response is a job object.

`DELETE /jobs/id/{id}`

Delete a job by identifier.

`GET /jobs/name/{name}`

Fetch a job by name.

The response is a job object.

`PUT /jobs/name/{name}`

Deploy a single job in the current project. If a job already exists with this name, it is updated; if not, a new job is created.

The request is a job specification object.

The response is a job object representing the job created or updated from the specifications.

If the dry-run query parameter is set, Eventline validates the job specification but does not deploy it.

`POST /jobs/id/{id}/rename`

Rename a job. The request is a JSON object containing the following fields:

name (string): The new name of the job.
description (optional string): The new description of the job.

Note	Renaming a job will affect its specification: if the job is deployed from a job specification file, you will have to update it manually.

`POST /jobs/id/{id}/enable`

Enable a job by identifier. Nothing is done if the job is already enabled.

`POST /jobs/id/{id}/disable`

Disable a job by identifier. Nothing is done if the job is already disabled.

`POST /jobs/id/{id}/execute`

Execute a job by identifier.

The request is a JSON object containing the following field:

parameters (object): The set of parameters to use for execution.

The response is a job execution object.

Job executions

`GET /job_executions/id/{id}`

Fetch a job execution by identifier.

The response is a job execution object.

`POST /job_executions/id/{id}/abort`

Abort a created or started job execution by identifier.

`POST /job_executions/id/{id}/restart`

Restart a finished job execution by identifier.

Events

`GET /events`

Fetch a paginated list of events.

The response is a page of event objects.

`GET /events/id/{id}`

Fetch an event by identifier.

The response is an event object.

`POST /events/id/{id}/replay`

Replay an event by identifier.

Identities

`GET /identities`

Fetch a paginated list of identities.

The response is a page of identity objects.

`POST /identities`

Create a new identity.

The request must be a JSON object containing the following field:

name (name): The name of the identity.
connector (string): the name of the connector.
type (string): the type of the identity.
data (object): the set of data associated with the identity.

The response is the identity object which was created.

Note	Identities requiring a web browser such as OAuth2-based identities cannot be created using the API.

`GET /identities/id/{id}`

Fetch a identity by identifier.

The response is a identity object.

`GET /identities/name/{name}`

Fetch a identity by name.

The response is a identity object.

`PUT /identities/id/{id}`

Update an existing identity.

The request must be a JSON object containing the following field:

name (name): The name of the identity.
connector (string): the name of the connector.
type (string): the type of the identity.
data (object): the set of data associated with the identity.

The response is the modified identity object.

`DELETE /identities/id/{id}`

Delete a identity by identifier.

12. HOWTO

This chapter contains various examples and ideas designed to help users accomplish specific tasks. Most of them come from questions asked by users; if you are not sure about how to do something, just ask!

12.1. How to use a private Docker image registry

Use a runner identity to authenticate against the Docker registry. For example, to use the GitHub image registry, you can use a github/token identity:

name: "my-job"
runner:
  name: "docker"
  parameters:
    image: "my-org/my-private-image:latest"
  identity: "my-github-token-identity"

Refer to the runner documentation for more information about identities supported for each runner.

12.2. How to use other languages than the shell

Code steps are executed as any other code file. If there is no Shebang, Eventline add the one configured by default in the project, which happens to be /bin/sh.

To use any other language, just include your own shebang.

For example, to run Python code:

name: "my-job"
steps:
  - code: |
      #!/usr/bin/python
      puts("Hello world!")

See the job documentation for more information.

12.3. How to use a SSH key in a job

SSH keys are stored in generic/ssh_key identities. The field used to authenticate is private_key.

Since identity fields are injected in jobs as files, you can access the private key at $EVENTLINE_DIR/identities/<identity-name>/private_key.

The simplest way is to link the files as the default SSH key. For example, if you are using a RSA SSH key:

ln -s $EVENTLINE_DIR/identities/<identity-name>/private_key ~/.ssh/id_rsa

You can of course use the private key file, for example for the -i option of ssh and scp.

12.4. How to clone a private GitHub repository

Private GitHub repositories are accessed with a SSH key. You can link the private key as the default SSH key, then simply call git clone:

ln -s $EVENTLINE_DIR/identities/<identity-name>/private_key ~/.ssh/id_rsa
git clone git@github.com:my-organization/my-repository.git

Eventline handbook

Preface

What is Eventline?

Features

Digging in

1. Getting started

1.1. Eventline

Installation

First connection

1.2. Evcli

Installation

Authentication

1.3. Writing your first job

2. Writing jobs

2.1. Definition files

2.2. Deployment

2.3. Export

2.4. Writing code

Code blocks

Commands

Scripts

2.5. Reference

Job specification

Trigger specification

Parameter specification

Filter specification

Runner specification

Step specification

3. Using identities

3.1. Lifecycle

3.2. Refresh

4. Runners

4.1. local

Configuration

Parameters

Identity

4.2. docker

Configuration

Parameters

Identity

4.3. ssh

OpenSSH server configuration

Configuration

Parameters

Identity

4.4. kubernetes

Execution

Configuration

Parameters

Identity

5. Executing jobs

5.1. Triggers

5.2. Events

Replay

5.3. Manual execution

5.4. Lifecycle

5.5. Timeouts

Abortion

Restart

5.6. Runtime environment

Filesystem

Execution context

Environment variables

6. Connectors

6.1. aws

Identities

access_key

6.2. dockerhub

Identities

password

token

6.3. eventline

Identities

api_key

6.4. generic

Identities

api_key

password

gpg_key

ssh_key

4.1. `local`

4.2. `docker`

4.3. `ssh`

4.4. `kubernetes`

6.1. `aws`

`access_key`

6.2. `dockerhub`

`password`

`token`

6.3. `eventline`

`api_key`

6.4. `generic`

`api_key`

`password`

`gpg_key`

`ssh_key`

`oauth2`

6.5. `github`

`oauth2`

`token`

`raw`

`repository_creation`

`repository_deletion`

`tag_creation`

`tag_deletion`

`branch_creation`

`branch_deletion`

`push`

6.6. `postgresql`

`password`

6.7. `slack`

`oauth2_bot`

`oauth2_user`

6.8. `time`

`tick`

`abort-job-execution`

`create-project`

`create-identity`

`delete-job`