Sleuth Custom Automations Documentation

Sleuth's custom automations framework lets you build your own automation rules to remove toil and streamline your development and deployment workflows.

There are two ways to build and deploy custom automations. The first is to define your rules in a YAML file and then insert that YAML file into each code repository where you want those rules to be evaluated (located at .sleuth/rules.yml). The advantage of this approach is that you can modify your custom automations "as code" (e.g. enforcing your SDLC process around changes to rules and ensuring preservation of historical states). The disadvantage is that it's more effort to deploy rules to multiple repositories (since each requires its own rules.yaml file) and to make changes to rules over time (since rules changes will be treated like any other code changes).

The second method is to use the custom automation template provided in Sleuth's automations Marketplace. This option allows you to quickly write a new automations (using YAML) and deploy it to multiple teams or projects, all from within Sleuth's UI. You can also modify or uninstall those automations quickly from within Sleuth's UI. This option is great for quickly experimenting with automations, putting them out to team quickly, tweaking them based on feedback, and if they aren't producing the intended effect, removing them as quickly as you added them. While this option streamlines experimentation by removing the need to manually add a rules.yaml files to multiple repositories or to issue pull requests each time you want to change those rules, it might not conform to some organizations' policies for managing "rules as code".

Whichever method you prefer, Sleuth automations are always comprised one or more triggers (e.g. a PR update, a code deployment, or a user-defined schedule), conditions (e.g. a code deployment is "healthy"), and actions (e.g. send a Slack message, trigger a build). This document describes how to express and combine the trigger, conditions, and actions supported by Sleuth's custom automations framework.

Here's an example of .sleuth/rules.yml:

rules:
    - stage-to-prod:
        description: Automatically promotes a healthy staging deployment to production
        conditions:
          - environment='Staging'
          - health='Healthy'
        actions:
          - auto_approve_build: test-and-deploy
          - add_to_deploy_message_thread: Build promoted automatically on a healthy staging deploy

    - quick-fixes:
        conditions:
          - environment='Staging'
          - pr_labels='quickfix'
        triggers:
          - deployed
        actions:
          - auto_approve_build: 'test-and-deploy'
          - add_to_deploy_message_thread: Deploy automatically promoted as a quickfix

A rule can be named whatever you want, stage-to-prod and quick-fixes in this example, and can have a description to display in the Sleuth interface.

For more examples, see our cookbook.

To validate your actions, open up a code deployment and find actions validator in the top right (cog wheel) menu.

Conditions

The conditions section contains zero or more expressions that all must match in order to execute the actions. An expression can contain joined expressions using or and and.

If an operator is applied to a list of values, the operator returns true if any of the values in the list match.

Lists can also be compared to integers, in this case we are looking at the number of items in the list.

String literal values must be single quoted (') as double quotes are generally reserved for the YAML parser.

Operators

> - Greater than
>= - Greater than or equal to
< - Less than
<= - Less than or equal to
= - Equal to. If either side is a list, it matches if those intersect.
!= - Not equal to. If either side is a list, it matches if those do not intersect.
<> - Not equal to. If either side is a list, it matches if those do not intersect.
~= - Matches the value. If the value is a string, the 'regex' matcher is assumed. If either side is a list, it matches if those intersect.
!~= - Doesn't match the value. If the value is a string, the 'regex' matcher is assumed. If either side is a list, it matches if those do not intersect.

Variables

all (string) - Matches all in the passed list, which can also be a variable [Team] [Pull request] [Set of deploys] [Set of pull requests] [Set of Work in progress Items] [Deploy] [Work in progress item] [Project]
Default triggers: /
author_email (string) - Deploy author email [Deploy]
Default triggers: deployed
base_branch (string) - PR base branch. Branch PR is opened against [Pull request]
Default triggers: pr_openedpr_edited
branch (string) - Deploy source branch (full name of the branch this deploy's latest commit is on) [Deploy]
Default triggers: deployed
coding_time (time period) - Coding time [Set of Work in progress Items] [Pull request] [Deploy] [Work in progress item]
Default triggers: wip_set_durationdeployed
commits (integer) - Num of commits [Deploy]
Default triggers: deployed
day_of_month (integer) - Day of month action execution (value between 1 and 28/29/30/31) [Team] [Pull request] [Set of deploys] [Set of pull requests] [Set of Work in progress Items] [Deploy] [Work in progress item] [Project]
Default triggers: /
day_of_week (string) - Day of week of action execution (Monday is 0 and Sunday is 6) [Team] [Pull request] [Set of deploys] [Set of pull requests] [Set of Work in progress Items] [Deploy] [Work in progress item] [Project]
Default triggers: /
deploy_message_reaction_minus_1 (integer) - Count of :-1: reactions to the deploy channel message [Deploy]
Default triggers: deploy_message_reaction
deploy_message_reaction_plus_1 (integer) - Count of :+1: reactions to the deploy channel message [Deploy]
Default triggers: deploy_message_reaction
deploy_message_reaction_rocket (integer) - Count of :rocket: reactions to the deploy channel message [Deploy]
Default triggers: deploy_message_reaction
deployed_for (time period) - Time passed since the deploy was registered [Deploy]
Default triggers: deployed_duration
deploying_time (time period) - Deploying time [Set of Work in progress Items] [Pull request] [Deploy] [Work in progress item]
Default triggers: wip_set_durationdeployed
deployment (string) - Deployment or change source name [Deploy]
Default triggers: /
drift_days_to_development (integer) - Drift to Development in in days [Deploy]
Default triggers: deployed
drift_days_to_production (integer) - Drift to Production in in days [Deploy]
Default triggers: deployed
drift_days_to_staging (integer) - Drift to Staging in in days [Deploy]
Default triggers: deployed
drift_to_development (integer) - Drift to Development in deploy count [Deploy]
Default triggers: deployed
drift_to_production (integer) - Drift to Production in deploy count [Deploy]
Default triggers: deployed
drift_to_staging (integer) - Drift to Staging in deploy count [Deploy]
Default triggers: deployed
environment (string) - Environment name [Pull request] [Deploy] [Work in progress item]
Default triggers: deployed
even_week (boolean) - Is the current week even? [Team] [Pull request] [Set of deploys] [Set of pull requests] [Set of Work in progress Items] [Deploy] [Work in progress item] [Project]
Default triggers: /
execute_once_per_pr (boolean) - Execute only once per PR [Pull request]
Default triggers: /
failed_rules (list of strings) - Rules that failed on the same deploy (rules with failed conditions are not included) [Deploy]
Default triggers: /
files (list of strings) - List of file paths in the target object [Pull request] [Deploy]
Default triggers: /
git_commit_format (string) - Git commit format matcher. Uses Gitlint - https://jorisroovers.com/gitlint/latest/ [Team] [Pull request] [Set of deploys] [Set of pull requests] [Set of Work in progress Items] [Deploy] [Work in progress item] [Project]
Default triggers: /
glob (string) - Glob pattern for matching paths [Team] [Pull request] [Set of deploys] [Set of pull requests] [Set of Work in progress Items] [Deploy] [Work in progress item] [Project]
Default triggers: /
health (string or none) - Health [Deploy]
Default triggers: deploy_health_received
issue_key_pattern (string) - The issue key regex pattern for the configured issue integration of the project [Pull request] [Deploy]
Default triggers: /
last_locked_by (string) - Email of the last user who locked the Code Deployment [Deploy]
Default triggers: /
locked (boolean) - Deployment is locked [Deploy]
Default triggers: /
magnitude (string) - Magnitude (ie. Small, Medium,...) [Deploy]
Default triggers: deployed
month (integer) - Month of action execution (value between 1 and 12) [Team] [Pull request] [Set of deploys] [Set of pull requests] [Set of Work in progress Items] [Deploy] [Work in progress item] [Project]
Default triggers: /
num_pull_requests (integer) - Number of pull requests in a deploy [Deploy]
Default triggers: /
other_pr_labels (list of strings) - Labels on pull requests other than the current one [Pull request]
Default triggers: pr_others_edited
pr_additions_count (integer) - Number of added or changed lines in a pull request [Pull request]
Default triggers: pr_openedpr_edited
pr_approver_emails (list of strings) - A list of emails of people that have approved the pull request [Pull request]
Default triggers: pr_openedpr_reviewed
pr_batch_size (string) - Batch size of a pull request. Options: ['SMALL', 'MEDIUM', 'LARGE', 'GIGANTIC'] [Pull request] [Work in progress item]
Default triggers: pr_openedpr_edited
pr_check_disabled (boolean) - PR check disabled [Pull request]
Default triggers: /
pr_coding_time (time period) - PR coding time [Pull request]
Default triggers: pr_open_duration
pr_commit_messages (list of strings) - List of commit messages in a pull request [Pull request]
Default triggers: pr_openedpr_edited
pr_deletions_count (integer) - Number of deleted lines in a pull request [Pull request]
Default triggers: pr_openedpr_edited
pr_description (string) - Pull request description [Pull request]
Default triggers: pr_openedpr_edited
pr_file_count (string) - Number of changed files in a pull request [Pull request]
Default triggers: pr_openedpr_edited
pr_files (list of strings) - List of changed files in a pull request [Pull request]
Default triggers: pr_openedpr_edited
pr_is_draft (boolean) - Pull request is marked as draft [Pull request] [Work in progress item]
Default triggers: pr_opened
pr_is_ignored (boolean) - Pull request is ignored from metrics [Pull request] [Work in progress item]
Default triggers: /
pr_labels (list of strings) - Pull request labels [Pull request] [Deploy]
Default triggers: pr_openedpr_edited
pr_last_update_event (float) - Days since the last update event for a pull request was received [Pull request]
Default triggers: /
pr_open_duration_days (float) - PR open duration expressed in days [Pull request]
Default triggers: pr_open_duration
pr_review_lag_time (time period) - PR review lag time [Pull request]
Default triggers: pr_open_duration
pr_review_time (time period) - PR review time [Pull request]
Default triggers: pr_open_duration
pr_title (string) - Pull request title [Pull request]
Default triggers: pr_openedpr_edited
project (string) - Project name [Deploy]
Default triggers: /
pull_request_author (list of strings) - Pull request author (email or username or name) [Pull request]
Default triggers: /
regex (string) - Regular expression [Team] [Pull request] [Set of deploys] [Set of pull requests] [Set of Work in progress Items] [Deploy] [Work in progress item] [Project]
Default triggers: /
review_lag_time (time period) - Review lag time [Set of Work in progress Items] [Pull request] [Deploy] [Work in progress item]
Default triggers: wip_set_durationdeployed
review_time (time period) - Review time [Set of Work in progress Items] [Pull request] [Deploy] [Work in progress item]
Default triggers: wip_set_durationdeployed
slack_channel_approval_status (string) - If deploy was approved via Slack [Deploy]
Default triggers: /
source_branch (string) - PR Source branch. Branch PR is opened from [Pull request]
Default triggers: pr_opened
stale_for (time period) - Pull request stale duration [Pull request]
Default triggers: pr_open_duration
state (string) - State of the Pull request (possibilities: closed, merged, open, unknown) [Pull request] [Work in progress item]
Default triggers: /
successful_rules (list of strings) - Rules that succeeded on the same deploy [Deploy]
Default triggers: /
tags (list of strings) - Deploy tags [Deploy]
Default triggers: deployed
todo_item_pattern (string) - Checkbox indicating a TODO item [Pull request]
Default triggers: /
utc_hour (integer) - UTC hour of action execution (value between 0 and 23) [Team] [Pull request] [Set of deploys] [Set of pull requests] [Set of Work in progress Items] [Deploy] [Work in progress item] [Project]
Default triggers: /

Matchers

sleuth_validate_config_format (boolean) - Validates the presence and format of Sleuth validate INI configuration. Can pass in 'required=false' to make it optional.

Time period units

We support the following format "dd:hh:mm:ss". The values are optional, but must be in the order of days, hours, minutes and seconds.

d - days
h - hours
m - minutes
s - seconds

Filters

The filters section is optional and contains zero or more expressions that filter a set of items.

It can only be used on sets of items like [Set of Work in progress Items] or [Set of pull requests], not on individual targets like [Deploy] or [Pull request].

The conditions are applied to the target (a set or an individual object) as a whole, while the filters are applied to each item in the set.

Here's an example:

rules:
    - deploying-time-goal:
        description: Notify us when the deploying time is over the goal
        triggers:
          - wip_set_duration
        filters:
          - state='merged'
          - environment='Production'
        conditions:
          - deploying_time>'0d:0h:20m:0s'
        actions:
          - slack_channel_message:
              channel: '#dev-notifications'
              message: 'Oh-oh, {{ authors_slack }} the deploying time is over the goal!'

The rule in the example above will be triggered every minute (because the triggers is wip_set_duration).

The rule will fetch all Work in progress items whose state is merged and which belong to the environment Production (because of the filters).

Then it will check if deploying_time for the set is more than 20 minutes and send a Slack notification, if it is so.

Actions

add_deploy_tag - Add a tag to the deploy
Parameters:
- value (string) - The tag name to add
add_pr_comment - Add a comment to the pull request
Parameters:
- comment (string) - Comment to add on a PR
add_pr_label - Add a label to the pull request
Parameters:
- value (string) - The label name to add
add_to_deploy_message_thread - Add a message to the deploy message thread
Parameters:
- value (string) - The markdown message to send
analytics_tracking - Analytics tracking
Parameters:
- analytics_event (string) - Analytics event to track
- extra_data (Dict) - Extra arguments passed into analytics event
auto_approve_build - Auto-approve build
Parameters:
- value (string) - The name of the build to approve
- build (string) - The name of the build to approve
- job (string) - The name of the job inside the build to approve
close_pr - Add a comment to the pull request
monitor_drift - Monitor the drift from a deploy to a target environment and perform actions when it is exceeded or returned
Parameters:
- environment (string) - The name of the target environment to measure against
- threshold (string) - The amount of drift time to trigger actions. Use the format '4h' or 4d'.
- on_over (ActionList) - A list of actions to execute the first time the drift threshold has been exceeded
- on_under (ActionList) - A list of actions to execute the first time the deploy is under the drift threshold
notify_mentioned_issues - Notify issues mentioned in commits or pull requests (Jira and Linear only)
project_lock - Project Lock
Parameters:
- action (string) - The locking action performed: either 'lock' on 'unlock'
remove_pr_label - Add a label to the pull request
Parameters:
- value (string) - The label name to add
remove_pr_label_by_prefix_filter - Remove PR labels by prefix filter, while keeping the optional label value
Parameters:
- label_to_keep (string) - The label name to keep
- prefix_filter (string) - Prefix filter to filter labels from
set_pr_check - Change the pull request status check
Parameters:
- name (string) - The name of the check
- status (string) - The status of the check
set_pull_request_ignore - Ignore this PR from metrics
Parameters:
- ignored (Boolean) - Should the PR be ignored (ignored:true) or un-ignored (ignored:false). Default: true.
slack_channel_approval - Display an approval on Slack and execute actions if confirmed
Parameters:
- name (string) - Human-readable name of approval, used for display in Sleuth UI
- channel (string) - Slack channel to display approval in, starting with #
- prompt (string) - Text to use as prompt, supports markdown
- on_success (ActionList) - A list of actions to execute on confirmation
- authors_only (Boolean) - Only accept approvals from deploy authors
- approvers (List) - A list of users, specified as Sleuth-registered email addresses, to restrict approvals
- votes_required (integer) - The number of votes required for a successful approval
slack_channel_message - Slack channel message
Parameters:
- channel (string) - The slack channel, starting with #
- message (string) - The markdown message to send
- analytics_event (string) - Analytics event to track
slack_personal_message - Slack personal message
Parameters:
- email (string) - The email of the user to message. Required if 'group' is not used.
- group (string) - The name of the group of users to target. Must be one of ['authors', 'deployer', 'reviewers']. Required if 'email' is not used.
- message (string) - The markdown message to send
teams_channel_message - Microsoft Teams channel message
Parameters:
- webhook_url (string) - URL of an incoming webhook for your Microsoft Teams channel
- message (string) - The markdown message to send
teams_deploy_message - Deploy message sent to a Microsoft Teams channel
Parameters:
- webhook_url (string) - URL of an incoming webhook for your Microsoft Teams channel
transition_mentioned_issues - Transition issues mentioned in commits or pull requests (Jira and Linear only)
Parameters:
- state (string) - The target transition state name
- state_id (string) - The target transition state ID for Linear (`state_id` will override `state` when set)
- resolution (string) - The optional resolution name, if applicable
trigger_build - Trigger build
Parameters:
- revision (string) - The revision to execute the build from
- parameters (Dict) - A map of parameters to pass to the workflow
update_mentioned_issues - Update issues mentioned in commits or pull requests (Jira only, and only for simple fields)
Parameters:
- field_name (string) - The name of the field to update
- field_value (string) - The value of the field
validate_pr_metrics - Validate PR metrics, if configured [beta]
Parameters:
- on_pass (ActionList) - A list of actions to execute on pass
- on_fail (ActionList) - A list of actions to execute on fail
- on_error (ActionList) - A list of actions to execute on error
webhook - Outgoing webhook
Parameters:
- value (string) - The webhook URL

Variables

authors_slack (string) - Authors in the form of Slack mentions [Set of Work in progress Items] [Pull request] [Deploy] [Work in progress item]
base_branch (string) - PR base branch. Branch PR is opened against [Pull request]
branch (string) - Deploy source branch (full name of the branch this deploy's latest commit is on) [Deploy]
coding_time (time period) - Coding time [Set of Work in progress Items] [Pull request] [Deploy] [Work in progress item]
deploy_date (string) - Date when the deploy occurred in ISO format (ie. 2000-01-01T12:00:00+00:00) [Deploy]
deploy_slack (string) - URL to the deploy in Slack format [Deploy]
deploy_teams (string) - URL to the deploy in Microsoft Teams format [Deploy]
deploy_url (string) - URL to the deploy in the Sleuth UI [Deploy]
deploying_time (time period) - Deploying time [Set of Work in progress Items] [Pull request] [Deploy] [Work in progress item]
deployment (string) - Deployment or change source name [Deploy]
drift_days_to_development (integer) - Drift to Development in in days [Deploy]
drift_days_to_production (integer) - Drift to Production in in days [Deploy]
drift_days_to_staging (integer) - Drift to Staging in in days [Deploy]
drift_to_development (integer) - Drift to Development in deploy count [Deploy]
drift_to_production (integer) - Drift to Production in deploy count [Deploy]
drift_to_staging (integer) - Drift to Staging in deploy count [Deploy]
env_drift_url (string) - Environment drift URL [Deploy]
environment (string) - Environment name [Pull request] [Deploy] [Work in progress item]
environment_slack (string) - Slack link to list of all deploys registered in this environment [Deploy]
goal_coding_time_remaining (string) - Time remaining to open the PR to reach the goal (humanized) [Pull request]
goal_deploying_time (string) - Deploying time (humanized) [Set of Work in progress Items] [Pull request] [Deploy] [Work in progress item]
goal_deploying_time_remaining (string) - Time remaining to deploy the future deploy to reach the goal (humanized) [Set of Work in progress Items] [Pull request] [Deploy] [Work in progress item]
goal_pr_coding_time (string) - PR coding time (humanized) [Pull request]
goal_pr_review_lag_time (string) - PR review lag time (humanized) [Pull request]
goal_pr_review_time (string) - PR review time (humanized) [Pull request]
goal_review_lag_time_remaining (string) - Time remaining for first review the PR to reach the goal (humanized [Pull request]
goal_review_time_remaining (string) - Time remaining to review the PR to reach the goal (humanized) [Pull request] [Work in progress item]
goal_target_time (string) - The goal time we are aiming for [Team] [Pull request] [Set of deploys] [Set of pull requests] [Set of Work in progress Items] [Deploy] [Work in progress item] [Project]
health (string or none) - Health [Deploy]
last_locked_by (string) - Email of the last user who locked the Code Deployment [Deploy]
pr_authors_slack (string) - PR author in the form of Slack mention [Pull request] [Deploy]
pr_batch_size (string) - Batch size of a pull request. Options: ['SMALL', 'MEDIUM', 'LARGE', 'GIGANTIC'] [Pull request] [Work in progress item]
pr_coding_time (time period) - PR coding time [Pull request]
pr_open_duration_days (float) - PR open duration expressed in days [Pull request]
pr_review_lag_time (time period) - PR review lag time [Pull request]
pr_review_time (time period) - PR review time [Pull request]
pr_reviewers_and_authors_slack (string) - PR reviewers and authors in the form of Slack mentions [Pull request]
pr_reviewers_slack (string) - PR reviewers in the form of Slack mentions [Pull request]
project (string) - Project name [Deploy]
prs_summary (string) - List of pull requests in Slack format [Set of Work in progress Items]
prs_summary_teams (string) - List of pull requests in MS Teams format [Set of Work in progress Items]
pull_request_id (string) - ID of the pull request [Pull request]
pull_request_slack (string) - URL to the pull request in Slack format [Pull request]
pull_request_teams (string) - URL to the pull request in MS Teams format [Pull request]
pull_request_url (string) - URL to the pull request [Pull request]
review_lag_time (time period) - Review lag time [Set of Work in progress Items] [Pull request] [Deploy] [Work in progress item]
review_time (time period) - Review time [Set of Work in progress Items] [Pull request] [Deploy] [Work in progress item]
revision (string) - SHA value of deploy revision [Deploy]
slack_channel_approval_status (string) - If deploy was approved via Slack [Deploy]
slack_dora_metrics_digest_weekly (string) - Weekly DORA Metrics Digest for Slack [Team] [Project]
slack_wip_metrics_digest (string) - WIP Metrics Digest for Slack [Team] [Project]
source_branch (string) - PR Source branch. Branch PR is opened from [Pull request]
stale_for (time period) - Pull request stale duration [Pull request]
state (string) - State of the Pull request (possibilities: closed, merged, open, unknown) [Pull request] [Work in progress item]
teams_dora_metrics_digest_weekly (string) - Weekly DORA Metrics Digest for Microsoft Teams [Team] [Project]
teams_wip_metrics_digest (string) - WIP Metrics Digest for Microsoft Teams [Team] [Project]
wip_prs_waiting_for_deploy_slack (string) - List of merged pull requests, waiting for deploy, in Slack format [Set of Work in progress Items]

Action variables may be used in various string parameters when configuring actions. They must be wrapped in double curly braces, e.g.: add_to_deploy_message_thread: '{{ authors_slack }}, your code was just deployed!'

Triggers

cron - Regular interval [Set of Work in progress Items]
cron_project - Regular interval for Project [Project]
cron_team - Regular interval for Team [Team]
deploy_health_received - Deploy health received [Deploy]
deploy_message_reaction - Deploy message reaction [Deploy]
deployed - Deployed [Deploy]
deployed_duration - Deployed duration [Deploy]
pr_edited - Pull request edited [Pull request]
pr_merged - Pull request merged [Pull request]
pr_open_duration - Pull request open duration [Pull request]
pr_opened - Pull request opened [Pull request]
pr_others_edited - Another open pull request edited [Pull request]
pr_reviewed - Pull request reviewed [Pull request]
wip_set_duration - Duration of WIP items [Set of Work in progress Items]