Setting up slack notification of detecting flaky tests using CircleCI(Orb)

This blog describes how to set up slack notification of detecting flaky tests using CircleCI(Orb)

• How to see flaky tests in Test Insights Dashboard
• About flaky-tests-notify Orb
• 1) Generate CircleCI token and Slack token
• 2) Create a CircleCI config file
• 3) Set up Scheduled Pipeline

How to see flaky tests in Test Insights Dashboard

In 2021, CircleCI released Test Insights Dashboard.

You can see flaky tests in Test Insights Dashboard so that your team can improve development performance by finding and fixing them quickly.

For more information, please refer to the blog post and documentation

• Introducing Test Insights with flaky test detection | CircleCI

• Test Insights - CircleCI

About flaky-tests-notify Orb

Usually, you should regularly check Test Insights Dashboard if there are flaky tests.

To find and fix flaky tests faster, I created a CircleCI Orb(Config package) to notify them via slack.

• CircleCI Developer Hub - tadashi0713/flaky-tests-notify

• GitHub - tadashi0713/flaky-tests-notify-orb

1) Generate CircleCI token and Slack token

To use this orb, you will need to generate a CircleCI token and Slack token.

Since this orb utilizes Slack Orb, please refer to the below document to generate a Slack token.

• Setup · CircleCI-Public/slack-orb Wiki · GitHub

CircleCI token is used to get flaky tests via API.

• Managing API Tokens - CircleCI

Please make sure to put these environment variables before moving to the next step.

• CIRCLE_TOKEN
• SLACK_ACCESS_TOKEN
• SLACK_DEFAULT_CHANNEL(Which slack channel to post)

2) Create a CircleCI config file

Before using this orb, please make sure that your organization allows uncertified Orb.

Next, create such a CircleCI config file using this orb.

version: '2.1'

orbs:
  flaky-tests-notify: tadashi/flaky-tests-notify@1.0.0

parameters:
  notify_flaky_tests:
    type: boolean
    default: false

workflows:
  notify_flaky_tests:
    jobs:
      - flaky-tests-notify/notify
    when: << pipeline.parameters.notify_flaky_tests >>

This notify_flaky_tests workflow does not trigger until notify_flaky_tests parameter(true) is passed.

We will pass this parameter using Scheduled Pipeline.

3) Set up Scheduled Pipeline

To run this job regularly and notify flaky tests via Slack, we will use Scheduled Pipelines.

• Getting started with scheduled pipelines | CircleCI

• Scheduled Pipelines - CircleCI

I added the parameter notify_flaky_tests so that this scheduled pipeline can trigger the notify_flaky_tests workflow.

If you configured Scheduled Pipelines correctly, CircleCI will trigger pipelines based on its settings.

If there are flaky tests, you will see a slack notification as below.