Tadashi Nemoto
Published on

Setting up slack notification of detecting flaky tests using CircleCI(Orb)


Flaky Tests Notify Orb

This blog describes how to set up slack notification of detecting flaky tests using CircleCI(Orb)

How to see flaky tests in Test Insights Dashboard

In 2021, CircleCI released Test Insights Dashboard.

You can see flaky tests in Test Insights Dashboard so that your team can improve development performance by finding and fixing them quickly.

Flaky Tests

For more information, please refer to the blog post and documentation

About flaky-tests-notify Orb

Usually, you should regularly check Test Insights Dashboard if there are flaky tests.

To find and fix flaky tests faster, I created a CircleCI Orb(Config package) to notify them via slack.

1) Generate CircleCI token and Slack token

To use this orb, you will need to generate a CircleCI token and Slack token.

Since this orb utilizes Slack Orb, please refer to the below document to generate a Slack token.

CircleCI token is used to get flaky tests via API.

Please make sure to put these environment variables before moving to the next step.

  • SLACK_DEFAULT_CHANNEL(Which slack channel to post)

Flaky Tests Env

2) Create a CircleCI config file

Before using this orb, please make sure that your organization allows uncertified Orb.

Uncertified Orb

Next, create such a CircleCI config file using this orb.

version: '2.1'

  flaky-tests-notify: tadashi/flaky-tests-notify@1.0.0

    type: boolean
    default: false

      - flaky-tests-notify/notify
    when: << pipeline.parameters.notify_flaky_tests >>

This notify_flaky_tests workflow does not trigger until notify_flaky_tests parameter(true) is passed.

We will pass this parameter using Scheduled Pipeline.

3) Set up Scheduled Pipeline

To run this job regularly and notify flaky tests via Slack, we will use Scheduled Pipelines.

I added the parameter notify_flaky_tests so that this scheduled pipeline can trigger the notify_flaky_tests workflow.

Scheduled Pipelines Parameter

If you configured Scheduled Pipelines correctly, CircleCI will trigger pipelines based on its settings.

Scheduled Pipelines

If there are flaky tests, you will see a slack notification as below.

Flaky Tests Notify Orb