- Tadashi Nemoto
This blog describes how to set up slack notification of detecting flaky tests using CircleCI(Orb)
- How to see flaky tests in Test Insights Dashboard
- About flaky-tests-notify Orb
- 1) Generate CircleCI token and Slack token
- 2) Create a CircleCI config file
- 3) Set up Scheduled Pipeline
How to see flaky tests in Test Insights Dashboard
In 2021, CircleCI released Test Insights Dashboard.
You can see flaky tests in Test Insights Dashboard so that your team can improve development performance by finding and fixing them quickly.
For more information, please refer to the blog post and documentation
About flaky-tests-notify Orb
Usually, you should regularly check Test Insights Dashboard if there are flaky tests.
To find and fix flaky tests faster, I created a CircleCI Orb(Config package) to notify them via slack.
1) Generate CircleCI token and Slack token
To use this orb, you will need to generate a CircleCI token and Slack token.
Since this orb utilizes Slack Orb, please refer to the below document to generate a Slack token.
CircleCI token is used to get flaky tests via API.
Please make sure to put these environment variables before moving to the next step.
SLACK_DEFAULT_CHANNEL(Which slack channel to post)
2) Create a CircleCI config file
Before using this orb, please make sure that your organization allows uncertified Orb.
Next, create such a CircleCI config file using this orb.
version: '2.1' orbs: flaky-tests-notify: firstname.lastname@example.org parameters: notify_flaky_tests: type: boolean default: false workflows: notify_flaky_tests: jobs: - flaky-tests-notify/notify when: << pipeline.parameters.notify_flaky_tests >>
notify_flaky_tests workflow does not trigger until
notify_flaky_tests parameter(true) is passed.
We will pass this parameter using Scheduled Pipeline.
3) Set up Scheduled Pipeline
To run this job regularly and notify flaky tests via Slack, we will use Scheduled Pipelines.
I added the parameter
notify_flaky_tests so that this scheduled pipeline can trigger the
If you configured Scheduled Pipelines correctly, CircleCI will trigger pipelines based on its settings.
If there are flaky tests, you will see a slack notification as below.