Test RCA - Flaky Tests¶
This guide describes how CloudAEye detects and analyzes flaky tests.
Overview¶
A flaky test is a test that fails intermittently without any actual code associated with the underlying test. Unlike reliable tests which produc consistent outcomes, flaky tests are inconsistent, making them difficult to trust.
Test Failure History¶
CloudAEye groups all flaky tests and shows their individual history.
The test's recent failure history is shown along with the following details:
- Success Rate: Percentage of time when this test ran successfully.
- Average Test Run Duration: Average time spent to run this test.
- Last Run Start Time: Start time of the last run.
- Latest Test Run Duration: Test duration from the latest run.
- Total Tipe Spent: Total time spent running this test.
- First Run: Date when this test was first ran.
Root Cause¶
CloudAEye analyzes the flaky tests and shows the root cause for each tests. This makes it very convenient to diagnose a flaky test.
Fix¶
CloudAEye shows the proposed fix for a flaky test. This makes it very convenient to resolve a flaky test.
Trends¶
CloudAEye shows the trends of flaky tests in a CI test job.
It shows any new flaky tests being detected and total number of flaky tests in the test sute for the CI job.
GitHub UI¶
CloudAEye groups the flaky tests in GitHub UI. It shows the root cause analysis and proposed fix for each flaky test.
