Integration Test Flakiness

How to connect to your repo

1. Fork this template
2. Authenticate with GitHub
3. Select a repo
4. Filter and check your data

This template helps you quickly identify which jobs in your GitHub Actions are flaky and re-running. Flaky tests can be a thorn in your team’s side, subtly but constantly wasting engineering time.

The example dataset for this template is showing 818 job runs from the sveltejs/svelte repository’s integration tests.

See this tutorial video for a quick overview on how to use this template.

Confirmed flakiness

Which jobs fail, but then pass on a re-run?

| Tests (14, macOS-latest) | 3154.1 (master), 3154.2 (master), 3154.3 (master) | | Tests (10, windows-latest) | 3167.1 (@feature/pass-source-to-preprocess) |

Possible flakiness

Which jobs fail, but we don’t have a re-run to compare it to? Excludes runs in which all jobs fail.

| Unit (ubuntu-latest) | 3157.1 (stylesTarget), 3156.1 (stylesTarget), 3130.1 (add-trusted-modifier) | | Tests (14, windows-latest) | 3152.1 (master), 3147.1 (v3.37.0), 3142.1 (tanhauhau/set-context-in-constructor) | | Tests (8, windows-latest) | 3150.1 (patch-5), 3144.1 (master), 3133.1 (add-trusted-modifier) | | Tests (14, ubuntu-latest) | 3156.1 (stylesTarget), 3130.1 (add-trusted-modifier) | | Tests (12, windows-latest) | 3141.1 (patch-1), 3139.1 (change-tracking) | | Tests (12, ubuntu-latest) | 3157.1 (stylesTarget) |

Below, see all jobs colored by conclusion. Click a run to go to its logs on GitHub.

How to connect to your repo

1. Fork this template

Your changes will not be saved until you fork.

2. Authenticate with GitHub

Create a GitHub personal access token with no scopes selected, or the repo scope if you need access to a private repository.

Save the token in a Secret named GITHUB_ACCESS_TOKEN and refresh this page. Treat the Secret like a password! Do not share or publish a notebook with a Secret; if you do, the Secret will refuse to load.

API key: ❌ Secret("GITHUB_ACCESS_TOKEN") not found. Add it in Settings.

Rate limit: No requests made yet.

3. Select a repo

If you’re not authenticated, we load cached results for the Svelte repository as a demonstration. Without authentication, you are rate-limited to 60 requests per hour; with a token, it goes up to 5,000 requests per hour. If you change the repo three times without authenticating, you’ll hit the rate limit.

repo = "sveltejs/svelte"

4. Filter and check your data

A repo may have several associated workflows; those workflows can be filtered by which branch of the repo they ran on.

Workflow CI (active) BranchAllmasterfix/derived-store-type-overloads@feature/pass-source-to-preprocessdeselect-unmatched-optionadd-trusted-modifierpatch-1stylesTargetpatch-2patch-5dynamic-elements-implementationv3.37.0tanhauhau/set-context-in-constructorfix/use-different-random-image-servicechange-trackingdependabot/npm_and_yarn/y18n-4.0.1v3.36.0gh-6066

Showing 818 job runs.

Appendix

Once you fork for your own authenticated use, you can simplify the data fetching code below by removing isSample.

isSample = true

workflows = Array(1) [Object]

runs = Array(50) [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, …]

branches = Set(17) {"master", "fix/derived-store-type-overloads", "@feature/pass-source-to-preprocess", "deselect-unmatched-option", "add-trusted-modifier", "patch-1", "stylesTarget", "patch-2", "patch-5", "dynamic-elements-implementation", "v3.37.0", "tanhauhau/set-context-in-constructor", "fix/use-different-random-image-service", "change-tracking", "dependabot/npm_and_yarn/y18n-4.0.1", "v3.36.0", "gh-6066"}

You can filter the jobs using the criteria of your choice:

jobs = Array(818) [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, …]

flaky = Array(4) [Object, Object, Object, Object]

flakyRates = Array(2) [Object, Object]

failedWithoutRetry = Array(14) [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object]

failureRates = Array(6) [Object, Object, Object, Object, Object, Object]

retriedJobs = Array(50) [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, …]

Helpers

prettyConclusion = ƒ(d)

color = Object {domain: Array(3), range: Array(3)}

timeParser = ƒ(e)

timeFormat = ƒ(e)