Make Gitlab CI a required PR check for all GitHub SIMP repos

Description

As a replacement for our Travis CI checks, we shold make the GitLab CI checks mandatory.

This will be a bit of a culture shift:

An authorized user on the gitlab.com/simp will need to push a PR over to the gitlab mirror in order to kick off the job (for security reasons, the CI pipeline the runs acceptance tests should not be triggered at-will by PRs from external contributors).
Not all github.com/simp repos are presently mirrored on gitlab.com/simp
Not all github.com/simp repos have GitLab CI pipelines yet

Acceptance Criteria

None

Activity

Show:

Liz Nemsick

January 4, 2021, 1:40 PM

Copy link to comment

Some component acceptance tests fail on GitLab, even though they run successfully on faster servers. For example, the rubygem-simp-cli tests will randomly fail with ssh connection failures. This transition is going to be a problem for those components.

Chris Tessmer

January 4, 2021, 10:44 PM

Copy link to comment

🤔 Components with intermittently-failing acceptance tests seem like they already have problems. What do we do with them now?

Chris Tessmer

January 4, 2021, 10:51 PM

Copy link to comment

Depending on the answer, we can formalize our expectations/response in the components' pipeline config:

If we're comfortable that a GitLab CI test might fail, we can add `allow_failure: true`, gate it behind a conditional (like we do with `<<: *with_SIMP_ACCEPTANCE_MATRIX_LEVEL_3`), or just remove it.
If we want to know when a test fails but have the chance to okay it anyway, we could add a (blocking) manual step that only appears after the failure.
If it's absolutely required that the test always runs and always passes: we can add a `retry` condition (or just accept the failure and hit "retry").

Chris Tessmer

January 4, 2021, 10:54 PM

Copy link to comment

(Although considering how time- and resource-intensive acceptance tests can be, I'd be wary of the `retry` condition until something like gitlab-ce#27988 is implemented.)

Chris Tessmer

January 5, 2021, 11:21 PM

Copy link to comment

Following today's discussion with , here are all the current simp GitLab CI pipelines that contain jobs with allow_failure: true, linked to their present commit refs for historical preservation: