As a replacement for our Travis CI checks, we shold make the GitLab CI checks mandatory.
This will be a bit of a culture shift:
An authorized user on the gitlab.com/simp will need to push a PR over to the gitlab mirror in order to kick off the job (for security reasons, the CI pipeline the runs acceptance tests should not be triggered at-will by PRs from external contributors).
Not all github.com/simp repos are presently mirrored on gitlab.com/simp
Not all github.com/simp repos have GitLab CI pipelines yet
Some component acceptance tests fail on GitLab, even though they run successfully on faster servers. For example, the rubygem-simp-cli tests will randomly fail with ssh connection failures. This transition is going to be a problem for those components.
🤔 Components with intermittently-failing acceptance tests seem like they already have problems. What do we do with them now?
Depending on the answer, we can formalize our expectations/response in the components' pipeline config:
If we're comfortable that a GitLab CI test might fail, we can add `allow_failure: true`, gate it behind a conditional (like we do with `<<: *with_SIMP_ACCEPTANCE_MATRIX_LEVEL_3`), or just remove it.
If we want to know when a test fails but have the chance to okay it anyway, we could add a (blocking) manual step that only appears after the failure.
If it's absolutely required that the test always runs and always passes: we can add a `retry` condition (or just accept the failure and hit "retry").
(Although considering how time- and resource-intensive acceptance tests can be, I'd be wary of the `retry` condition until something like gitlab-ce#27988 is implemented.)
Following today's discussion with , here are all the current simp GitLab CI pipelines that contain jobs with allow_failure: true, linked to their present commit refs for historical preservation:
https://github.com/simp/simp-doc/blob/daf1bca/.gitlab-ci.yml
https://github.com/simp/pupmod-simp-simpcat/blob/ab51f4e/.gitlab-ci.yml
https://github.com/simp/pupmod-simp-pam/blob/459bf19/.gitlab-ci.yml
https://github.com/simp/pupmod-simp-simp_logstash/blob/f75080e/.gitlab-ci.yml
https://github.com/simp/pupmod-simp-rsyslog/blob/f5802f2/.gitlab-ci.yml
https://github.com/simp/rubygem-simp-beaker-helpers/blob/0855d22/.gitlab-ci.yml
https://github.com/simp/rubygem-simp-rake-helpers/blob/75e3665/.gitlab-ci.yml
https://github.com/simp/pupmod-simp-pupmod/blob/3962ea1/.gitlab-ci.yml
https://github.com/simp/pupmod-simp-simp_pki_service/blob/12f9713/.gitlab-ci.yml
https://github.com/simp/pupmod-simp-simp_elasticsearch/blob/2e2b2e2/.gitlab-ci.yml
https://github.com/simp/simp-integration_tests/blob/0927cff/.gitlab-ci.yml
https://github.com/simp/pupmod-simp-simplib/blob/9da0f56/.gitlab-ci.yml
https://github.com/simp/pupmod-simp-iptables/blob/384e828/.gitlab-ci.yml
https://github.com/simp/pupmod-simp-simp_grafana/blob/839fcae/.gitlab-ci.yml
https://github.com/simp/pupmod-simp-simp_kubernetes/blob/a0fc9ec/.gitlab-ci.yml