Verified Commit 3f813ed4 authored by Bennet Bleßmann's avatar Bennet Bleßmann
Browse files

Add CI Readme and adjust some comments

parent 8de0eb5a
Pipeline #1322 passed with stages
in 259 minutes and 57 seconds
......@@ -51,7 +51,7 @@ stages:
- doc # build lib and manual
- package_release # build distribution
- upload_release # create tar for webserver
- tag_release # create release
- tag_release # create release
- curry-lang # commit updated download data to curry-lang.org repo
- curry-lang-ci # trigger curry-lang.org repo CI
......@@ -301,12 +301,7 @@ upload_page_tar:
- "${PUBLIC_HTML}"
# TODO add artifact with link to the packages uploaded in the 'upload_release' job
# we would like to add links to:
# pakcs-${VERSION}-amd64.tar.gz (produced by bundle_tar job)
# pakcs-${VERSION}-src.tar.gz (produced by bundle_tar job)
# Manual.pdf (produced by make_manual job)
#
# TODO release asset links accessible without being logged in see [Issue 289848](https://gitlab.com/gitlab-org/gitlab/-/issues/289848)
# run for release runs triggered by version tag
tag_release:
stage: tag_release
......@@ -370,9 +365,9 @@ link_release:
script:
- ci/update_version.sh release
# run for nightly and release runs
# the commit done by the link_nightly job should be enough once https://gitlab.com/gitlab-org/gitlab/-/issues/246784 is fixed
# it may still be wanted to keep it this way to keep the pipelines linked
downstream:
stage: curry-lang-ci
rules:
......
CI Readme
=========
## Files
```text
. -- repo root
|
+-- .gitlab-ci.yml -- Gitlab CI file, the root of all CI
+-- .tarignore -- Files ignored for distribution (`make dist`)
+-- Makefile -- The Projects Makefile invoked by CI
|
+-- ci
| |
| +-- before_make.sh -- run before most CI jobs to setup the basic environment
| +-- build_docker_image.sh -- script to build the Dockerfile-swi image and tag it as pakcs-swi-ci
| +-- cypm_helper.sh -- script with helper function for interacting with cypm
| +-- Dockerfile-swi -- Dockerfile for the Docker Image used by most CI jobs
| +-- gitlab_release.sh -- used to create the gitlab (nightly) releases
| +-- make_download.sh -- script run to test the website download
| +-- README.md -- this Readme explaining the CI setup
| +-- release_helper.sh -- script with helper function to setup the release environment
| +-- update_version.sh -- script to update the latest release/nightly referenced on curry-lang.org
| +-- upload_release.sh -- script to create a new generic package for releases and nightlies
```
## Configuration
### Repo Source
Most Configuration is Configured in one of the Files listed above.
Things that one may want to override is where possible configured via Variables in .gitlab-cyí.yml
#### .gitlab-ci.yml Docker
The default docker image used is defined by the top-level `image` entry,
as of writing this is `skgland/pakcs-swi-ci:latest` which is build from the Dockerfile `Dockerfile-swi`
The `tag_release` job uses a different image as it needs access to the gitlab release-cli.
All jobs will run on any available runner that is tagged with the `docker2` tag,
where possible they will run concurrently.
As the `link_nightly` and `link_release` jobs can interfere with each other,
they are configured to not run concurrently by sharing a `resource_group`
#### .gitlab-ci.yml Variables
- `DONWLOAD_VERSION`\: Used by `make_download.sh` to download the distribution
from `http://www.informatik.uni-kiel.de/~pakcs/download/packs-${DOWNLOAD_VERSION}-<src|amd64-Linux>.tar.gz`
for the nightly tests
- `CPM_VERSION`\: Used by `before_make.sh` to configure the cypm directories,
can't ask cypm as it might not be available at this point, and we don't want it to write to wrong places
- `CURRY_LANG_USER`\: The username for the Access Token (see Gitlab Section) [Username Doc](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html)
- `CURRY_LANG_EMAIL` \: The email address for the Access Token, Username without first `_` @ domain, domain is `example.com`
unless configured by a Gitlab Admin, no clue how I found that out
- `CURRY_LANG_PROJECT`\: The Repo ID for the curry-lang.org repo, assumed to be in the same gitlab instance
- `CURRY_LANG_BRANCH` \: The branch to update for releases/nightlies in the curry-lang.org repo
- `URRY_CHECK_VERSION`, `CURRY_DOC_VERSION`, `MARKDOWN_VERSION`\: the version of the corresponding package to install using cypm
##### `gitlab-ci.yml` job `run_download_make` Variables
- `CURRY_CHECK_VERSION` override as this may need a different version than the other jobs
##### Values that usually don't need to be changed
`LANG` and `LC_ALL` cause problems when not set or set to incompatible values, therefore they are
set to known working values.
The variables `STACK_ROOT`, `CABAL_DIR`, `BIN_INSTALL_DIR` are there to move those folders to a place where they can be cached,
it should not be necessary to override them.
As this project contains submodules those are checkedout recursively by instructing Gitlab via
`GIT_SUBMODULE_STRATEGY`, this should not be changed as long as this project uses submodules.
#### `ùpdate_version.sh`
Here the content of the `./data/versions/packs/{latest,latest-nightly,v${VERSION}}.version`
including the download link bases.
Also, the commit messages to curry-lang.org are defined here.
#### `release_helper.sh`
Defined the names used for tags and the generic packages.
Also, defined the mapping of paths to filenames for the files that are added to a release.
`LOCAL_FILE_NAME` contains the name of the file on disc relative to the `CI_PROJECT_DIR`.
`ÙPLOAD_FILE_NAMES` contains the corresponding name for the file upload.
#### `make_download.sh`
Has a hard coded download url
#### `gitlab_release.sh`
Contains the release name, description and tag
#### `before_make.sh`
Configures some Paths for cypm to have them under the `CI_PROJECT_DIR` so that we can cache them
### Gitlab
All variables defined in the .gitlab-ci script can
be overridden with Project CI Variables, Schedule Variables or Variables in Manual runs
#### Access Token
The access token to commit to the curry-lang.org repo **MUST NOT** be configured as part of the repo source,
but in one of the positions mentioned above for override Variables.
Usually this should be set in `packs >> Setting >> CI/CD Settings >> Variables` and
there `CURRY_LANG_ORG_ACCESS_TOKEN` should contain the access token `Masked` should be set,
as we don't use protected branches `Protected` should **NOT** be set.
To get an access token for the curry-lang.org repo go to `curry-lang.org >> Project Access Tokens`
and create an access token with at least `write_repository` permission, though more permissions may be required.
When changing Access Tokens change the variables `CURRY_LANG_USER`, `CURRY_LANG_EMAIL` accordingly.
#### Schedules
##### Nightly Release Build
For the Schedule Variables `BUILD_NIGHTLY` should be set to `yes`
##### Nightly Download Test
For the Schedule Variables `TEST_DOWNLOAD` should be set tp `yes`
## CI Runs
### Commit
On commit a Pipeline runs unless the commit message contains `[skip ci]`.
For commit only the jobs `run_make` and `run_test` are executed, to keep the CI time short.
### Tag
On tag a Pipeline runs unless the commit message contains `[skip ci]`!
For tags the jobs:
- `run_make`
- `run_test`,`check_version`
- `make_manual`
- `bundle_tar`
- `upload_release`, `upload_page_tar`
- `tag_release`
- `link_release`
- `downstream`
are (potentially) executed.
### Schedule
#### Nightly Test
Only the job `run_download_make` is run, to test the distribution download
The pipeline is run with Variable `TEST_DOWNLOAD` set to `yes`
#### Nightly Release Build
For nightly release build pipeline the jobs:
- `run_make`
- `run_test`
- `make_manual`
- `bundle_tar`
- `upload_nightly`
- `link_nightly`
- `downstream`
are executed.
The pipeline is run with Variable `BUILD_NIGHTLY` set to `yes`
### Manual
Manuel Pipelines are started from the Gitlab Web Interface under `packs >> CI/CD > Pipelines >> Run Pipeline`,
than the branch/tag that should be run can be selected and override Variables can be set.
#### Nightly Release Build
When a Nightly Build should be retried one can also just retry the existing nightly run
or manually trigger the scheduled pipeline under `packs >> CI/CD > Schedules >> Test Nightly > Play`.
To Manually Test the nightly release, create a manual Pipeline with Variable `BUILD_NIGHTLY` set to `yes`,
usually the `CURRY_LANG_BRANCH` variable should also be overridden,
note that the corresponding branch needs to exist
in the curry-lang.org repo before the pipeline is started.
#### Nightly Download Test
Manually trigger the scheduled pipeline under `packs >> CI/CD > Schedules >> Test Download > Play`
#### Release Build
To create a release, create the tag `v<packs-version>` with `<packs-version>` replaced by the version that the release is for.
To test most of the release cycle, create a Manual pipline and set the Variable `TEST_RELEASE` to `yes` or create a tag
that does not potentially match a past or futur version.
## CI Job Dependencies
Gitlab requires that job dependencies are acyclic (reasonable) and that jobs may only depend on jobs of prior stages.
While keeping this in mind the jobs are separated further in semantic stages.
```text
make | run_make
| |
| +-------------+---------+------------+------------+
| | | | | |
test | | | | run_test check_version run_download_make
| | | |
| | | +----------+-------------------+
| | v v v v
| | | | | |
doc | | | make_manual | |
| | | | | |
| | +----<----+---->-----|-------------------|-+
package_release | | bundle_tar v | | |
| | | | | | |
| | | | | | |
| | +---------+---->----+----------+ | |
| | | v | |
| | | +---------+----+--------------+ | |
| | | | | | | |
upload_release | | | upload_release upload_nightly upload_page_tar | |
| | | | | |
| | | +-------<------+-------------------------+ |
| | | | | |
tag_release | | v tag_release v |
| | | | | |
| | +---------|--------------+---------------------------+
| | | |
| +-------------+ |
curry-lang | link_release link_nightly
| | |
| +--------------+
| |
curry-lang-ci | curry-lang-ci
|
```
## Makefile Interaction
All CI invocations of make should set `CI_BUILD=yes`,
this should cause most if not all warnings to be errors.
For example fail if the tool to build the documentation or run the test is missing,
rather than warn.
Also, this flag should cause more verbose output, for better debugging of traces for failed CI runs.
Files/Folders that should not be packed for distribution (`make dist`)
should be listed in the `.tarignore`, files that only be ignored for some distribution artifacts
are listed directly in the Makefile
\ No newline at end of file
......@@ -42,6 +42,15 @@ function release_helper_init() {
# but gitlab restricts the version to the regex /\A\d+.\d+.\d+.\z/
# so we add it as a suffix to the package name and the file name
# this sadly means we create a package per nightly
# TODO when [Issue 273034](https://gitlab.com/gitlab-org/gitlab/-/issues/273034) is fixed
# use a SemVer lable instead of creating a package per nightly
# could also drop the package_suffix entirely than releases and nightly would be under the same package
#
# package_suffix="-nightly"
# version="${version}-nightly-${CI_COMMIT_SHORT_SHA}"
#
# release_tag and name_suffix stay as is
package_suffix="-nightly-${CI_COMMIT_SHORT_SHA}"
name_suffix="-nightly-${CI_COMMIT_SHORT_SHA}"
......@@ -65,7 +74,7 @@ function release_helper_init() {
"docs/Manual.pdf"
)
# name given to the file in the package and for the release asset
# name given to the file in the package and for the release asset link
UPLOAD_FILE_NAMES=(
"${full_name}-src.tar.gz"
"${full_name}-${arch}.tar.gz"
......
......@@ -34,6 +34,10 @@ function finish() {
# update the latest nightly version
function nightly() {
# These links will expire as the artifacts are by default only kept for a limited time
# but as these are nightlies this should be fine.
# When it is possible to lik to the generic packages, we create it would be best to change to links to point there
# as the job id would then also be irrelevant
BUNDLE_DOWNLOAD_URL="https://git.ps.informatik.uni-kiel.de/curry/pakcs/-/jobs/${BUNDLE_JOB_ID}/artifacts/raw"
MANUAL_DOWNLOAD_URL="https://git.ps.informatik.uni-kiel.de/curry/pakcs/-/jobs/${MANUAL_JOB_ID}/artifacts/raw"
......@@ -56,6 +60,7 @@ NIGHTLY_VERSION
# commit updated file
git add "${NIGHTLY_FILE}"
# regarding the mentioned issue it may still be wanted to keep it this way to keep the pipelines linked
git commit -F - <<COMMIT_MSG
[Pakcs CI] Update Nightly
......@@ -75,7 +80,6 @@ function release() {
DOWNLOAD_URL="https://git.ps.informatik.uni-kiel.de/curry/pakcs/${release}/downloads"
# replace the latest.version file
install -D /dev/stdin "${LATEST_FILE}" <<LATEST_VERSION
---
# THIS FILE IS GENERATED AUTOMATICALLY BY THE PACKS RELEASE PIPELINE
......@@ -110,6 +114,7 @@ CURRENT_VERSION
# commit updated file
git add "${LATEST_FILE}" "${VERSION_FILE}"
# regarding the mentioned issue it may still be wanted to keep it this way to keep the pipelines linked
git commit -F - <<COMMIT_MSG
[Pakcs CI] Update Release
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment