jlab-cluster/helm-charts
3
0 Fork 0
Code Issues 15 Pull Requests 3 Actions Packages Wiki Activity

PROBE: validate-charts on Gitea Actions (throwaway, do not merge) #1

Closed
jlab-bot wants to merge 4 commits from ci/gitea-validate-probe into main
pull from: ci/gitea-validate-probe
merge into: jlab-cluster:main
Conversation 0 Commits 4 Files Changed 5
+425 -6
5 changed files with 425 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/renovate.yml
+108
View File
@@ -0,0 +1,108 @@
# Renovate, self-hosted (issue #225).
#
# On github.com Renovate arrived as the `jlab-renovate` GitHub App and this
# repo only ever carried renovate.json5. Gitea has no app model and no hosted
# Renovate, so after the move the bot has to be *run* by this repo: a scheduled
# job with its own token. That is the only change to how dependencies are
# updated - renovate.json5 stays the single source of what gets updated, and
# everything in it works unchanged on the Gitea platform.
#
# What the maintainer has to create once on the instance (the token this
# workflow cannot create for itself):
#
# RENOVATE_TOKEN Gitea access token of the bot account, scopes
# read:user, write:issue, write:repository. Renovate opens
# the PRs and the dependency dashboard issue with it, so
# the PRs will be authored by whoever owns the token.
# GITHUB_COM_TOKEN Read-only github.com PAT (public_repo / no scopes at all
# is enough). NOT optional here: every `github-releases` and
# `github-tags` lookup in renovate.json5 - and the whole
# built-in github-actions manager, which resolves
# actions/checkout and friends on github.com - hits
# github.com's unauthenticated rate limit within one run
# otherwise. On the App this came for free; here it does not.
# README already names this for the `no-result` symptom.
#
# There is deliberately no `allowedCommands` to set up by hand any more: the
# postUpgradeTasks hook renovate.json5 needs (ci/renovate-bump-chart.sh, which
# recomputes `version:` after an appVersion change) is allow-listed by the
# RENOVATE_ALLOWED_COMMANDS env below, because this workflow *is* the instance
# config now. Without it Renovate skips the task silently and every chart PR
# stays red on the version-bump check - see README, "The version-bump hook".
name: Renovate
on:
schedule:
# 03:40 UTC, once a day. Renovate's own `timezone` in renovate.json5
# (Europe/Berlin) governs the PR schedules; this cron is plain UTC.
- cron: "40 3 * * *"
workflow_dispatch:
inputs:
dryRun:
description: "Dry run: 'full' looks up and renders everything but writes nothing"
type: choice
default: "full"
options: ["full", "lookup", "extract", "disabled"]
logLevel:
description: "Renovate log level"
type: choice
default: "info"
options: ["info", "debug"]
jobs:
renovate:
name: Renovate
# The guard that keeps this workflow inert on github.com. The file reaches
# GitHub as well (that is where the migration PR is merged), and a
# scheduled run there would authenticate a *Gitea* token against a Gitea
# endpoint from a runner that has neither - failing nightly for a reason
# nobody is watching for. GITHUB_SERVER_URL is the one context value that
# differs between the two platforms without naming our instance: Gitea
# sets it to its own base URL. Delete this line once github.com is a
# push mirror without Actions.
if: github.server_url != 'https://github.com'
# No Docker socket needed: the Gitea runner starts every job in a container
# itself, and `container:` is the runner's job to create, not the job's.
# Measured on the probe PR of #225 - the same distinction that puts
# install-test on ubuntu-docker leaves this job on ubuntu-latest.
runs-on: ubuntu-latest
# The official image rather than `npx renovate`: it ships the git, node and
# package-manager versions Renovate expects, and pulling ~200 MB from npm
# on every scheduled run is the alternative. Renovate's built-in
# github-actions manager keeps this tag current (proved by dry run, see the
# migration PR) - it reads `container:` images, not only `uses:`.
container: ghcr.io/renovatebot/renovate:44.87.1
steps:
- name: "Checkout Repository"
uses: actions/checkout@v7
- name: "Run Renovate"
env:
# Platform. The endpoint is derived from the run itself so the
# instance address stays out of this repository, the same rule the
# publish workflow follows for the registry URL.
RENOVATE_PLATFORM: gitea
RENOVATE_ENDPOINT: ${{ github.server_url }}/api/v1
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
GITHUB_COM_TOKEN: ${{ secrets.GITHUB_COM_TOKEN }}
# Exactly this repository. Autodiscovery would enrol every repo the
# token can see, and the token is an account token, not an app
# installation scoped to one repo.
RENOVATE_AUTODISCOVER: "false"
RENOVATE_REPOSITORIES: ${{ github.repository }}
# Who the commits are by. The e-mail must not be a real address of a
# person - Gitea attributes a commit to the account whose verified
# address matches, and these commits are the bot's.
RENOVATE_GIT_AUTHOR: "Renovate Bot <renovate-bot@noreply.invalid>"
# The instance-level half of renovate.json5's postUpgradeTasks. A
# command not listed here is skipped WITHOUT an error message.
RENOVATE_ALLOWED_COMMANDS: '["^ci/renovate-bump-chart\\.sh"]'
# Manual runs default to a dry run; the schedule is the only thing
# that writes on its own.
RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dryRun || 'null' }}
LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.logLevel || 'info' }}
run: renovate
.github/workflows/validate-charts.yml
+60 -5
View File
@@ -199,11 +199,25 @@ jobs:
uses: actions/checkout@v7
- name: "Install Helm"
uses: azure/setup-helm@v5
# A plain release download rather than aquasecurity/setup-trivy, measured
# on the Gitea probe run of #225. That action runs actions/checkout
# against aquasecurity/trivy on github.com and hands it `github.token` -
# which on Gitea is a GITEA token. github.com answers 401, checkout
# retries twice and ends with "fatal: could not read Username for
# 'https://github.com': terminal prompts disabled", an error naming
# neither the action nor the token. Its actions/cache step is dead on the
# instance as well ("Cache action is only supported on GHES version >=
# 3.5"), and fetching the nested actions cost 4m30s before the job even
# started. One tarball behaves the same on both platforms, and the
# Renovate comment above still keeps TRIVY_VERSION current.
- name: "Install Trivy"
uses: aquasecurity/setup-trivy@v0.3.1
with:
version: ${{ env.TRIVY_VERSION }}
cache: true
run: |
curl -sSfL -o /tmp/trivy.tar.gz \
"https://github.com/aquasecurity/trivy/releases/download/${TRIVY_VERSION}/trivy_${TRIVY_VERSION#v}_Linux-64bit.tar.gz"
mkdir -p "$HOME/.local/bin"
tar -xzf /tmp/trivy.tar.gz -C "$HOME/.local/bin" trivy
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
"$HOME/.local/bin/trivy" --version
# Findings land in the job summary, not only in the log: exceptions with
# their reasons come from ci/trivy/trivyignore.yaml, the trusted-registry
# list from ci/trivy/data/ksv0125.yaml.
@@ -214,7 +228,23 @@ jobs:
name: "Install-test ${{ matrix.chart }}"
needs: [detect-changed-charts, build-chart]
if: needs.detect-changed-charts.outputs.any-install == 'true'
runs-on: ubuntu-latest
# NOT ubuntu-latest, and the one job in this repo that deviates. This is
# the only job that needs a Docker socket - it creates a k3d cluster, and
# k3d is a docker client. On GitHub's hosted runners every label carries
# Docker, so the distinction never came up; on the Gitea instance this repo
# is moving to (#225) the labels mean different machines: `ubuntu-latest`
# has no socket, `ubuntu-docker` has. Measured on the probe PR, not
# assumed - on ubuntu-latest the step "Create k3d cluster" fails with
# "Cannot connect to the Docker daemon at unix:///var/run/docker.sock",
# an error that points at k3d and never at the runner label. Every other
# job here (lint/package, the quantity render, the Trivy config scan) was
# measured green on ubuntu-latest and stays there.
#
# The label is unknown to GitHub Actions. While this repo still builds on
# github.com the job therefore stays queued - that is the intended state
# for the window in which CI is being handed over, and it is the reason
# this line must not be "corrected" back to ubuntu-latest.
runs-on: ${{ github.server_url == 'https://github.com' && 'ubuntu-latest' || 'ubuntu-docker' }}
# Blocking since Renovate automerges patch/minor updates: a chart that no
# longer starts must stop the PR before it reaches main and publishes. This
# ends the stabilization phase of issue #22 - the job reported failures
@@ -288,10 +318,35 @@ jobs:
: "${FREESAILARR_WG_KEY:?empty - check op://GitHub-Actions/freesailarr-mullvad-test-config/credential}"
: "${FREESAILARR_WG_ADDRESSES:?empty - check op://GitHub-Actions/freesailarr-mullvad-test-config/ip-address}"
echo "Both VPN values arrived non-empty."
- name: "PROBE runner facts (remove before merge)"
run: |
echo "labels/runner: ${RUNNER_NAME:-?} os=${RUNNER_OS:-?} arch=${RUNNER_ARCH:-?}"
echo "nproc: $(nproc)"
free -m || true
df -h / || true
echo "--- docker ---"
docker version || echo "NO DOCKER CLIENT/DAEMON"
docker info --format '{{.ServerVersion}} storage={{.Driver}} cpus={{.NCPU}} mem={{.MemTotal}}' || echo "docker info FAILED"
echo "--- toolchain ---"
node --version || echo "no node"
npx --version || echo "no npx"
python3 --version || echo "no python3"
git --version; curl --version | head -1
- name: "Install k3d"
run: curl -sSfL https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | TAG="$K3D_VERSION" bash
- name: "Create k3d cluster"
run: k3d cluster create install-test --image "$K3S_IMAGE" --wait --timeout 180s
- name: "PROBE why did k3s not come up (remove before merge)"
if: failure()
run: |
docker ps -a
echo "=== docker logs k3d-install-test-server-0 ==="
docker logs k3d-install-test-server-0 2>&1 | tail -100 || true
echo "=== docker info ==="
docker info || true
echo "=== host cgroup ==="
cat /proc/self/cgroup || true
ls /sys/fs/cgroup | head -30 || true
# Every chart reaching this step is one the exclusion filter let through,
# so the script's SKIP branch is never taken here - it exists for local
# runs. The variables below are empty for every chart but the one that
ci/migrate-issues-to-gitea.sh
Executable
+256
View File
@@ -0,0 +1,256 @@
#!/usr/bin/env bash
# Copy this repository's GitHub issues into the Gitea repository, over the API.
#
# Part of the move to JLab-Git (#225). The Gitea repo already exists and keeps
# its history, so Gitea's own "migrate from GitHub" is not available - it
# insists on creating the repository itself. What is left is this: read the
# issues from GitHub, write them to Gitea.
#
# WHAT THIS SCRIPT CANNOT DO - read this before running it
#
# Three losses are inherent to the API path, not bugs to be fixed later. They
# are named here because each of them is invisible in the result: the migrated
# issue looks complete, and only the detail is gone.
#
# 1. COMMENT AUTHORSHIP. Every issue and every comment is created by the
# token's user. Gitea has no "create as another user" outside the admin
# sudo API, and the GitHub accounts do not exist on the instance anyway.
# The script therefore prepends an attribution line to each body naming
# the original author and date. A thread of three people still reads as
# three people - but the avatar, the @mention and every "authored by"
# filter say the token's user wrote all of it.
#
# 2. CROSS-REFERENCES BETWEEN ISSUES. Gitea hands out numbers sequentially,
# and issues and pull requests share one counter on both systems, so
# GitHub's #218 does not become Gitea's #218. Every "#<n>" inside a body,
# a comment, a values.yaml, a NOTES.txt or a commit message then points at
# a different entry - silently, because a wrong number still renders as a
# valid link. This script does NOT rewrite them. It writes the mapping to
# --map-file (old<TAB>new, one per line) so the rewrite can be decided and
# done separately; #225 lists the options and none of them is decided.
#
# 3. ATTACHMENTS. Images and files in a body live behind GitHub's
# user-content CDN. The markdown is copied verbatim, so the links keep
# working exactly as long as the GitHub repo stays reachable - and break
# the day it does not. Re-uploading them is a separate job (download,
# POST to Gitea's attachment API, rewrite the link) and deliberately not
# part of this one.
#
# Smaller ones, for completeness: milestones, assignees, reactions, review
# threads and the closed pull requests are not migrated. Pull requests are
# skipped entirely - a PR without its branch and diff is a dead entry, and
# every branch it referred to is merged or gone.
#
# USAGE
#
# GITEA_URL=https://<instance> GITEA_TOKEN=<token> \
# ci/migrate-issues-to-gitea.sh --dry-run
#
# --state open|closed|all which issues (default: open)
# --limit N stop after N issues (a trial run on real data)
# --map-file PATH where old->new lands (default: issue-map.tsv)
# --source OWNER/REPO GitHub source (default: jk-cluster/helm-charts)
# --target OWNER/REPO Gitea target (default: jlab-cluster/helm-charts)
#
# The instance address comes from the environment, never from this file - the
# same rule the publish workflow follows for the registry URL.
#
# IDEMPOTENCE. The map file is also the guard: an issue already listed there is
# skipped. Delete the file and the next run creates everything a second time,
# so keep it next to the migration and do not regenerate it from scratch.
#
# REQUIREMENTS. `gh` authenticated against github.com (in this setup that means
# `op plugin run -- gh`, otherwise the identity falls back to the wrong user),
# plus `curl`, `jq` and a Gitea token with write access to issues on the target
# repo.
set -euo pipefail
DRY_RUN=0
STATE=open
LIMIT=0
MAP_FILE=issue-map.tsv
SOURCE_REPO=jk-cluster/helm-charts
TARGET_REPO=jlab-cluster/helm-charts
while [ $# -gt 0 ]; do
case "$1" in
--dry-run) DRY_RUN=1 ;;
--state) STATE="${2:?--state needs open|closed|all}"; shift ;;
--limit) LIMIT="${2:?--limit needs a number}"; shift ;;
--map-file) MAP_FILE="${2:?--map-file needs a path}"; shift ;;
--source) SOURCE_REPO="${2:?--source needs OWNER/REPO}"; shift ;;
--target) TARGET_REPO="${2:?--target needs OWNER/REPO}"; shift ;;
-h | --help) sed -n '2,66p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $1" >&2; exit 2 ;;
esac
shift
done
case "$STATE" in
open | closed | all) ;;
*) echo "--state must be open, closed or all (got: $STATE)" >&2; exit 2 ;;
esac
for tool in gh curl jq; do
command -v "$tool" >/dev/null 2>&1 || {
echo "missing required tool: $tool" >&2
exit 1
}
done
# The target address is required even for a dry run: a dry run that skips the
# address proves less than it appears to, and the run would then differ from
# the real one in more than the writes.
: "${GITEA_URL:?set GITEA_URL to the instance base URL, e.g. https://git.example.com}"
GITEA_URL="${GITEA_URL%/}"
if [ "$DRY_RUN" -eq 0 ]; then
: "${GITEA_TOKEN:?set GITEA_TOKEN to a token with issue write access on $TARGET_REPO}"
fi
API="$GITEA_URL/api/v1"
# The token goes into a curl config file rather than onto the command line, so
# it never appears in the process list - the same reason publish-helm-chart.yml
# pipes its options into `curl --config -`. Here it is a file and not stdin,
# because stdin carries the JSON body. mktemp creates it 0600.
CURL_CFG="$(mktemp)"
trap 'rm -f "$CURL_CFG"' EXIT
if [ "$DRY_RUN" -eq 0 ]; then
printf 'header = "Authorization: token %s"\n' "$GITEA_TOKEN" >"$CURL_CFG"
fi
# --fail-with-body turns 401/403/422 into a non-zero exit and still prints the
# response. Without it curl exits 0 on all of them and the migration would
# report success having written nothing.
gitea() {
local method="$1" path="$2"
if [ "$method" = GET ]; then
curl --config "$CURL_CFG" --silent --show-error --fail-with-body \
--request GET "$API/$path"
else
curl --config "$CURL_CFG" --silent --show-error --fail-with-body \
--request "$method" \
--header "Content-Type: application/json" \
--data-binary @- "$API/$path"
fi
}
say() { printf '%s\n' "$*"; }
[ "$DRY_RUN" -eq 1 ] && say "DRY RUN - nothing is written to $GITEA_URL"
say "source: github.com/$SOURCE_REPO target: $GITEA_URL/$TARGET_REPO"
say "state: $STATE map file: $MAP_FILE"
say ""
touch "$MAP_FILE"
# --- labels ------------------------------------------------------------------
#
# A label that does not exist on the target is dropped by Gitea's issue API
# without an error, so they are created on demand below. Names are compared
# case-sensitively, which is how both systems store them.
existing_labels=""
if [ "$DRY_RUN" -eq 0 ]; then
existing_labels="$(gitea GET "repos/$TARGET_REPO/labels?limit=100" | jq -r '.[].name')"
fi
# --- issues ------------------------------------------------------------------
#
# --paginate walks every page; PRs are filtered out here rather than by an API
# flag, because GitHub's issues endpoint returns them too and only the
# .pull_request key tells them apart. Oldest first, so the Gitea numbers come
# out in the same order as the originals even though the values differ.
issues_json="$(gh api --paginate \
-H "Accept: application/vnd.github+json" \
"repos/$SOURCE_REPO/issues?state=$STATE&per_page=100&sort=created&direction=asc" |
jq -s 'add | map(select(.pull_request == null))')"
total="$(jq 'length' <<<"$issues_json")"
say "found $total issue(s) on GitHub (pull requests excluded)"
say ""
count=0
while read -r number; do
[ -n "$number" ] || continue
if grep -qE "^${number} " "$MAP_FILE"; then
say "#$number already migrated (in $MAP_FILE) - skipped"
continue
fi
if [ "$LIMIT" -gt 0 ] && [ "$count" -ge "$LIMIT" ]; then
say "--limit $LIMIT reached, stopping"
break
fi
count=$((count + 1))
issue="$(jq --argjson n "$number" '.[] | select(.number == $n)' <<<"$issues_json")"
title="$(jq -r '.title' <<<"$issue")"
state="$(jq -r '.state' <<<"$issue")"
labels="$(jq -c '[.labels[].name]' <<<"$issue")"
# The provenance line is the only edit made to a body. It carries the old
# number, which is what makes an un-rewritten "#<n>" elsewhere traceable at
# all, and it names the original author because the API cannot.
body="$(jq -r '
"_Migriert von GitHub: " + .html_url + " - urspruenglich von @" + .user.login
+ " am " + (.created_at | split("T")[0]) + "._\n\n"
+ (.body // "")' <<<"$issue")"
say "#$number $title [$state, labels: $labels]"
if [ "$DRY_RUN" -eq 1 ]; then
comment_count="$(jq -r '.comments' <<<"$issue")"
suffix=""
[ "$state" = closed ] && suffix=", then close it"
say " would create the issue, $comment_count comment(s)$suffix"
continue
fi
# Create any label this issue needs and the target does not have yet.
while read -r label; do
[ -n "$label" ] || continue
if ! grep -qxF "$label" <<<"$existing_labels"; then
jq -n --arg n "$label" '{name: $n, color: "#ededed"}' |
gitea POST "repos/$TARGET_REPO/labels" >/dev/null
existing_labels="$existing_labels"$'\n'"$label"
say " created label: $label"
fi
done < <(jq -r '.[]' <<<"$labels")
created="$(jq -n --arg t "$title" --arg b "$body" --argjson l "$labels" \
'{title: $t, body: $b, labels: $l}' |
gitea POST "repos/$TARGET_REPO/issues")"
new_number="$(jq -r '.number' <<<"$created")"
say " -> $GITEA_URL/$TARGET_REPO/issues/$new_number"
# Comments in order. Each gets the same attribution header as the body; see
# loss (1) in the header of this file.
while read -r comment; do
[ -n "$comment" ] || continue
comment_body="$(jq -r '
"_@" + .user.login + " am " + (.created_at | split("T")[0])
+ " (migriert von GitHub)._\n\n" + (.body // "")' <<<"$comment")"
jq -n --arg b "$comment_body" '{body: $b}' |
gitea POST "repos/$TARGET_REPO/issues/$new_number/comments" >/dev/null
done < <(gh api --paginate -H "Accept: application/vnd.github+json" \
"repos/$SOURCE_REPO/issues/$number/comments?per_page=100" |
jq -s -c 'add // [] | .[]')
if [ "$state" = closed ]; then
jq -n '{state: "closed"}' |
gitea PATCH "repos/$TARGET_REPO/issues/$new_number" >/dev/null
say " closed"
fi
printf '%s\t%s\n' "$number" "$new_number" >>"$MAP_FILE"
done < <(jq -r '.[].number' <<<"$issues_json")
say ""
if [ "$DRY_RUN" -eq 1 ]; then
say "dry run finished - $count issue(s) would have been created"
else
say "finished - $count issue(s) created, mapping appended to $MAP_FILE"
say "the mapping is the input for the #<n> rewrite, which this script does not do"
fi
cyberchef/Chart.yaml
+1 -1
View File
@@ -5,7 +5,7 @@ icon: https://raw.githubusercontent.com/gchq/CyberChef/master/src/web/static/ima
type: application
version: 9.0.1+up11.4.0
version: 9.0.2+up11.4.0
appVersion: "11.4.0"
maintainers:
issue-map.tsv
View File
Whitespace-only changes.