Merge pull request 'post/blog-upgrade' (#5) from post/blog-upgrade into main

Reviewed-on: #5

.woodpecker/build.yml

@@ -1,6 +1,6 @@
-pipeline:
+steps:
   build:
-    image: python:3.11-slim
+    image: python:3.12-slim
     commands:
       - apt-get update
       - apt-get install -y --no-install-recommends build-essential gcc make rsync openssh-client && apt-mark auto build-essential gcc make rsync openssh-client

Dockerfile

@@ -1,5 +1,5 @@
 
-FROM python:3.11-slim
+FROM python:3.12-slim
 
 RUN apt-get update
 RUN apt-get install -y --no-install-recommends build-essential gcc make && apt-mark auto build-essential gcc make

content/Blog_Upgrade.md

@@ -0,0 +1,203 @@
+Title: Upgrading my Gitea/Woodpecker Setup
+Date: 2024-03-16 09:52
+Category: Devops
+Tags: cicd,til
+Slug: ciupgrade
+Authors: Okusanya David
+Summary: About upgrading my gitea/woodpecker setup
+
+### OUTLINE
+
+* [Reasons](#upgrade-gitea-cicd-reasons)
+* [Plan of action](#upgrade-gitea-cicd-plan)
+*     [Gitea](#upgrade-gitea-cicd-gitea)
+*     [Woodpecker](#upgrade-gitea-cicd-woodpecker)
+* [Observations](#upgrade-gitea-cicd-observations)
+
+### REASONS {#upgrade-gitea-cicd-reasons}
+
+So I am using [gitea-1.19.4](https://docs.gitea.com/1.19/category/installation) as well as [Woodpecker-0.15.x](https://woodpecker-ci.org/docs/0.15/intro). The current version(s) of gitea and woodpecker are 1.21.x and 2.3.x respectively. 
+
+I wanted to upgrade to these versions because of:  
+
+(a) Gitea 1.21.x comes with [Gitea Actions](https://docs.gitea.com/usage/actions/overview) enabled by default. THis is similar to [Github Actions](https://github.com/features/actions) which I want to learn as we use that at $DAYJOB.
+
+(b) I want to upgrade to a new version every year. This is to make upgrades manageable and repeatable so by doing it more frequently, I would have a runbook for how to upgrade.
+
+(c) These versions (1.19.x, 0.x.x) are no longer actively maintained, which could mean that they are stable but not receiving security fixes which is something I want to avoid.
+
+(d) Woodpecker 2.3.x adds the ability to use [Forego](https://forgejo.org/) which is a community fork of Gitea and runs [Codeberg](https://codeberg.org/). I may switch to Forego in the future as I may want to hack on it and contribute back to it.
+
+### PLAN OF ACTION {#upgrade-gitea-cicd-plan}
+
+Just persuing the [Gitea changelong](https://github.com/go-gitea/gitea/blob/main/CHANGELOG.md) and [Woodpecker changelog](https://github.com/woodpecker-ci/woodpecker/blob/main/CHANGELOG.md) gives me the sense that a lot has changed between versions. This might mean that the upgrade might be seamless or not. So my plan of action is this:
+
+(a) Copy over my data from the two instances to my local computer
+
+(b) Upgrade Gitea. (Ideally, I would have loved to resize the instances but I am too lazy to do so)
+
+(c) Upgrade Woodpecker (Ideally, I would have loved to resize the instances but I am too lazy to do so)
+
+(d) Test, If botched, then recreate new instances (what a pain in the ass, that would be)
+
+#### GITEA {#upgrade-gitea-cicd-gitea}
+
+For the Gitea instance, I would mainly be upgrading to 1.21.x and with the Woodpecker instance, I would be upgrading to 2.2.x. For Gitea,you would:
+
+- Backup database
+- Backup Gitea config
+- Backup Gitea data files in APP_DATA_PATH
+- Backup Gitea external storage (eg: S3/MinIO or other storages if used)
+
+The steps I have used are taken from [here](https://github.com/go-gitea/gitea/blob/release/v1.21/contrib/upgrade.sh) and [here](https://archive.is/hwRvo).
+
+
+(a) Define some environment variables
+
+```sh
+$ export GITEABIN="/usr/local/bin/gitea"
+$ export GITEACONF="/etc/gitea/app.ini"
+$ export GITEAUSER="git"
+$ export GITEAWORKPATH="/var/lib/gitea"
+```
+
+(a) Get current version
+
+```sh
+$ sudo --user "${GITEAUSER}" "${GITEABIN}" --config "${GITEACONF}" --work-path "${GITEAWORKPATH}" --version | cut -d ' ' -f 3
+```
+
+(b) Get gitea version to install
+
+```sh
+$ export GITEAVERSION=$(curl --connect-timeout 10 -sL https://dl.gitea.com/gitea/version.json | jq -r .latest.version)
+```
+
+(c) Download version from (b) and uncompress
+
+```sh  
+$ binname="gitea-${GITEAVERSION}-linux-amd64"
+$ binurl="https://dl.gitea.com/gitea/${giteaversion}/${binname}.xz"
+$ curl --connect-timeout 10 --silent --show-error --fail --location -O "$binurl{,.sha256,.asc}"
+$ sha256sum -c "${binname}.xz.sha256"
+$ rm "${binname}".xz.{sha256,asc}
+$ xz --decompress --force "${binname}.xz"
+```
+
+(d) Flush Gitea queues so that you can properly backup
+
+```sh
+$ sudo --user "${GITEAUSER}" "${GITEABIN}" --config "${GITEACONF}" --work-path "${GITEAWORKPATH}" manager flush-queues
+```
+
+(e) Stop gitea
+
+```sh
+$ sudo systemctl stop gitea
+```
+
+(f) Dump database using `pg_dump`
+
+```sh
+$ pg_dump -U $USER $DATABASE > gitea-db.sql
+```
+
+(g) Dump the gitea config and files
+
+```sh
+$ mkdir temp
+$ chmod 777 temp # necessary otherwise the dump fails
+$ cd temp
+$ sudo --user "${GITEAUSER}" "${GITEABIN}" --config "${GITEACONF}" --work-path "${GITEAWORKPATH}" dump --verbose
+```
+
+(h) Move the downloaded gitea binary to the new location
+
+```sh
+$ sudo cp -f "${GITEABIN}" "${GITEABIN}.bak"
+$ sudo mv -f "${binname}" "${GITEABIN}"
+```
+
+(i) Restart Gitea
+
+```sh
+$ sudo chmod +x "${GITEABIN}"
+$ sudo systemctl start gitea
+$ sudo systemctl status gitea
+```
+
+To restore as taken from [here](https://docs.gitea.com/administration/backup-and-restore)
+```sh
+$ sudo systemctl stop gitea
+$ sudo cp -f "${GITEABIN}.bak" "${GITEABIN}"
+$ cd temp
+$ unzip gitea-dump-*.zip
+$ cd gitea-dump-*
+$ sudo mv app.ini "${GITEACONF}"
+$ sudo mv data/* "${GITEAWORKPATH}/data/"
+$ sudo mv log/* "${GITEAWORKPATH}/log/"
+$ sudo mv repos/* "${GITEAWORKPATH}/gitea-repositories/"
+$ sudo chown -R gitea:gitea /etc/gitea/conf/app.ini "${GITEAWORKPATH}"
+$ cp ../gitea-db.sql .
+$ psql -U $USER -d $DATABASE < gitea-db.sql
+$ sudo systemctl start gitea
+$ sudo systemctl stop gitea
+# Regenerate repo hooks
+$ sudo --user "${GITEAUSER}" "${GITEABIN}" --config "${GITEACONF}" --work-path "${GITEAWORKPATH}" admin regenerate hooks
+```
+
+#### WOODPECKER {#upgrade-gitea-cicd-woodpecker}
+
+(a) Stop agent and server
+
+```sh
+$ sudo systemctl stop woodpecker
+$ sudo systemctl stop woodpecker-agent
+```
+
+(b) Backup database with `pg_dump`
+
+```sh
+$ pg_dump -U woodpecker woodpeckerdb > woodpecker-db.sql
+$ pg_dump -U $USER $DATABASE > woodpecker-db.sql
+$ sudo cp /etc/woodpecker-agent.conf woodpecker-agent.conf.bak
+$ sudo cp /etc/woodpecker.conf woodpecker.conf.bak
+```
+
+(c) Download the binaries
+
+```sh
+$ woodpeckerversion=2.3.0
+$ binurl="https://github.com/woodpecker-ci/woodpecker/releases/download/v${woodpeckerversion}/"
+$ curl --connect-timeout 10 --silent --show-error --fail --location -O "${binurl}woodpecker{,-agent,-cli,-server}_${woodpeckerversion}_amd64.deb"
+$ ls
+woodpecker-cli_2.3.0_amd64.deb 
+woodpecker-agent_2.3.0_amd64.deb
+woodpecker-server_2.3.0_amd64.deb
+```
+
+(d) Install the new ones
+
+```sh
+$ sudo dpkg -i ./woodpecker-*.deb
+```
+
+(e) Restart woodpecker
+
+```sh
+$ sudo systemctl start woodpecker
+$ sudo systemctl start woodpecker-agent
+```
+
+(f) Delete back up files
+
+```sh
+$ sudo rm /etc/woodpecker-agent.conf.bak /etc/woodpecker.conf.bak
+```
+
+Unfortunately I did not find any restore functions for woodpecker. SO I did it blindly.
+
+
+### OBSERVATIONS {#upgrade-gitea-cicd-observations}
+
+On the whole, the upgrade of Gitea was very smooth as there was an ability to restore if I failed. Both softwares though was easy to upgrade. I initially panicked during the Woodpecker upgrade after receiving errors like [this](https://github.com/woodpecker-ci/woodpecker/blob/main/cmd/agent/core/config.go#L74). After I read [this](https://github.com/woodpecker-ci/woodpecker/blob/5d98a717acf386477f7b676aac5e78005d5c4c4a/cmd/agent/core/flags.go#L55-L59), I saw that those errors did not affect the running of my Woodpecker instance.