h+h lab cloud server

Small hosting for non-critical websites:

1. h+h lab landing
2. Development preview of Slow Reader
3. Browserslist REPL
4. Sitnik personal website
5. Logux website
6. Nano Stores website
7. Self-hosted Plausible Analytics
8. Grafana with Slow Reader production charts

Stack:

• Ubuntu 26.04 LTS + Canonical Livepatch
• Ansible
• Rootless Podman/Quadlet for each service
• gVisor for the pull request previews
• Caddy web server and load balancer

Goals

1. Automatic updates for low maintenance.
2. Accept downtime, but keep it short.
3. No backups or duplication needed.

Deploy Changes

Deploy, entering the server user’s password when prompted:

ansible-playbook site.yml --user ai --ask-become-pass

The playbook is idempotent and preserves the container serving each domain.

Services

All services run under systemd with auto-restart.

Each website has a separate user and can run multiple apps. Each app has its own image, containers, domain, and deployment.

Web Service

Web services use public Podman images. Containers are read-only, have no capabilities, and limit memory, CPU, and processes to protect the server.

Images should define a HEALTHCHECK (or custom health_cmd for OCI images). Podman kills unresponsive containers; systemd restarts them. Old deployment images are removed nightly.

After publishing an image:

1. GitHub Actions sends an HTTP request; we verify its GitHub origin and repository.
2. Podman pulls and starts the image, checks its health, and switches the domain to it.
3. The API waits for deployment and returns its log. Broken images fail the workflow.

Database and Tooling

Databases and tools like Redis also run in Podman, with rolling tags such as :9 updated automatically by Podman tools.

Each database belongs to one website and exposes no host port. It listens only on a private Podman network inside that user’s network namespace.

Analytics

Plausible Community Edition runs as three containers of the plausible user: Plausible itself, PostgreSQL for accounts and settings, and ClickHouse for the events. The databases expose no host port, listen only on a private Podman network, and have generated passwords.

All three images use rolling tags and update automatically. Plausible migrates both databases on every start. PostgreSQL and ClickHouse versions follow the Docker Compose file of Plausible; check it before changing them.

Dashboards

Grafana runs as one container of the grafana user, with SQLite in a volume. Its image uses the latest tag and updates automatically.

It reads VictoriaMetrics on the Slow Reader server with the built-in Prometheus data source: Slow Reader Server for server metrics and Slow Reader App for users and events.

Alerts go to Telegram (dashboards_telegram_token and dashboards_telegram_chat). Rules are in rules.alerting.yml.j2.

Pull Request Previews

A pull request can run a single image on its own subdomain, such as preview-42.slowreader.hplush.dev.

Because pull request code is unreviewed, previews have:

• A separate user, no database, no shared network, and no host access.
• A shared memory limit for all previews.
• Routes written by the root preview-route wrapper, never the preview user. The wrapper validates the PR number and port, then uses its own Caddy template.
• gVisor, a userspace kernel, to handle syscalls. Escaping a regular container can require only a host kernel bug; previews require a gVisor bug as well.

Network syscalls use the host kernel within each preview’s own network namespace, not the host’s network namespace. The user’s firewall rules still apply. This is needed because gVisor’s network stack cannot use pasta’s tap device, which would leave published ports unreachable.

gVisor costs memory and network throughput. Websites and databases keep the default runtime because we build their images ourselves.

Previews stop when their PR closes. A daily timer also removes previews not redeployed for max_days days (default: 30), limiting their lifetime even if the cleanup workflow fails.

Internal Web API

The deploy API is a custom Node.js HTTP server. Its source files live on the server and run in an automatically updated Node.js image.

Files

• inventory.yml: server address and SSH account.
• requirements.txt: Ansible CLI versions.
• requirements.yml: Ansible collection versions.
• .vault-pass: Ansible Vault password; create locally, excluded from Git.
• group_vars/all.yml: server-wide settings.
• websites/: one config per website, named after its domain.
• previews/: one config per preview type, named after its parent domain.
• site.yml: playbook calling all roles.
• roles/base/: updates, Livepatch, fail2ban, firewall, swap, Podman, gVisor, users, journal limits, and daily image cleanup.
• roles/caddy/: Caddy and domain configs.
• roles/api/: internal web API for GitHub Actions.
• roles/analytics/: Plausible Analytics with PostgreSQL and ClickHouse.
• roles/dashboards/: Grafana and its dashboards.
• roles/web/: website user, two containers, and deploy script.

Prepare the Server

1. Create a cheap server with 4 GB memory and Ubuntu 26.04 LTS.

2. Create firewall rules:

   • Public: ICMP, TCP 80, TCP 443, UDP 443 (HTTP/3 QUIC)
   • Admin IP only: TCP 22

3. Add A and AAAA DNS records for hplush.dev.

4. Add CNAME records for cloud, api.cloud, lens, and dashboard pointing to hplush.dev.

5. Update the system:

   ssh root@cloud.hplush.dev
   apt update && apt upgrade -y
   sudo reboot now

6. Create the admin user:

   ssh root@cloud.hplush.dev
   adduser ai
   usermod -aG sudo ai
   mkdir -p /home/ai/.ssh
   cp /root/.ssh/authorized_keys /home/ai/.ssh/
   chown -R ai:ai /home/ai/.ssh
   chmod 700 /home/ai/.ssh
   chmod 600 /home/ai/.ssh/authorized_keys
   exit

7. Create known_hosts:

   ssh-keyscan cloud.hplush.dev > known_hosts
   ssh-keygen -lf known_hosts

8. Generate the Ansible Vault password in .vault-pass:

   pnpm dlx nanoid --size 32 > .vault-pass
   chmod 600 .vault-pass

9. Encrypt the Ubuntu Pro token and add the output to group_vars/all.yml:

   ansible-vault encrypt_string --name ubuntu_pro_token 'YOUR_TOKEN'

10. Deploy:

    ansible-playbook site.yml --user ai --ask-become-pass

11. Right after the first deploy, open lens.hplush.dev and register the Plausible admin account.

12. Open dashboard.hplush.dev and log in as ai with the password from the server:

    sudo cat /home/grafana/.local/state/secrets/admin-password

Add a Website

1. Copy websites/hplush.dev.yml to websites/YOUR_DOMAIN.yml.
2. Set the user, image, allowed GitHub repository/workflow/branch, container port, and a free pair of host ports.
3. Point the domain’s A and AAAA records to the server and deploy changes.

Default container limits: 512 MB memory, one CPU, and 512 processes.

Caddy requests a Let's Encrypt certificate on the first request; HTTPS requires working DNS.

Deploy a Website from GitHub Actions

After publishing an image, get a GitHub OIDC token and call the deploy API. Each app deploys independently through its domain endpoint, which accepts only its configured workflow:

permissions:
  id-token: write
concurrency:
  group: deploy-hplush.dev
  cancel-in-progress: false
steps:
  # Some steps of preparing the image
  - name: Deploy image
    uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
    with:
      script: |
        let token = await core.getIDToken('https://api.cloud.hplush.dev')
        let response = await fetch(
          'https://api.cloud.hplush.dev/deploy/hplush.dev',
          { method: 'POST', headers: { authorization: `Bearer ${token}` } }
        )
        let answer = await response.text()
        if (response.ok) core.info(answer)
        else core.setFailed(`${response.status} ${response.statusText}: ${answer}`)

See full example.

Add Pull Request Previews

1. Copy previews/slowreader.hplush.dev.yml and configure it.
2. Point wildcard A and AAAA records for *.YOUR_DOMAIN to the server.
3. Create a deploy-preview label; apply it to PRs only after basic review.
4. Use a pull_request workflow without permissions to build a Docker image and upload it as an artifact.
5. Use a workflow_run workflow to download the artifact, push it with a preview tag, and request deployment. The server verifies the workflow:

# Download artifact from pull_request workflow and push it with tag
- name: Deploy the preview
  uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
  with:
    script: |
      let token = await core.getIDToken('https://api.cloud.hplush.dev')
      let response = await fetch(
        `https://api.cloud.hplush.dev/deploy/preview-${process.env.PR}.slowreader.hplush.dev`,
        { method: 'POST', headers: { authorization: `Bearer ${token}` } }
      )
      let answer = await response.text()
      if (response.ok) core.info(answer)
      else core.setFailed(`${response.status} ${response.statusText}: ${answer}`)

On PR close, a pull_request workflow with type closed triggers a workflow_run workflow that sends DELETE to the same endpoint:

- name: Clean the preview
  uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
  with:
    script: |
      let token = await core.getIDToken('https://api.cloud.hplush.dev')
      let response = await fetch(
        `https://api.cloud.hplush.dev/deploy/preview-${process.env.PR}.slowreader.hplush.dev`,
        { method: 'DELETE', headers: { authorization: `Bearer ${token}` } }
      )
      let answer = await response.text()
      if (response.ok) core.info(answer)
      else core.setFailed(`${response.status} ${response.statusText}: ${answer}`)

Both requests wait for the result; a preview startup failure fails the workflow.

See examples:

1. preview-prepare.yml
2. preview-deploy.yml
3. preview-close.yml
4. preview-clean.yml

Analytics Accounts

Only the first account, which you register after the first deploy, needs no invitation.

Plausible has no email, so an invited person gets no link. Invite them on the site’s “People” settings page, then get the link from the database:

sudo -u plausible podman exec plausible-db psql -U plausible -c \
  'select invitation_id from guest_invitations'

Send them https://lens.hplush.dev/register/invitation/INVITATION_ID.

Reset a forgotten password from the server:

sudo -u plausible podman exec -it plausible /app/bin/plausible remote

Plausible.Repo.get_by!(Plausible.Auth.User, email: "EMAIL")
|> Plausible.Auth.User.set_password("NEW_PASSWORD")
|> Plausible.Repo.update!()

Maintenance

Automatic updates:

• unattended-upgrades installs packages nightly.
• needrestart restarts services using old libraries.
• Livepatch patches the running kernel.
• Every day at 04:00 UTC the server reboots for a new kernel, but only if Livepatch can’t protect the running kernel.

To reboot earlier:

ssh ai@cloud.hplush.dev
sudo needrestart -r l
sudo reboot # if needed

If the kernel needs no reboot, restart everything listed, including user managers. Websites will be down for a few seconds:

sudo needrestart -b -r l | awk -F': ' '/^NEEDRESTART-SVC/{print $2} /^NEEDRESTART-SESS/{split($2,u," ");c="id -u "u[1];c|getline i;close(c);print "user@"i".service"}' | sort -u | xargs -r sudo systemctl restart

Debug

Find failed services:

systemctl --failed

Services run as separate users; read their logs in the system journal:

sudo journalctl -u caddy
sudo journalctl _SYSTEMD_USER_UNIT=api.service
sudo journalctl _SYSTEMD_USER_UNIT=deploy-hplush.service
sudo journalctl _SYSTEMD_USER_UNIT=hplush-blue.service
sudo journalctl _SYSTEMD_USER_UNIT=slowreader-db.service
sudo journalctl _SYSTEMD_USER_UNIT=preview-42.service
sudo journalctl _SYSTEMD_USER_UNIT=plausible.service
sudo journalctl _SYSTEMD_USER_UNIT=plausible-events.service
sudo journalctl _SYSTEMD_USER_UNIT=grafana.service

Units use app names: slowreader-app has slowreader-app-blue.service and deploy-slowreader-app.service.

Open a website’s database:

sudo -u slowreader podman exec -it slowreader-db psql -U slowreader

Open the databases of Plausible:

sudo -u plausible podman exec -it plausible-db psql -U plausible
sudo -u plausible podman exec -it plausible-events clickhouse-client --database plausible_events_db --ask-password

Their passwords are in /home/plausible/.local/state/secrets/.

Reset the Grafana password:

read -rs PASSWORD
printf %s "$PASSWORD" | sudo -u grafana podman exec -i grafana \
  grafana cli admin reset-admin-password --password-from-stdin
unset PASSWORD

Deploy manually by creating a request file:

sudo touch /var/lib/deploy/hplush.dev/requests/manual

Start or stop a preview manually:

echo 'deploy 42' | sudo tee /var/lib/deploy/previews/slowreader.hplush.dev/requests/manual
echo 'clean 42' | sudo tee /var/lib/deploy/previews/slowreader.hplush.dev/requests/manual

The deploy script removes the request file, writes the result to /var/lib/deploy/hplush.dev/results/manual, and logs it to the journal.