You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Go to file
forest f53765f5e7 get rid of concourse pipelines 1 year ago
ansible comment out goatcounter vars_prompt 2 years ago
builds Make prom-collect generic 3 years ago
docs add "Debugging & OAuth Tokens" section to forge docs 3 years ago
element-web-custom-favicon add custom matrix favicon setup 2 years ago
howto add web application to 'how to invite a new user' page 3 years ago
rules change postgres offsite backup from critical to info 2 years ago Add new memberz 3 years ago

Cyberia operations handbook

This project provides guidance for those wonderful operations members who are technically "on the hook" for all Cyberia-related production services, especially

  • Capsul
  • Matrix
  • Forge
  • Jitsi (Cafe)
  • Mumble

Our current list of operations members is:

  • fack
  • forest
  • j3s
  • plantdaddy
  • skh
  • vvesley


For information on how to run ansible, see the in the ansible folder.


Operations provides around the clock shared support. We are all technically on-call 24/7/365.

Things to keep an eye on

All operations members must:

  • have a Matrix account and join and
  • have a Forge account


Check alerts on Prometheus

This handbook contains the above alert definitions, and they are pulled down into Prometheus automatically when updated.

You are welcome to adjust, add, and remove alerts


Grafana graphs are currently available here

Informational alerts

  severity: info

Informational alerts are free to be ignored, but indicate curious happenings in our infrastructure. Operations members are not expected to look at or react to these.

Critical alerts

  severity: critical

Critical alerts are those that indicate a failing of one or more of our services.

When a critical alert fires, we should respond with a sense of urgency.

If a critical alert fires and there was no associated outage, it is our shared responsibility to eliminate that alert. All critical alerts must be actionable. If a critical alert can be resolved by a cronjob, it should be resolved by a cronjob and removed as an alert.


Don't panic. If you are feeling overwhelmed, please contact another operations member about the issue at hand.


We will communicate in the Matrix channel. If that channel is down, we will communicate via

Communicate your activity. If you are bouncing a machine, please notify #ops that you are bouncing a machine. If you are reloading a service, tell #ops so that people don't step on each others toes. This is also important to establish a timeline.