Go to file
forest fb80e7cb91 try adding nginx to monitoringservers playbook .. ? 2024-02-06 17:16:53 -06:00
ansible try adding nginx to monitoringservers playbook .. ? 2024-02-06 17:16:53 -06:00
builds update submit-builds 2023-01-04 02:44:53 +00:00
docs add "Debugging & OAuth Tokens" section to forge docs 2021-04-22 10:06:58 -05:00
element-web-custom-favicon add custom matrix favicon setup 2021-09-19 17:38:42 -05:00
howto changed docs from riot to element 2022-09-28 11:55:31 -05:00
rules what;jegrwg 2023-05-31 22:18:45 -07:00
README.md changed docs from riot to element 2022-09-28 11:55:31 -05:00


Cyberia operations handbook

This project provides guidance for those wonderful operations members who are technically "on the hook" for all Cyberia-related production services, especially

  • Capsul
  • Matrix
  • Forge
  • Jitsi (Cafe)
  • Mumble

Our current list of operations members is:

  • fack
  • forest
  • j3s
  • plantdaddy
  • skh
  • vvesley


For information on how to run ansible, see the README.md in the ansible folder.


Operations provides around the clock shared support. We are all technically on-call 24/7/365.

Things to keep an eye on

All operations members must:

  • have a Matrix account and join #ops:cyberia.club and #services:cyberia.club
  • have a Forge account


Check alerts on Prometheus

This handbook contains the above alert definitions, and they are pulled down into Prometheus automatically when updated.

You are welcome to adjust, add, and remove alerts


Grafana graphs are currently available here

Informational alerts

  severity: info

Informational alerts are free to be ignored, but indicate curious happenings in our infrastructure. Operations members are not expected to look at or react to these.

Critical alerts

  severity: critical

Critical alerts are those that indicate a failing of one or more of our services.

When a critical alert fires, we should respond with a sense of urgency.

If a critical alert fires and there was no associated outage, it is our shared responsibility to eliminate that alert. All critical alerts must be actionable. If a critical alert can be resolved by a cronjob, it should be resolved by a cronjob and removed as an alert.


Don't panic. If you are feeling overwhelmed, please contact another operations member about the issue at hand.


We will communicate in the #ops:cyberia.club Matrix channel. If that channel is down, we will communicate via cafe.cyberia.club/ops

Communicate your activity. If you are bouncing a machine, please notify #ops that you are bouncing a machine. If you are reloading a service, tell #ops so that people don't step on each others toes. This is also important to establish a timeline.