aboutsummaryrefslogtreecommitdiff

Cyberia operations handbook

This project provides guidance for those wonderful operations members who are technically "on the hook" for all Cyberia-related production services, especially

  • Capsul
  • Matrix
  • Forge
  • Jitsi

Our current list of operations members is:

  • j3s
  • skh
  • aimless
  • cassact
  • plantdaddy

On-Call

Operations provides around the clock shared support. We are all technically on-call 24/7/365.

Things to keep an eye on

All operations members must:

  • have a Matrix account and join #ops:cyberia.club and #services:cyberia.club
  • have a Forge account
  • subscribe to the ops mailing list
  • subscribe to the ops todo tracker

Alerts

Check alerts on Prometheus

This handbook contains the above alert definitions, and they are pulled down into Prometheus automatically when updated.

You are welcome to adjust, add, and remove alerts

Graphs

Grafana graphs are currently available here

Informational alerts

labels:
  severity: info

Informational alerts are free to be ignored, but indicate curious happenings in our infrastructure. Operations members are not expected to look at or react to these.

Critical alerts

labels:
  severity: critical

Critical alerts are those that indicate a failing of one or more of our services.

When a critical alert fires, we should respond with a sense of urgency.

If a critical alert fires and there was no associated outage, it is our shared responsibility to eliminate that alert. All critical alerts must be actionable. If a critical alert can be resolved by a cronjob, it should be resolved by a cronjob and removed as an alert.

Incidents

Don't panic. If you are feeling overwhelmed, please contact another operations member about the issue at hand.

Communication

We will communicate in the #ops:cyberia.club Matrix channel. If that channel is down, we will communicate via cafe.cyberia.club/ops

Communicate your activity. If you are bouncing a machine, please notify #ops that you are bouncing a machine. If you are reloading a service, tell #ops so that people don't step on each others toes. This is also important to establish a timeline.

References

Capsul

Prometheus

Matrix

Email

Forge

Git