| ansible | ||
| builds | ||
| docs | ||
| element-web-custom-favicon | ||
| howto | ||
| rules | ||
| README.md | ||
Cyberia operations handbook
This project provides guidance for those wonderful operations members who are technically "on the hook" for all Cyberia-related production services, especially
- Capsul
- Matrix
- Forge
- Jitsi (Cafe)
- Mumble
Our current list of operations members is:
- fack
- forest
- j3s
- plantdaddy
- skh
- vvesley
Ansible
For information on how to run ansible, see the README.md in the ansible folder.
On-Call
Operations provides around the clock shared support. We are all technically on-call 24/7/365.
Things to keep an eye on
All operations members must:
- have a Matrix account and join #ops:cyberia.club and #services:cyberia.club
- have a Forge account
- subscribe to the ops mailing list
- subscribe to the ops todo tracker
Alerts
Check alerts on Prometheus
This handbook contains the above alert definitions, and they are pulled down into Prometheus automatically when updated.
You are welcome to adjust, add, and remove alerts
Graphs
Grafana graphs are currently available here
Informational alerts
labels:
severity: info
Informational alerts are free to be ignored, but indicate curious happenings in our infrastructure. Operations members are not expected to look at or react to these.
Critical alerts
labels:
severity: critical
Critical alerts are those that indicate a failing of one or more of our services.
When a critical alert fires, we should respond with a sense of urgency.
If a critical alert fires and there was no associated outage, it is our shared responsibility to eliminate that alert. All critical alerts must be actionable. If a critical alert can be resolved by a cronjob, it should be resolved by a cronjob and removed as an alert.
Incidents
Don't panic. If you are feeling overwhelmed, please contact another operations member about the issue at hand.
Communication
We will communicate in the #ops:cyberia.club Matrix channel. If that channel is down, we will communicate via cafe.cyberia.club/ops
Communicate your activity. If you are bouncing a machine, please notify #ops that you are bouncing a machine. If you are reloading a service, tell #ops so that people don't step on each others toes. This is also important to establish a timeline.