cyberia/capsul-flask
9
6
Fork 8
Code Issues 21 Pull requests 5 Projects Releases Wiki Activity

Capsul outage mitigation: need a way to shutdown the server #13

New issue
Open
opened 2021-09-18 07:40:43 +00:00 by j3s · 3 comments
j3s commented 2021-09-18 07:40:43 +00:00
Owner
Copy link

Can we change the default behaviour we observed during the incident? (it tries to shut down the Vms 1 by 1 with a long timeout).

Like, could we make it shut them off 10 at a time with a much shorter timeout?

Maybe we monkey-patch all the default shutdown commands to insert our own vm-halting logic? This is related to the ACPI gracefull shutdown of VMs task.

Can we change the default behaviour we observed during the incident? (it tries to shut down the Vms 1 by 1 with a long timeout). Like, could we make it shut them off 10 at a time with a much shorter timeout? Maybe we monkey-patch all the default shutdown commands to insert our own vm-halting logic? This is related to the ACPI gracefull shutdown of VMs task.
forest commented 2021-12-16 17:13:57 +00:00
Owner
Copy link

forest (he/him)
we should also implement #13

Capsul outage mitigation: need a way to shutdown the server
How do we wanna approach that? My 1st thought was to add a button to the admin panel that stops all VMs
Obviously modifying the default behaviour would be better because then you can't forget to shut them down -- it's built in

f0x
you can (partly) override an existing systemd service
adding conf files in /etc/systemd/system/.service.d/ apparently

techieb0y
'drop-ins' is the name of that feature, it's kinda handy

Set up systemd drop in:

mkdir -p /etc/systemd/system/libvirt-guests.service.d/
echo '
# NOTE this is a systemd "drop-in" which modifies the ExecStart and ExecStop properties of the existing libvirt-guests
 service unit 

[Service]
ExecStart=/opt/capsul-flask/capsulflask/shell_scripts/libvirt-guests.sh start
ExecStop=/opt/capsul-flask/capsulflask/shell_scripts/libvirt-guests.sh stop
' > /etc/systemd/system/libvirt-guests.service.d/libvirt-guests.conf

create modified version of default libvirt-guests script:

cp /usr/lib/libvirt/libvirt-guests.sh /opt/capsul-flask/capsulflask/shell_scripts/libvirt-guests.sh
and then edit /opt/capsul-flask/capsulflask/shell_scripts/libvirt-guests.sh to do virsh destroy instead of virsh shutdown
> **forest (he/him)** > we should also implement https://git.cyberia.club/cyberia/capsul-flask/issues/13 > > Capsul outage mitigation: need a way to shutdown the server > How do we wanna approach that? My 1st thought was to add a button to the admin panel that stops all VMs > Obviously modifying the default behaviour would be better because then you can't forget to shut them down -- it's built in > **f0x** > you can (partly) override an existing systemd service > adding conf files in /etc/systemd/system/<unit-name>.service.d/ apparently > **techieb0y** > 'drop-ins' is the name of that feature, it's kinda handy Set up systemd drop in: ``` mkdir -p /etc/systemd/system/libvirt-guests.service.d/ echo ' # NOTE this is a systemd "drop-in" which modifies the ExecStart and ExecStop properties of the existing libvirt-guests service unit [Service] ExecStart=/opt/capsul-flask/capsulflask/shell_scripts/libvirt-guests.sh start ExecStop=/opt/capsul-flask/capsulflask/shell_scripts/libvirt-guests.sh stop ' > /etc/systemd/system/libvirt-guests.service.d/libvirt-guests.conf ``` create modified version of default libvirt-guests script: ``` cp /usr/lib/libvirt/libvirt-guests.sh /opt/capsul-flask/capsulflask/shell_scripts/libvirt-guests.sh and then edit /opt/capsul-flask/capsulflask/shell_scripts/libvirt-guests.sh to do virsh destroy instead of virsh shutdown ```
forest commented 2021-12-16 17:15:26 +00:00
Owner
Copy link

^ that should be part of our deployment plan for the next deployment

^ that should be part of our deployment plan for the next deployment
forest commented 2021-12-17 01:36:21 +00:00
Owner
Copy link

We tried to deploy the systemd drop-in for fixing the shutdown process of libvirt-guests, but it was still calling the old script not our new one.. systemctl status was showing the drop in, but it wasn't actually overriding the ExecStop script we had specified.

We tried to deploy the systemd drop-in for fixing the shutdown process of libvirt-guests, but it was still calling the old script not our new one.. systemctl status was showing the drop in, but it wasn't actually overriding the ExecStop script we had specified.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
rathouse
self-service-change-email
development
jes/add-nixos
docker-test
serial-console-access
ptyhunter
console-access
dhcp-wip
j3s/update-distros
deleteme
autonomic
autonomic-yolocolo
tests-with-logs
tests3
tests2
autonomic-webapi
docs-reshuffle2
mock-hub-create
optional-btcpay
generated-prices
tests
autonomic-public-api-tests
autonomic-docker-api
managed-ips
asd
multiple-hosts
forest-wip
0.1.0
0.0.0
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: cyberia/capsul-flask#13
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?