From 48435cbbea55bdf03e0d49159f0789a5bcb51ea6 Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 00:25:30 -0600 Subject: [PATCH 01/20] blog update: dec 2021 capsul maintenance --- site/blog.txt | 2 + ...l-rollin-onward-with-a-web-application.txt | 4 +- .../20211217-capsul-maintenance-updates.txt | 273 ++++++++++++++++++ site/blog/blog.xml | 10 +- 4 files changed, 286 insertions(+), 3 deletions(-) create mode 100644 site/blog/20211217-capsul-maintenance-updates.txt diff --git a/site/blog.txt b/site/blog.txt index f908903..71e5915 100644 --- a/site/blog.txt +++ b/site/blog.txt @@ -9,6 +9,8 @@ | | +-------------------------------------------------------+ +- Capsul - Rumors Of My Demise Have Been Greatly Exagerated + - COVIDaware MN app investigation - Capsul rollin' onward with a Web Application diff --git a/site/blog/20200520-capsul-rollin-onward-with-a-web-application.txt b/site/blog/20200520-capsul-rollin-onward-with-a-web-application.txt index b2c048e..f5d320a 100644 --- a/site/blog/20200520-capsul-rollin-onward-with-a-web-application.txt +++ b/site/blog/20200520-capsul-rollin-onward-with-a-web-application.txt @@ -215,6 +215,6 @@ verbose logging goes slightly against the "we don't collect any more data about you than we need to" mantra. If you would like to take a peek at the code, it's -hosted on forge: +hosted on our git server: -https://giit.cyberia.club/~forest/capsul-flask \ No newline at end of file +https://git.cyberia.club/cyberia/capsul-flask \ No newline at end of file diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt new file mode 100644 index 0000000..5195274 --- /dev/null +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -0,0 +1,273 @@ + + CAPSUL + + rumors of my demise have been greatly exagerated + + +Forest 2021-12-17 + + WHAT HAPPENED TO THE CRYPTOCURRENCY PAYMENT OPTION? + +Life happens. Cyberia Computer Club has been hustling +and bustling to build out our new in-person space in +Minneapolis, MN: + +https://wiki.cyberia.club/hypha/cyberia_hq/faq + +Hackerspace, computer lab, clubhouse, we aren't sure what +to call it yet, but we're extremely excited to finish +with the rennovations and move in! + +In the meantime, something went wrong with the physical +machine hosting our BTCPay server instance and we didn't +have anywhere convenient to move it, nor time to replace +it, so we simply disabled cryptocurrency payments +temporarily in September 2021. + +Many of yall have emailed us asking "what gives??", +and i'm glad to finally be able to announce that + +"the situation has been dealt with", + +we have a brand new server and the blockchain syncing +process is complete, cryptocurrency payments in bitcoin, +litecoin, and monero are back online now! + + --> https://capsul.org/payment/btcpay <-- + + + THAT ONE TIME CAPSUL WAS ALMOST fsync'd TO DEATH + +Guess what? Yall loved capsul so much, you wore our disks +out. Well, almost. + +We use redundant solid state disks + the ZFS file system +for your capsul's block storage needs, and it turns out +that some of our users like to write files. A lot. + +Over time, SSDs will wear out, mostly dependent on how +many writes hit the disk. Baikal, the server behind +capsul.org, is a bit different from a typical desktop +computer, as it hosts about 100 virtual machines, each +with thier own list of application processes, for over 50 +individual capsul users, each of which may be providing +services to many other individuals in turn. + +The disk-wear-out situation was exacerbated by our +geographical separation from the server; we live in +Minneapolis, MN, but the server is in Georgia. We wanted +to install NVME drives to expand our storage capacity +ahead of growing demand, but when we would mail PCI-e to +NVME adapters to CyberWurx, our datacenter colocation +provider, they kept telling us the adapter didn't fit +inside the 1U chassis of the server. + +At one point, we were forced to take a risk and undo the +redundancy of the disks in order to expand our storage +capacity and prevent "out of disk space" errors from +crashing your capsuls. It was a calculated risk, trading +certain doom now for the potential possibility of doom +later. + +Well, time passed while we were busy with other projects, +and those non-redundant disks started wearing out. +According to the "smartmon" monitoring indicator, they +reached about 25% lifespan remaining. Once the disk +theoretically hit 0%, it would become read-only in order +to protect itself from total data loss. +So we had to replace them. + +https://picopublish.sequentialread.com/files/smartmon_dec2021.png + +We were so scared of what could happen if we slept on +this that we booked a flight to Atlanta for maintenance. +We wanted to replace the disks in person, and ensure we +could restore the ZFS disk mirroring feature. + +We even custom 3d-printed a bracket for the tiny PCI-e +NVME drive that we needed in order to restore redundancy +for the disks, just to make 100% sure that the +maintenance we were doing would succeed; would maintain +stability for everyone who has placed thier trust in us +and voted with thier shells, investing thier time and +money on virtual machines that we maintain on a volunteer +basis. + +https://picopublish.sequentialread.com/files/silly-nvme-bracket2.jpg + +We also made sure to test the ZFS disk mirroring feature +in practice by carefully removing a disk from the server +while it's running: + +https://picopublish.sequentialread.com/files/zfs_disk_replacement/ + +While we could have theoretically done this maintenance +remotely with the folks at CyberWurx performing the +physical parts replacement per a ticket we open with +them, we wanted to be sure we could meet the timeline +that the disks had set for **US**. That's no knock on +CyberWurx, moreso a knock on us for yolo-ing this server +into "production" with tape and no test environment :D + +Well, the reality is we are vounteer supported. Right now +the payments that the club receives from capusl users +don't add up to enough to compensate (make ends meet for) +your average professional software developer or sysadmin, +at least if local tech labor market stats are to be +believed. + +We are all also working on other things, we can't devote +all of our time to capsul. But we do care about capsul, +we want our service to live, because we use it ourselves +and the club benefits from it . + +We want it to be easy and fun to use, while also staying +easy and fun to maintain. A system that's agressively +maintained will be a lot more likely to remain maintained +when it's no one's job to come in every weekday for that. + +That's why we also decided to upgrade to the latest +stable Debian major version on baikal while we were +there. We encountered no issues during the upgrade +besides a couple of initial omissions in our package +source lists. The installer also notified us of several +configuration files we had modified, presenting us with +a git-merge-ish interface that displayed diffs and +allowed us to decide to keep our changes, replace our +file with the new version, or merge the two manually. + +I can't speak more accurately about it than that, as +j3s did this part and I just watched :) + + LOOKING TO THE FUTURE + +We wanted to upgrade to this new Debian version because +it had a new major version of QEMU, supporting virtio-blk +storage devices that can pass-through file system discard +commands to the host operating system. + +We didn't see any benefits right away, as the vms +stayed defined in libvirt as their original machine types, +either pc-i440fx-3.1 or one from the pc-q35 family. + +After returning home, we noticed that when we created +a new capsul, it would come up as the pc-i440fx-5.2 +machine type and the main disk on the guest would display +discard support in the form of a non-zero DISC-MAX size +displayed by the `lsblk -D` command: + +localhost:~# sudo lsblk -D +NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO +sr0 0 0B 0B 0 +vda 512 512B 2G 0 + +Most of our capsuls were pc-i440fx ones, and we upgraded +them to pc-i440fx-5.2, which finally got discards working +for the grand majority of capsuls. + +If you see discard settings like that on your capsul, +you should also be able to run `fstrim -v /` on your +capsul which saves us disk space on baikal: + +welcome, cyberian ^(;,;)^ +your machine awaits + +localhost:~# sudo lsblk -D +NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO +sr0 0 0B 0B 0 +vda 512 512B 2G 0 + +localhost:~# sudo fstrim -v / +/: 15.1 GiB (16185487360 bytes) trimmed + +^ Please do this if you are able to! + +However, some of the older vms were the pc-q35 family of +QEMU machine type, and while I was able to get one of +ours to upgrade to pc-i440fx-5.2, discard support still +did not show up in the guest OS. We're not sure what's +happening there yet. + +We also improved capsul's monitoring features, we began +work on proper configuration-as-code style diffing +functionality, so we get notified if any key aspects of +your capsuls are out of whack. In the past this had been +an issue, with DHCP leases expiring during maintenance +downtimes and capsuls stealing each-others assigned IP +addresses when we turn everything back on. + +Also, capsul-flask now includes an admin panel with +1-click-fix actions built in, leveraging this data: + +https://git.cyberia.club/cyberia/capsul-flask/src/branch/main/capsulflask/admin.py#L36-L202 + +https://picopublish.sequentialread.com/files/admin-panel.jpg + +I acknowledge that this is a bit of a silly system, +but it's an artifact of how we do what we do. Capsul +is always in motion, and the web app was built on the +idea of "providing a button for" any manual action +that would have to be taken, either by a user or by an +admin, while operating the old command-line and email- +based version of "cvm", which later capsul. Everything +there was once an experimental feature. + +When we learn more about how our system works, we +expand what's possible for capsul, and we come up +with new ways to organize data, things, and +intelligently direct the venerable virtualization +software our service is built on. + +We do have a lot of ideas about how to continue +making capsul easier for everyone involved, things +like: + +1. Metered billing w/ stripe, so you get a monthly bill + with auto-pay to your credit card, and you only pay + for the resources you use, similar to what service + providers like Backblaze do. + + (Note: of course we would also allow you to + pre-pay with cryptocurrency if you wish) + +2. Looking into rewrite options for some parts of the + system: + perhaps driving QEMU from capsul-flask + directly instead of going through libvirt, + and perhaps rewriting the web application in golang + instead of sticking with flask. + +3. IO throttling your vms: + As I mentioned before, the vms wear out the disks + fast. We had hoped that enabling discards would help + with this, but it appears that it hasn't done much + to decrease the growth rate of the smartmon wearout + indicator metric. + So, most likely we will have to enforce some form of + limit on the amount of disk writes your capsul can + perform while it's running day in and day out. + 80-90% of capsul users will never see this limit, + but our heaviest writers will be required to either + change thier software so it writes less, or pay more + money for service. In any case, we'll send you a + warning email long before we throttle your capsul's + disk. + + +And last but not least, Cybera Computer Club voted to use +a couple thousand of the capsulbux we've recieved in +payment to purchase a new server, allowing us to expand +the service ahead of demand and improve our processes +all the way from hardware up. (No tape this time!) + +https://picopublish.sequentialread.com/files/baikal2/ + + +Forest 2021-12-17 + + + + + + + diff --git a/site/blog/blog.xml b/site/blog/blog.xml index 55ceedf..661b1b7 100644 --- a/site/blog/blog.xml +++ b/site/blog/blog.xml @@ -33,7 +33,7 @@ 20/05/2020: Capsul rollin' onward with a Web Application - A necessary post + Forest describes his thought process when building the capsul.org web application https://cyberia.club/blog/20200520-capsul-rollin-onward-with-a-web-application https://cyberia.club/blog/20200520-capsul-rollin-onward-with-a-web-application Wed, 20 May 2020 00:00:00 +0000 @@ -47,5 +47,13 @@ Fri, 27 Nov 2020 13:46:13 UTC + + 17/12/2021: Capsul - Rumors Of My Demise Have Been Greatly Exagerated + Forest regails you with tales of maintenance, links to pictures and video, and hopes for the future of capsul + https://cyberia.club/blog/20211217-capsul-maintenance-updates + https://cyberia.club/blog/20211217-capsul-maintenance-updates + Sat, 18 Dec 2021 00:00:00 +0000 + + -- 2.40.1 From a66112da2176ff652fc6b07cde426673980e7eb9 Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 01:02:24 -0600 Subject: [PATCH 02/20] add "what is this" and "can I help" --- .../20211217-capsul-maintenance-updates.txt | 58 ++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index 5195274..f1ee8a0 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -6,6 +6,12 @@ Forest 2021-12-17 + WHAT IS THIS? + +If you're wondering what capsul is, see my previous post: + +https://cyberia.club/blog/20200520-capsul-rollin-onward-with-a-web-application.html + WHAT HAPPENED TO THE CRYPTOCURRENCY PAYMENT OPTION? Life happens. Cyberia Computer Club has been hustling @@ -237,7 +243,11 @@ like: and perhaps rewriting the web application in golang instead of sticking with flask. -3. IO throttling your vms: +3. JSON API designed to make it easier to manage capsuls + in code, scripts, or with an infrastructure-as-code + tool like Terraform. + +4. IO throttling your vms: As I mentioned before, the vms wear out the disks fast. We had hoped that enabling discards would help with this, but it appears that it hasn't done much @@ -263,6 +273,50 @@ all the way from hardware up. (No tape this time!) https://picopublish.sequentialread.com/files/baikal2/ + + CAN I HELP? + +Yes! We are not the only ones working on capsul these +days. For example, another group, https://coopcloud.tech/, +has forked capsul-flask and set up thier own instance at + +https://yolo.servers.coop/ + +Thier source code repository is here +(not sure this is the right one): + +https://git.autonomic.zone/3wordchant/capsul-flask + +Having more people setting up instances of capsul-flask +really helps us, whether folks are simply testing or +aiming to run it in production like we do. + +Unfortunately we don't have a direct incentive to +work on making capsul-flask easier to set up until folks +ask us how to do it. Autonomic helped us a lot as they +made thier way through our terrible documentation and +asked for better organization / clarification along the +way, leading to much more expansive and organized README +files. + +They also gave a great shove in the right direction when +they decided to contribute most of a basic automated +testing implementation and the beginnings of a JSON API +at the same time. They are building a command line tool +called abra that can create capsuls upon the users +request, as well as many other things like installing +applications. I think it's very neat :) + +Also, just donating or using the service helps support +cyberia.club, both in terms of maintaing capsul.org and +reaching out and supporting our local community. + +We accept donations via either a credit card (stripe) +or in Bitcoin, Litecoin, or Monero via our BTCPay server: + +https://cyberia.club/donate + + Forest 2021-12-17 @@ -271,3 +325,5 @@ Forest 2021-12-17 + + -- 2.40.1 From 8aacef5bee23c35bfa04d8a2151f55eff3d0437e Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 01:10:18 -0600 Subject: [PATCH 03/20] add support email address to blog post --- .../20211217-capsul-maintenance-updates.txt | 20 +++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index f1ee8a0..fdc6872 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -264,14 +264,19 @@ like: disk. -And last but not least, Cybera Computer Club voted to use -a couple thousand of the capsulbux we've recieved in -payment to purchase a new server, allowing us to expand -the service ahead of demand and improve our processes -all the way from hardware up. (No tape this time!) +And last but not least, Cybera Computer Club Congress +voted to use a couple thousand of the capsulbux we've +recieved in payment to purchase a new server, allowing +us to expand the service ahead of demand and improve our +processes all the way from hardware up. + +(No tape this time!) https://picopublish.sequentialread.com/files/baikal2/ +Shown: Dell PowerEdge R640 1U server with two +10-core xeon silver 4114 processors and 256GB of RAM. +(Upgradable to 768GB!!) CAN I HELP? @@ -289,7 +294,7 @@ https://git.autonomic.zone/3wordchant/capsul-flask Having more people setting up instances of capsul-flask really helps us, whether folks are simply testing or -aiming to run it in production like we do. +aiming to run it in production like we do. Unfortunately we don't have a direct incentive to work on making capsul-flask easier to set up until folks @@ -316,6 +321,9 @@ or in Bitcoin, Litecoin, or Monero via our BTCPay server: https://cyberia.club/donate +As always, you may contact us at: + +support@cyberia.club Forest 2021-12-17 -- 2.40.1 From a7ddea81bd96542add63bf6aa2c0ec131084fa73 Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 01:12:28 -0600 Subject: [PATCH 04/20] conf as code --> inf as code --- site/blog/20211217-capsul-maintenance-updates.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index fdc6872..95aff81 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -195,7 +195,7 @@ did not show up in the guest OS. We're not sure what's happening there yet. We also improved capsul's monitoring features, we began -work on proper configuration-as-code style diffing +work on proper infrastructure-as-code style diffing functionality, so we get notified if any key aspects of your capsuls are out of whack. In the past this had been an issue, with DHCP leases expiring during maintenance -- 2.40.1 From 7693a133d21ac504543b1d7385159e3a271698fa Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 02:04:00 -0600 Subject: [PATCH 05/20] clean up unfinished verbaige --- site/blog/20211217-capsul-maintenance-updates.txt | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index 95aff81..6759a77 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -211,12 +211,12 @@ https://picopublish.sequentialread.com/files/admin-panel.jpg I acknowledge that this is a bit of a silly system, but it's an artifact of how we do what we do. Capsul -is always in motion, and the web app was built on the -idea of "providing a button for" any manual action -that would have to be taken, either by a user or by an -admin, while operating the old command-line and email- -based version of "cvm", which later capsul. Everything -there was once an experimental feature. +is always changing and evolving, and the web app was +built on the idea of "providing a button for" any +manual action that would have to be taken, either by +a user or by an admin, while operating the old +command-line and email-based version of "cvm", +which later became capsul. When we learn more about how our system works, we expand what's possible for capsul, and we come up -- 2.40.1 From 1f7ecee1cfcec9cff8686df2db8126aa7dc0ca90 Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 02:14:12 -0600 Subject: [PATCH 06/20] fixing links --- site/blog/20211217-capsul-maintenance-updates.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index 6759a77..9c2f0ef 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -282,7 +282,7 @@ Shown: Dell PowerEdge R640 1U server with two CAN I HELP? Yes! We are not the only ones working on capsul these -days. For example, another group, https://coopcloud.tech/, +days. For example, another group, https://coopcloud.tech/ has forked capsul-flask and set up thier own instance at https://yolo.servers.coop/ @@ -323,7 +323,7 @@ https://cyberia.club/donate As always, you may contact us at: -support@cyberia.club + mailto:support@cyberia.club Forest 2021-12-17 -- 2.40.1 From ada1f44224e1280bcdfb20f3b046c9b4a22421d4 Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 02:20:15 -0600 Subject: [PATCH 07/20] fix links regex to support URLS less than 50 chars long :( --- make | 1 + site/blog/20211217-capsul-maintenance-updates.txt | 7 ++++--- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/make b/make index 1695d9a..3d3eced 100755 --- a/make +++ b/make @@ -51,6 +51,7 @@ while read -r page; do # link https sed -E "s|^(https[:]//[^ )]{50})([^ )]*)|\1|g" | + sed -E "s|^(https[:]//[^ )]{1,50})|\0|g" | # link mailto sed -E "s|(mailto:[^ ]*)|\1|g" | # color hexcodes diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index 9c2f0ef..879ef37 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -209,6 +209,7 @@ https://git.cyberia.club/cyberia/capsul-flask/src/branch/main/capsulflask/admin. https://picopublish.sequentialread.com/files/admin-panel.jpg + I acknowledge that this is a bit of a silly system, but it's an artifact of how we do what we do. Capsul is always changing and evolving, and the web app was @@ -272,7 +273,7 @@ processes all the way from hardware up. (No tape this time!) -https://picopublish.sequentialread.com/files/baikal2/ +https://picopublish.sequentialread.com/files/baikal2 Shown: Dell PowerEdge R640 1U server with two 10-core xeon silver 4114 processors and 256GB of RAM. @@ -282,10 +283,10 @@ Shown: Dell PowerEdge R640 1U server with two CAN I HELP? Yes! We are not the only ones working on capsul these -days. For example, another group, https://coopcloud.tech/ +days. For example, another group, https://coopcloud.tech has forked capsul-flask and set up thier own instance at -https://yolo.servers.coop/ +https://yolo.servers.coop Thier source code repository is here (not sure this is the right one): -- 2.40.1 From f04b82d7e35c6c53184878f45c0e5a4e4748f466 Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 02:21:15 -0600 Subject: [PATCH 08/20] fix spelling --- site/blog/blog.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/blog/blog.xml b/site/blog/blog.xml index 661b1b7..45da997 100644 --- a/site/blog/blog.xml +++ b/site/blog/blog.xml @@ -48,7 +48,7 @@ - 17/12/2021: Capsul - Rumors Of My Demise Have Been Greatly Exagerated + 17/12/2021: Capsul - Rumors Of My Demise Have Been Greatly Exaggerated Forest regails you with tales of maintenance, links to pictures and video, and hopes for the future of capsul https://cyberia.club/blog/20211217-capsul-maintenance-updates https://cyberia.club/blog/20211217-capsul-maintenance-updates -- 2.40.1 From 8d5c84172b526132df890495ec8c99275e491e8a Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 02:24:14 -0600 Subject: [PATCH 09/20] forgot to save blog.txt --- site/blog.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/blog.txt b/site/blog.txt index 71e5915..7ee6f47 100644 --- a/site/blog.txt +++ b/site/blog.txt @@ -9,7 +9,7 @@ | | +-------------------------------------------------------+ -- Capsul - Rumors Of My Demise Have Been Greatly Exagerated +- Capsul - Rumors Of My Demise Have Been Greatly Exaggerated - COVIDaware MN app investigation -- 2.40.1 From 2de3734cb173ddd85aade98a1309fb14b68b215b Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 03:25:15 -0600 Subject: [PATCH 10/20] fix wording and space before period --- site/blog/20211217-capsul-maintenance-updates.txt | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index 879ef37..edc953e 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -1,7 +1,7 @@ CAPSUL - rumors of my demise have been greatly exagerated + rumors of my demise have been greatly exaggerated Forest 2021-12-17 @@ -124,8 +124,8 @@ believed. We are all also working on other things, we can't devote all of our time to capsul. But we do care about capsul, -we want our service to live, because we use it ourselves -and the club benefits from it . +we want our service to live, mostly because we use it +ourselves, but also because the club benefits from it. We want it to be easy and fun to use, while also staying easy and fun to maintain. A system that's agressively @@ -209,7 +209,6 @@ https://git.cyberia.club/cyberia/capsul-flask/src/branch/main/capsulflask/admin. https://picopublish.sequentialread.com/files/admin-panel.jpg - I acknowledge that this is a bit of a silly system, but it's an artifact of how we do what we do. Capsul is always changing and evolving, and the web app was -- 2.40.1 From da4b1dfad4bd565294f8263b72fb1a9b58d661df Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 03:40:14 -0600 Subject: [PATCH 11/20] permalink to gitea --- site/blog/20211217-capsul-maintenance-updates.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index edc953e..8c616ec 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -205,7 +205,7 @@ addresses when we turn everything back on. Also, capsul-flask now includes an admin panel with 1-click-fix actions built in, leveraging this data: -https://git.cyberia.club/cyberia/capsul-flask/src/branch/main/capsulflask/admin.py#L36-L202 +https://git.cyberia.club/cyberia/capsul-flask/src/commit/b013f9c9758f2cc062f1ecefc4d7deef3aa484f2/capsulflask/admin.py#L36-L202 https://picopublish.sequentialread.com/files/admin-panel.jpg -- 2.40.1 From a914b07fd0b907405429ab5ded0710a3b022238b Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 03:42:50 -0600 Subject: [PATCH 12/20] fix hyphenation issue --- site/blog/20211217-capsul-maintenance-updates.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index 8c616ec..5833abf 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -195,7 +195,7 @@ did not show up in the guest OS. We're not sure what's happening there yet. We also improved capsul's monitoring features, we began -work on proper infrastructure-as-code style diffing +work on proper infrastructure-as-code-style diffing functionality, so we get notified if any key aspects of your capsuls are out of whack. In the past this had been an issue, with DHCP leases expiring during maintenance -- 2.40.1 From d05558e761549ec2bf6b7694c10ba055d50c8789 Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 03:48:43 -0600 Subject: [PATCH 13/20] source code link front and center --- site/blog/20211217-capsul-maintenance-updates.txt | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index 5833abf..734dc8a 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -12,6 +12,10 @@ If you're wondering what capsul is, see my previous post: https://cyberia.club/blog/20200520-capsul-rollin-onward-with-a-web-application.html +For the capsul source code, see here: + +https://git.cyberia.club/cyberia/capsul-flask + WHAT HAPPENED TO THE CRYPTOCURRENCY PAYMENT OPTION? Life happens. Cyberia Computer Club has been hustling @@ -212,10 +216,10 @@ https://picopublish.sequentialread.com/files/admin-panel.jpg I acknowledge that this is a bit of a silly system, but it's an artifact of how we do what we do. Capsul is always changing and evolving, and the web app was -built on the idea of "providing a button for" any -manual action that would have to be taken, either by -a user or by an admin, while operating the old -command-line and email-based version of "cvm", +built on the idea of simply "providing a button for" +any manual action that would have to be taken, +either by a user or by an admin, while operating the +old command-line and email-based version of "cvm", which later became capsul. When we learn more about how our system works, we -- 2.40.1 From f43e1a19ff4f284d3d797b9e66ba88d3607eb24a Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 03:55:16 -0600 Subject: [PATCH 14/20] whoom --- site/blog/20211217-capsul-maintenance-updates.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index 734dc8a..121f3a5 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -60,7 +60,7 @@ many writes hit the disk. Baikal, the server behind capsul.org, is a bit different from a typical desktop computer, as it hosts about 100 virtual machines, each with thier own list of application processes, for over 50 -individual capsul users, each of which may be providing +individual capsul users, each of whoom may be providing services to many other individuals in turn. The disk-wear-out situation was exacerbated by our @@ -97,7 +97,7 @@ could restore the ZFS disk mirroring feature. We even custom 3d-printed a bracket for the tiny PCI-e NVME drive that we needed in order to restore redundancy for the disks, just to make 100% sure that the -maintenance we were doing would succeed; would maintain +maintenance we were doing would succeed & maintain stability for everyone who has placed thier trust in us and voted with thier shells, investing thier time and money on virtual machines that we maintain on a volunteer -- 2.40.1 From c84eb82fc0b29209494a2ad3d11b646ae3458e20 Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 04:05:36 -0600 Subject: [PATCH 15/20] whom --- site/blog/20211217-capsul-maintenance-updates.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index 121f3a5..296bfa5 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -60,7 +60,7 @@ many writes hit the disk. Baikal, the server behind capsul.org, is a bit different from a typical desktop computer, as it hosts about 100 virtual machines, each with thier own list of application processes, for over 50 -individual capsul users, each of whoom may be providing +individual capsul users, each of whom may be providing services to many other individuals in turn. The disk-wear-out situation was exacerbated by our -- 2.40.1 From 4a6220a922e0f1e655aa1b60d227ef72b17f38d8 Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 12:42:52 -0600 Subject: [PATCH 16/20] Add NVME formatted size was too small for ZFS story --- .../20211217-capsul-maintenance-updates.txt | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index 296bfa5..7e27261 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -105,6 +105,36 @@ basis. https://picopublish.sequentialread.com/files/silly-nvme-bracket2.jpg +Unfortunately, "100% sure" was still not good enough, the +new NVME drive didn't work as a ZFS mirroring partner +at first ⁠— the existing NVME drive was 951GB, and the +one we had purchased was 931GB. It was too small and ZFS +would not accept that. f0x suggested: + +> [you could] start a new pool on the new disk, +> zfs send all the old data over, then have an +> equally sized partition on the old disk then add +> that to the mirror + +But we had no idea how to do that exactly or how long it +would take & we didn't want to change the plan at the +last second, so instead we ended up taking the train from +the datacenter to Best Buy to buy a new disk instead. + +The actual formatted sizes of these drives are typically +never printed on the packaging or even mentioned on PDF +datasheets online. When I could find an actual number +for a model, it was always the lower 931GB. +So, we ended up buying a "2TB" drive as it was the only +one BestBuy had which we could guarantee would work. + +So, lesson learned. If you want to use ZFS mirroring and +maybe replace a drive later, make sure to choose a fixed +partition size which is slightly smaller than the +typical avaliable space on the size of drive you're +using, in case the replacement drive was manufactured +with slightly less avaliable formatted space!!! + We also made sure to test the ZFS disk mirroring feature in practice by carefully removing a disk from the server while it's running: -- 2.40.1 From 48c217f26ed0009baed0858cb0cb8dc0cb9d706e Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 13:17:13 -0600 Subject: [PATCH 17/20] tweak the link shortening regex to include the end of the file name --- make | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/make b/make index 3d3eced..7f23e9d 100755 --- a/make +++ b/make @@ -50,10 +50,10 @@ while read -r page; do "../site/$page" | # link https - sed -E "s|^(https[:]//[^ )]{50})([^ )]*)|\1|g" | + sed -E "s|^(https[:]//[^ )]{32})([^ )]{3,1000})([^ )]{15})|\1...\3|g" | sed -E "s|^(https[:]//[^ )]{1,50})|\0|g" | # link mailto - sed -E "s|(mailto:[^ ]*)|\1|g" | + sed -E "s|mailto:([^ ]*)|\1|g" | # color hexcodes sed -E 's|(#[a-f0-9]{6})|\1|g' | # color cyberia -- 2.40.1 From 0b110b72a0bef7045bf243407e04563376c83032 Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 13:17:30 -0600 Subject: [PATCH 18/20] finally rewrite the awkward cvm history paragraph --- .../20211217-capsul-maintenance-updates.txt | 68 ++++++++++++------- 1 file changed, 42 insertions(+), 26 deletions(-) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index 7e27261..193c74e 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -8,11 +8,11 @@ Forest 2021-12-17 WHAT IS THIS? -If you're wondering what capsul is, see my previous post: +If you're wondering "what is capsul?", see: https://cyberia.club/blog/20200520-capsul-rollin-onward-with-a-web-application.html -For the capsul source code, see here: +For the capsul source code, navigate to: https://git.cyberia.club/cyberia/capsul-flask @@ -24,18 +24,18 @@ Minneapolis, MN: https://wiki.cyberia.club/hypha/cyberia_hq/faq -Hackerspace, computer lab, clubhouse, we aren't sure what -to call it yet, but we're extremely excited to finish -with the rennovations and move in! +Hackerspace, lab, clubhouse, we aren't sure what to call +it yet, but we're extremely excited to finish with the +rennovations and move in! In the meantime, something went wrong with the physical -machine hosting our BTCPay server instance and we didn't -have anywhere convenient to move it, nor time to replace -it, so we simply disabled cryptocurrency payments +machine hosting our BTCPay server and we didn't have +anywhere convenient to move it, nor time to replace it, +so we simply disabled cryptocurrency payments temporarily in September 2021. Many of yall have emailed us asking "what gives??", -and i'm glad to finally be able to announce that +and I'm glad to finally be able to announce that "the situation has been dealt with", @@ -85,7 +85,7 @@ According to the "smartmon" monitoring indicator, they reached about 25% lifespan remaining. Once the disk theoretically hit 0%, it would become read-only in order to protect itself from total data loss. -So we had to replace them. +So we had to replace them before that happened. https://picopublish.sequentialread.com/files/smartmon_dec2021.png @@ -105,8 +105,8 @@ basis. https://picopublish.sequentialread.com/files/silly-nvme-bracket2.jpg -Unfortunately, "100% sure" was still not good enough, the -new NVME drive didn't work as a ZFS mirroring partner +Unfortunately, "100% sure" was still not good enough, +the new NVME drive didn't work as a ZFS mirroring partner at first ⁠— the existing NVME drive was 951GB, and the one we had purchased was 931GB. It was too small and ZFS would not accept that. f0x suggested: @@ -128,14 +128,15 @@ for a model, it was always the lower 931GB. So, we ended up buying a "2TB" drive as it was the only one BestBuy had which we could guarantee would work. -So, lesson learned. If you want to use ZFS mirroring and -maybe replace a drive later, make sure to choose a fixed -partition size which is slightly smaller than the -typical avaliable space on the size of drive you're -using, in case the replacement drive was manufactured -with slightly less avaliable formatted space!!! +So, lesson learned the hard way. If you want to use ZFS +mirroring and maybe replace a drive later, make sure to +choose a fixed partition size which is slightly smaller +than the typical avaliable space on the size of drive +you're using, in case the replacement drive was +manufactured with slightly less avaliable formatted +space!!! -We also made sure to test the ZFS disk mirroring feature +Once mirroring was restored, we made sure to test it in practice by carefully removing a disk from the server while it's running: @@ -149,7 +150,7 @@ that the disks had set for **US**. That's no knock on CyberWurx, moreso a knock on us for yolo-ing this server into "production" with tape and no test environment :D -Well, the reality is we are vounteer supported. Right now +The reality is we are vounteer supported. Right now the payments that the club receives from capusl users don't add up to enough to compensate (make ends meet for) your average professional software developer or sysadmin, @@ -188,7 +189,7 @@ commands to the host operating system. We didn't see any benefits right away, as the vms stayed defined in libvirt as their original machine types, -either pc-i440fx-3.1 or one from the pc-q35 family. +either pc-i440fx-3.1 or a type from the pc-q35 family. After returning home, we noticed that when we created a new capsul, it would come up as the pc-i440fx-5.2 @@ -222,13 +223,17 @@ localhost:~# sudo fstrim -v / ^ Please do this if you are able to! +You might also be able to enable an fstrim service or +timer which will run fstrim to clean up and optimize +your disk periodically. + However, some of the older vms were the pc-q35 family of QEMU machine type, and while I was able to get one of ours to upgrade to pc-i440fx-5.2, discard support still did not show up in the guest OS. We're not sure what's happening there yet. -We also improved capsul's monitoring features, we began +We also improved capsul's monitoring features; we began work on proper infrastructure-as-code-style diffing functionality, so we get notified if any key aspects of your capsuls are out of whack. In the past this had been @@ -248,9 +253,13 @@ but it's an artifact of how we do what we do. Capsul is always changing and evolving, and the web app was built on the idea of simply "providing a button for" any manual action that would have to be taken, -either by a user or by an admin, while operating the -old command-line and email-based version of "cvm", -which later became capsul. +either by a user or by an admin. + +At one point, back when capsul was called "cvm", +_everything_ was done by hand over email and the +commandline, so of course anything that reduced the +amount of manual administration work was welcome, +and we are still working on that today. When we learn more about how our system works, we expand what's possible for capsul, and we come up @@ -357,7 +366,14 @@ https://cyberia.club/donate As always, you may contact us at: - mailto:support@cyberia.club +mailto:support@cyberia.club + +Or on matrix: + +#services:cyberia.club + +For information on what matrix chat is and how to use it, +see: https://cyberia.club/matrix Forest 2021-12-17 -- 2.40.1 From 8e3b5689d6b18aea0260c0dce454b59c312c1d89 Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 13:20:00 -0600 Subject: [PATCH 19/20] delete ', things,' --- site/blog/20211217-capsul-maintenance-updates.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index 193c74e..e24aaa4 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -263,9 +263,9 @@ and we are still working on that today. When we learn more about how our system works, we expand what's possible for capsul, and we come up -with new ways to organize data, things, and -intelligently direct the venerable virtualization -software our service is built on. +with new ways to organize data and intelligently +direct the venerable virtualization software our +service is built on. We do have a lot of ideas about how to continue making capsul easier for everyone involved, things -- 2.40.1 From 4731ea0fcc318aa105b18b4cf03399f7e7f7784a Mon Sep 17 00:00:00 2001 From: forest Date: Sat, 18 Dec 2021 13:30:49 -0600 Subject: [PATCH 20/20] "agile development" buzzword --- .../20211217-capsul-maintenance-updates.txt | 32 +++++++++---------- 1 file changed, 15 insertions(+), 17 deletions(-) diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt index e24aaa4..5e7d1ec 100644 --- a/site/blog/20211217-capsul-maintenance-updates.txt +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -258,14 +258,22 @@ either by a user or by an admin. At one point, back when capsul was called "cvm", _everything_ was done by hand over email and the commandline, so of course anything that reduced the -amount of manual administration work was welcome, +amount of manual administration work was welcome, and we are still working on that today. -When we learn more about how our system works, we -expand what's possible for capsul, and we come up -with new ways to organize data and intelligently -direct the venerable virtualization software our -service is built on. +When we build new UIs and prototype features, we learn +more about how our system works, we expand what's +possible for capsul, and we come up with new ways to +organize data and intelligently direct the venerable +virtualization software our service is built on. + +I think that's what the "agile development" buzzword from +professional software development circles was supposed to +be about: freedom to experiment means better designs +because we get the opportunity to experience some of the +consequences before we fully commit to any specific +design. A touch of humility and flexibility goes a +long way in my opinion. We do have a lot of ideas about how to continue making capsul easier for everyone involved, things @@ -280,8 +288,7 @@ like: pre-pay with cryptocurrency if you wish) 2. Looking into rewrite options for some parts of the - system: - perhaps driving QEMU from capsul-flask + system: perhaps driving QEMU from capsul-flask directly instead of going through libvirt, and perhaps rewriting the web application in golang instead of sticking with flask. @@ -376,12 +383,3 @@ For information on what matrix chat is and how to use it, see: https://cyberia.club/matrix Forest 2021-12-17 - - - - - - - - - -- 2.40.1