diff --git a/make b/make index 1695d9a..7f23e9d 100755 --- a/make +++ b/make @@ -50,9 +50,10 @@ while read -r page; do "../site/$page" | # link https - sed -E "s|^(https[:]//[^ )]{50})([^ )]*)|\1|g" | + sed -E "s|^(https[:]//[^ )]{32})([^ )]{3,1000})([^ )]{15})|\1...\3|g" | + sed -E "s|^(https[:]//[^ )]{1,50})|\0|g" | # link mailto - sed -E "s|(mailto:[^ ]*)|\1|g" | + sed -E "s|mailto:([^ ]*)|\1|g" | # color hexcodes sed -E 's|(#[a-f0-9]{6})|\1|g' | # color cyberia diff --git a/site/blog.txt b/site/blog.txt index f908903..7ee6f47 100644 --- a/site/blog.txt +++ b/site/blog.txt @@ -9,6 +9,8 @@ | | +-------------------------------------------------------+ +- Capsul - Rumors Of My Demise Have Been Greatly Exaggerated + - COVIDaware MN app investigation - Capsul rollin' onward with a Web Application diff --git a/site/blog/20200520-capsul-rollin-onward-with-a-web-application.txt b/site/blog/20200520-capsul-rollin-onward-with-a-web-application.txt index b2c048e..f5d320a 100644 --- a/site/blog/20200520-capsul-rollin-onward-with-a-web-application.txt +++ b/site/blog/20200520-capsul-rollin-onward-with-a-web-application.txt @@ -215,6 +215,6 @@ verbose logging goes slightly against the "we don't collect any more data about you than we need to" mantra. If you would like to take a peek at the code, it's -hosted on forge: +hosted on our git server: -https://giit.cyberia.club/~forest/capsul-flask \ No newline at end of file +https://git.cyberia.club/cyberia/capsul-flask \ No newline at end of file diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt new file mode 100644 index 0000000..5e7d1ec --- /dev/null +++ b/site/blog/20211217-capsul-maintenance-updates.txt @@ -0,0 +1,385 @@ + + CAPSUL + + rumors of my demise have been greatly exaggerated + + +Forest 2021-12-17 + + WHAT IS THIS? + +If you're wondering "what is capsul?", see: + +https://cyberia.club/blog/20200520-capsul-rollin-onward-with-a-web-application.html + +For the capsul source code, navigate to: + +https://git.cyberia.club/cyberia/capsul-flask + + WHAT HAPPENED TO THE CRYPTOCURRENCY PAYMENT OPTION? + +Life happens. Cyberia Computer Club has been hustling +and bustling to build out our new in-person space in +Minneapolis, MN: + +https://wiki.cyberia.club/hypha/cyberia_hq/faq + +Hackerspace, lab, clubhouse, we aren't sure what to call +it yet, but we're extremely excited to finish with the +rennovations and move in! + +In the meantime, something went wrong with the physical +machine hosting our BTCPay server and we didn't have +anywhere convenient to move it, nor time to replace it, +so we simply disabled cryptocurrency payments +temporarily in September 2021. + +Many of yall have emailed us asking "what gives??", +and I'm glad to finally be able to announce that + +"the situation has been dealt with", + +we have a brand new server and the blockchain syncing +process is complete, cryptocurrency payments in bitcoin, +litecoin, and monero are back online now! + + --> https://capsul.org/payment/btcpay <-- + + + THAT ONE TIME CAPSUL WAS ALMOST fsync'd TO DEATH + +Guess what? Yall loved capsul so much, you wore our disks +out. Well, almost. + +We use redundant solid state disks + the ZFS file system +for your capsul's block storage needs, and it turns out +that some of our users like to write files. A lot. + +Over time, SSDs will wear out, mostly dependent on how +many writes hit the disk. Baikal, the server behind +capsul.org, is a bit different from a typical desktop +computer, as it hosts about 100 virtual machines, each +with thier own list of application processes, for over 50 +individual capsul users, each of whom may be providing +services to many other individuals in turn. + +The disk-wear-out situation was exacerbated by our +geographical separation from the server; we live in +Minneapolis, MN, but the server is in Georgia. We wanted +to install NVME drives to expand our storage capacity +ahead of growing demand, but when we would mail PCI-e to +NVME adapters to CyberWurx, our datacenter colocation +provider, they kept telling us the adapter didn't fit +inside the 1U chassis of the server. + +At one point, we were forced to take a risk and undo the +redundancy of the disks in order to expand our storage +capacity and prevent "out of disk space" errors from +crashing your capsuls. It was a calculated risk, trading +certain doom now for the potential possibility of doom +later. + +Well, time passed while we were busy with other projects, +and those non-redundant disks started wearing out. +According to the "smartmon" monitoring indicator, they +reached about 25% lifespan remaining. Once the disk +theoretically hit 0%, it would become read-only in order +to protect itself from total data loss. +So we had to replace them before that happened. + +https://picopublish.sequentialread.com/files/smartmon_dec2021.png + +We were so scared of what could happen if we slept on +this that we booked a flight to Atlanta for maintenance. +We wanted to replace the disks in person, and ensure we +could restore the ZFS disk mirroring feature. + +We even custom 3d-printed a bracket for the tiny PCI-e +NVME drive that we needed in order to restore redundancy +for the disks, just to make 100% sure that the +maintenance we were doing would succeed & maintain +stability for everyone who has placed thier trust in us +and voted with thier shells, investing thier time and +money on virtual machines that we maintain on a volunteer +basis. + +https://picopublish.sequentialread.com/files/silly-nvme-bracket2.jpg + +Unfortunately, "100% sure" was still not good enough, +the new NVME drive didn't work as a ZFS mirroring partner +at first ⁠— the existing NVME drive was 951GB, and the +one we had purchased was 931GB. It was too small and ZFS +would not accept that. f0x suggested: + +> [you could] start a new pool on the new disk, +> zfs send all the old data over, then have an +> equally sized partition on the old disk then add +> that to the mirror + +But we had no idea how to do that exactly or how long it +would take & we didn't want to change the plan at the +last second, so instead we ended up taking the train from +the datacenter to Best Buy to buy a new disk instead. + +The actual formatted sizes of these drives are typically +never printed on the packaging or even mentioned on PDF +datasheets online. When I could find an actual number +for a model, it was always the lower 931GB. +So, we ended up buying a "2TB" drive as it was the only +one BestBuy had which we could guarantee would work. + +So, lesson learned the hard way. If you want to use ZFS +mirroring and maybe replace a drive later, make sure to +choose a fixed partition size which is slightly smaller +than the typical avaliable space on the size of drive +you're using, in case the replacement drive was +manufactured with slightly less avaliable formatted +space!!! + +Once mirroring was restored, we made sure to test it +in practice by carefully removing a disk from the server +while it's running: + +https://picopublish.sequentialread.com/files/zfs_disk_replacement/ + +While we could have theoretically done this maintenance +remotely with the folks at CyberWurx performing the +physical parts replacement per a ticket we open with +them, we wanted to be sure we could meet the timeline +that the disks had set for **US**. That's no knock on +CyberWurx, moreso a knock on us for yolo-ing this server +into "production" with tape and no test environment :D + +The reality is we are vounteer supported. Right now +the payments that the club receives from capusl users +don't add up to enough to compensate (make ends meet for) +your average professional software developer or sysadmin, +at least if local tech labor market stats are to be +believed. + +We are all also working on other things, we can't devote +all of our time to capsul. But we do care about capsul, +we want our service to live, mostly because we use it +ourselves, but also because the club benefits from it. + +We want it to be easy and fun to use, while also staying +easy and fun to maintain. A system that's agressively +maintained will be a lot more likely to remain maintained +when it's no one's job to come in every weekday for that. + +That's why we also decided to upgrade to the latest +stable Debian major version on baikal while we were +there. We encountered no issues during the upgrade +besides a couple of initial omissions in our package +source lists. The installer also notified us of several +configuration files we had modified, presenting us with +a git-merge-ish interface that displayed diffs and +allowed us to decide to keep our changes, replace our +file with the new version, or merge the two manually. + +I can't speak more accurately about it than that, as +j3s did this part and I just watched :) + + LOOKING TO THE FUTURE + +We wanted to upgrade to this new Debian version because +it had a new major version of QEMU, supporting virtio-blk +storage devices that can pass-through file system discard +commands to the host operating system. + +We didn't see any benefits right away, as the vms +stayed defined in libvirt as their original machine types, +either pc-i440fx-3.1 or a type from the pc-q35 family. + +After returning home, we noticed that when we created +a new capsul, it would come up as the pc-i440fx-5.2 +machine type and the main disk on the guest would display +discard support in the form of a non-zero DISC-MAX size +displayed by the `lsblk -D` command: + +localhost:~# sudo lsblk -D +NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO +sr0 0 0B 0B 0 +vda 512 512B 2G 0 + +Most of our capsuls were pc-i440fx ones, and we upgraded +them to pc-i440fx-5.2, which finally got discards working +for the grand majority of capsuls. + +If you see discard settings like that on your capsul, +you should also be able to run `fstrim -v /` on your +capsul which saves us disk space on baikal: + +welcome, cyberian ^(;,;)^ +your machine awaits + +localhost:~# sudo lsblk -D +NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO +sr0 0 0B 0B 0 +vda 512 512B 2G 0 + +localhost:~# sudo fstrim -v / +/: 15.1 GiB (16185487360 bytes) trimmed + +^ Please do this if you are able to! + +You might also be able to enable an fstrim service or +timer which will run fstrim to clean up and optimize +your disk periodically. + +However, some of the older vms were the pc-q35 family of +QEMU machine type, and while I was able to get one of +ours to upgrade to pc-i440fx-5.2, discard support still +did not show up in the guest OS. We're not sure what's +happening there yet. + +We also improved capsul's monitoring features; we began +work on proper infrastructure-as-code-style diffing +functionality, so we get notified if any key aspects of +your capsuls are out of whack. In the past this had been +an issue, with DHCP leases expiring during maintenance +downtimes and capsuls stealing each-others assigned IP +addresses when we turn everything back on. + +Also, capsul-flask now includes an admin panel with +1-click-fix actions built in, leveraging this data: + +https://git.cyberia.club/cyberia/capsul-flask/src/commit/b013f9c9758f2cc062f1ecefc4d7deef3aa484f2/capsulflask/admin.py#L36-L202 + +https://picopublish.sequentialread.com/files/admin-panel.jpg + +I acknowledge that this is a bit of a silly system, +but it's an artifact of how we do what we do. Capsul +is always changing and evolving, and the web app was +built on the idea of simply "providing a button for" +any manual action that would have to be taken, +either by a user or by an admin. + +At one point, back when capsul was called "cvm", +_everything_ was done by hand over email and the +commandline, so of course anything that reduced the +amount of manual administration work was welcome, +and we are still working on that today. + +When we build new UIs and prototype features, we learn +more about how our system works, we expand what's +possible for capsul, and we come up with new ways to +organize data and intelligently direct the venerable +virtualization software our service is built on. + +I think that's what the "agile development" buzzword from +professional software development circles was supposed to +be about: freedom to experiment means better designs +because we get the opportunity to experience some of the +consequences before we fully commit to any specific +design. A touch of humility and flexibility goes a +long way in my opinion. + +We do have a lot of ideas about how to continue +making capsul easier for everyone involved, things +like: + +1. Metered billing w/ stripe, so you get a monthly bill + with auto-pay to your credit card, and you only pay + for the resources you use, similar to what service + providers like Backblaze do. + + (Note: of course we would also allow you to + pre-pay with cryptocurrency if you wish) + +2. Looking into rewrite options for some parts of the + system: perhaps driving QEMU from capsul-flask + directly instead of going through libvirt, + and perhaps rewriting the web application in golang + instead of sticking with flask. + +3. JSON API designed to make it easier to manage capsuls + in code, scripts, or with an infrastructure-as-code + tool like Terraform. + +4. IO throttling your vms: + As I mentioned before, the vms wear out the disks + fast. We had hoped that enabling discards would help + with this, but it appears that it hasn't done much + to decrease the growth rate of the smartmon wearout + indicator metric. + So, most likely we will have to enforce some form of + limit on the amount of disk writes your capsul can + perform while it's running day in and day out. + 80-90% of capsul users will never see this limit, + but our heaviest writers will be required to either + change thier software so it writes less, or pay more + money for service. In any case, we'll send you a + warning email long before we throttle your capsul's + disk. + + +And last but not least, Cybera Computer Club Congress +voted to use a couple thousand of the capsulbux we've +recieved in payment to purchase a new server, allowing +us to expand the service ahead of demand and improve our +processes all the way from hardware up. + +(No tape this time!) + +https://picopublish.sequentialread.com/files/baikal2 + +Shown: Dell PowerEdge R640 1U server with two +10-core xeon silver 4114 processors and 256GB of RAM. +(Upgradable to 768GB!!) + + + CAN I HELP? + +Yes! We are not the only ones working on capsul these +days. For example, another group, https://coopcloud.tech +has forked capsul-flask and set up thier own instance at + +https://yolo.servers.coop + +Thier source code repository is here +(not sure this is the right one): + +https://git.autonomic.zone/3wordchant/capsul-flask + +Having more people setting up instances of capsul-flask +really helps us, whether folks are simply testing or +aiming to run it in production like we do. + +Unfortunately we don't have a direct incentive to +work on making capsul-flask easier to set up until folks +ask us how to do it. Autonomic helped us a lot as they +made thier way through our terrible documentation and +asked for better organization / clarification along the +way, leading to much more expansive and organized README +files. + +They also gave a great shove in the right direction when +they decided to contribute most of a basic automated +testing implementation and the beginnings of a JSON API +at the same time. They are building a command line tool +called abra that can create capsuls upon the users +request, as well as many other things like installing +applications. I think it's very neat :) + +Also, just donating or using the service helps support +cyberia.club, both in terms of maintaing capsul.org and +reaching out and supporting our local community. + +We accept donations via either a credit card (stripe) +or in Bitcoin, Litecoin, or Monero via our BTCPay server: + +https://cyberia.club/donate + +As always, you may contact us at: + +mailto:support@cyberia.club + +Or on matrix: + +#services:cyberia.club + +For information on what matrix chat is and how to use it, +see: https://cyberia.club/matrix + +Forest 2021-12-17 diff --git a/site/blog/blog.xml b/site/blog/blog.xml index 55ceedf..45da997 100644 --- a/site/blog/blog.xml +++ b/site/blog/blog.xml @@ -33,7 +33,7 @@ 20/05/2020: Capsul rollin' onward with a Web Application - A necessary post + Forest describes his thought process when building the capsul.org web application https://cyberia.club/blog/20200520-capsul-rollin-onward-with-a-web-application https://cyberia.club/blog/20200520-capsul-rollin-onward-with-a-web-application Wed, 20 May 2020 00:00:00 +0000 @@ -47,5 +47,13 @@ Fri, 27 Nov 2020 13:46:13 UTC + + 17/12/2021: Capsul - Rumors Of My Demise Have Been Greatly Exaggerated + Forest regails you with tales of maintenance, links to pictures and video, and hopes for the future of capsul + https://cyberia.club/blog/20211217-capsul-maintenance-updates + https://cyberia.club/blog/20211217-capsul-maintenance-updates + Sat, 18 Dec 2021 00:00:00 +0000 + +