diff --git a/make b/make
index 1695d9a..7f23e9d 100755
--- a/make
+++ b/make
@@ -50,9 +50,10 @@ while read -r page; do
"../site/$page" |
# link https
- sed -E "s|^(https[:]//[^ )]{50})([^ )]*)|\1|g" |
+ sed -E "s|^(https[:]//[^ )]{32})([^ )]{3,1000})([^ )]{15})|\1...\3|g" |
+ sed -E "s|^(https[:]//[^ )]{1,50})|\0|g" |
# link mailto
- sed -E "s|(mailto:[^ ]*)|\1|g" |
+ sed -E "s|mailto:([^ ]*)|\1|g" |
# color hexcodes
sed -E 's|(#[a-f0-9]{6})|\1|g' |
# color cyberia
diff --git a/site/blog.txt b/site/blog.txt
index f908903..7ee6f47 100644
--- a/site/blog.txt
+++ b/site/blog.txt
@@ -9,6 +9,8 @@
| |
+-------------------------------------------------------+
+- Capsul - Rumors Of My Demise Have Been Greatly Exaggerated
+
- COVIDaware MN app investigation
- Capsul rollin' onward with a Web Application
diff --git a/site/blog/20200520-capsul-rollin-onward-with-a-web-application.txt b/site/blog/20200520-capsul-rollin-onward-with-a-web-application.txt
index b2c048e..f5d320a 100644
--- a/site/blog/20200520-capsul-rollin-onward-with-a-web-application.txt
+++ b/site/blog/20200520-capsul-rollin-onward-with-a-web-application.txt
@@ -215,6 +215,6 @@ verbose logging goes slightly against the "we don't
collect any more data about you than we need to" mantra.
If you would like to take a peek at the code, it's
-hosted on forge:
+hosted on our git server:
-https://giit.cyberia.club/~forest/capsul-flask
\ No newline at end of file
+https://git.cyberia.club/cyberia/capsul-flask
\ No newline at end of file
diff --git a/site/blog/20211217-capsul-maintenance-updates.txt b/site/blog/20211217-capsul-maintenance-updates.txt
new file mode 100644
index 0000000..5e7d1ec
--- /dev/null
+++ b/site/blog/20211217-capsul-maintenance-updates.txt
@@ -0,0 +1,385 @@
+
+ CAPSUL
+
+ rumors of my demise have been greatly exaggerated
+
+
+Forest 2021-12-17
+
+ WHAT IS THIS?
+
+If you're wondering "what is capsul?", see:
+
+https://cyberia.club/blog/20200520-capsul-rollin-onward-with-a-web-application.html
+
+For the capsul source code, navigate to:
+
+https://git.cyberia.club/cyberia/capsul-flask
+
+ WHAT HAPPENED TO THE CRYPTOCURRENCY PAYMENT OPTION?
+
+Life happens. Cyberia Computer Club has been hustling
+and bustling to build out our new in-person space in
+Minneapolis, MN:
+
+https://wiki.cyberia.club/hypha/cyberia_hq/faq
+
+Hackerspace, lab, clubhouse, we aren't sure what to call
+it yet, but we're extremely excited to finish with the
+rennovations and move in!
+
+In the meantime, something went wrong with the physical
+machine hosting our BTCPay server and we didn't have
+anywhere convenient to move it, nor time to replace it,
+so we simply disabled cryptocurrency payments
+temporarily in September 2021.
+
+Many of yall have emailed us asking "what gives??",
+and I'm glad to finally be able to announce that
+
+"the situation has been dealt with",
+
+we have a brand new server and the blockchain syncing
+process is complete, cryptocurrency payments in bitcoin,
+litecoin, and monero are back online now!
+
+ --> https://capsul.org/payment/btcpay <--
+
+
+ THAT ONE TIME CAPSUL WAS ALMOST fsync'd TO DEATH
+
+Guess what? Yall loved capsul so much, you wore our disks
+out. Well, almost.
+
+We use redundant solid state disks + the ZFS file system
+for your capsul's block storage needs, and it turns out
+that some of our users like to write files. A lot.
+
+Over time, SSDs will wear out, mostly dependent on how
+many writes hit the disk. Baikal, the server behind
+capsul.org, is a bit different from a typical desktop
+computer, as it hosts about 100 virtual machines, each
+with thier own list of application processes, for over 50
+individual capsul users, each of whom may be providing
+services to many other individuals in turn.
+
+The disk-wear-out situation was exacerbated by our
+geographical separation from the server; we live in
+Minneapolis, MN, but the server is in Georgia. We wanted
+to install NVME drives to expand our storage capacity
+ahead of growing demand, but when we would mail PCI-e to
+NVME adapters to CyberWurx, our datacenter colocation
+provider, they kept telling us the adapter didn't fit
+inside the 1U chassis of the server.
+
+At one point, we were forced to take a risk and undo the
+redundancy of the disks in order to expand our storage
+capacity and prevent "out of disk space" errors from
+crashing your capsuls. It was a calculated risk, trading
+certain doom now for the potential possibility of doom
+later.
+
+Well, time passed while we were busy with other projects,
+and those non-redundant disks started wearing out.
+According to the "smartmon" monitoring indicator, they
+reached about 25% lifespan remaining. Once the disk
+theoretically hit 0%, it would become read-only in order
+to protect itself from total data loss.
+So we had to replace them before that happened.
+
+https://picopublish.sequentialread.com/files/smartmon_dec2021.png
+
+We were so scared of what could happen if we slept on
+this that we booked a flight to Atlanta for maintenance.
+We wanted to replace the disks in person, and ensure we
+could restore the ZFS disk mirroring feature.
+
+We even custom 3d-printed a bracket for the tiny PCI-e
+NVME drive that we needed in order to restore redundancy
+for the disks, just to make 100% sure that the
+maintenance we were doing would succeed & maintain
+stability for everyone who has placed thier trust in us
+and voted with thier shells, investing thier time and
+money on virtual machines that we maintain on a volunteer
+basis.
+
+https://picopublish.sequentialread.com/files/silly-nvme-bracket2.jpg
+
+Unfortunately, "100% sure" was still not good enough,
+the new NVME drive didn't work as a ZFS mirroring partner
+at first — the existing NVME drive was 951GB, and the
+one we had purchased was 931GB. It was too small and ZFS
+would not accept that. f0x suggested:
+
+> [you could] start a new pool on the new disk,
+> zfs send all the old data over, then have an
+> equally sized partition on the old disk then add
+> that to the mirror
+
+But we had no idea how to do that exactly or how long it
+would take & we didn't want to change the plan at the
+last second, so instead we ended up taking the train from
+the datacenter to Best Buy to buy a new disk instead.
+
+The actual formatted sizes of these drives are typically
+never printed on the packaging or even mentioned on PDF
+datasheets online. When I could find an actual number
+for a model, it was always the lower 931GB.
+So, we ended up buying a "2TB" drive as it was the only
+one BestBuy had which we could guarantee would work.
+
+So, lesson learned the hard way. If you want to use ZFS
+mirroring and maybe replace a drive later, make sure to
+choose a fixed partition size which is slightly smaller
+than the typical avaliable space on the size of drive
+you're using, in case the replacement drive was
+manufactured with slightly less avaliable formatted
+space!!!
+
+Once mirroring was restored, we made sure to test it
+in practice by carefully removing a disk from the server
+while it's running:
+
+https://picopublish.sequentialread.com/files/zfs_disk_replacement/
+
+While we could have theoretically done this maintenance
+remotely with the folks at CyberWurx performing the
+physical parts replacement per a ticket we open with
+them, we wanted to be sure we could meet the timeline
+that the disks had set for **US**. That's no knock on
+CyberWurx, moreso a knock on us for yolo-ing this server
+into "production" with tape and no test environment :D
+
+The reality is we are vounteer supported. Right now
+the payments that the club receives from capusl users
+don't add up to enough to compensate (make ends meet for)
+your average professional software developer or sysadmin,
+at least if local tech labor market stats are to be
+believed.
+
+We are all also working on other things, we can't devote
+all of our time to capsul. But we do care about capsul,
+we want our service to live, mostly because we use it
+ourselves, but also because the club benefits from it.
+
+We want it to be easy and fun to use, while also staying
+easy and fun to maintain. A system that's agressively
+maintained will be a lot more likely to remain maintained
+when it's no one's job to come in every weekday for that.
+
+That's why we also decided to upgrade to the latest
+stable Debian major version on baikal while we were
+there. We encountered no issues during the upgrade
+besides a couple of initial omissions in our package
+source lists. The installer also notified us of several
+configuration files we had modified, presenting us with
+a git-merge-ish interface that displayed diffs and
+allowed us to decide to keep our changes, replace our
+file with the new version, or merge the two manually.
+
+I can't speak more accurately about it than that, as
+j3s did this part and I just watched :)
+
+ LOOKING TO THE FUTURE
+
+We wanted to upgrade to this new Debian version because
+it had a new major version of QEMU, supporting virtio-blk
+storage devices that can pass-through file system discard
+commands to the host operating system.
+
+We didn't see any benefits right away, as the vms
+stayed defined in libvirt as their original machine types,
+either pc-i440fx-3.1 or a type from the pc-q35 family.
+
+After returning home, we noticed that when we created
+a new capsul, it would come up as the pc-i440fx-5.2
+machine type and the main disk on the guest would display
+discard support in the form of a non-zero DISC-MAX size
+displayed by the `lsblk -D` command:
+
+localhost:~# sudo lsblk -D
+NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
+sr0 0 0B 0B 0
+vda 512 512B 2G 0
+
+Most of our capsuls were pc-i440fx ones, and we upgraded
+them to pc-i440fx-5.2, which finally got discards working
+for the grand majority of capsuls.
+
+If you see discard settings like that on your capsul,
+you should also be able to run `fstrim -v /` on your
+capsul which saves us disk space on baikal:
+
+welcome, cyberian ^(;,;)^
+your machine awaits
+
+localhost:~# sudo lsblk -D
+NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
+sr0 0 0B 0B 0
+vda 512 512B 2G 0
+
+localhost:~# sudo fstrim -v /
+/: 15.1 GiB (16185487360 bytes) trimmed
+
+^ Please do this if you are able to!
+
+You might also be able to enable an fstrim service or
+timer which will run fstrim to clean up and optimize
+your disk periodically.
+
+However, some of the older vms were the pc-q35 family of
+QEMU machine type, and while I was able to get one of
+ours to upgrade to pc-i440fx-5.2, discard support still
+did not show up in the guest OS. We're not sure what's
+happening there yet.
+
+We also improved capsul's monitoring features; we began
+work on proper infrastructure-as-code-style diffing
+functionality, so we get notified if any key aspects of
+your capsuls are out of whack. In the past this had been
+an issue, with DHCP leases expiring during maintenance
+downtimes and capsuls stealing each-others assigned IP
+addresses when we turn everything back on.
+
+Also, capsul-flask now includes an admin panel with
+1-click-fix actions built in, leveraging this data:
+
+https://git.cyberia.club/cyberia/capsul-flask/src/commit/b013f9c9758f2cc062f1ecefc4d7deef3aa484f2/capsulflask/admin.py#L36-L202
+
+https://picopublish.sequentialread.com/files/admin-panel.jpg
+
+I acknowledge that this is a bit of a silly system,
+but it's an artifact of how we do what we do. Capsul
+is always changing and evolving, and the web app was
+built on the idea of simply "providing a button for"
+any manual action that would have to be taken,
+either by a user or by an admin.
+
+At one point, back when capsul was called "cvm",
+_everything_ was done by hand over email and the
+commandline, so of course anything that reduced the
+amount of manual administration work was welcome,
+and we are still working on that today.
+
+When we build new UIs and prototype features, we learn
+more about how our system works, we expand what's
+possible for capsul, and we come up with new ways to
+organize data and intelligently direct the venerable
+virtualization software our service is built on.
+
+I think that's what the "agile development" buzzword from
+professional software development circles was supposed to
+be about: freedom to experiment means better designs
+because we get the opportunity to experience some of the
+consequences before we fully commit to any specific
+design. A touch of humility and flexibility goes a
+long way in my opinion.
+
+We do have a lot of ideas about how to continue
+making capsul easier for everyone involved, things
+like:
+
+1. Metered billing w/ stripe, so you get a monthly bill
+ with auto-pay to your credit card, and you only pay
+ for the resources you use, similar to what service
+ providers like Backblaze do.
+
+ (Note: of course we would also allow you to
+ pre-pay with cryptocurrency if you wish)
+
+2. Looking into rewrite options for some parts of the
+ system: perhaps driving QEMU from capsul-flask
+ directly instead of going through libvirt,
+ and perhaps rewriting the web application in golang
+ instead of sticking with flask.
+
+3. JSON API designed to make it easier to manage capsuls
+ in code, scripts, or with an infrastructure-as-code
+ tool like Terraform.
+
+4. IO throttling your vms:
+ As I mentioned before, the vms wear out the disks
+ fast. We had hoped that enabling discards would help
+ with this, but it appears that it hasn't done much
+ to decrease the growth rate of the smartmon wearout
+ indicator metric.
+ So, most likely we will have to enforce some form of
+ limit on the amount of disk writes your capsul can
+ perform while it's running day in and day out.
+ 80-90% of capsul users will never see this limit,
+ but our heaviest writers will be required to either
+ change thier software so it writes less, or pay more
+ money for service. In any case, we'll send you a
+ warning email long before we throttle your capsul's
+ disk.
+
+
+And last but not least, Cybera Computer Club Congress
+voted to use a couple thousand of the capsulbux we've
+recieved in payment to purchase a new server, allowing
+us to expand the service ahead of demand and improve our
+processes all the way from hardware up.
+
+(No tape this time!)
+
+https://picopublish.sequentialread.com/files/baikal2
+
+Shown: Dell PowerEdge R640 1U server with two
+10-core xeon silver 4114 processors and 256GB of RAM.
+(Upgradable to 768GB!!)
+
+
+ CAN I HELP?
+
+Yes! We are not the only ones working on capsul these
+days. For example, another group, https://coopcloud.tech
+has forked capsul-flask and set up thier own instance at
+
+https://yolo.servers.coop
+
+Thier source code repository is here
+(not sure this is the right one):
+
+https://git.autonomic.zone/3wordchant/capsul-flask
+
+Having more people setting up instances of capsul-flask
+really helps us, whether folks are simply testing or
+aiming to run it in production like we do.
+
+Unfortunately we don't have a direct incentive to
+work on making capsul-flask easier to set up until folks
+ask us how to do it. Autonomic helped us a lot as they
+made thier way through our terrible documentation and
+asked for better organization / clarification along the
+way, leading to much more expansive and organized README
+files.
+
+They also gave a great shove in the right direction when
+they decided to contribute most of a basic automated
+testing implementation and the beginnings of a JSON API
+at the same time. They are building a command line tool
+called abra that can create capsuls upon the users
+request, as well as many other things like installing
+applications. I think it's very neat :)
+
+Also, just donating or using the service helps support
+cyberia.club, both in terms of maintaing capsul.org and
+reaching out and supporting our local community.
+
+We accept donations via either a credit card (stripe)
+or in Bitcoin, Litecoin, or Monero via our BTCPay server:
+
+https://cyberia.club/donate
+
+As always, you may contact us at:
+
+mailto:support@cyberia.club
+
+Or on matrix:
+
+#services:cyberia.club
+
+For information on what matrix chat is and how to use it,
+see: https://cyberia.club/matrix
+
+Forest 2021-12-17
diff --git a/site/blog/blog.xml b/site/blog/blog.xml
index 55ceedf..45da997 100644
--- a/site/blog/blog.xml
+++ b/site/blog/blog.xml
@@ -33,7 +33,7 @@
20/05/2020: Capsul rollin' onward with a Web Application
- A necessary post
+ Forest describes his thought process when building the capsul.org web application
https://cyberia.club/blog/20200520-capsul-rollin-onward-with-a-web-application
https://cyberia.club/blog/20200520-capsul-rollin-onward-with-a-web-applicationWed, 20 May 2020 00:00:00 +0000
@@ -47,5 +47,13 @@
Fri, 27 Nov 2020 13:46:13 UTC
+
+ 17/12/2021: Capsul - Rumors Of My Demise Have Been Greatly Exaggerated
+ Forest regails you with tales of maintenance, links to pictures and video, and hopes for the future of capsul
+ https://cyberia.club/blog/20211217-capsul-maintenance-updates
+ https://cyberia.club/blog/20211217-capsul-maintenance-updates
+ Sat, 18 Dec 2021 00:00:00 +0000
+
+