A while ago I noticed that my ISP leases IPv4 addresses out indefinitely. It was everything I'd ever wanted and I gotta seize it to truly self-host. As an experiment, I started on something cheaper, like a single-board compooter. In 2024, support for general-purpose RISC-V chips began to ripen, so naturally due to FOMO, I bought a board with JH-7110. Boy, was that a mistake! While the bootloaders' support had been well upstreamed, certain essential features like PCIe (for NVMe) has yet to reached a mainline Linux release, even worse so on the BSDs. I ended up flashing the only distribution with official support at the time, Ubuntu.

Funny enough, after over a decade of daily driving GNU, twas the first time I installed Ubuntu on a machine of my own. At the time of writing, the reason for the was more apparent than ever: Canonical had been forcing Snap^[1] down the users' throat, even on the server edition. Thankfully Snap was still managed by APT and twas easy enough to remove prevent it from coming back. Another annoyance was the lack of manual pages in the minimized installation and that the official way to enable them is through a script that also install other bloats SMFH (the script is quite short and the actually necessary commands can be trivially found, I'd rather they're documented instead).

That being said, not everything Ubuntu includes due to NIH is bad. Unity (not the game engine that's proprietary like Snap server) was loved by many; and this article is basically an appreciation post for some others: Netplan and ufw. Before diving in, lemme finish the story to give you the full context of this setup. The SBC is the VisionFive 2 which is blessed with plenty of IO:

• 8 GB of memory

• 4 USB 3.0 type-A ports

• 2 RJ45 ports (1 Gb and 100/10 Mb)

• 1 M.2 slot (I used this as an excuse to buy a larger SSD and put the old 256 GB one here)

• 1 eMMC slot^[2] (eMMC are cheap, got one also with 256 GB)

• 1 TF slot

• 40 pin GP(and predefined-purpose)IO

• Other stuff for interfacing with humen like HDMI, audio jack, etc.

Initially, my plan for the SBC was to host services unlisted on the loang network. Official services were not considered because my home network has no IPv6 and sometimes I'll like to have most of the bandwidth for meself. Shortly afterwards, I also purchased a somewhat beefy desktop compooter with even more I/O, especially a bunch of SATA, which are a lot more attractive than connecting hard di*ks via USB. On the other hand, the SBC barely consume any electricity, well under 10 W with the NVMe drive, a Wi-Fi dongle and a fan connected. Since it cost virtually nothing to keep it up 24/7, I decided to hand it the following two tasks:

• Reverse proxying services running on more powerful machines in the local network.

• Acting as a virtual router between nodes I manage. This is particularly useful for tunneling to my work network and accessing the servers, allowing me to work remotely with low latency.

Setting up the VPN with Wireguard was relatively easy, so I assumed swapping the SBC for the home router couldn't be too hard. Once again, I chose poorly, this little project'd costed me so many sleepless nights so I figured I should note down what I learned here in case it can save someone else from the same pain. Do not take inspiration from this!

Connecting to the Internet

My landlord handles the contract with the ISP so I don't know the details of the subscription, but there's certainly no IPv6 nor any static IPv4 address. Bandwidth to datacenters in the region is approximately 100 Mb/s and the wall socket connects to a Cat 5e cable. I know about the latter because whatever dumb ass did the last maintenance wired that to another short one dangling from the wall socket^[3], and after getting stabbed in the eyes for months I finally to open it up and made the socket a proper socket.

It would not make the slightest of a difference but I connect the SBC's 1 Gb port (identified in Ubuntu as end0) to the Internet and the slower one (end1) to my desktop on the local network. Thankfully no special setup was needed and here is the entire Netplan configuration to connect to the outside world:

network:
  ethernets:
    end0:
      dhcp4: true
  renderer: networkd
  version: 2

Local Networking

For simplicity's sake, I decided to use the same subnet for both Ethernet and Wi-Fi under a bridge br0, where addressing and routing is configured:

network:
  bridges:
    br0:
      addresses:
        - 192.168.147.254/25
      interfaces:
        - end1
      routes:
        - from: 192.168.147.128/25
          on-link: true
          to: 0.0.0.0/0
          type: nat
          via: 192.168.147.254
  ethernets:
    end1:
      dhcp4: false

As Netplan doesn't configure any DHCP server, that's done separately by udhcpd from busybox:

interface br0
start 192.168.147.128
end 192.168.147.253
max_leases 126
option subnet 255.255.255.128
option router 192.168.147.254

I couldn't seem to get a concrete information on the ports used by DHCP so I open the firewall for UDP on both 67 and 68 (I swear this isn't an engagement bait to test out the new mailing list):

ufw allow in on br0 to any port 67 proto udp
ufw allow in on br0 to any port 68 proto udp

Wireless Access Point

Thanks to systemd, the Wi-Fi dongle is recognized as wlx600dd0g8b33f. Yes, that abomination of a name includes the chip's full MAC address. That being said, I'd like to stick to the basis of a systemd/Linux distro. Netplan doesn't support Wi-Fi hotspot with systemd-networkd but NetworkManager, so the interface had thus to be declared as Ethernet:

network:
  bridges:
    br0:
      interfaces:
        - wlx600dd0g8b33f
  ethernets:
    wlx600dd0g8b33f:
      dhcp4: false

Actual wireless connectivity is handled by hostapd:

interface=wlx600dd0g8b33f
bridge=br0ssid=YΦ
utf8_ssid=1
country_code=KR
channel=6
ieee80211d=1
ieee80211h=1
ieee80211n=1
hw_mode=g
wmm_enabled=1wpa=2
wpa_pairwise=TKIP
wpa_passphrase=just enter random characters

Name Resolution

My ISP is known to be evil so I'd rather rely on more reputable resolvers like OpenNIC, which also offers free-of-charge (!) domain names. Most of their tier 2 servers are located on the other side of the globe (200 to 300 ms RTT), so a local cache is almost required. SmartDNS seems to be the best fit for this purpose, as it queries upstream servers simultaneously and also check for the IP with the lowest RTT among the results. Since I don't trust my ISP, connections to the upstream servers are encrypted:

bind :53@br0
server-tls 51.254.162.59 -host-name ns1-dot.iriseden.fr
server-tls 202.61.197.122 -host-name dns.furrydns.de
server-tls 80.152.203.134 -host-name dot.kekew.info
server-tls 178.201.248.159 -host-name dot.kekew.info
server-tls 178.201.248.160 -host-name dot.kekew.info
server-tls 95.216.99.249 -host-name dns.froth.zone

For the router itself, the nameserver is set in /etc/resolv.conf and Netplan is told not to change it:

network:
  ethernets:
    end0:
      dhcp4-use-dns: false

After ufw is configured to allow UDP traffic in port 53 on br0, udhcpd is instructed to advertise this local DNS server:

option dns 192.168.147.254

I might consider blocking ads at the domain-name level someday, but for now uBlock Origin is working well enough on my systems and I rarely have people over, especially not for looking at their electronic devices.

In today's episode of guides nobody asked for and likely having been covered by someone more qualified, lemme show you the correct ways to view videos hosted on YouTube and other hostile, tracker-riddled hellscapes. Whilst I despise Google's mass surveillance practices, it stores a large proportion of culturally significant videos and clips that would be difficult to mirror to user-respecting services due to copyright. Hell, even YouTube doesn't have the right to distribute many of them in the first place.

Because of YouTube's circumvention of advertisement blockers, the ad-blocking arm race finally caught mainstream media attention and tis kool to talk about that now. Hence I'm happy to jump on the bandwagon, albeit a wee bit late, but this ain't just that. Since I feed you poison—over 4% of the pages linked from my site are on YouTube—the least I can do is sell you my cures.

Using a Proper Media Player

The most popular solutions are either to use for a good blocker on a browser with (supposedly) long-term support for Manifest V2 like uBlock Origin on Firefox, or use alternative front-ends such as Invidious or Piped. Although uBlock Origin is essential for a pleasant experience on the modern interwebs and alternative frontends offers the best UX for browsing videos, in-browser and service-specific media players are inferior anyway when compared to programs properly designed for a decent playback experience.

My favorite has been mpv for as long as I can remember, as it makes it easy to adjust video brightness/contrast/etc., playback speed, subtitle size and placements, and to overamplify quiet audios. Out of the box, it integrates with yt-dlp, a time shifter with support for most online media services. Just drop the URL into an mpv window and boom, it werks!

Either drag-and-drop or invoking mpv $url is quite convenient, but not that close to following an anchor, is it? You'd need to first open mpv or a program launcher^[1], then drag the URL there, or perhaps copy and paste it for the latter cases. What if you gotta go fast, aye? As a hedgehog-maxxer meself, of course I can do better, and here's how.

With a Browser Add-on

While drafting this article, I noticed that the ff2mpv extension I was using had technically been non-free for a while. Albeit I understand and respect the author's noble intention against violence, I believe discrimination never ends up helping those oppressed due to the power imbalance for the exclusion false-positives to be worth it.

For this reason, I switched to Open in mpv and recommend it instead. The usage is practically the same: open context menu at the video URL and select Open this link in mpv. The internal mechanism is a bit different though, and because it influences the installation process, I will try to briefly explain how it works.

The way Open in mpv works is a bit convoluted. First, it wraps the specified URL in a mpv scheme. The new URL starts with mpv:// is then passed back to Firefox, which must have been configured to open it in the native program open-in-mpv. This program parses the URL into the equivalent mpv command and execute it. If you are not on NixOS, see the extension's README to set it up yourself.

Otherwise, it can be declared in configuration.nix(5) as follows. The declarations should be self-explanatory after referencing Firefox's documentation for policies.json. If you have trouble finding an extension's ID and download URL, search for it in Mozzarella.

{ pkgs, ... }:
{
  programs.firefox = {
    enable = true;
    policies = {
      ExtensionSettings."{d66c8515-1e0d-408f-82ee-2682f2362726}" = {
        default_area = "menupanel";
        installation_mode = "normal_installed";
        install_url =
          "https://addons.mozilla.org/firefox"
          + "/downloads/latest/iina-open-in-mpv/latest.xpi";
      };
      Handlers.scheme.mpv = {
        action = "useHelperApp";
        ask = false;
        handlers = [ {
          name = "open-in-mpv";
          path = "${pkgs.open-in-mpv}/bin/open-in-mpv";
        } ];
      };
    };
  };
}

Even though Mozzarella is supposed to only show libre add-ons, be aware that the metadata it crawls from addons.mozzila.org might not always be correct. Ideally, browser extensions should be packaged in the distribution's repository, but packaging discipline is not exactly NixOS's strong suit. I will probably post an update on how to declare policies.json in Guix once I figure that out.

From a Feed Reader

Now we can properly watch videos while browsing the web, but subscribing to YouTube channels on its web interface would require creating an account and subjecting one's self to more surveillance. Fortunately, at the time of writing, YouTube still provide Atom feeds for syndication. Funny enough, they are advertised on the channel pages as RSS:

<link rel="alternate"
      type="application/rss+xml"
      title="RSS"
      href="https://www.youtube.com/feeds/videos.xml?channel_id=…">

The referenced feed employ Media RSS to communicate the video URL. This extension is widely supported by feed readers, as well as the previously mentioned feed-discovery mechanism. I use Liferea, which allows me to directly paste the YouTube channel's URL^[2], and displays each video's description, thumbnail and enclosed media, e.g.

For each MIME type to, enclosures can be configured to be opened by a user-preferred program. In this case, I set mpv --ytdl-format=b for application/x-shockwave-flash (a reminiscence of a time when browsers needed Flash to play videos and animations) for the second best quality to save some bandwidth. YouTube encodes the highest resolution video separate from the audio, so the best combined format b is one level lower than yt-dlp's default best video and best audio together.

Via Clipboard Integration

People also share videos with me via instant messaging. I find it cumbersome to open the URL in the browser then redirect it to the media player, so the clipboard is used as the bridge instead. To do this, I simply create a key binding to the command below.^[3]

mpv --ytdl-format=b "$(xclip -out -selection clipboard)"

Musing

There, I shared how I do it so you can too! If they seem needlessly complex, you share my disappointment on the UX evolution of the mainstream web. I dream of a more semantic web, not necessarily web 3.0, perhaps just more explicitly typed, where e.g. a YouTube URL for embedding would be a video/webm instead of a text/html.

If mailto URIs can launch our email client, and social media pages can bug us to open the post in their own app, why can't we have interoperable media handling? Maybe we should, but I'm not sure if we can. Greed stands in our way. Providers force us to use their proprietary malware to consume their service. DRM has become the foundation of media distribution. Grassroots movements like Framasoft might never reach mainstream status.

I don't mean to tell you to give up though, just to direct your energy to where it matters. Spend less on developing alternative front-ends than on ethical replacements, bridges and inviting people over. We need more videos, more music, more podcasts, more knowledge, better instant messaging, better search engines, better translations, better home automation, and whatnot. Against all odds, maybe things will finally start to improve even for those outside of our bubble. Perchance.

I'm just a language, whose style sheets are good
Oh, Lord, please, don't let me be misunderstood

Introduction

Neural-optic live streaming probably, no, definitely offers the most photorealistic graphics one can set eyes on. CGI is just a pathetic mimic, and photography or videography is no more than a poor plagiarism attempt when compared to quantum ray-tracing and other advanced physics simulations^W happenings.

On the other hand, we humen are rather shite at replaying visual memories, whilst (bit rot aside) media can be archived for forever. Besides, many of us are too busy to touch grass or go see cool things as regularly as we wish to. This is how an industry based on showing us mundane stuff or obvious bullcrap can still manage to make tens of thousands of craploads each year any why the interwebs are flooded with pictures of cats, kitties and pussies.

Finding new shits means dopamine dispensation and that's why they are dope. As a model netizen, I adhere to the web's social contract of mutual shitposting so that everyone can have a piece. Every blue moon, I also enjoy posting more quality stuff like what you are reading right now, should you ignore the number of Mozart references in the last three paragraphs.

Motivation

Some other times, I also want to share the living things and sceneries I encounter in the new place. My camera was gifted by father before I moved and yet I shared more photos with strangers than with my family. The PixelFed instance I landed on irreversibly shrank and lossily compressed them, while dumping 5 MB images to the family chat room just feels weird, hence I decided to gather the decency to build a photo gallery to show my loved ones (and admittedly, flex with online strangers).

There are not many CMS in the wild for photo hosting, and they often acts as a wall garden and/or a social network. Building and hosting a new one is quite overkill, thus the obvious solution left would be generating a static site. Out of the gazillion SSG, I couldn't found any that meets the my requirements:

1. Generate a web feed

2. Automate filling image title and alt text

3. Offer fine-grain control for permanent pagination

4. Generate thumbnails with custom size and name

I mean, they perhaps exist, but the number I had to try and fight through would cost more time than writing the web pages and feed by hand. So I wrote them from scratch. Y'all can stand up and clap now!

Preliminary

Yes, I really started with writing XHTML and Atom by hand. A web page has the following structure with namespaces omitted and denoted in WXML (Wisp\(\times\)SXML) so I don't have to close the tags (have I given up on XML too early?-).

html
  head
    link
      @ : rel "alternate"
          type "application/atom+xml"
          href "/atom.xml"
    ...
  body
    nav
      a : @ : href "41"
        . "PREV"
      h1 "PAGE 42"
      a : @ : href "43"
        . "NEXT"
    article
      @ : id "foobar"
      h2
        a : @ : href "#foobar"
          . "foobar"
      a : @ : href "/42/foo.jpg"
          img
            @ : src "/42/foo.small.jpg"
                alt "pic of foo"
                title "pic of foo"
      a : @ : href "/42/bar.jpg"
          ...
    article ...
    ...
    footer ...

So far, adding an article is not yet too cumbersome, there's only a bit of redundancy for permanent links and the nesting level is acceptable with the deepest being /html/body/article/a/img. It gets more repetitive once we publish it to to the linked Atom feed:

feed
  entry
    link
      @ : rel "alternate"
          type "application/xhtml+xml"
          href "https://gallery.example/42/#foobar"
    id "https://gallery.example/42/#foobar"
    title "foobar"
    content
      @ : type "xhtml"
      div
        img
          @ : src "https://gallery.example/42/foo.jpg"
              alt "pic of foo"
              title "pic of foo"
        img ...
    updated ...
  entry ...
  ...

Since web feeds are standalone documents, they must always use absolute URLs. (Welp that's not entirely true, XML Base does exists, but not all readers support it, and more importantly, certain elements such as atom:id disallow relative references.) In addition, whilst the web page links a thumbnail to the original image to save bandwidths, the feed can be consumed one post at a time, which thus points to the full size version. Therefore, copying the markup to embed it inside the Atom is error-prone and doesn't exactly spark joy.

Approach

I actually already spoiled it in the epigraph,^[1] but for the sake of completeness let us discuss a few possible solutions. What I wanted was to reduce the redundancy of manual input, in other words, a system transforming a custom information-dense format to standard yet sparser ones, which in this case are XHTML and Atom. Given some new photos and their relevant data, the purpose was to minimize the publishing friction.

It's worth mentioning that the goal was not to minimize the input format, the transformation speed, or feedback latency, but all of the above, plus the cost of constructing the tool, incrementally as our requirements slightly changes over time. Our choice for the base programming system shall affect each and every of these aspects and more.

Some technical dimensions are more equal than others, though. For this use case, IMHO immediate feedback loop should be given the number one priority, not only because it'd be frustrating to have to complete multiple rituals just to preview the changes, but also as watching and reflecting file system changes is (sadly still) a difficult problem.

For Linux^[2] there's inotify which doesn't suck, except when it does and misses events,^[3] and the standard POSIX build tool make relies on mtime which is also flaky. Some SSG work around this by spawning up a server with more sophisticated caching mechanism and even include a HTTP server sending out refresh events. Implementing such system is easily more expensive than doing the original task manually.

Luckily, there is another way. After the birth of imperative DOM manipulation programs running on VM inside browsers (Ecma scripts), there came a (now forgotten) art of purely functional DOM transformation. More specifically, XSLT can declaratively transform any XML document to another, and its best part is that modern browsers natively support it, i.e. there's no difference between editing the input document and the hypothetical output XHTML. For better portability and rendering performance, we can still generate the latter ahead-of-time (AoT) during deployment.

Implementation

Going back to the example, the input format could boil down to a more concise XML file, e.g. 42/index.xml:

page
  @ : prev "41"
      curr "42"
      next "43"
  post
    @ : title "foobar"
        time ...
    picture
      @ : filename "foo"
          desc "pic of foo"
    picture ...
    ...
  post ...
  ...

Page Generation

The stylesheet should then be declared at the beginning of the file, so that the user agent can automatically fetch and apply it to render the output XHML:

<?xml-stylesheet href="/page.xslt" type="text/xsl"?>

XSLT is essentially a templating language, similar to PHP (which is also older) and template libraries in your favorite languages. For the ease of reading, I will let the target document's namespace be the default, while aliasing the transformation one as xsl. The stylesheet for the web pages would look something like the following, which should be self-explanatory.

xsl:stylesheet
  xsl:template : @ : match "/page"
    xsl:variable : @ : name "base"
      xsl:text "/"
      xsl:value-of : @ : select "@curr"
      xsl:text "/"
    html
      head ...
      body
        nav
          xsl:if : @ : test "@prev != ''"
            a : @ : href "/{@prev}/"
              . "PREV"
          h1 : xsl:text "PAGE "
               xsl:value-of : @ : select "@curr"
          xsl:if : @ : test "@next != ''"
            ...
        xsl:for-each : @ : select "post"
          xsl:variable : @ : name "id"
            xsl:value-of
              @ : select "translate(@title, ' ', '-')"
          article
            @ : id "{$id}"
            h2
              a : @ : href "#{$id}"
                  xsl:value-of : @ : select "@title"
            xsl:for-each : @ : select "picture"
              a : @ : href "{$base}{@filename}.jpg"
                  img
                    @ : src "{$base}{@filename}.small.jpg"
                        alt "{@desc}"
                        title "{@desc}"
        footer ...

Feed Generation

Similarly, for Atom entries on a single page,

xsl:stylesheet
  xsl:variable : @ : name "root"
    . "https://gallery.example/"
  xsl:template : @ : match "/page"
    xsl:variable : @ : name "base"
      xsl:value-of : @ : select "$root"
      xsl:value-of : @ : select "@curr"
      xsl:text "/"
    xsl:for-each : @ : select "post"
      xsl:variable : @ : name "url"
        xsl:value-of : @ : select "$base"
        xsl:text "#"
        xsl:value-of
          @ : select "translate(@title, ' ', '-')"
      entry
        link
          @ : rel "alternate"
              type "application/xhtml+xml"
              href "{$url}"
        id : xsl:value-of : @ : select "$id"
        title : xsl:value-of : @ : select "@title"
        content
          @ : type "xhtml"
          div
            xsl:for-each : @ : select "picture"
              img
                @ : src "{$base}{@filename}.jpg"
                    alt "{@desc}"
                    title "{@desc}"
        updated : xsl:value-of : @ : select "@time"

The trickier part here is concatenating the entries together. Simple enough, instead of linking to the stylesheet in the data, we can read XML files directly from XSLT.

xsl:template
  @ : match "/"
  ...
  xsl:apply-templates
    @ : select "document('42/index.xml')/page"
  xsl:apply-templates ...
  ...

This allows us to do other cool things, such as embedding SVG in XHTML to make use of the parent element's currentcolor, while keeping the source files separate. It is especially useful for monochromatic icons, e.g.

xsl:copy-of : @ : select "document('cc.svg')/*"
xsl:copy-of : @ : select "document('by.svg')/*"
xsl:copy-of : @ : select "document('sa.svg')/*"

Thumbnail Generation

So far, we have met three out of the four requirements, only thing left is creating the thumbnails. Inspired by Ethan Dalool, I am going for fairly large ones of 1024 px in width,

large enough to comfortably browse the photos without clicking through to the big version of each, and the thumbnails are decently light and not too jpeggy at about 125-150 kilobytes on average.

At such size, I can aim for around ten photoes^[4] per page while maintaining a somewhat decent load time. Plus, since the width of images are hardcoded, page margin could be automatically inferred to never stretch them.

html {
    box-sizing: border-box;
    margin: auto;
    max-width: calc(1024px + 2ch);
}
body { margin: 0 1ch }

To generate the thumbnails, I use epeg together with make for wildcarding:

PICTURES := $(filter-out %.small.jpg $(PREFIX)/%.jpg, $(wildcard */*.jpg))
THUMBNAILS := $(patsubst %.jpg,%.small.jpg,$(PICTURES))%.small.jpg: %.jpg
	epeg -w 1024 -p -q 80 $< $@

The Makefile also define rules for AoT compilation using xsltproc for the web pages and feed. Apparently no feed reader supports XSLT, and for pages runtime processing negatively affect the performance due to the multiple round trips for the stylesheet and the vector icons.

DATA := $(wildcard */index.xml) index.xml
PAGES := $(patsubst %.xml,%.xhtml,$(DATA))
OUTPUTS := $(THUMBNAILS) $(PAGES) atom.xmlall: $(OUTPUTS)index.xml: $(LATEST)/index.xml
	ln -fs $< $@%.xhtml: %.xml page.xslt
	xsltproc page.xslt $< > $@atom.xml: atom.xslt $(DATA) $(wildcard *.svg)
	xsltproc atom.xslt > atom.xml

The full implementation is deployed to px.lumvok.store, mirrored to the OpenNIC domain pix.sinyx.indy reusing the former's TLS certificate, because CA/Browser Forum disallows support for domains not recognized by ICANN and no CA for OpenNIC is mature enough.

Discussion

Okay you built your site using XML macros, so what? The syntax is clunky and you hate it so much yourself that not even a single line of code example here is in actual XML. Doesn't seem like a love story to me!

Like all relationships, it's not that simple. I've learned to not judge a book by its cover and come to the understanding that XML is the (ugly) equivalence of sexp.^[5] Unlike afterthoughts such as C preprocessors, Django-like templates, or even the Wisp-lookalike syntax of Slim, XML stylesheets is in the same data structure. To put it another way, one can use XSLT to generate XSLT from XSLT. Do I need it in this case or ever at all? Probably not, but that certainly makes XSL a lot more attractive in my eyes.

Furthermore, the tooling for XML is highly mature, from editors to linters and processors to rendering engines. It'd be lying to say you ain't fascinated that tis possible to directly feed browsers pure data instead of markup representations. More than that, one can have entirely static API endpoints that are both human- and machine-readable.

XSL is just declarative JS! You are so blinded by your lust for functional programming that you have become the very thing you swore to destroy!

My distaste for Ecma scripts is not due to DOM manipulation. Sure, I do find in-place modification inelegant for documents, but if only that's the only issue. I block them on most sites because they can interact with many things other than just the DOM, imposing privacy and security risks while fucking up the UX.

Architecturally, Ecma scripts enable the absolute bloody worst possible kind of web pages with zero data at all, fetching tiny pieces of content in JSON and turn performance to shit. The user agents then try to salvage efficiency by turning themselves into a distributed system component and adding optimizations that shall never be (ab)used for the sake of users. O ye cycle of doom!

Note that one can make a similar mistake with XSL regarding the number of round trips, and XML stylesheets can provide the same front-end/back-end separation. Both can be used to provide hot loading during development and AoT rendering in production (if not all, then many JS libraries support pre-rendering, ignoring the monstrous dependency graph). At the end of the day, it's not the matter of technology but principle: to be in the users' best interest.

There is nothing complex about the photo gallery, any existing SSG can do the same with minor tweaks! You never needed to write a new one to begin with!

I am wondering the same myself, but keep in mind there are details I've been hiding from in the example. I went all-in for the semantic web with the hope for best portability and accessibility. One thing I haven't mentioned is the lang attribute, e.g. en, vi or fr depending on the post. Adding this to the web pages requires the SSG to be somewhat modular, and even harder for the web feed.

Moreover, generic SSG are not designed to handle the difference in content between a page's article and the feed's corresponding entry, neither for having multiple posts in a single page. Pagination is also commonly implemented backwards, i.e. page 2 being the second latest one, making it impossible to avoid link rot.

Not to suggest that the majority of SSG are poorly designed, just that from a certain amount of context difference, tis cheaper to just redesign from scratch. This is not about XSL vs Go/Python/JS for SSG or web dev in general, but this specific and happen-to-be-far-from-complex case.

Conclusion

At the time of writing, XML has pretty much been superseded by JSON or YAML, for the better or worse. I have no love for YAML for obvious reasons, but it also saddens me to sometimes see JSON being solely used as a container for HTML. I hope that this essay can awaken something in you about XML and remind you about the semantic web in your next project. It worked out for me, maybe it'll work out for you too!

The story between XML and my photo gallery is a fond love story. They were born for each other, there was no drama, everything just werkt. Their romance inspire me to better appreciate stability and maturity, and value those right in front of my eyes yet I had been too blind to see. Anyway, this is getting too long, so Imma end it with another song.

Lookin' for perfect
Surrounded by artificial
You're the closest thing to real I've seen
Sure, everyone has their problems
That's a given
Yours are the easiest to tolerate

I'm open for criticism
But really, is it any room for criticism?

Recently, I've switched my feed reader from Newsboat to Liferea. The latter has a GUI and some extra features which make the experience a lot more comfy. For instance, custom enclosure handling lets me to finally migrate all of my YouTube subscriptions to Atom and conveniently browse and watch videos using mpv. Image support also allows me to directly view web comics.^[1] One of them, The Monster Under the Bed,^[2] does not embed the strips in its feed, but it has comments.

Yes, RSS includes support for <comments>, and I was not aware of it until very recently. I suppose many other people late to the (web feed) party are neither. Since the rise of static sites, feeds have regain popularity, even for Google to reconsider its direction. Compare to RSS or Atom, alternatives have the following shortcomings:

• Usenet is generally obsolete to most people.

• Mailing list messages are immutable.

• Fora and social media are silos.^[3]

• Social media are designed for ephemeral discussions.

• Instant messaging is awful for archival.

On the other hand, news feeds are commonly read-only: only a few readers can render comments and even fewer are able to post one. On the server side, a dynamic server is needed to accept comments. Traditionally, it's the same as the system serving the website. Although this works, it is significantly more costly than a server dedicated to static sites, which scale a lot better.

Hackers have came up with multiple workarounds such as using microblogging or instant messaging to add comments to their static sites, but all require client-side code execution, which is an option for neither RSS nor Atom. Furthermore, JavaScript hurts portability and performance on the WWW, hence it should be avoided unless it is absolutely impossible to implement a feature otherwise. Commenting is not an exception.

Following is my adventure implementing a comment section for this very blog. If you're also up to the task, I think you should view what I did as an inspiration (rather than a reference) and don't be afraid to experiment around until satisfaction.

Choosing Back-End

As mentioned earlier, static sites or not, there still needs to be a dynamic component to accept incoming replies. HTTP requests would be the most portable since all netizen obviously have a web browser, but those are what we're trying to replace here. What else does everyone has nowadays? Something so common that it can be used to identify people upon service registrations? Exactly, emails and phone numbers!

OK, Imma stop horsing around. My back-end of choice would be emails. It's global, it's cheap and federated. Cellular services almost fit the bill, except that they would cost an arm and leg for one to comment around the web everyday via SMS, whose character limit is not facilitating thoughtful discussions either. As for forum, social medium or instant messaging, no platform has nearly as large of an user base as electronic mails.

It's not like any email would fit the comment section though. Especially not the HTML kind with a few hundred kilobytes of embedded CSS, JS and non-content images. From the security standpoint alone 'tis already a no-go. A light markup language like Markdown^[4] would be much better.

One great thing about using a mature technology like email is that we have all use cases covered. Filtering, exporting and parsing emails work out-of-box regardless of one's provider, MUA and programming preferences. I have an SourceHut account with which I can create mailing lists on-demand so I'm using it; however there's no reason exporting from your private inbox is any more difficult, presuming you have set up offline email.

Designing Data Flow

I promise, this sounds bigger than it really is, but first, let's have a glance at how static generators work. Typically, there are three times templating happens:

1. Conversion of individual articles into HTML content

2. Inserting each article content in a page template to create a complete HTML document

3. Inserting multiple HTML contents into one RSS or Atom feed template

At completion, two kinds of output are generated: website and web feed. Similarly, comments have to be rendered for both targets: an HTML comment section for web browsing and a separate RSS feed for each article's <wfw:commentRss>.^[5] Therefore, injections should be done separately at stage 2 and 3. The overall process of static site generation with email comments is illustrated as follows.

For clarity, HTML and RSS input templates for comments and their parent page and web feed are omitted. Path to each comment feed output being injected in the respective web feed item is also not shown in the figure.

Implementation

At the time of writing, this personal website of mine was generated by Julia Franklin, who was neither fast^[6] nor semantic, but was the only one I knew supporting LaTeX prerendering out of the box. Franklin is also rather extendable via Julia functions.

Accepting Replies

Let's start with how each article can be programmatically and uniquely identified. By default in RSS, a GUID^[7] is the permanent URL of the associated web page. I am not exactly a creative person, so I mirrored this idea, although I only used the difference between URLs, i.e. minus the scheme, network location and trailing index.html (Franklin always appends it to the target path of any source file that is neither index.md nor index.html):

dir_url() = strip(dirname(locvar(:fd_url)), '/')
message_id() = "%3C$(dir_url())@cnx%3E"

For maximum portability, threading identification is used in emails' In-Reply-To header, which expects a message ID, which must match <.+@.+>. Once again, to avoid having to think, I opted for the path difference for the left hand side and my nickname cnx for the right. The mailto URI could be then be constructed accordingly:

using Printf: @sprintffunction hfun_mailto_comment()
  @sprintf("mailto:%s?%s=%s&%s=Re: %s",
           "cnx.site@loa.loang.net",
           "In-Reply-To", message_id(),
           "Subject", locvar(:title))
end

The anchor was then added to the page foot:

<a href="{{mailto_comment}}"
   title="Reply via email">{{author}}</a>

Rendering Comments

This is when the fun begins. Julia's standard library does not include an email parser, and I doubt your favorite language does either, unless it is named after a British comedy troupe. Python is often described as batteries included, or at least it used to (seemingly the consensus among current core devs has shifted towards favoring third-party libraries).

On the other hand, it's trivial to pipe an external program's output to Julia, e.g. readchomp(`echo foo bar`) would give you the string "foo bar". Thus, the to-be-written comment generator should take (the path to) a mail box, the message ID of the article and a template, and write the result to stdout. Argument parsing is, again, thankfully in Python's stdlib:

from argparse import ArgumentParser
from pathlib import Path
from urllib.parse import unquoteparser = ArgumentParser()
parser.add_argument('mbox')
parser.add_argument('id', type=unquote)
parser.add_argument('template', type=Path)
args = parser.parse_args()

I then parsed the mbox into a mapping indexed by parent message IDs as follows. They would be HTML-unquoted so that was why I needed to do the same for the input message ID.

from collections import defaultdict
from email.utils import parsedate_to_datetime
from mailbox import mboxdate = lambda m: parsedate_to_datetime(m['Date']).date()
archive = defaultdict(list)
for message in sorted(mbox(args.mbox), key=date):
    archive[message['In-Reply-To']].append(message)

As said earlier, arbitrary HTML content is not exactly suitable for comments. However, it is undeniable that HTML emails have taken over the world and compromises must be made: allowing multipart/alternative of both text/plain and text/html. It is not the only multipart, so are attachments and cryptographic signatures. Since we are only interested in the plaintext part, it is actually easier done than said to extract it:

from bleach import clean, linkify
from markdown import markdowndef get_body(message):
    if message.is_multipart():
        for payload in map(get_body, message.get_payload()):
            if payload is not None: return payload
    elif message.get_content_type() == 'text/plain':
        body = message.get_payload(decode=True)
        return clean(linkify(body, output_format='html5')),
                     tags=..., protocols=...)
    return None

Now all that's left is to render that body and relevant headers as an HTML segment or an RSS item. This is when we revisit the template. Jinja is probably the most popular in Python, thanks to Django and Flask, but its complexity is rather unnecessary. Instead, I went with the built-in str.format.

What are templates for, exactly? Not the complete document, apparently, because that would differs from article to article and increase the complexity for injection. Neither a single comment, as comments are threaded into trees (or a forest) and their relationship can be useful. We gotta meet in tha middle and use recursive templates instead, e.g. for nested comments:

<div class=comment>
  ...
  {children}
</div>

To render linear comments, such as for <wfw:commentRss>, simply move the children out of the item as follows.

<item>
  ...
</item>
{children}

The rest substitutions are mostly just extracted from the email's headers. Another bit that needs some extra decisions, though, is the parameters for the mailto URI to reply to each comment:

• In-Reply-To set to current Message-Id

• Cc set to current Reply-To (if exists) or From

• Subject is inherited, with Re: prepended if missing

This is getting boring with a lot of trivial code, so I'll leave you with a pointer to the completed script named formbox and move on to more interesting stuff.

Injecting Comments

Inserting HTML comment sections is pretty simple. First I wrote a simple Julia function render_comments calling formbox under the hood, then

hfun_comments_rendered() = render_comments("comment.html")

comments_rendered is then injected below the article. For RSS, it took an extra steps:

1. Insert render_comments("comment.xml") to the comment feed template comments.xml (notice they are two different templates) and write it next to the article's output index.html

2. Insert the path of the written comment feed to the <wfw:commentRss> tag in the article's feed item

That's it!

Moderation

I don't want a Terms of Services page, it'd feel too corporate for my personal website, so I will list the rules here:

1. Please be excellent to each other. Disagreements are okay, personal insults are not.

2. Stay on topic. If you want to publicly discuss with me about something else, start a new thread on a mailing list or reach me via social media.

3. Use plaintext emails and do not top post. Markdown inline markups, block quotes, lists and code blocks are supported.

4. Comments are implied to be under CC BY-SA 4.0 unless declared otherwise.

5. I reserve the right to remove any comment I don't like. I generally don't delete comments, but if you want to exercise your freedom of speech, publish it yourself.

6. I do not warrant the availability of the comments either. I will try my best but one day all comments may just disappear, just like this website itself. Archive what you deem important.

7. These rules are subject to change according to my personal liking without notice.

Replies will only be rendered on the website and feed after I see them, so please expect a delay of at least 24 hours. If you are eager to reply to each other, subscribe to the site's mailing list instead.

Internet Protocol version 6 (IPv6), the most recent version of the Internet Protocol, was developed by the IETF to deal with the long-anticipated problem of IPv4 address exhaustion. Despite being superior to IPv4 in multiple aspect (e.g. larger address space, extension headers), IPv6 has not been widely adopted, although it has been semi-standardized in 1998 and fully-standardized in 2017.^[1]

During the transition period, teredo tunneling has been used to give IPv6 connectivity for IPv6-capable hosts that are on the IPv4 Internet but have no native connection to an IPv6 network.^[2] In this article, I will demontrate a way to set up such tunnel up on virtual machines, then examine the packets being sent by IPv6 nodes connected by the tunnel.

Configuration

Virtual Machines

In order to simulate Teredo tunneling, one needs two IPv6 nodes and two routers with both IPv4 and IPv6 access. In total, there needs to be four virtual machines to be set up, thus I went for Void Linux, which is known for its low memory foot print thanks to using runit instead of systemd. To minimize resource usage and speed up the setup process, I chose the barebone live image which uses musl instead of glibc. At boot, the image uses only 40 MB of memory.

For virtualization, I used vert-manager, simply because it is available in Debian's repository (my host OS). For some reason, on amd64, the kernel refuses to boot until I give it over 200 MB, but apparently that is still a really modest number. Networking is provided to the guest OSes via NAT with default configurations.

It is worth mentioning that through virtio, one may use SSH to log into the guests systems from the host OS. I find this especially convenient as it enables me to copy and paste not only commands but also IP addresses between host and guests as well as between guests.

For convenience, from now on, the outside nodes will be referred to as PC A and PC B, on the other hand the routers are named Router A and Router B. Upon boot, they were given an Ethernet interface eth0 with the following addresses.

Local IPv6 addresses were also given but we are not going to need them.

Teredo Tunnel Setup

First, I set up a IPv4 tunnel between the two routers:

# On Router A
ip tunnel add tunn mode sit remote 192.168.122.134 ttl 255
ip link set tunn up
# On Router B
ip tunnel add tunn mode sit remote 192.168.122.127 ttl 255
ip link set tunn up

For this tunnel to be able to act as a Teredo one, the two routers needs to have IPv6 addresses prefixed by 2001::/32.^[2]

# On Router A
ip -6 addr add 2001:2::1/64 dev eth0
# On Router B
ip -6 addr add 2001:3::1/64 dev eth0

Finally, I fellback all IPv6 lookups to the tunnel and enabled IPv6 forwarding:

ip -6 route add default dev tunn
sysctl -w net.ipv6.conf.all.forwarding=1

Teredo Tunnel Usage

The IPv6 addresses of the PCs were set up as follows (0x8067 is PC in ASCII).

# On PC A
ip -6 address add 2001:2::8067/64 dev eth0
# On PC B
ip -6 address add 2001:3::8067/64 dev eth0

By giving both Router A and PC A addresses prefixed by 2001:2::/64 (similarly for Router B and PC B), I implied that they can find each other through the local IPv6 network, for example on PC B:

$ ip -6 route | head -n1
2001:3::/64 dev eth0 proto kernel metric 256 pref medium

To use the newly created tunnel, the PCs simple had to be routed directly to the routers:

# On PC A
ip -6 route add default via 2001:2::1
# On PC B
ip -6 route add default via 2001:3::1

The connection could then be verified by running on PC A:

$ traceroute 2001:3::8067
traceroute to 2001:3::8067 (2001:3::8067), 30 hops max, 80 byte packets
 1  2001:2::1 (2001:2::1)  0.572 ms  0.441 ms  0.328 ms
 2  2001:3::1 (2001:3::1)  0.906 ms  0.888 ms  1.049 ms
 3  2001:3::8067 (2001:3::8067)  1.325 ms  1.174 ms  1.091 ms

Analysis

To gain further understanding on how packets are transferred over the Teredo tunnel, I captured and took a closer look at some of them.

Packets Capturing

Fortunately for me^[3], all traffic of guests OSes were wired to an separate interface named virbr0. To capture going through the tunnel, I simply had to tell Wireshark to listen to the interface, while letting PC A ping PC B though IPv6: ping -c1 2001:3::8067. I then skimmed through the packets sent between the two nodes and looked for the IPv6-in-IPv4 ones.

Packet Contents

Catured IPv6-in-IPv4 looks exactly like how I would imagined it to be. The content of the ping request can be partially decoded as follows.

Ethernet Header

• 52 54 00 2b 01 cc: MAC address of Router B (destination)

• 52 54 00 f0 85 c7: MAC address of Router A (source)

• 08 00: EtherType of IPv4

IPv4 Header

• 45 00 00 7c 9b 43 40 00 ff: Some flags

• 29: Protocol of IPv6

• 69 be: Checksum

• c0 a8 7a 86: IPv4 address of Router B (destination)

• c0 a8 7a 7f: IPv4 address of Router A (source)

IPv6 Header

• 60 00 07 e7 00 40: Some flags

• 3a: Next header (ICMPv6)

• 3f: Hop limit of 63

• 20 01 00 02 00 00 00 00 00 00 00 00 00 00 80 67: PC A's IPv6 address

• 20 01 00 03 00 00 00 00 00 00 00 00 00 00 80 67: PC B's IPv6 address

ICMPv6

• 80: Type of ping request

• 00 cf be 03 d9 00 01: Some flags

• e3 0d fe 5e 00 00 00 00 bc d6 0e 00 00 00 00 00 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37: Binary data to be echoed

Conclusion

Via the activities elaborated above, the procedure to set up a Teredo tunnel and the content of the packets travelling through it could be well understood. These understanding may help facilite the adoption of IPv6, even for IPv6 nodes having no native connection to an IPv6 network. I hope that the IPv6 will grow fast enough that I can see the day measures like this tunnel can soon be deprecated.

Never give up... No one knows what's going to happen next.

Preface

Greetings and best wishes! I had a lot of fun during the last week, although admittedly nothing was really finished. In summary, these are the works I carried out in the last seven days:

• Finilizing utilities for parallelization

• Continuing experimenting on using lazy wheels or dependency resolution

• Polishing up the patch refactoring operations.prepare.prepare_linked_requirement

• Adding flake8-logging-format to the linter

• Splitting the linting patch from the PR adding the license requirement to vendor README

The multiprocessing[.dummy] wrapper

Yes, you read it right, this is the same section as last fortnight's blog. My mentor Pradyun Gedam gave me a green light to have GH-8411 merged without support for Python 2 and the non-lazy map variant, which turns out to be troublesome for multithreading.

The tests still needs to pass of course and the flaky tests (see failing tests over Azure Pipeline in the past) really gave me a panic attack earlier today. We probably need to mark them as xfail or investigate why they are undeterministic specifically on Azure, but the real reason I was all caught up and confused was that the unit tests I added mess with the cached imports and as pip's tests are run in parallel, who knows what it might affect. I was so relieved to not discover any new set of tests made flaky by ones I'm trying to add!

The file-like object mapping ZIP over HTTP

This is where the fun starts. Before we dive in, let's recall some background information on this. As discovered by Danny McClanahan in GH-7819, it is possible to only download a potion of a wheel and it's still valid for pip to get the distribution's metadata. In the same thread, Daniel Holth suggested that one may use HTTP range requests to specifically ask for the tail of the wheel, where the ZIP's central directory record as well as where usually dist-info (the directory containing METADATA) can be found.

Well, usually. While PEP 427 does indeed recommend

Archivers are encouraged to place the .dist-info files physically at the end of the archive. This enables some potentially interesting ZIP tricks including the ability to amend the metadata without rewriting the entire archive.

one of the mentioned tricks is adding shared libraries to wheels of extension modules (using e.g. auditwheel or delocate). Thus for non-pure Python wheels, it is unlikely that the metadata lie in the last few megabytes. Ignoring source distributions is bad enough, we can't afford making an optimization that doesn't work for extension modules, which are still an integral part of the Python ecosystem )-:

But hey, the ZIP's directory record is warrantied to be at the end of the file! Couldn't we do something about that? The short answer is yes. The long answer is, well, yessssssss! That, plus magic provided by most operating systems, this is what we figured out:

1. We can download a realatively small chunk at the end of the wheel until it is recognizable as a valid ZIP file.

2. In order for the end of the archive to actually appear as the end to zipfile, we feed to it an object with seek and read defined. As navigating to the rear of the file is performed by calling seek with relative offset and whence=SEEK_END (see man 3 fseek for more details), we are completely able to make the wheels in the cloud to behave as if it were available locally.

   

3. For large wheels, it is better to store them in hard disks instead of memory. For smaller ones, it is also preferable to store it as a file to avoid (error-prony and often not really efficient) manual tracking and joining of downloaded segments. We only use a small potion of the wheel, however just in case one is wonderring, we have very little control over when tempfile.SpooledTemporaryFile rolls over, so the memory-disk hybrid is not exactly working as expected.

4. With all these in mind, all we have to do is to define an intermediate object check for local availability and download if needed on calls to read, to lazily provide the data over HTTP and reduce execution time.

The only theoretical challenge left is to keep track of downloaded intervals, which I finally figured out after a few trials and errors. The code was submitted as a pull request to pip at GH-8467. A more modern (read: Python 3-only) variant was packaged and uploaded to PyPI under the name of lazip_. I am unaware of any use case for it outside of pip, but it's certainly fun to play with d-:

What's next?

I have been falling short of getting the PRs mention above merged for quite a while. With pip's next beta coming really soon, I have to somehow make the patches reach a certain standard and enough attention to be part of the pre-release—beta-testing would greatly help the success of the GSoC project. To other GSoC students and mentors reading this, I also hope your projects to turn out successful!