The Germination

To understand my 2020, we have to travel back a few months, when it all started. No, not that thing beginning at the end of '19. I am talking about my 2020 experience, remember?

The story started in October 1810 in the not-so-little city of Munich, Germany. Alright, it sounds like I lied about the 2019 and my story part, but bear with me, it's all connected. Anyhow, some Bavarian couple got married and threw a big party. People like parties, so naturally they celebrated the anniversaries, year after year until it became a tradition known to in English as the Oktoberfest.

Over two centuries years later, on the wedding day of another Bavarian couple,^[1] DigitalOcean began to an annual PR campaign on the same month called Hacktoberfest. I know, to many of you maintaining projects on GitHub (and more recently GitLab.com), the name might not remind you of something festive, but it really opened a new chapter in my life.

Back to the future in 2019, it was my first year taking part in the event. The premise was that one would receive a t-shirt after having filed at least four GitHub Pull Requests™.^[2] Unlike plethora, this does not sound like it was a lot, yet more than I ever had done. Getting out of my comfort zone was the first baby step, opening various opportunities in the upcoming months and perhaps, years.

The Fruition

Probably what I benefited the most from participating in Hacktoberfest was learning to not be afraid of communicating with complete strangers maintaining the software I use. Stepping into 2020, I started to do a larger variety of stuff in Python, which made installing libraries happen on a regular basis. The international Internet connection from home at the time was unstable and usually downloads from the package warehouse was a few kBps and that definitely did not help. A few moments later, I found myself on PyPA's IRC channel discussing strategies to speed up pip downloading.

After several days of on-off conversations (mostly I was asking questions to fill in the blanks), a proposal was under draft: I was an undergrad sophomore and had been eyeing on Google Summer of Code (GSoC) for quite a while. Applying for pip wasn't the plan, but rather Octave, the first big project I have contributed code to.^[3] Now thinking about it, it was a better choice since I was more comfortable with pip's tech stack. The rest of the story was already noted down so I won't be retelling it here.

The Disease

When the world had been battling SARS-CoV-2 for a few months, Việt Nam was barely affected. By refusing inbound travelers and temporary switching to work/study-from-home, the number of cases and deaths was neglible and by the end of summer we were virtually back to normal. I hated that most organizations, my university included, straight up offered big techs our data without a second thought, and was thankful online learning did not last.

Like many others, I spent that summer rarely leaving the house. I was grateful of GSoC for keeping me busy and giving me the opportunity to socialize with new cool people. It was impossible for me to catch the virus, I thought. I was not wrong though, but I got something else: dengue fever. The fever wasn't too bad, I was high as a kite for half a week, but never critical. The aftermath, however, was much less pleasant.

For the next week, I was in a living hell because of a throat infection. I'd had sore throats before, quite regularly in fact, often at least once every few months, but they had been a mere inconvenience. Usually, all I'd gotta do had been to person up, swallow a few times and get on with my day. This was different. Everything hurt like a bitch. The slightest texture or flavor could cause minutes of pain. For the first time, I experience throat lozenges being the opposite of soothing.

For the entire week, I survived on undercooked scrambled eggs and mushy porridge. I had to take α-chymotrypsin^[4] before every meal and was practically microdosing it throughout the day to be able to drink water. You can't imagine how happy I was when I could finally eat rice again. While the infection was not directly caused by dengue (it only weakened my immune system), the trauma was enough to make me finally care about home mosquito eradication. Guess who learnt it the hard way!

The Profit

GSoC gave me in stipend 3000 USD, minus Payoneer fees and shitty currency exchange "tax". That was the largest sum I'd ever had in my hands. Because of the low cost of living in Việt Nam,^[5] I no longer completely financially dependent on my parents. I could pay my own school fees (scholarship would give back the money months after paying), hang out more with friends (we had zero-COVID for a while, remember?), tip free software projects and services I had (and have) been using for years.

More importantly, I could buy myself future e-waste. I got a Model M so that I no longer need to change keyboard every year, a lefty Ploopy to ease my traffic-accident-injured right wrist that's prolly never gonna fully heal, a new DAP to replace my dead walk man, my first phone and perhaps some other things. No worries, I'm still daily driving them today, they ain't ended up in the landfill (yet).

The Migrations

Admittedly, the first freedesktop.org smartphone caught my eye was actually the Librem 5, which I could afford neither the time nor the money for. I know, the terminology sounds ridiculous, but Linux would include Android and GNU'd exclude postmarketOS. Anyway, Purism, the company behind the Librems, has seriously invested in adaptive GUI and federated services. My first ActivityPub account was provided by Librem One.

It was not the first time I use a federated service. I've used email for as long as I can remember and begun to use Matrix intensively since I entered university. So what (were there to be) changed? At the time, my online presence^[6] was primarily inside surveillance capitalist walled gardens. I was mostly active(ly posting) on bird site socializing with people I acquainted during my GSoC and publishing my development/shitpost^[7] videos to YouTube.

Nothing on fedi really caught my eyes, until I got (hyped up for getting) my PinePhone. Its software landscape was incredibly fast moving back then. Most peripherals were barely working. Desktop programs were being ported for narrower screens using brand new convergent libraries. Many developers were contracted by Purism or sponsored by Pine64, a large fraction of whom are free software purists, rejecting spyware disguised as social media. Never before, hanging out in chat rooms^[8] and the Fediverse were the absolutely best ways to keep up with life-quality-changing updates.

Like with desktop-handheld convergence, I was impressed with Fediverse's interoperability between multiple media formats, from (micro)blogs to picture albums to videos. Imagine being able to share and comment on a YouTube directly from Twitter! Shortly, I registered for a PeerTube account and migrate all my videos there. The longer I stayed on fedi, the more cool stuff I found and the more satisfied I was. Fast forward over two years, I have deleted or permanently logged out of most; only quiddit^[9] is left.

One thing led to another, Martijn Braam's apps introduced me to SourceHut, which embraces email for federation and focuses on useful stuff like SSH for CI, instead of trying to be a social media or relicense the projects it hosts. I have moved most of the software I maintain from GitHub to sr.ht, but the network effect is too strong: I still have to stick around with the former to contribute to software I regularly use.

However, it's unlikely that most of those growing up with GitHub, especially inexperienced contributors, will be willing to adapt to a workflow revolving around mailing lists for such kind of forge to become mainstream again. On the bright side, I start to seeing more larger projects hosting their development platform, and I am watching forge federation with great interest.

The Moral

At this point, you probably wonder, what I am trying to tell from all these random rambling. Welp, nothing. My life is not like the movies, there ain't no plot, no meaning. The whole point of this log is to bridge the gap between /blog and /blog/2020/gsoc. 2020 was indeed positively life-changing for me, tho/so I can't expect most of y'all'll be able to relate. 2023 is already underway, and I hope we will all have a year we can look back to the same way I did in this post. Perchance.

Earlier this year I enrolled in a master's programme^[1] at UNIST and joined the Programming Languages and Software Engineering lab (PLaSE) as a student researcher. The stipend covers the school fees and living expenses, and I'm given an academic freedom to choose what to work on and take risks. I will review the life here in detail in another post, but (SPOILER ALERT!) overall I'm quite content with it.

That being said, PLaSE is new and small, we only do research on software engineering and don't do its name justice. Because of that, in the first year here I decided to do each day of Advent of Code^[2] in a language I'd never used in competitive programming (CP) before.

Here was my blacklist going in, chronologically: Pascal, Python, Scheme, C, C++, Common Lisp, Lua, Raku, Go, Rust and Zig. I am only proficient in over half of the listed languages, but dura lex, sed lex, I'd already had my CP first time with the rest.

To try any new language, all I have to do is dropping into an ephemeral shell with its implementation using nix-shell or guix shell without the fear of bloating up my systems. I'm running NixOS on my laptop with nixpkgs being one of the largest downstream repositories, including everything but the kitchen sink. On the work desktop, I installed Guix System which has a decent set of packages and nix service in case something is missing. Every update, I run garbage collection and get rid of all unnecessary software, i.e. those not declared in my config.

Day One

The first day should have been the warm up so I challenged myself with using POSIX utilities. This is a bit irony though as the majority of my time spent outside of the editor or a web browser is inside a (Bourn-again) shell.

The problem was indeed simple, involving only finding the maxima among the sums of newline-separated numbers. I used sed(1p) to turn the input into dc(1) eypressions, and sort(1p) and tail(1p) for picking the largest sum. Probably the most interesting part was that the summation was reusable to grade an assignment for a course I was a teaching assistant for.

Day Two

The second problem didn't ramp up much in difficulty. It only called for some rather simple arithmetic, and the input format's regularity convinced me to finally give Hare a try.

For just a taste, Hare is boring in a good way. I was excited for the tagged union of error which can include and propagate any debugging information, but unfortunately it wasn't needed for programs of such complexity (nor that errors are ever handled in CP). I'm looking forward to a chance to write more Hare in the future.

Day Three

The task for day 3 was literally day \(1 + 2\) in scope. I went for another better C that is Nim. My first impression with it wasn't positive: Nim insists on considering each source file as a module and does not allow hyphens in identifier name, so filenames mustn't have any hyphen either. This had led me to piping the source code to nim c - and executing ~/.cache/nim/stdinfile_d/stdinfile to keep my solution naming convention. nim r - wouldn't have worked either since the convention also consists of reading the input from stdin.

On the bright side, uniform function call syntax, identifier case-(and underscore-)insensitivity and optional parentheses allowed me to dodge parentheses in calls and camelCasing altogether. Although I love Lisp and don't have any problem with brackets, I think their placement in ALGOL style hurts the readability of nested calls and camelCase is just objectively bad, pun^[3] unintended.

Day Four

The forth problem wasn't any harder, only requiring simple logic operations and summation. To save time, I opted for Julia, which I was kinda sorta familiar with in building this site (at the time this is published at least). Like Nim, it has higher-order functions and a (reference) compiler capable of producing fast binaries.

Day Five

The next day's task was finally a breath of fresh air with matrix parsing and LIFO (literal) stacks. It begged for a regular expression parser,^[4] hence I mined a tiny bit of Ruby for the task. Ruby had been designed to be an object-oriented Perl, and expectedly it feels very similar to Raku. To an extend, I was also able to avoid ALGOL-style call do quite a Lisp impression.

When I was looking for a second language to learn after the peak of my CP career in middle school, I was choosing between those with garbage-collection that are most popularly used in free software at the time, namely Perl, Python and Ruby. Perl was ruled out due to my fear of sigils and I picked up Python as I didn't want to be a weeaboo. Sometimes I wonder how my side projects would have turned out to be had I chosen differently.

Day Six

The sixth problem essentially asked for maintaining a finite queue of English letters until it is distinct. The most efficient way to do this is employing bit shifting for the FIFO and a bit set for the letters. I implemented that literally in the Better C subset of D.

Although the language is around my age and influenced the big names like modern C++, Swift and Zig,^[5] its documentation is pretty underwhelming and inconsistent. For instance, the 128-bit integer type cent is documented as a basic data type, however it only exists in the core.int128 library with more cumbersome usage (and doesn't work with dmd -betterC).

Like with Nim, D compilers also don't allow hyphens in source filenames, so I had to pipe the code to dmd -of=a.out - (the executable name would be randomized otherwise).

Day Seven

On the first Wednesday of the month of celebration, the problem was parsing cd and ls-like invocation and output to reconstruct a directory tree and do, uh, tree stuff. Janet's PEG module was much more delightful for parsing than regular expression on steroids like Raku's grammar.

Writing imperative S-expressions felt dirty, though it's IMHO a quite better take than Lua, understandably as it was originally a redesign of Fennel.

Day Eight

The eighth problem could be efficiently solved via dynamic programming on multidimensional arrays so I used Fortran for array programming. There's not much to say other than that it werkt and, ah yea, dynamic allocation didn't seem worth the effort so I ended up hardcoding the sizes.

Day Nine

The ninth task was about sparse matrix transformation. Naturally I used hash table in Tcl for this purpose and the solution was straightforward enough. I am planning on extending a video game's level configuration to be programmable and the top contenders are now Lua/Fennel, Janet and Tcl. No idea when I'll get to it, but I'mma keep ya posted.

Day Ten

On day 10, I needed to build a less-than-basic^[6] calculator. I thought using AWK would spice things up a bit, but it actually simplified the solution. Instead of having to read and parse each operation, the script is executed for each input line, even allowing interleaving matching. Therefore, the behavior specification could be followed closely without any significant effort on adapting the logic for the language.

I used to think of AWK as just a more verbose sed(1). I was wrong and am glad that I was. I guess AWK can come in pretty handy for similar real-world usages, such as log processing or moderately complex transformation of textual data.

May Day

Oops!… I did it again.^[2] If you thought because I published this right after Christmas it must be a complete advent journal, I have played you for absolute fools! The later problems were increasingly parsing heavy, and while I still had languages I wanted to try, none left was designed for text processing. I was also busy in meatspace at the time thus I couldn't find the time to write byte-level parsers in languages I didn't know.

I didn't try really hard nor got really far, but in the end maybe the real treasure was the experiences I had along the way. I suppose the contact hypothesis might be true, at least in this context; my prejudice against many languages had been cleared away even after surface-level interactions. You should probably also give it a try, who knows, it could be much gayer than you'd expect!

Late April 2021, Việt Nam witnessed the beginning of the fourth wave of SARS-CoV-2 after a few months without any community case. Soon enough, students are told to not come to their schools' campus. This happens when I was an intern at USTH ICTLab, so I was advised to work remotely as well. I asked for this at the start of the internship but my supervisor was rather reluctant, since there was multiple interns working together and communication in person might be most effective. Working from home was beneficial to me in a few important ways:

1. I had a three-monitor setup at home and a more comfortable space.

2. I could have be more flexible working hours at home.

3. I did not have to bike back and forth to the lab (which is 4 km away) twice a day^[1], which could be exhausting in the hot summer.

Thanks to the last point, I also sweat a lot less and as I no longer had to maintain a public appearance, I decided to give #nopoo a try. I had been aware of such practice for quite a few years, but had never thought of actually implementing it until I saw Johnny Harris' vlog, which I can only describe as intriguing. TL;DW the journalist maintained that generally shampoos washed away his scalp's natural oil, and in combination with other hair products made the scalp itchy and unhealthy. His solution was to drop the use of all products completely and so far it had been working for him.^[2]

Well, my head was itchy sometimes (still itchy at the time of writing), alors, pour la patrie, les sciences et la gloire, let's do it!

Day One

I was going full no poo, no soap, no baking soda, no vinegar, just water, raw water. Everything was going as expected, my hair was not as fluffy as usual after washing, but it was easier to get in shape. I didn't really style my hair. Not as a fashion statement, I was (still am!) just rather lazy. Usually this wasn't an issue, unless when my hair was long, it tent to cover my forehead, ears and eyes, which was arguably an uncomfortable experience. Having the hair stay in place was indeed a blessing!

Day Two

My hair started to feel thicker and running hands through it no longer felt simulating. On the bright side it looked fabulous and did not itch.

Day Four

My hair and scalp began to feel greasy. I guess it was because I did not wash it thoroughly that day. With just water one would need to take more effort scrubbing the hair and especially the scalps to return them to a comfortable state. Plus my mentality got worse so my perceived experience could be exaggeratedly negative.

Day Five

I worked out and paid more attention to the hair washing process. It felt noticeably better.

Day Six

The brief revival of my mental health did not last very long:^[3] later that day I was completely autopiloting and accidentally poo'ed myself. It felt fluffy again but I was disappointed that things did not go as planned.

Day Seven

I decided to cut my hair. I had been doing so for a decade when I wrote this, but I got neither better nor faster at it, so it only happens twice or thrice a year. Of course I had to poo myself after to get rid of all the tiny pieces.

Day Eleven

Fast forward a few days it started to feel greasy again, but this time the hair was shorter so it was less of an issue. I began to apply saline to the hair after washing and somehow it helped a lot in improving the situation. Saline was also my solution for face acne in my teenage year (along with finger nails and pillowcase hygiene).

Day Fifteen

At this point the experience had become more stable. My scalp still itched occasionally but seemly less often than when I was poo'ing more regularly. The hair stayed in shape with merely any effort (I didn't even use a comb).

Overall, the difference is barely noticeable otherwise but I think I will be continuing holding my poo for another while, probably in long term. Do not let my experience speak for you, however, try it yourself if you are interested, but keep observing the effect objectively.

In the summer of 2020, I worked with the contributors of pip, trying to improve the networking performance of the package manager. Admittedly, at the end of the internship period, the benchmark said otherwise; though I really hope the clean-up and minor fixes I happened to be doing to the codebase over the summer, in addition to the implementation of parallel utils and lazy wheel, might actually help the project.

Personally, I learned a lot: not just about Python packaging and networking stuff, but also on how to work with others. I am really grateful to @pradyunsg (my mentor), @chrahunt, @uranusjr, @pfmoore, @brainwane, @sbidoul, @xavfernandez, @webknjaz, @jaraco, @deveshks, @gutsytechster, @dholth, @dstufft, @cosmicexplorer and @ofek. While this feels like a long shout-out list, it really isn't. These people are the maintainers, the contributors of pip and/or other Python packaging projects, and more importantly, they have been more than helpful, encouraging and patient to me throughout my every activities, showing me the way when I was lost, fixing me when I was wrong, putting up with my carelessness and showing me support across different social media.

To best serve the community, below I have tried my best to document what I have done, how I've done it and why I've done it for over the last three months. At the time of writing, some work is still in progress, so these also serve as a reference point for myself and others to reason about decisions in relevant topics.

The Main Story

The storyline can be divided into the following four main acts.

Act One: Parallelization Utilities

In this first act, I ensured the portibility of parallelization measures for later use in the final act. Multithreading and multiprocessing map were properly fellback on platforms without full support.

• GH-8320: Add utilities for parallelization (close GH-8169)

• GH-8538: Make utils.parallel tests tear down properly

• GH-8504: Parallelize pip list --outdated and --uptodate (using GH-8320)

Act Two: Lazy Wheels

As proposed by @cosmicexplorer in GH-7819, it is possible to only download a portion of a wheel to obtain metadata during dependency resolution. Not only that this would reduce the total amount of data to be transmitted over the network in case the resolver needs to perform heavy backtracking, but also it would create a synchronization point at the end of the resolution progress where parallel downloading can be applied to the needed wheels (some wheels solely serve their metadata during dependency backtracking and are not needed by the users).

• GH-8467: Add utitlity to lazily acquire wheel metadata over HTTP

• GH-8584: Revise lazy wheel and its tests

• GH-8681: Make range requests closer to chunk size (help GH-8670)

• GH-8716 and GH-8730: Disable caching for range requests

Act Three: Late Downloading

During this act, the main works were refactoring to integrate the lazy wheel into pip's codebase and clean up the way for download parallelization.

• GH-8411: Refactor operations.prepare.prepare_linked_requirement

• GH-8629: Abstract away AbstractDistribution in higher-level resolver code

• GH-8442, GH-8532 and GH-8588 (later reworked by @chrahunt in GH-8685): Use lazy wheel to obtain dependency information for the new resolver

• GH-8743: Test hash checking for fast-deps

• GH-8804: Check download directory before making range requests

Act Four: Batch Downloading in Parallel

The final act is mostly about the UI of the parallel download. My work involved around how the progress should be displayed and how other relevant information should be reported to the users.

• GH-8710: Revise method fetching metadata using lazy wheels

• GH-8722: Dedent late download logs (fix GH-8721)

• GH-8737: Add a hook for batch downloading

• GH-8771: Parallelize wheel download

The Side Quests

In order to keep the wheel turning (no pun intended) and avoid wasting time waiting for the pull requests above to be reviewed, I decided to create even more PRs (as I am typing this, many of the patches listed below are nowhere near being merged).

• GH-7878: Fail early when install path is not writable

• GH-7928: Fix rst syntax in Getting Started guide

• GH-7988: Fix tabulate col size in case of empty cell

• GH-8137: Add subcommand alias mechanism

• GH-8143: Make mypy happy with beta release automation

• GH-8248: Fix typo and simplify ireq call

• GH-8332: Add license requirement to _vendor/README.rst

• GH-8423: Nitpick logging calls

• GH-8435: Use str.format style in logging calls

• GH-8456: Lint src/pip/_vendor/README.rst

• GH-8568: Declare constants in configuration.py as such

• GH-8571: Clean up Configuration.unset_value and nit __init__

• GH-8578: Allow verbose/quiet level to be specified via config files and environment variables

• GH-8599: Replace tabs by spaces for consistency

• GH-8614: Use monkeypatch.setenv to mock environment variables

• GH-8674: Fix tests/functional/test_install_check.py, when run with new resolver

• GH-8692: Make assertion failure give better message

• GH-8709: List downloaded distributions before exiting (fix GH-8696)

• GH-8759: Allow py2 deprecation warning from setuptools

• GH-8766: Use the new resolver for test requirements

• GH-8790: Mark tests using remote svn and hg as xfail

• GH-8795: Reformat a few spots in user guide

The Plot Summary

Every Monday throughout the Summer of Code, I summarized what I had done in the week before in the form of either a short blog or an (even shorter) check-in. These write-ups often contain handfuls of popular culture references and was originally hosted on Python GSoC.

• First Check-In

• Unexpected Things When You're Expecting

• Second Check-In

• The Wonderful Wizard of O'zip

• Third Check-In

• I'm Not Drowning On My Own

• Fourth Check-In

• I've Walked 500 Miles…

• Fifth Check-In

• Sorting Things Out

• Sixth Check-In

• Parallelizing Wheel Downloads

• Final Check-In

• Outro