As we mature, the dependency graph matures with us.

Exposition

In the occasional fights between system and language packagers, I'm known to take the downstream camp. As a user, there are lots of things I take for granted. I install the stuff I need, occasionally upgrade the system, and everything gets updated. Vulnerability in a library used by multiple programs? Its patched version gets swapped in within a few hours (given it's not vendored or pinned). Most distributions even apply hardening flags that some bugs aren't even exploitable in the first place. They create a safe place for me to comfortably express myself at work and at home.

Recently on my work computer, I've switched to Guix System, which has yet many packages. Looking into the way to package programs I use and ongoing efforts, I realized the colossal number of transitive dependencies of certain software and the impracticality for a user union (i.e. a distro) to maintain such set of micro packages in every language.

Confrontation

This gave me a more serious thought on software sustainability. Such topic often reminds us of energy consumption, modularity, development model, or even style (clean code). End-users (including self-hosts), on the other hand, ask the following questions to decide upon installing and keeping a piece of software:

• Can I trust installing this won't do anything funny to my machine?

• How much effort I need to prevent people from doing funny things to my machine if the software includes something that gets on the front page of some magazines tomorrow?

• How much of my limited resources will it take to run or simply exist?

There are certain intersections in concerns of enterprises and users, however it's worth noticing that distributions are almost exclusively optimized to cater for the users' need. Not only they fight for the users, they are the users. Suppose you don't want to write yellow-glowing programs, you should make the life of downstream package maintainers easier. No, it does not count if you push them to give in to run go mod vendor or download from NPM recursively.

Resolution

So how do I write software that is easy to package, you may ask. If you followed the articles linked above, you've probably already figured that out. It's less about what you write and more about what you use. When someone complains a program is difficult to build from source, certainly it's not about how hard it is to type, say make install, but acquiring the dependencies for that to run successfully and the result will work.

Lower the number of dependencies will absolutely help. To put it bluntly, you can't have a problem with dependencies if there's none of them. This sounds like reinventing the wheel, but if the use case is common enough, you might find what you need in the standard library.^[1] I've been restricting myself from using third-party libraries for new side projects and it actually worked for my most recent ones:

• Phylactery, a static comics web server on Go with CBZ parsing and concurrent request handling

• Fead, an SSG plugin in Python for advertising others’ feed with parallel HTTP request, parsing of RSS 2 and Atom and CLI argument parsing

Even for such simple use cases, there are still many libraries in the wild that can handle more data formats, are more convenient to use or more performant. On the other hand, the amount of maintenance needed to keep the programs safe indefinitely for a user is much lower thanks to the small dependency footprint.

What I'm asking you to give a try in the advent days^[2] is not as drastic. Look through your works, find a library you require for a small portion of its power, or something can be implemented specifically for your project using reasonable effort (w.r.t. the whole codebase). This is not just for the sake of maintainability: being less general, the new implementation can likely outperform the replaced public library.

In many cases, you will find yourself making use of the standard library. Standards make life much easier, if only people can come up with an agreement. Or maybe they don't have to. Maybe each could choose among the utilities libraries. At the end of the day, it's the total number of packages that can have bugs to be reported upstream and patched that matters.

That being said, please keep an eye on the standard library the same way you (should) watch your other dependencies, just in case what you need is finally added. Worry not of backward incompatibility, users of LTS systems are content with older versions of your software.

Fall and Catastrophe

Just kidding, I'm offering answers, not tragedies. Winter is coming, join me in a De-Dependency December and fight for the users!

In the summer of 2020, I worked with the contributors of pip, trying to improve the networking performance of the package manager. Admittedly, at the end of the internship period, the benchmark said otherwise; though I really hope the clean-up and minor fixes I happened to be doing to the codebase over the summer, in addition to the implementation of parallel utils and lazy wheel, might actually help the project.

Personally, I learned a lot: not just about Python packaging and networking stuff, but also on how to work with others. I am really grateful to @pradyunsg (my mentor), @chrahunt, @uranusjr, @pfmoore, @brainwane, @sbidoul, @xavfernandez, @webknjaz, @jaraco, @deveshks, @gutsytechster, @dholth, @dstufft, @cosmicexplorer and @ofek. While this feels like a long shout-out list, it really isn't. These people are the maintainers, the contributors of pip and/or other Python packaging projects, and more importantly, they have been more than helpful, encouraging and patient to me throughout my every activities, showing me the way when I was lost, fixing me when I was wrong, putting up with my carelessness and showing me support across different social media.

To best serve the community, below I have tried my best to document what I have done, how I've done it and why I've done it for over the last three months. At the time of writing, some work is still in progress, so these also serve as a reference point for myself and others to reason about decisions in relevant topics.

The Main Story

The storyline can be divided into the following four main acts.

Act One: Parallelization Utilities

In this first act, I ensured the portibility of parallelization measures for later use in the final act. Multithreading and multiprocessing map were properly fellback on platforms without full support.

• GH-8320: Add utilities for parallelization (close GH-8169)

• GH-8538: Make utils.parallel tests tear down properly

• GH-8504: Parallelize pip list --outdated and --uptodate (using GH-8320)

Act Two: Lazy Wheels

As proposed by @cosmicexplorer in GH-7819, it is possible to only download a portion of a wheel to obtain metadata during dependency resolution. Not only that this would reduce the total amount of data to be transmitted over the network in case the resolver needs to perform heavy backtracking, but also it would create a synchronization point at the end of the resolution progress where parallel downloading can be applied to the needed wheels (some wheels solely serve their metadata during dependency backtracking and are not needed by the users).

• GH-8467: Add utitlity to lazily acquire wheel metadata over HTTP

• GH-8584: Revise lazy wheel and its tests

• GH-8681: Make range requests closer to chunk size (help GH-8670)

• GH-8716 and GH-8730: Disable caching for range requests

Act Three: Late Downloading

During this act, the main works were refactoring to integrate the lazy wheel into pip's codebase and clean up the way for download parallelization.

• GH-8411: Refactor operations.prepare.prepare_linked_requirement

• GH-8629: Abstract away AbstractDistribution in higher-level resolver code

• GH-8442, GH-8532 and GH-8588 (later reworked by @chrahunt in GH-8685): Use lazy wheel to obtain dependency information for the new resolver

• GH-8743: Test hash checking for fast-deps

• GH-8804: Check download directory before making range requests

Act Four: Batch Downloading in Parallel

The final act is mostly about the UI of the parallel download. My work involved around how the progress should be displayed and how other relevant information should be reported to the users.

• GH-8710: Revise method fetching metadata using lazy wheels

• GH-8722: Dedent late download logs (fix GH-8721)

• GH-8737: Add a hook for batch downloading

• GH-8771: Parallelize wheel download

The Side Quests

In order to keep the wheel turning (no pun intended) and avoid wasting time waiting for the pull requests above to be reviewed, I decided to create even more PRs (as I am typing this, many of the patches listed below are nowhere near being merged).

• GH-7878: Fail early when install path is not writable

• GH-7928: Fix rst syntax in Getting Started guide

• GH-7988: Fix tabulate col size in case of empty cell

• GH-8137: Add subcommand alias mechanism

• GH-8143: Make mypy happy with beta release automation

• GH-8248: Fix typo and simplify ireq call

• GH-8332: Add license requirement to _vendor/README.rst

• GH-8423: Nitpick logging calls

• GH-8435: Use str.format style in logging calls

• GH-8456: Lint src/pip/_vendor/README.rst

• GH-8568: Declare constants in configuration.py as such

• GH-8571: Clean up Configuration.unset_value and nit __init__

• GH-8578: Allow verbose/quiet level to be specified via config files and environment variables

• GH-8599: Replace tabs by spaces for consistency

• GH-8614: Use monkeypatch.setenv to mock environment variables

• GH-8674: Fix tests/functional/test_install_check.py, when run with new resolver

• GH-8692: Make assertion failure give better message

• GH-8709: List downloaded distributions before exiting (fix GH-8696)

• GH-8759: Allow py2 deprecation warning from setuptools

• GH-8766: Use the new resolver for test requirements

• GH-8790: Mark tests using remote svn and hg as xfail

• GH-8795: Reformat a few spots in user guide

The Plot Summary

Every Monday throughout the Summer of Code, I summarized what I had done in the week before in the form of either a short blog or an (even shorter) check-in. These write-ups often contain handfuls of popular culture references and was originally hosted on Python GSoC.

• First Check-In

• Unexpected Things When You're Expecting

• Second Check-In

• The Wonderful Wizard of O'zip

• Third Check-In

• I'm Not Drowning On My Own

• Fourth Check-In

• I've Walked 500 Miles…

• Fifth Check-In

• Sorting Things Out

• Sixth Check-In

• Parallelizing Wheel Downloads

• Final Check-In

• Outro