Scaling Software Modularization with Topkg

There is one issue that bazaars, library operating systems like MirageOS or random left-padists face: how to scale extreme software modularization. Scaling not from a software composition perspective; this could be readily solved by using the type system and the functional programming techniques everyone’s heard about. But scaling from a purely bureaucratic point of view.

The bureaucracy of making sure main issues are being taken care of. The bureaucracy of making sure changes to the software are being clearly communicated to dependents. The bureaucracy of making sure the software is correctly versioned and is able to precisely self-describe as such. The bureaucracy of making sure the software is a good citizen of the eco-system and that it follows its evolution. The bureaucracy of being able to quickly push fixes and let dependents test them as if they were released. The bureaucracy of making flawless releases: publishing correct and reproducible tarballs, publishing documentation and submitting the software package to the OCaml opam repository.

All this has clearly nothing to do with the fun of programming and calls for efficient, standardized, “don’t make me think” procedures and conventions. To give some perspective I myself formally publish 28 software packages, the MirageOS project directly manages 93 packages, any two minutes administrative task on each of these respectively represent one and three hours gone – dreadful hours. Any non-project specific bit that has to be committed to these repositories is maintenance burden and a curse.

This is where topkg fits in. If you want to learn more about it, make sure to read the documentation basics and the tool manual (opam install topkg-care && topkg help). To sum up, you give topkg enough information so that it is able to determine what needs to be installed in the current build environment. From these bits topkg will infer targets to invoke in your build system – rather than try to generate one – and produce a manifest file to let the package manager install your package.

A few additional conventions and metadata bits you specify allow to lint the package for good practices and release your software with the topkg tool according to a well defined script. Read all the details in topkg help release but the gist of it is:

topkg browse issues # Review remaining outstanding issues
topkg status        # Review the changes since last version
topkg log edit      # Write the release notes
topkg log commit    # Commit the release notes
topkg tag           # Tag the distribution with a version
topkg distrib       # Create the distribution archive
topkg publish       # Publish it on the WWW with its documentation
topkg opam pkg      # Create an opam package
topkg opam submit   # Submit it to OCaml's opam repository

The last four steps can be performed via a single invocation to
topkg bistro.

topkg is far from being a perfect system, it tries to compose with the existing build infrastructure. It is a step towards what is needed from a packaging workflow perspective to scale modularity: the act of releasing should be an inane, swift, robust and painless process.

If you are interested in programming and sharing your results, give topkg a try. No left-pad function left behind.

Related Posts