nomeata’s mind shares - Debian

The carbondioxide footprint of Debian's Haskell packages

mail@joachim-breitner.de (nomeata) — Wed, 24 Apr 2013 13:36:02 +0200

By now, Debian ships quite a lot of Haskell packages (~600). Because of GHC's ABI volatility, whenever we upload a new version of a library, we have to rebuild all libraries that depend on that. In particular, if we upload a new version of the compiler itself, we have to rebuild all Haskell library packages. So we have to rebuild stuff a lot. Luckily, Debian has a decent autobuilding setup so that I just need to tell it what to rebuild, and the rest happens automatically (including figuring out the actual order to build things).

I was curious how much we use the buildd system compared to other packages, and also how long the builders are busy building Haskell packages. All the data is in a postgresql database on buildd.debian.org, so with some python and javascript code, I can visualize this. The graphs show the number of all uploads by autobuilder on the amd64 architecture, with haskell uploads specially marked, and the second graph does the same for the build time. You can select time ranges and get aggregate statistics for that time span.

During the last four days a complete rebuild was happening, due to the upload of GHC 7.6.3. During these 2 days and 18 hours building 537 packages took 48 hours of build time and produced 15kg of CO₂. That is 94% of all uploads and 91% the total build time. The numbers are lower for the whole of last year: 52% of uploads, 31% of build time and 57kg of CO₂. (The CO₂ numbers are very rough estimates.)

Note that amd64 is a bit special, as most packages are uploaded on this architecture by the developers, so no automatic builds are happening. On other architectures have, every upload of a (arch:any) package is built, so the share of Haskell packages will be lower. Unfortunately, at the moment the database does not provide me with a table across all architectures (and I was too lazy to make it configurable yet).

Going to FOSDEM after all

mail@joachim-breitner.de (nomeata) — Thu, 31 Jan 2013 21:03:50 +0100

Earlier this week, things were looking different, and I did have to cancel a Haskell talk in Freiburg on Monday due to illness. But I’m back on track and will be travelling to Brussels tomorrow.

I’ll be holding a talk on how we package Haskell in Debian, on Suday at 15:30. I hope it will be useful to Debian users (who will better understand the packaging), other Debian Developers (who’ll learn about the peculiarities of Haskell and the implications for the Debian infrastructure), other distro’s maintainers (to compare best practices) and Haskell developers (to learn about the needs and worries of downstream packages). The talk will be based on my DebConf 11 talk on the same topic. I’m also happy to answer questions about Haskell, Haskell in Debian or any other topic that you want to hear my opinion about, so just talk to me during FOSDEM.

In related news: GHC 7.6.2 was uploaded to Debian experimental the day it was released; the rebuilding of all libraries is still in progress (~370 of ~570 done).

Creating a Debian source package without unpacking the source

mail@joachim-breitner.de (nomeata) — Sat, 13 Oct 2012 15:03:46 +0200

There may be a situation where you want to create a Debian source package without having to unpack the upstream source. One case is where you store your debian/ directory in a version control system of its own and to build your package, you pass the .dsc file to a tool like pbuilder or some remote builder. Now a modern debian source package is just a text file (the .dsc file) that references two tarballs, one being the upstream source and the other being the Debian directory. Thanks to Niels Thykier on #debian, I found the magic invocation of dpkg-source that allows you to create the .dsc directly from the two tarballs, the control file and the changelog:

> apt-get source --download-only -t experimental haskell-mtl
Reading package lists... Done
Building dependency tree       
Reading state information... Done
NOTICE: 'haskell-mtl' packaging is maintained in the 'Darcs' version control system at:
http://darcs.debian.org/darcs/pkg-haskell/haskell-mtl
Need to get 18.8 kB of source archives.
Get:1 http://http.debian.net/debian/ experimental/main haskell-mtl 2.1.2-1 (dsc) [1616 B]
Get:2 http://http.debian.net/debian/ experimental/main haskell-mtl 2.1.2-1 (tar) [13.7 kB]
Get:3 http://http.debian.net/debian/ experimental/main haskell-mtl 2.1.2-1 (diff) [3509 B]
Fetched 18.8 kB in 0s (22.3 kB/s)    
Download complete and in download only mode
> cp ./haskell-mtl_2.1.2-1.dsc ./haskell-mtl_2.1.2-1.dsc-orig
> tar xzf haskell-mtl_2.1.2-1.debian.tar.gz
> dpkg-source -cdebian/control -ldebian/changelog --format="3.0 (custom)" --target-format="3.0 (quilt)" -b / haskell-mtl_2.1.2.orig.tar.gz haskell-mtl_2.1.2-1.debian.tar.gz
dpkg-source: info: using source format `3.0 (custom)'
dpkg-source: info: building haskell-mtl in haskell-mtl_2.1.2-1.dsc
> diff  -u haskell-mtl_2.1.2-1.dsc haskell-mtl_2.1.2-1.dsc-orig
--- haskell-mtl_2.1.2-1.dsc	2012-10-13 15:09:00.031695856 +0200
+++ haskell-mtl_2.1.2-1.dsc-orig	2012-10-13 15:08:47.267696329 +0200
@@ -1,3 +1,6 @@
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA1
+
 Format: 3.0 (quilt)
 Source: haskell-mtl
 Binary: libghc-mtl-dev, libghc-mtl-prof, libghc-mtl-doc
@@ -23,3 +26,11 @@
 Files: 
  943c110524d96126bfa0e61f7df1ebcd 13723 haskell-mtl_2.1.2.orig.tar.gz
  dda5ded58a8d009ecddeed68058ca787 3509 haskell-mtl_2.1.2-1.debian.tar.gz
+
+-----BEGIN PGP SIGNATURE-----
+Version: GnuPG v1.4.12 (GNU/Linux)
+
+iEYEARECAAYFAlBzRQIACgkQ9ijrk0dDIGwymwCfW87jwCv5ZvAXPBwh0LbvYLP9
+n+wAn3TCJwPRuBDEWRYmcv3Opn1Kb9+F
+=/Llc
+-----END PGP SIGNATURE-----

As you can see, the resulting .dsc file is identical to the original one, so this approach works. The parameter to -b does not matter as long as it is some existing directory. Of course you can easily shoot yourself and others in the foot, almost as easily as hand-editing the .dsc file... but if you know what you are doing, then this can be quite handy.

Next to do: Write a tool that takes an unpacked Debian directory as an argument, packs it, finds the original tarball using uscan and creates the .dsc file. Would this be something that people would want in the devscripts package?

10 years of using Debian

mail@joachim-breitner.de (nomeata) — Sat, 12 May 2012 12:00:00 +0200

Today, it must have been exactly 10 years that I started using Debian. The story of how I came to Debian shows some of its strengths, so I’ll use this occasion to share it.

I spent the first half of 2002 as an high-school exchange student in Wenatchee, USA. I was already a user of Linux at that time: I made my first contact roughly in 1996 and did my first installation at home two years later, but all that time I was dual-booting and my main system was a well-arranged Windows 98. The machine was a regular tower PC, but nevertheless I put the computer into my trunk when I flew to the US. It took away most of the space, and I had to put some of my cloths inside the case.

So I was there, happily using my Windows and my manually set up “Linux From Scratch” until one day the inevitable happened; inevitable at least until you start doing backups: On April 30th, my hard drive crashed, and took the two systems together with 4 years of personal data with it.

Two weeks later I had a new hard drive and was pondering my options. I did plan to install Windows again; at that time Windows XP was just released. But I wanted a German version of Windows, which would be hard to get there. Also, I did not want to use Linux from Scratch any more, and wanted to make a well-founded choice of a distribution. On the other hand, I really wanted to get my machine up and running quickly, to be able to read my mail more comfortably. I had heard that Debian had good support for network installations (downloading a full 700MB CD was something to avoid at that time), so I grabbed some netinst images, burned a CD, and quickly installed Debian.

I was planning to use the system for about two weeks. I did not pay any particular attention to the setup. Heck, I even picked from my Simpsons sidekick machine naming scheme one that I would not miss being used up (“barney”). Nevertheless, I was using this installation for many years (and many upgrades), until I eventually switched to using laptops. In fact, that very installation is still on the machine somewhere and works. I did install Windows XP a few weeks later as well, but hardly used it. So May 12th of 2002 was when I turned into a full-time Linux and Debian user.

I soon became interested in Debian the project and started to contribute. But that is another story for another ten year anniversary blog post, most likely on October 21, 2013...

Some late recognition of metainit

mail@joachim-breitner.de (nomeata) — Sat, 03 Dec 2011 22:19:11 +0100

Four years ago, during DebConf 7 in Edinburgh, I had the idea of creating init scripts not by hand, but rather describe them in a declarative way that handles most common cases and generate the real init scripts from them. Back than, this was partly motivated by the varying quality of init scripts and the advent of an alternative init system, upstart, for which proper service files could be generated as well.

I implemented the idea and uploaded it as metainit to Debian, but it did not take off. Only one package uses it, and this package is not used much either. The project basically fell dormant.

Yesterday, I got this mail:

Hi Joachim,

I'm not very familiar with Debian ecosystem, however our company has recently switched to exclusively using Debian based servers, please forgive me if there exists a better channel for sending patches.

We need (in house) to create a lot of trivial services and the sysv init system seems an ideal way to manage them, and metainit an great way to keep the config simple.

However all our scripts require a heartbeat, the way we normally do this is with a bunch of checkers launched from cron. Metainit was ideal, but lacked the simple 'status' task found in most init scripts.

This small patch adds that task, I hope you and others find it usefull.

Regards,

together with the mentioned patch. So it turned out that the idea was a useful one after all, although it found its use on our user’s machines, and not within the Debian packages, as I originally anticipated. Nice!

During the recent discussions about the latest init system, systemd, there were more thoughts about generating init scripts/upstart service descriptions/systemd units from a declarative description, and if metainit had taken off back then, things would be easier now. So if anyone feels like reviving the project: Please do.

Help wanted maintaining link-monitor-applet

mail@joachim-breitner.de (nomeata) — Sat, 15 Oct 2011 16:16:47 +0200

Jean-Yves Lefort’s link-monitor-applet, an applet for the GNOME panel, is a great tool for those with often-changing and possibly unreliable network connection. It allows me to see, at one glance, whether I can reach the router, the internet and/or my VPN. Furthermore, the latency display tells me whether there is a point in working over SSH right now. I often observe other people, e.g. on conferences, to stare for a while at a non-loading web page, then fire up a terminal window and run random ping commands, while I immediately see that the network is down and can relax.

Unfortunately, Jean-Yves seems to have stopped working on link-monitor-applet and does not reply to bug reports. This was not a big deal as long as I could use the last released version with minor patches of mine, but now gnome-panel 3 is coming up and all applets need porting to GTK3. Yesterday and today I worked on that, and got it so far that the applet is fully functional, although the visual styling is not perfect yet, because I’m not sure how to do what Jean-Yves did with GtkStyle and GdkGC now with GtkStyleContext and Cairo. The code is in the gnome3 branch of my git repository.

For this issue, and any further maintenance work, I’m looking for someone who is willing to take upstream responsibility for link-monitor-applet, e.g. making sure it works with the latest GNOME (well, GNOME in fallback mode) and that bug reports are handled. The code base is not too big, and both the non-standard build system and the gob-enhanced C code are fine to work with. I’ll still be happy to serve as the maintainer of the Debian package of link-monitor-applet.

Yesod packaged for Debian

mail@joachim-breitner.de (nomeata) — Sun, 25 Sep 2011 22:39:04 +0200

As Clint writes on his blog, we have now Yesod, a Haskell web framework, packaged in Debian. Despite his humble choice of words, it was mostly his work, and it is quite some work, given that Yesod has a huge number of dependencies that were not yet packaged.

So if you want to use Yesod on Debian, and prefer pre-built libraries without dependency problems and clean installation/deinstallation with dpkg over having always latest and greatest, this should provide you with a solid base.

SAT-solving the testing transition problem

mail@joachim-breitner.de (nomeata) — Tue, 26 Jul 2011 11:23:35 +0200

It is half-time for me at DebCamp/DebConf11 in Banja Luka, Bosnia-Herzegovina, and that is a good time to write about my work so far. I started with some finger exercises, such as the HTML copying for gnome-terminal (as blogged previously) and cosmetic changes to the buildd.debian.org /status website. After that, I began my main project: A SAT-based solver for the testing transition problem.

In Debian, new software first enters a repository called unstable, where it is tested. Once some requirements are fulfilled (package has been in unstable for usually at least 10 days and has no new release critical bugs), it is entitled to enter the repository called testing, which will eventually form a new stable Debian release. But some packages need to migrate together, usually because either requires the new version of the other. Also, no packages in testing ought to become uninstallable by some seemingly unrelated change.

The software that decides these things is called britney. It is sufficiently good in making sure nothing bad happens, but not smart enough to figure out what to do in case more than two packages need to migrate simultaneously. My goal is to improve this. Now, the the various requirements can all be expressed as predicative formulas, and a lot of research has been going into writing good solvers for such problems, called SAT-solvers. Hence my plan was to only specify what we expect from a testing migration, but leave the search for a solution to such a general purpose and highly optimized program.

My initial progress was good and I had some result after two days of hacking, and the approach is promising. Because the data sets are pretty large (1.5GB of input data, the final SAT problem has 1.8 million clauses with 250.000 variables), so I learned quite a bit about profiling and optimizing Haskell programs, and that parsec is slower than working with BS.lines and BS.split to parse simple input. I also used the FFI to use dpkg code directly to compare Debian version numbers.

At some point I noticed that I actually want to solve PMAX-SAT problems: Given two set of clauses, hard clauses and soft clauses, find a variable assignment that fulfills the hard clauses and as many of the soft clauses as possible. Unfortunately, there are no fast Free PMAX-SAT solvers around. The ones that I found and that were fast enough that I could use them, msuncore by Joao Marques-Silva at the University College Dublin and MiniMaxSat by Federico Heras at the Universitat Politècnica de Catalunya, are only available as statically linked binaries. It is a shame that this is acceptable by the academic community; imagine mathematicians would stop including the proofs in their papers, and only share their theorems.

So if you happen to have written a PMAX-SAT solver and can solve this instance (in weighted DIMACS format) in less than five minutes, and want to brag that the Debian project is using your code, then please release it under a Free license (e.g. GPL or BSD) and tell me about it!

The code of my SAT-based solver is, of course, available, though slightly unpolished. Parts of it (notably the interface to picosat and the PMAX-SAT solvers) might be of general interest and I plan to put them on Hackage as a separate license.

Update: I have contacted the authors of the SAT solvers mentioned above, and they have reconfirmed that they have no intention of releasing the source. Now I put my hope in maxsatz2009 by Chumin LI, which is GPL but was not able to cope with my large instance directly; I think I need to change the memory management.

How forky may one maintain a Debian package?

mail@joachim-breitner.de (nomeata) — Sat, 17 Jul 2010 21:59:47 +0200

I maintain most of my Debian packages because I use them myself. Sometimes, I have some needs that go slightly beyond what is currently offered by the software. This is not a problem: Debian ships Free Software and I can program, therefore I can patch the software to also do what I want it to do. Trying to be a good member of the Free Software community, I then submit the patch to the upstream author. If he accepts the patch (which is usually the case), everything is fine. But what if he does not reply to the report or rejects it because he does not want this feature (although the patch is technically fine)? I see two options:

I could continue to use a privately patched and built version of the package, while separately building packages for Debian. This way, Debian ships the software as intended by the upstream maintainer while I can use the features I need. On the other hand, I would not be using the version that I upload to Debian, which is not good, and it causes double work when a a new version is released.
I could upload a package to Debian that contains my patch. The technical infrastructure to add patch in Debian packages has always been there... I would actually use the package as it is in Debian and only manage one line of versions. But would I be abusing my powers as a Debian maintainer? If I were not the maintainer, I could not make this decision by myself (this happend with my patch to nagstamon). Plus it could have a negative effect on the Debian-upstream relationship.

How do other Debian Developers handle such issues? The actual case I’m considering is a feature enhancement for link-monitor-applet (but I only just wrote the patch, so it does not yet fall in the category “upstream does not reply”).

bluetile in Debian

mail@joachim-breitner.de (nomeata) — Sat, 12 Jun 2010 20:44:10 +0200

I just packaged and uploaded Jan Vornberger’s window manager bluetile to Debian. This very nice piece of software brings the benefits of a tiling window manager to users who prefer to use the mouse and who don’t want to learn a new programing language to configure their window manager. Bluetile uses the xmonad libraries and extends them with an easy to use and discoverable user interface.

Serna XML editor uploaded to Debian

mail@joachim-breitner.de (nomeata) — Mon, 25 Jan 2010 12:28:33 +0100

The XML-Editor Serna by Syntext has been published as Free Software a few months ago. This was very good news, because there was a lack of a good free XML editors with a good graphical view on DocBook documents, which I needed to recommend to users of zpub. Therefore, I investigated packaging Serna for Debian. I had to patch a few things to make it compile on and64 and to use components shipped by Debian where possible. Today, I could finally close the RFP bug filed by W. Martin Borgert, as the serna package was accepted by the ftp-masters. The first bug (SEGFAULT on startup on lenny) is already filed. I hope this is a good sign, as it shows that there is interest in the package.

For my packaging workflow, I used git-svn to import the upstream SVN branch into a git repository. I then use git-dpm by Bernhard R. Link to manage my changes as patches in the new 3.0 (quilt) debian source package. I must say that I prefer this approach to git-buildpackage, as there is only one git branch to publish. I hope that Bernhard uploads git-dpm to Debian soon.

Serna is quite a big software project and uses stuff that I know little about (Qt, C++ with python interaction etc.). Also, the package currently bundles the DITA-OT package, which should rather be packaged separately. Therefore, I’d be glad if co-maintainers would join the effort.

DebConf mugshots view statistics

mail@joachim-breitner.de (nomeata) — Sat, 09 Jan 2010 15:02:29 +0100

In a comment to my previous blog post, I linked to the DebConf photo gallery page with the DebConf8 mugshots. Felix Brandt, a friend of mine, noticed the per-picture view statistics there and plotted them, differentiating between male and female. He finds that the number of views on an image gives a fairly good indication of the sex (or gender?) of the person in question:

The most notable exception is the image of bubulle. He does not even look feminine. Maybe it’s because he’s like a mother to us Debianers, always kind and always helpful? :-)

This observation fits my experience when I created a top-100-statistic of individual picture page views of my personal gallery: I got a collection of pretty much all pictures of girls in bikinis, lying at some beach, across the various pages and years, and hardly any other picture. I won’t post these top-100 here, as I don’t want to additionally increase the effect...

Already dead

mail@joachim-breitner.de (nomeata) — Sat, 25 Jul 2009 22:47:30 +0200

This year’s game of Assassins at DebConf 9 was quick. I did not even had a chance to check who my target would have been, and already I got touched by a sock...

Stronger GPG key generated

mail@joachim-breitner.de (nomeata) — Fri, 10 Jul 2009 11:07:49 +0200

In time for the upcoming DebConf keysigning party I followed the crowd and generated a new 4096-bit RSA GPG key, which you can find on the keyserver near you. But I plan to use my old key for a while, until the new one is properly integrated in the Web of Trust. I still have a batch of cards with the old key on it as well, so I won’t revoke that too soon.

pub   4096R/F0FBF51F 2009-07-10
      Key fingerprint = 1A46 087F 955D 93C5 7C60  571B 3D90 8AB3 F0FB F51F
uid                  Joachim Breitner <mail@joachim-breitner.de>
uid                  Joachim Breitner <joachim.breitner@itomig.de>
uid                  Joachim Breitner (Jabber-ID) <nomeata@joachim-breitner.de>
uid                  Joachim Breitner <nomeata@debian.org>
sub   4096R/6F927969 2009-07-10

I’m also considering to play the keysigning game a bit less intense. I reached a rank of about 20 two months ago with my old key and had my fun.

darcswatch uploaded to hackage

mail@joachim-breitner.de (nomeata) — Wed, 21 Jan 2009 23:17:32 +0100

I just made my first upload to hackage (not counting uploads I did during my work in Dresden). Don Steward repeatedly told me to package and upload darcswatch, so I just did that. Thanks to Gwern Branwen to contribute the first cabal file.

There is little documentation on how to set up darcswatch yourself, so if you actually want to try it out, you most likely will have to get in touch with me. Note that you can just use the installation on http://darcswatch.nomeata.de/.

If you, for some reason, feel like hacking on darcswatch, I’d be interested in memory reduction. I currently process 916 mails containing 1867 patches and 47MB, as well as 13 repositories, some of which are rather large, and the numbers are increasing.