nomeata’s mind shares - Haskell

The carbondioxide footprint of Debian's Haskell packages

mail@joachim-breitner.de (nomeata) — Wed, 24 Apr 2013 13:36:02 +0200

By now, Debian ships quite a lot of Haskell packages (~600). Because of GHC's ABI volatility, whenever we upload a new version of a library, we have to rebuild all libraries that depend on that. In particular, if we upload a new version of the compiler itself, we have to rebuild all Haskell library packages. So we have to rebuild stuff a lot. Luckily, Debian has a decent autobuilding setup so that I just need to tell it what to rebuild, and the rest happens automatically (including figuring out the actual order to build things).

I was curious how much we use the buildd system compared to other packages, and also how long the builders are busy building Haskell packages. All the data is in a postgresql database on buildd.debian.org, so with some python and javascript code, I can visualize this. The graphs show the number of all uploads by autobuilder on the amd64 architecture, with haskell uploads specially marked, and the second graph does the same for the build time. You can select time ranges and get aggregate statistics for that time span.

During the last four days a complete rebuild was happening, due to the upload of GHC 7.6.3. During these 2 days and 18 hours building 537 packages took 48 hours of build time and produced 15kg of CO₂. That is 94% of all uploads and 91% the total build time. The numbers are lower for the whole of last year: 52% of uploads, 31% of build time and 57kg of CO₂. (The CO₂ numbers are very rough estimates.)

Note that amd64 is a bit special, as most packages are uploaded on this architecture by the developers, so no automatic builds are happening. On other architectures have, every upload of a (arch:any) package is built, so the share of Haskell packages will be lower. Unfortunately, at the moment the database does not provide me with a table across all architectures (and I was too lazy to make it configurable yet).

Evaluation-State Assertions in Haskell

mail@joachim-breitner.de (nomeata) — Wed, 06 Feb 2013 10:33:03 +0100

I have just uploaded a new version of ghc-heap-view to Hackage that provides “Evaluation state assertions” in the module GHC.AssertNF.

Imagine you are writing a web application in Haskell that sports a global number-of-visitors counter in an IORef Int. For every request, you call modifyIORef (+1). Eventually, you notice your very popular web site to hog more and more memory. So you browse to the internal page that shows the counter, and you have to wait for a long time until you eventually see the result (or get a stack overflow). The reason: The applications of (+1) were not performed until you looked at the number; instead, a long chain of such computation first filled your heap and then your stack.

So you have learned the hard way that you might want to avoid space leaks, and want calculations to be done during the request that caused them, and want the IORef to always contain fully evaluated data. So you stumble about modifyIORef' in Data.IORef and indeed, this fixes your problem.

Later, you notice that you want to count POST and GET requests separately. You change the type to IORef (Int, Int) and call modifyIORef' (first (+1)) or modifyIORef' (second (+1)). And suddenly, the space leak is back (which you only notice after the next push to the real site, because your local tests never caused enough requests to make it noticeable). So you not only want to fix it, you also want to ensure that it does not break again.

In other words, you want to ensure the policy that values stored in an IORef are always in normal form. You achieve this with the following alternative to modifyIORef':

modifyIORef'Assert :: IORef a -> (a -> a) -> IO ()
modifyIORef'Assert ref f = do
    x <- readIORef ref
    let x' = f x
    x' `seq` return ()
    assertNF x'
    writeIORef ref x'

Using this instead of modifyIORef' will print this warning to standard error output right the first time you call modifyIORef'Assert (first (+1)):

Parameter not in normal form: 2 thunks found:
let x1 = (S# 1,S# 1)
in _bh (_thunk x1 (_bco (S# 1)),_sel x1)

(Otherwise, the program runs as usual.) So obviously, you need to use a strict variant of first (or strict pairs):

first' :: (a -> b) -> (a, c) -> (b, c)
first' f (x,y) = let { x' = f x; r = (x', y) } in x' `seq` r `seq` r

With this, the warning goes away. Whenever you now change the type of the IORef or modify it in a too-lazy-way, you can be sure that you’ll be warned about it, before the space leak itself becomes noticeable.

In the production code, you might want to disable the check. For that, simply put disableAssertNF somewhere in your main function.

Why is this better than just calling deepseq in modifyIORef'Assert? Because this way, the code still creates unwanted thunks that are then evaluated before storing them in the IORef, whereas with assertNF you are told about the thunks and can prevent them from being created in the first place. Also, assertNF does not add a type class constraint.

This is just one example application for assertNF (and its variants assertNFNamed, which includes a name in the warning to better spot the cause, and $assertNFHere, which uses Template Haskell to include the current source code position in the warning), and I hope that there are more. If you happen to make use of it, I’d like to hear your story.

“Haskell Bytes” again

mail@joachim-breitner.de (nomeata) — Thu, 10 Jan 2013 23:24:26 +0100

Janis Voigtländer has invited me again to give a lecture in his Advanced Functional Programming course at the University of Bonn. Like last time, when I talked about performance analysis in Haskell, I chose a topic that is more on implementationishy side to contrast the theoretic content of the rest of the course. I recycled my talk on the memory representation of GHC that I have held at last year’s Meta Main-Rhein Chaos Days. As usual I provide a transcript of the talk as intended (not as held). It is slightly revised over the old version and I have added a section on unboxed values in constructors, which I have nevertheless skipped today.

Circle Packing

mail@joachim-breitner.de (nomeata) — Sun, 16 Dec 2012 19:04:17 +0100

For an upcoming introductory Haskell talk of mine (next Tuesday in Karlsruhe – please come and join if you will) I was looking into data visualization as a example application. With libraries like gloss, getting some nice vector graphics up and running within one talk is quite possible.

For some reason, I want to arrange multiple circles of varying size so that they do not overlap and use as little space as possible. My first approach was to use general purpose optimizers such as cmaes, Numeric.GSL.Minimization from the hmatrix package and force-layout. I especially liked the interface of cmaes, which is able to minimize a function of, say, type [(String, Double, Double)] -> Double by automatically finding the values of type Double in the input, as described by Takayuki Muranushi. That would have looked great in the talk. Unfortunately, none of these libraries gave sufficient good results in a reasonable amount of time for this problem.

I then reviewed some of the academic literature on the circle packing problem and there were some algorithms proposed, but no simple one and none of the papers had a concise pseudo-code description that I could transfer to Haskell.

So eventually I set out to implement a relatively dump and slow greedy algorithm myself: I place one circle after another, starting with the largest, and among the valid positions where a new circle touches two already placed circles, I choose the positions closest to the center of mass. The result, available in the circle-packing package, looks surprisingly well, here visualized using the diagrams library (See the documentation of Optimisation.CirclePacking for the code used to generate that image):

Now showing just this image is not a very good way to demonstrate my code. A few years ago, I might have created a CGI script that would dynamically generate images based on your input. But not in 2012: Since my code is pure Haskell without fancy features like type classes, I can use the fay compiler to generate JavaScript code from it. Add a little Haskell code to interact with the HTML5 canvas and now you can interactively try out circle-packing in your browser.

Oh, and while I am talking about neat tricks: You can put vector graphics in your haddock documentation, as I have done for Optimisation.CirclePacking, using the syntax <<<data:image/svg+xml;base64,PD94bWwgdmV...c3ZnPg==>>.

Calculating the internal rate of return with hledger

mail@joachim-breitner.de (nomeata) — Fri, 07 Dec 2012 23:22:41 +0100

For more than ten years now, I am a regular and happy user of GnuCash, the free accountancy program. Especially the integrated online banking support for HBCI that allows me not only to fetch my transactions, but also to transfer money using a cryptographic makes it a great tool. Nevertheless, it has shortcomings, especially when it comes to reports. And for some reason I never felt the urge to hack on that C/Guile code (although I recently hacked a bit on the UI).

Joey Hess’ blog post about Simon Michael’s hledger made me look at that command line based accountancy tool. It certainly won’t replace Gnucash for me, but luckily ledger, of which hledger is a re-implementation, can read GnuCash’s file format and convert it to a ledger file that hledger can read as well. So now I can do analysis on my financial data in Haskell – what would you want more?

One report that I was missing was the internal rate of investment of some account. This is the annual rate of a hypothetical fixed interest-bearing savings account that would, given the same deposits and payouts, result in the same final value. I now implemented this as the hledger-irr command and – as you would expect – you can find it on hackage. See the package description for documentation and example output.

It is very fresh, hardly tested code, so don’t base important decisions on it yet. Also, it needs work to work correctly with multiple currencies or things with varying prices, and I am not yet sure what to do there.

Update: Simon asked me to elaborate the difference to hledger-interest by Peter Simons (on which my code was initial based, but eventually I rewrote everything but the option parsing code). The difference is that hledger-interest answers the question „I have this investment with this yearly interest rate (which may even change), how large are the interest payments going to be.“ I imagine a usecase is that you have borrowed some money to a friend at a certain (surely amicable) rate and need to figure out what he owes you, even when you gave him the money mid year and he payed you some back shortly after easter. hledger-irr serves a different purpose: Here you already know your investment payments, gains and losses and you want to have a number describing “how good” your investment was. One usecase might be that, at the end of a live of paying into a pension fund, you ask yourself: “Given all the feeds and commissions, was this a good idea or should I just have invested the money myself?” Then the internal rate of investment tells you how lucrative an alternative fixed-rate investment would have had to been to match your pension fund investment. (Or course this is not the only aspect that you should look at when comparing investments, as it totally ignores risk and possible insurance elements of the investment.)

c't features Haskell

mail@joachim-breitner.de (nomeata) — Wed, 24 Oct 2012 20:49:20 +0200

This week’s issue of c't, one of Germany’s largest and most influential computer magazine, has a six page introductory article about Haskell (c't 2012, Issue 23, pages 182-187). The author Harad Bögeholz demonstrates its features and strength by implementing a Slitherlink puzzle solver. As I have helped him a bit by checking the code and article for untypical Haskell code and general advice, I happen to be named at the end of the article – nice.

Dennis Felsing’s thesis on ghc-vis submitted

mail@joachim-breitner.de (nomeata) — Tue, 02 Oct 2012 09:39:01 +0200

Last week, my first student ever submitted his bachelor thesis „Visualization of Lazy Evaluation and Sharing“ about a tool he created that allows you to investigate the heap of a running Haskell program, including unevaluated values and their sharing properties, in a visual and interactive manner. You can see it in action at the end of the video of my lightning talk about ghc-dup at the Haskell Implementors Workshop this year, and read more about it on Dennis’ ghc-vis web page. I’m quite happy about the result and I think I can get used to have other people implement my ideas and needs :-)

Dennis said he is willing to continue the maintenance of the tool, so give it a shot and bombard him with bug reports and feature requests!

The might applicative left fold

mail@joachim-breitner.de (nomeata) — Fri, 28 Sep 2012 18:40:00 +0200

Just about three years ago, I created the “automatic rule-based time tracker” (arbtt), a tool that quietly runs in the background, records your current windows and their titles, and allows you to later process this data using, well, rules. By now, I have almost half a million records, spanning 317 days of me working at the computer. Processing this amount of data has become very memory-hungry: It takes 1G of RAM to process my data. Other users, with a presumably higher sampling rate, report even worse performance, which finally made me approach the problem again.

Naturally, one wonders why it is such a memory hog, after all it takes just one run over the data to produce a report. The data itself is a lazy list so the garbage collector should the be able to drop the seen elements immediately.

But arbtt is able to print several reports in one run; there is even an option (--each-category) that creates a number of reports based on the data itself. The code was structured that first a number of values that are possibly used by several reports are first created one and then re-used by the reports. And then there is the issue of the filters: The user specifies a what predicates are interesting (e.g., those where he was active) and some reports need to know not only which samples were matched, but also distinguish the consecutive runs of selected samples.

Lots of reasons that made it hard to not use the list of of samples several times, which causes it to be completely retained in memory. The idea of completely rewriting everything in a much more imperative way was not appealing. But luckily Haskell’s good abstraction possibilities can help here.

What is the “correct” way to traverse a lazy list and calculate some information about it? It is a strict left fold, i.e. an algorithm that can be represented as arguments to

foldl' :: (s -> x -> s) -> s -> [x] -> s

For those who are new to folds: The first argument is a function that processes one element from the list, returning the new state of the processing. The second argument is the initial state. And foldl' can apply such an algorithm to a list to return a final state.

So obviously I had to express my reports as left folds. To think clearer about it, I gave them a name. I also added a finishing step to the mix, which is handy:

data LeftFold x a = forall s. LeftFold {
    start :: s,
    process :: s -> x -> s,
    finish :: s -> a
    }

A value of type LeftFold x a is then an algorithm that traverses a list of xes and returns something of type a. The forall s. means that the user of such a processor must not make any assumptions about the type of the state. I also have a function that runs such a list processor, simply by invoking foldl' with the arguments stored in the LeftFold constructor:

runLeftFold :: LeftFold x a -> [x] -> a
runLeftFold (LeftFold st1 p1 f1) xs = f1 (foldl' p1 st1 xs)

Now, as a Haskell programmer, I am searching for monads everywhere, and it really would have been nice if this were a monad, as they are very easy to compose. But unfortunately, they do not form a monad. If they did then I could decide after running one processor and looking at the result how I would want to process the list now, but this obviously contradicts with the desire to traverse the list only once.

But there is the monad’s smaller ugly sibling, the applicative functor. One can still somewhat nicely compose them to form more complex algorithms, just with more restrictions. Here is my instance:

instance Functor (LeftFold x) where
    fmap f (LeftFold st1 p1 f2) = LeftFold st1 p1 (f . f2)

instance Applicative (LeftFold x) where
    pure x = LeftFold () const (const x)
    LeftFold st1 p1 f1 <*> LeftFold st2 p2 f2 = LeftFold {
        start   =                   st1 :!: st2,
        process = \(s1 :!: s2) x -> p1 s1 x :!: p2 s2 x,
        finish  = \(s1 :!: s2)   -> f1 s1 (f2 s2)
        }

The :!: operator is a strict tuple constructor. This is important, as otherwise the fold would be come a lazy one, I would be building up huge numbers of thunks and performance would be very bad. For the same reason every list processor should make sure that its state does not build thunks once it is in weak head normal form.

Now I am at a abstraction level that I feel comfortable working in, by writing functions that combine the list processors in various ways. Probably the most important one for me is already present in the libraries (here with a specialized type):

sequenceA :: [LeftFold x a] -> LeftFold x [a]

This is the whole trick: Now I can, based on the user input, run any number and selection of reports in one single list traversal. I find it very elegant that “under the hood”, the strict tuples are nested at precisely the right depth depending on the length of the list.

As hinted at before, I do some interesting stuff that was previously represented as lists of lists of values. Expressing these as left folds was a bit tricky, but again, the complexity is contained in the combinators that I wrote (although it still gets confusing, check out processIntervalReport in Stats.hs):

monoidFold :: Monoid m => LeftFold m m
mapElems :: LeftFold y a -> (x -> y) -> LeftFold x a 
filterWith :: (x -> Bool) -> LeftFold (Bool :!: x) a -> LeftFold x a
onSelected :: LeftFold x a -> LeftFold (Bool :!: x) a
onJusts :: LeftFold x a -> LeftFold (Maybe x) a
onAll :: LeftFold x a -> LeftFold (Bool :!: x) a
runOnGroups :: (x -> x -> Bool) -> LeftFold x y -> LeftFold y z -> LeftFold x z
runOnIntervals :: LeftFold x y -> LeftFold y z -> LeftFold (Bool :!: x) z
lfLength :: LeftFold x Int
lfFirst :: LeftFold x (Maybe x)
lfLast :: LeftFold x (Maybe x)
toList :: LeftFold x [x]
concatFold :: LeftFold [x] [x]

As a result, the program now only requires 10MB and runs 18% faster.

If you wonder why, in the definition of runLeftFold, I do not apply the finish function strictly: That is intentional; I am actually passing the result of a processor that calculates some often required data (e.g. total time recorded) as an argument to all the list processors. (Near the end of the main function in stats-main.hs) This “loop” works fine as long as I am using these only in the finish functions.

There is one downside to this approach: I cannot easily calculate stuff on demand and share it between two reports. Previously, I could just bind this to a variable, pass it to all reports, only if at least one is interested it would be evaluated and if several are interested, the result would be shared. I do not have a good solution for this yet.

The code above is in the self-contained module LeftFold.hs in the arbtt sources.

Several People have had this idea before, here is one blog post by Max Rabkin.

“Haskell Bytes”

mail@joachim-breitner.de (nomeata) — Wed, 12 Sep 2012 21:53:08 +0200

Last Saturday I have held a talk at the Meta Main-Rhein Chaosdays 2012 in Darmstadt, an event of the local Chaos Computer Club groups and not unsimilar to Karlsruhe’s yearly GPN. I was asked to give an talk on a advanced, Haskell related topic, so I took my audience on a guided tour through the memory of a running Haskell program, explained the different types of closures and their layout, and showed how info tables are used to allow the garbage collector to uniformly treat the objects. We also used that knowledge to roughly predict the memory footprint of a program processing a large file using the String data type.

Besides showing raw bytes (using my ghc-heap-view package and a small utility function) I included two visualizations: One is the video of a copying garbage collector at work that I blogged about last week, and the other is a nice tool that a student of mine creates for his bachelor thesis: It allows you to visualize the heap structure of a (potentially partially evaluated) value from GHCi and interactively evaluate parts of the structure – if you want, even thunks that are “hidden” behind other thunks, something that you cannot do from Haskell code. The following picture is a screen shot of ghc-vis visualizing the evaluation of the famous two-line primes definition:

The slides of my talk do not cover everything that was included, but the transcript does. It is in German, so as before, if you want me to translate it, then (convince your professor or employer to) invite me to hold the talk again. The ghc-vis tool can be found on Hackage, more information about it is on the author’s website.

A copying garbage collector animated

mail@joachim-breitner.de (nomeata) — Mon, 03 Sep 2012 15:00:22 +0200

Next weekend I will hold an advanced Haskell talk at the Meta Rhein-Main ChaosDays in Darmstadt. It will mostly be a guided tour through the main memory of a running Haskell program, showing how things are represented and how they change over time. For this I cannot avoid to touch the issue of garbage collection, so I created a short animation:

Your browser does not seem to support WebM videos. Try the YouTube link below.

(If you do not see it above, then you can try on YouTube.)

It is created with Synfig, feel free to use the sources. It seems that Synfig, although it has some features that allow you to link position in various objects, or calculate with them, is not really meant for programmers. Take the arrows: They are all quite similar, and defined by starting point, end point, and how much the initial segment sticks out to the right. I could even create them so that when I move the first or last point, the rest adjusts automatically. So one would expect that I can combine these shapes in one re-usable object that has these three values as parameters, but unfortunately, that is not the case. I guess next time I want to animate something as structured as this, I use either tikz (and generate one page per frame, convert them to an image and then to a video) or a Haskell library like gloss or diagrams.

Update: I made some minor improvements to the video. Youtube does not allow me to change the video, so you’ll see the new version only here.

ghc-heap-view: Complete referential opacity

mail@joachim-breitner.de (nomeata) — Tue, 13 Mar 2012 10:52:00 +0100

During the last week, I created ghc-heap-view, a library to investigate the actual memory representation of Haskell values. It is inspired by vacuum and the GHCi debugger, but goes beyond them by allowing the user to look inside thunks and functions and see what other values they refer to. Let me demonstrate it by running the included demo:

ghc-heap-view-demo

Here are a four different lists, where the first three are already evaluated.
The first one, l, was defined as a top level constant as

> l = [1,2,3]

and is now found at 0x00000000006d1750/2 (where the /2 is the pointer tag information) and fully evaluated:

    ConsClosure {info = StgInfoTable {ptrs = 2, nptrs = 0, tipe = CONSTR_STATIC, srtlen = 1}, 
                 ptrArgs = [0x00000000006d16e0/1,0x00000000006d1730/2],
                 dataArgs = [], descr = "ghc-prim:GHC.Types.:"}

The second one, l2, is locally defined

> let l2 = 4:l

and now found at 0x00007fdce19fe4b0/2. See how the cons-cell references l!

    ConsClosure {info = StgInfoTable {ptrs = 2, nptrs = 0, tipe = CONSTR_2_0, srtlen = 1},
                 ptrArgs = [0x00000000006dca50/1,0x00000000006d1750/2],
                 dataArgs = [],
                 descr = "ghc-prim:GHC.Types.:"}

And the binding

> args <- map length `fmap` getArgs

evaluates to the “one”, global empty list at 0x00000000006db640/1:

    ConsClosure {info = StgInfoTable {ptrs = 0, nptrs = 0, tipe = CONSTR_NOCAF_STATIC, srtlen = 0},
                 ptrArgs = [],
                 dataArgs = [],
                 descr = "ghc-prim:GHC.Types.[]"}

And now we have, at 0x00007fdce19fe4c8, the concatenation of them, but unevaluated:

> let x = l ++ l2 ++ args

The thunk keeps a reference to l2 and args, but not l, as that is at a static address, unless you are running this in GHCi:

    ThunkClosure {info = StgInfoTable {ptrs = 2, nptrs = 0, tipe = THUNK_2_0, srtlen = 1},
                  ptrArgs = [0x00007fdce19fe4b0/2,0x00000000006db640/1],
                  dataArgs = []}

Now to some more closure types. m and m' locally bound of type the unboxed type Int#, with values 42 resp. 23.

> let f = \x n -> take (I# m + I# x) n ++ args
      t = f m' l2

So here is (0x00007fdce1937d50/2), referencing its free variables args and 42:

    FunClosure {info = StgInfoTable {ptrs = 1, nptrs = 1, tipe = FUN_1_1, srtlen = 65553},
                ptrArgs = [0x00000000006db640/1],
                dataArgs = [42]}

And t is a thunk that applies f (also referenced here) to an unboxed value (23) and l2:

    ThunkClosure {info = StgInfoTable {ptrs = 2, nptrs = 1, tipe = THUNK, srtlen = 0},
                  ptrArgs = [0x00007fdce19fe4b0/2,0x00007fdce1937d50/2],
                  dataArgs = [23]}

Lastly, here is the standard example for self reference:

> let x = id (:) () x

This is what x (0x00007fdce1947940) looks like, at least without -O:

    ThunkClosure {info = StgInfoTable {ptrs = 0, nptrs = 0, tipe = THUNK, srtlen = 1},
                  ptrArgs = [],
                  dataArgs = []}

So it is unevaluated. Let us evaluate it using seq. Now we have, still at 0x00007fdce1947940:

    IndClosure {info = StgInfoTable {ptrs = 1, nptrs = 0, tipe = BLACKHOLE, srtlen = 0},
                indirectee = 0x00007fdce194cc98/2}

The thunk was replaced by an indirection. If we look at the target, 0x00007fdce194cc98/2, we see that it is a newly created cons-cell referencing the original location of x:

    ConsClosure {info = StgInfoTable {ptrs = 2, nptrs = 0, tipe = CONSTR_2_0, srtlen = 1},
                 ptrArgs = [0x00000000006db620/1,0x00007fdce1947940],
                 dataArgs = [],
                 descr = "ghc-prim:GHC.Types.:"}

After running the garbage collector (performGC), we find that the address of x is now 0x00007fdce19f30d0/2 and that the self-reference is without indirections:

    ConsClosure {info = StgInfoTable {ptrs = 2, nptrs = 0, tipe = CONSTR_2_0, srtlen = 1},
                 ptrArgs = [0x00000000006db620/1,0x00007fdce19f30d0/2],
                 dataArgs = [],
                 descr = "ghc-prim:GHC.Types.:"}

Future plans

The output of ghc-heap-view is not really pretty yet; even the indentation in this blog post was added manually by me, so this really needs a pretty printer providing a nicer, possibly more compact representation, including something like what vacuum provides. Maybe vacuum can be ported to use this library, and also include the thunk’s and function’s references in the output. Maybe also the GHCi debugger can be extended to show more information about unevaluated expressions using this. Internally, the library is not very polished yet either. It only handles those closures types that I have seen so far, and is likely to break horribly if run in a threaded or debugging enabled runtime.

How it works

Obviously, this is not standard Haskell 98 code, but rather deep trickery involving the GHC API and some C code. Initially I tried to use the API that vacuum and the GHCi debugger rely on, which is an operation

unpackClosure# :: a -> (# Addr#, Array# b, ByteArray# #)

which takes any Haskell value and returns the address to its info table, the pointers and the non-pointer-data in the closure. Unfortunately, it was not complete in that it was meant only for data closures and will for other closure types, e.g. thunks, return no data and no pointers (as can be seen in the code). So I implemented my own version of this operation:

foreign import prim "slurpClosurezh" slurpClosure# :: Any -> (# Addr#, ByteArray#, Array# b #)

where the returned ByteArray# contains the complete closure, including extra information fields such as the arity of a function. The Array# is again the list of pointers in the closure. At first glance, this is a duplication, as the pointers are of course also contained in the ByteArray#. But as soon as the GHC runtime reigns again, a garbage collector run can happen, the referenced values will move somewhere else, and the words that once were pointers in the ByteArray# become useless. But the corresponding entries in the Array# are updated by the garbage collector, as it knows that these are pointers, and not just words. This way, we get both a faithful copy of the closure on the heap and useful references to the contained data. Here is a demonstration of this effect:

$ ghci -XMagicHash -package ghc-heap-view
GHCi, version 7.4.1: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
[..]
Loading package ghc-7.4.1 ... linking ... done.
Loading package ghc-heap-view-0.1 ... linking ... done.
Prelude> let {a = [1,2,3,4]; b = 5:a}
Prelude> :m + GHC.HeapView 
Prelude GHC.HeapView> rawHeapData <- getClosureRaw b
Prelude GHC.HeapView> rawHeapData 
(0x000000004080d658,[1082185320,140040739366568,140040739365928],[0x00007f5dc68626a8,0x00007f5dc6862428])
Prelude GHC.HeapView> System.Mem.performGC
Prelude GHC.HeapView> rawHeapData 
(0x000000004080d658,[1082185320,140040739366568,140040739365928],[0x00007f5dc41b3ad8,0x00007f5dc41b3b28])

The function rawHeapData is a thin wrapper around slurpClosure# which turns the primitive array in normal lists. Note that the second component of the triple is unchanged, but the third is updated by the garbage collector. Of course this means that the Show instance for the data type that ghc-heap-view uses to reference values is not referential transparent either.

The foreign function import above is of type “prim”, i.e. does not call a C function but rather a Cmm function. Cmm is a reduced C that GHC uses internally to compile the Haskell code to, and most primitive operations are implemented in this language – although I do quickly call regular C from my Cmm code to do the more complicated stuff, mainly figuring out what words of the closure are pointers.

The knowledgeable reader might notice that I am passing a boxed value of type Any to the foreign function. This is currently not possible with foreign prim functions, and to actually use that code, you need the patch in GHC ticket #5931. But you can use ghc-heap-view without that as well (and the Cabal package will by default use that path), using the following hack to obtain the pointer to a Haskell value on the Heap as an unboxed type that can pass to the primitive operation:

foreign import prim "slurpClosurezh" slurpClosure'# :: Word#  -> (# Addr#, ByteArray#, Array# b #)
data Ptr' a = Ptr' a
aToWord# :: Any -> Word#
aToWord# a = case Ptr' a of mb@(Ptr' _) -> case unsafeCoerce# mb :: Word of W# addr -> addr
slurpClosure# :: Any -> (# Addr#, ByteArray#, Array# b #)
slurpClosure# a = slurpClosure'# (aToWord# a)

This works because a Word and a Ptr' have the same closure layout, only differing in the fact that one stores an a, and the other stores a Word#.

Once we obtained the raw representation of the closure, we do the parsing in Haskell. Using the info table and the raw closure, we have enough information to tell which words have to be replaced by the appropriate pointer (which might already have been updated by the garbage collector) in the pointers list.

This work was supported by a scholarship from the Deutsche Telekom Stiftung.

GHC 7.4.1 speeds up arbtt by a factor of 22

mail@joachim-breitner.de (nomeata) — Sun, 26 Feb 2012 17:25:00 +0100

More than two years ago I wrote arbtt, a tool that silently records what programs you are using and allows you to do statistics on that data later, based on rules that you define afterwards, hence the name automatic rule based time tracker. I wasn’t doing much with it recently (the last release has been half a year ago), but it nevertheless was running on my machine and by now has tracked a total time span of 248 days in 350000 records.

Yesterday, I had a use for it again: measuring the time spent creating a certain document with LaTeX. So I added a rule to my categorize.cfg and ran arbtt-stats. I knew that it was not very fast, and that my data set has grown considerably since I last used it. And indeed, it took more than 6 minutes to process the data and spit out the result.

Since I’m currently working on the GHC 7.4.1 transition in Debian anyways, I decided to check what happens if I compile the code with that version of the Haskell compiler, instead of the previous version 7.0.4. And behold: The whole process took merely 17.3 seconds to complete! At first I did not believe it, but the result was identical, both binaries were built with the same option, i.e. no profiling enabled or anything like that. Wouldn’t you also like to have such speed ups for free, just by waiting for someone else to improve their work?

I tried to find out the reason for the speed up and created profiling output from both the old and the new binary. The old binary spends 83% of the time in Categorize.checkRegex, which basically just call Text.Regex.PCRE.Light.match. Since the version of pcre-light is the same in both binaries, I conclude that the Foreign Function Interface that GHC provides to interact with C libraries (libpcre in this case) is much faster now, although I do not find any mention in the release notes. And even if I do not count the 83% time spent in checkRegex, the code from the new compiler is still 2.7 times faster. Thanks, GHC devs, great work!

Guest lecture on Haskell performance

mail@joachim-breitner.de (nomeata) — Tue, 06 Dec 2011 22:16:28 +0100

Today, I had some business in Bonn, so Janis Voigtländer invited me to hold a guest lecture in his advanced functional programming course, maybe telling the students more about “real world” use of Haskell, given that he teaches mostly the theoretical foundations. I chose to discuss an experience I made while implementing SAT-Britney, namely that for the sake of good runtime and memory performance, it may be neither ideal to evaluate a list fully lazily, nor fully strict, but to have something in between. For the talk, I demonstrated this by a small example, showed how to use the GHCi debugger to get some insight in the evaluation order of things, how to benchmark the program and read the ghc statistics and a profiling chart, and finally how to get the most out of GHC’s List Fusion. I wanted to also touch on QuickCheck, but skipped it to finish in time. The full text of my talk (including graphs) is available, but in German; if you want an English version convince your prof to invite me for a guest lecture as well.

Afterwards I sat down with one of the students of the course, Helmut Grohne, who happened to take part in the Debian bug squashing party last weekend and had filed a release critical bug against the Haskell grammar generator frown (no better link available, homepage seems to be down) to debug the problem. It quickly became clear that the bug was introduced by me (can you spot it?) when trying to make it compile with a modern GHC – slightly embarrassing for me.

First contribution to a basic Haskell library

mail@joachim-breitner.de (nomeata) — Tue, 22 Nov 2011 21:15:36 +0100

While working on SAT-Britney, I made heavy use of the IntSet data type provided by the basic Haskell library “containers”. Since memory consumption was a problem, I looked at its implementation, which is a binary tree, and wondered whether this could be improved for dense sets by using a machine sized word as a bit map to represent a continuous part of the integers, so I started to implement it. The effect on my program was not as strong as I had hoped for, but nevertheless, the code made its way back into containers – after a thorough review and a considerable amount of further improvements by Milan Straka. I’m getting sucked deeper and deeper into the Haskell community... (which means that the Haskell community does not suck, of course.)

stm-stats published for factis research

mail@joachim-breitner.de (nomeata) — Sun, 09 Oct 2011 22:28:56 +0200

Factis research has hired me to make sure their code gets contributed back to the Haskell community. As a first step, I have followed a request of someone in the audience of Stefan Wehr’s talk at HaL 6 and prepared stm-stats – seem my post at the factis research blog about that.