Planet Exgentoo

A while ago Exherbo came up with a solution for the “large number of repositories” problem. It turns out this solution works rather well, and I’m now convinced that anyone rushing off and assuming that Git on its own will magically solve anything other than CVS sucking is going in the wrong direction.

Paludis also has ways of tracking packages installed by hand. Whilst useful on Gentoo, this really comes into play on Exherbo, where we don’t want to end up with a massive unmaintainable tree full of packages used by only two people.

The logical next step is tracking packages that don’t exist yet.

An unimaginative person might think that the way to solve this would be to have a bug for each requested package. But that gets messy — it’s not integrated with the package manager, and requires considerable extra effort from the user. A while ago Bernd suggested something more interesting: tracking unwritten packages in a repository in a similar way to how we do unavailable packages.

The implications:

• Most obviously, we can keep track of things we want that haven’t been written yet in a way that doesn’t involve leaving the package manager to look things up.
• We can also use it to track version bumps that we know will take a while. Doing paludis --query '>=hugescarypackage-4', say, would show that it’s being worked upon.
• We can also depend upon things that don’t exist, rather than leaving incomplete dependency strings around. This is fine in at least two cases — if a dependency is conditional upon an option that should probably be implemented but isn’t yet, we can add the option and make it unusable. And we can handle obscure suggested dependencies (e.g. git has lots of optional dependencies upon weird perl modules, so we can say “if you want support for git-send-email with SSL, you need to write such and such a package”).
• Bored developers could simply paludis --query '*/*::unwritten' and get ideas for what to do.

Adding support to Paludis for this only took a couple of hours. So now Exherbo developers can use this:

format = unwritten
location = /var/db/paludis/repositories/unwritten
sync = git://git.exherbo.org/unwritten.git
importance = -100

And see this:

$ paludis -q genesis
* sys-apps/genesis
    unwritten:               (1.0)X* {:0}
    Description:             The daddy of all init systems
    Homepage:                http://www.exherbo.org/
    Comment:                 We need an init system.
    Masked by unwritten:     Package has not been written yet

As with unavailable, installing behaves sensibly:

$ paludis -pi genesis
Building target list...
Building dependency list...
Query error:
  * In program paludis -pi genesis:
  * When performing install action from command line:
  * When executing install task:
  * When building dependency list:
  * When adding PackageDepSpec 'sys-apps/genesis':
  * All versions of 'sys-apps/genesis' are masked. Candidates are:
    * sys-apps/genesis-1.0:0::unwritten: Masked by unwritten (Package has not been written yet)

Those interested in the repository format can browse the tree. The observant might notice that the file format is quite similar to the one used by unavailable repositories, and wonder whether I was feeling lazy and swiped a load of code rather than implementing a new repository from scratch.

In the spirit of silly buzzwords, one could argue that this increases distributivity and democratisation of the repository, since there’s nothing to stop motivated users from creating their own wishlist repositories. It wouldn’t be hard to make a simple web interface that lets users request and vote on packages and version bumps, automatically generating a repository that anyone can use. But fortunately Exherbo is a dictatorship and has no users, so we don’t have to put up with that kind of nonsense.

EAPI 2 has been approved by the Gentoo Council and so can now be used in ebuilds. The first package manager with support was Paludis 0.30.1; Portage support came along with 2.2_rc11.

EAPI 2 consists purely of extensions to EAPI 1. The new features are:

• SRC_URI arrows
• Use dependencies
• !! blockers
• src_prepare
• src_configure, and a different default src_compile
• default_ phase functions and the default function
• doman language support

Formal definitions can be found in PMS; an overview of each feature will follow in subsequent posts.

First picture from Open Source Days:

This is the final post in a series on EAPI 2.

The doman helper is one of those pesky little beasts that makes specifying EAPI behaviour formally such a nuisance (although it is nowhere near as horrible as dohtml). EAPI 2 makes it even peskier.

I’ll try that again.

The doman helper makes writing ebuilds substantially easier by automagically doing the right thing when installing manual pages, freeing the developer from having to care about manual sections. EAPI 2 makes doman even more useful by making it aware of language codes as well as sections.

The painful details are available in PMS, but basically this will now ‘do the right thing’:

doman foo.1 foo.en.1 foo.en_GB.1

Previously only the first of the items would go to the right place.

This one’s a Gentoo innovation; see bug 222439 for its history. It was shamelessly stolen for exheres-0, but was too late for kdebuild-1.

This post is part of a series on EAPI 2.

With EAPIs 0 and 1, if you want to add something to, say, src_unpack, you have to manually write out the default implementation and then add your code. This is easy to screw up — developers are highly prone to getting the quoting wrong and forgetting which functions do and do not want a || die on the end.

EAPI 2 makes the default implementation of phase functions available as functions themselves. These functions are named default_src_unpack, default_src_configure and so on.

Typing out default_src_compile in full is pointless, though (especially since it’s illegal to call phase functions or default phase functions from other phase functions). So we also introduce the special default function, which calls whichever default_ phase function is appropriate for the phase we’re in. Thus:

src_compile() {
    default
    if useq extras ; then
        emake extras || die "splat"
    fi
}

Both features first appeared in exheres-0.

An alternative proposal (I think it came from the Pkgcore camp) was to make all EAPI default implementations available through functions named like eapi0_src_compile, eapi1_src_compile and eapi2_src_compile. This proposal was rejected because various Paludis people moaned about it not making sense or having any legitimate use cases (the ‘obvious’ use cases don’t work if you think them through), and no-one stood up to defend it.

This is post five in a series describing EAPI 2.

EAPI 2 splits src_compile into src_configure and src_compile. Like src_prepare, it’s mostly a convenience thing to reduce copying default implementations, although in this case it also makes it easier to hook in code in between configure and make being run.

The default src_configure implementation behaves like this:

src_configure() {
    if [[ -x ${ECONF_SOURCE:-.}/configure ]]; then
        econf
    fi
}

This is the first half of EAPI 1’s src_compile, not the non-ECONF_SOURCE-aware EAPI 0 version.

The default src_compile implementation is reduced accordingly:

src_compile() {
    if [[ -f Makefile ]] || [[ -f GNUmakefile ]] || [[ -f makefile ]]; then
        emake || die "emake failed"
    fi
}

The split configure / compile setup was first used in exheres-0, which uses more elaborate default implementations. Like src_prepare, it was considered but rejected for kdebuild-1 because of eclass difficulties.

This is post four in a series describing EAPI 2.

EAPI 2 has a new phase function called src_prepare. It is called after src_unpack, and can be used to apply patches, do sed voodoo and so on. The default implementation does nothing.

This function is purely for convenience. It gets rather tedious copying out the default implementation of src_unpack just to add a patch in somewhere.

src_prepare was first introduced in exheres-0 (which has a more elaborate default implementation). It was considered but rejected for kdebuild-1 because making best use of it requires eclass awareness, and the packages using kdebuild-1 had to share eclasses with the main Gentoo tree.

This is part three of a series of posts describing EAPI 2.

Blockers are a nuisance for end users. It’s rarely obvious how to fix them or what they mean, and getting it wrong can leave systems unusable.

There have been various proposals on how to fix this. For exheres-0, we’re going to go with something like this:

DEPENDENCIES="
    !app-misc/superfrozbinator [[
        description = [ Can only have one frozbinator installed at once ]
        resolution = uninstall-blocked-after
        url = [ http://explain.example.org/?only-one-frozbinator ]
    ]]
    !dev-libs/icky [[
        description = [ Having icky installed breaks the build process ]
        resolution = [ manual ]
        url = [ http://explain.example.org/?myfroz-hates-icky ]
    ]]"

The user can then be presented with a list of things that would need to be uninstalled to resolve blockers, along with clear descriptions of why they need to do so. Once the user has explicitly accepted the uninstalls, the package manager could then safely perform the installs.

Unfortunately, annotations aren’t something that can be implemented for Portage any time soon. Instead, Portage has gone with a fairly horrible and dangerous semi-automatic block resolution system that sometimes removes blocked packages automatically (often screwing up the user’s system in the process). Whilst doing so, Portage changed the meaning of EAPI 0 / 1 blockers from “this must not be installed when we do the build” to “this must be uninstalled after we do the build”.

EAPI 2 introduces a new kind of blocker using double exclamation marks, like !!app-misc/other. This goes back to the old meaning of “this must not be installed when we do the build”, keeping !app-misc/other for “this must be uninstalled after we do the build”.

This does not, unfortunately, make the user any safer, but it does allow packages that really can’t have something installed at build time to say so.

This is the first item in a series of posts describing EAPI 2.

Some upstreams use annoyingly named tarballs. Most commonly, they don’t include either the package name or the version in the filename. Because DISTDIR is a flat directory, this causes problems — the tree must not use two different tarballs with the same name. Previously, the solution to horrible upstream naming was to manually mirror the tarball with a new filename; this was considered excessively icky.

There have been two sane solutions proposed for this over time. The one we didn’t use was to define a DISTDIR_SUBDIR variable, and do all downloads into there. This would have made the A variable quite a bit messier, and complicated sharing certain tarballs between packages.

The arrows solution was something I came up with for early Paludis experimental EAPIs, and was adopted for kdebuild-1 and from there into 2; it’s also always been present in exheres-0. It works like this:

SRC_URI="http://example.com/stupid-named/1.23/stupid.tar.bz2 -> stupid-1.23.tar.bz2"

or using variables:

SRC_URI="http://example.com/stupid-named/${PV}/${PN}.tar.bz2 -> ${P}.tar.bz2"

This tells the package manager to look at the URL on the left of the arrow, but save to the filename on the right.

Mirroring effects are slightly subtle. Consider:

SRC_URI="mirror://foo/${PN}/${PV}.tar.bz2 -> ${P}.tar.bz2"

The package manager will look both on mirror://foo/ and mirror://gentoo/ for the download. When looking on foo, the raw filename must be used, but when looking on gentoo, the rewritten filename is used.

Anyone using arrows on mirror://gentoo/ URIs gets stabbed.

Arrows make another proposed but rejected EAPI feature irrelevant: there was a proposal floating around (I think it originated with drobbins, but I can’t find an original source) to make unpack ignore ;sf=tbz2 and ;sf=tgz suffixes on filenames, for interoperability with gitweb. Arrows are a more general solution.

Implementation-wise, anyone still using a lexer-based parser will need a single token of lookahead for this. Apparently this causes minor inconveniences in some broken programming languages that only support what C++ calls input iterators; I consider this a good thing, because it might make people either use a better iterator model or stop using lexers.

This is the second post in a series of posts describing EAPI 2.

Use dependencies have been needed for a very long time. They eliminate most of the built_with_use errors you see during pkg_setup, replacing them with an error that is seen at pretend-install time.

The first two real world trials of use dependencies were with Exherbo’s exheres-0 and Gentoo’s kdebuild-1. It became apparent that an awful lot of packages would end up with dependencies like:

blah? ( app-misc/foo[blah] ) !blah? ( app-misc/foo )
monkey? ( app-misc/foo[monkey] ) !monkey? ( app-misc/foo[-monkey] )
fnord? ( app-misc/foo ) !fnord? ( app-misc/foo[-fnord] )

Syntactically, that’s rather inconvenient. For exheres-0 and kdebuild-1, we added the following syntax:

[opt]

The flag must be enabled.

[opt=]

The flag must be enabled if the flag is enabled for the package with the dependency, or disabled otherwise.

[opt!=]

The flag must be disabled if the flag is enabled for the package with the dependency, or enabled otherwise.

[opt?]

The flag must be enabled if the flag is enabled for the package with the dependency.

[opt!?]

The flag must be enabled if the use flag is disabled for the package with the dependency.

[-opt]

The flag must be disabled.

[-opt?]

The flag must be disabled if the flag is enabled for the package with the dependency.

[-opt!?]

The flag must be disabled if the flag is disabled for the package with the dependency.

Dependencies could be combined by specifying multiple blocks, as in foo/bar[baz][monkey?].

For EAPI 2, Zac decided to go with an arbitrarily different syntax:

[opt]

The flag must be enabled.

[opt=]

The flag must be enabled if the flag is enabled for the package with the dependency, or disabled otherwise.

[!opt=]

The flag must be disabled if the flag is enabled for the package with the dependency, or enabled otherwise.

[opt?]

The flag must be enabled if the flag is enabled for the package with the dependency.

[!opt?]

The flag must be disabled if the use flag is disabled for the package with the dependency.

[-opt]

The flag must be disabled.

And to combine use dependencies, one uses a comma, as in foo/bar[baz,monkey?].

In both cases, the slot dependency must go before the dependency, so foo/bar:1[baz], not foo/bar[baz]:1. The use dependency goes after any version restrictions, so >=foo/bar-2.1:2[baz].

In both cases, it is illegal to reference a use flag that does not exist (including USE_EXPAND flags that are not explicitly listed in IUSE). So foo/bar[opt] when any version of foo/bar does not have opt in IUSE is illegal and has undefined behaviour, as is foo/baz[opt?] if either the owning package or foo/baz has no opt. For cases where only some versions of a package have a flag, use dependencies can be combined with version or slot restrictions.

From an implementation perspective: the package manager should not try to automatically solve unmet use dependencies. The package manager doesn’t know the impact of changing a use flag (changing some flags makes a system unbootable), so it can’t simply override the user’s choice. (Paludis will suggest an automatic reinstall if and only if the user has already modified their use.conf, so you don’t need to manually reinstall a dependency if you’re ok with altering the flags with which it is built.)

Paludis 0.30.1 has been released:

• EAPI 2 support.

I am using Mutt as my e-mail client and I have done that ever since I started using Unix. I have for many months been annoyed by my very ugly theme that I created when I was around 13, but I have never actually spend a few minutes on making something shiny that didn’t piss me off.

Inspired from Ciaran McCreesh’s Vim theme called Inkpot, I decided to create an inkpot’ish theme for Mutt and here is the result that you can put in your .muttrc:

color   attachment  color30 color80
color   header      color10 color80 "^(From|Subject|cc|date|To|X-Spam-Level|User-Agent|X-Mailer):"
color   signature   color39 color80
color   tree        color26 color80
color   message     color26 color80
color   status      color85 color81
color   normal      color78 color80
color   error       color79 color64
color   indicator   color80 color73
color   markers     color26 color80
color   index       color64 color80 ~D
color   index       color64 color80 ~F
color   index       color30 color80 ~T
color   tilde       color80 color80

color   body        color10 color80 "(http|https|ftp|news|telnet|finger|irc)://[^ \">\t\r\n]*"
color   body        color10 color80 "mailto:[-a-z_0-9.]+@[-a-z_0-9.]+"
color   body        color26 color80 "[;:=][-][)/(|]"

color   quoted      color52 color80
color   quoted1     color22 color80
color   quoted2     color71 color80

Note: This probably only works in urxvt since that’s what I am using

There is, of course, a screenshot:

Mutt

I’ve tidied up my Exherbo supplemental repository (which is what Gentoo people confusingly call an ‘overlay’) and published it for the foolish.

Current goodness is:

• hddtemp. This is mostly as an experiment in handling regularly updated data files.
• detox
• An updated imlib2 nicked from rbrown, who needs to lrn2src_uri
• Fluxbox SCM, based upon (and I feel very very dirty for doing this) ahf’s 1.0.0 packages.

Expect these to be neglected and / or broken irregularly.

Paludis 0.28.1 has been released:

• Work around compiler issues that eventually lead to unpack dying when it shouldn’t on certain EAPIs on certain systems (ticket:622).

Let’s say we have a Makefile using some target-specific variables:

$ cat Makefile
foo = global
b : foo = for-b

all : d
        @cat d

a :
        @echo a:$(foo) > a

b : a
        @{ cat a ; echo b:$(foo) ; } > b

c : a
        @{ cat a ; echo c:$(foo) ; } > c

d : b c
        @cat b c > d

Then we can get different results depending upon how we make the ‘all’ target:

$ rm a b c d ; make
a:for-b
b:for-b
a:for-b
c:global
$ rm a b c d ; make a b c d all
a:global
b:for-b
a:global
c:global

This is considered a feature:

There is one more special feature of target-specific variables: when you define a target-specific variable that variable value is also in effect for all prerequisites of this target, and all their prerequisites, etc. (unless those prerequisites override that variable with their own target-specific variable value).

In practice, it’s a pain in the ass. Using Quagmire, you’re supposed to be able to do this:

noinst_SHARED_LIBRARIES = libone.so libtwo.so libthree.so

libone.so_SOURCES = one.c
libone.so : LDFLAGS = -Wl,-soname,libone.so
libone.so_LIBS = libthree.so

libtwo.so_SOURCES = two.c
libtwo.so : LDFLAGS = -Wl,-soname,libtwo.so
libtwo.so_LIBS = libthree.so

libthree.so_SOURCES = three.c

Which leads to some rather bizarre behaviour:

$ make clean all &>/dev/null ; scanelf -S libthree.so
 TYPE   SONAME FILE
ET_DYN libone.so libthree.so 

$ make clean libone.so &>/dev/null ; scanelf -S libthree.so
 TYPE   SONAME FILE
ET_DYN libone.so libthree.so 

$ make clean libtwo.so &>/dev/null ; scanelf -S libthree.so
 TYPE   SONAME FILE
ET_DYN libtwo.so libthree.so 

$ make clean libthree.so &>/dev/null ; scanelf -S libthree.so
 TYPE   SONAME FILE
ET_DYN  libthree.so

I can think of a few not very nice ways of getting around this. Hopefully someone will come up with something better…

I’ve moved my configuration files repository from Subversion to Git, since I’m sick of trying to decide whether my laptop or desktop should hold the ‘master’ copy. For the lulz, I shall also push to GitHub at irregular intervals, so you shall no longer need to pester me to upload newer versions of my bashrc / vimrc / whatever to webspace somewhere. Instead, you can just:

git clone git://github.com/ciaranm/dotfiles-ciaranm.git

Or browse them online.

We’ll see whether this lasts…

Here’s my variation on the “Git and Subversion status in the bash prompt” thing:

ps_scm_f() {
    local s=
    if [[ -d ".svn" ]] ; then
        local r=$(svn info | sed -n -e '/^Revision: \([0-9]*\).*$/s//\1/p' )
        s="(r$r$(svn status | grep -q -v '^?' && echo -n "*" ))"
    else
        local d=$(git rev-parse --git-dir 2>/dev/null ) b= r= a=
        if [[ -n "${d}" ]] ; then
            if [[ -d "${d}/../.dotest" ]] ; then
                if [[ -f "${d}/../.dotest/rebase" ]] ; then
                    r="rebase"
                elif [[ -f "${d}/../.dotest/applying" ]] ; then
                    r="am"
                else
                    r="???"
                fi
                b=$(git symbolic-ref HEAD 2>/dev/null )
            elif [[ -f "${d}/.dotest-merge/interactive" ]] ; then
                r="rebase-i"
                b=$(<${d}/.dotest-merge/head-name)
            elif [[ -d "${d}/../.dotest-merge" ]] ; then
                r="rebase-m"
                b=$(<${d}/.dotest-merge/head-name)
            elif [[ -f "${d}/MERGE_HEAD" ]] ; then
                r="merge"
                b=$(git symbolic-ref HEAD 2>/dev/null )
            elif [[ -f "${d}/BISECT_LOG" ]] ; then
                r="bisect"
                b=$(git symbolic-ref HEAD 2>/dev/null )"???"
            else
                r=""
                b=$(git symbolic-ref HEAD 2>/dev/null )
            fi

            if git status | grep -q '^# Changed but not updated:' ; then
                a="${a}*"
            fi

            if git status | grep -q '^# Changes to be committed:' ; then
                a="${a}+"
            fi

            if git status | grep -q '^# Untracked files:' ; then
                a="${a}?"
            fi

            b=${b#refs/heads/}
            b=${b// }
            [[ -n "${r}${b}${a}" ]] && s="(${r:+${r}:}${b}${a:+ ${a}})"
        fi
    fi
    s="${s}${ACTIVE_COMPILER}"
    s="${s:+${s} }"
    echo -n "$s"
}

This is added to PS1 as normal.

For Subversion, it gives us the revision, and a * if anything’s been modified.

For Git, we get quite a bit more. If we’re in the middle of a merge, interactive rebase or am, we get told so. If we’re in the middle of a bisect, we get question marks, since I’ve not had to bisect anything since I wrote that code and I haven’t implemented anything fancy for it. We also get the branch, and a combination of * for changed but not updated files, + for changed and committed files and ? for untracked files.

It’s generally sufficiently fast to not be annoying, although the initial cd into a large project can take a few seconds. It’s probably possible to drop to a single ‘git status’ call if performance matters, although the kernel does a pretty good job of speeding up subsequent runs.

Being sick of wireless disconnecting every time it rained, I got my hands upon a cheap Edimax EW-7728In PCI 802.11n card. This uses a RaLink rt2860 (PDF) chip, which has vendor-supplied open source drivers available. Getting it to work with Linux 2.6.26 is slightly non-trivial, however.

First, we need the drivers. At the time of writing, that means 2008_0522_RT2860_Linux_STA_v1.6.1.0.tar.bz2.

Next, we need to fix a couple of things. First, the Makefile is dumb, and tries to install into /tftpboot:

From 82dc3a8737e4f97311f4f4fccd79ea55a319f1ea Mon Sep 17 00:00:00 2001
From: Ciaran McCreesh <ciaran.mccreesh@googlemail.com>
Date: Mon, 14 Jul 2008 16:47:00 +0100
Subject: [PATCH] Get your filthy paws off my /tftpboot

---
 Makefile |    2 --
 1 files changed, 0 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index 72bc933..aed3b00 100644
--- a/Makefile
+++ b/Makefile
@@ -80,11 +80,9 @@ LINUX:
 ifneq (,$(findstring 2.4,$(LINUX_SRC)))
        cp -f os/linux/Makefile.4 $(RT28xx_DIR)/os/linux/Makefile
        make -C $(RT28xx_DIR)/os/linux/
-       cp -f $(RT28xx_DIR)/os/linux/rt$(CHIPSET)sta.o /tftpboot
 else
        cp -f os/linux/Makefile.6 $(RT28xx_DIR)/os/linux/Makefile
        make  -C  $(LINUX_SRC) SUBDIRS=$(RT28xx_DIR)/os/linux modules
-       cp -f $(RT28xx_DIR)/os/linux/rt$(CHIPSET)sta.ko /tftpboot
 endif

 clean:
--
1.5.6.2

Next, dev->nd_net should now be dev_net(dev):

From 0878b37a40e2a7f466a74938920ff3751917eec3 Mon Sep 17 00:00:00 2001
From: Ciaran McCreesh <ciaran.mccreesh@googlemail.com>
Date: Mon, 14 Jul 2008 16:48:50 +0100
Subject: [PATCH] dev->nd_net is now dev_net(dev)

---
 os/linux/rt_main_dev.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/os/linux/rt_main_dev.c b/os/linux/rt_main_dev.c
index 24604da..6a3471d 100644
--- a/os/linux/rt_main_dev.c
+++ b/os/linux/rt_main_dev.c
@@ -669,7 +669,7 @@ static NDIS_STATUS rt_ieee80211_if_setup(struct net_device *dev, PRTMP_ADAPTER p

 #if LINUX_VERSION_CODE >= KERNEL_VERSION(2,5,0)
 #if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,24)
-        device = dev_get_by_name(dev->nd_net, slot_name);
+        device = dev_get_by_name(dev_net(dev), slot_name);
 #else
                device = dev_get_by_name(slot_name);
 #endif
--
1.5.6.2

Finally, the driver is unusably noisy. Unless you want fifty-odd lines of kernel debug informatione every five seconds:

From 19c7a6895333624566775541fbc836e0c9208225 Mon Sep 17 00:00:00 2001
From: Ciaran McCreesh <ciaran.mccreesh@googlemail.com>
Date: Mon, 14 Jul 2008 16:47:43 +0100
Subject: [PATCH] You shut your dirty whore mouth

---
 include/rt_linux.h |    4 ----
 1 files changed, 0 insertions(+), 4 deletions(-)

diff --git a/include/rt_linux.h b/include/rt_linux.h
index dfe4ab5..079d256 100644
--- a/include/rt_linux.h
+++ b/include/rt_linux.h
@@ -325,10 +325,6 @@ extern ULONG    RTDebugLevel;

 #define DBGPRINT_RAW(Level, Fmt)    \
 {                                   \
-    if (Level <= RTDebugLevel)      \
-    {                               \
-        printk Fmt;               \
-    }                               \
 }

 #define DBGPRINT(Level, Fmt)    DBGPRINT_RAW(Level, Fmt)
--
1.5.6.2

There’s no Makefile install target, so you’ll need to sudo cp os/linux/*.ko /lib/modules/`uname -r`/kernel/ then sudo depmod -a before you can sudo modprobe rt2860sta. You’ll also need to sudo mkdir -p /etc/Wireless/RT2860STA/ then cp RT2860STA.dat /etc/Wireless/RT2860STA/, even though we’re not using that file for any configuration.

Gentoo’s init scripts, if you’re using them, try to be overly clever, and won’t be able to bring up the interface. So use something like this as your /etc/init.d/net.ra0:

#!/sbin/runscript

depend() {
    need localmount
    after bootmisc hostname net.lo net.lo0
    use isapnp isdn pcmcia usb wlan
}

start() {
    bash -x -c 'ifconfig ra0 up'
    bash -x -c 'iwconfig ra0 essid giant-space-monkey key "aa:bb:cc:dd:ee:ff:00:11:22:33:44:55:66" freq 2.417G'
    bash -x -c 'dhcpcd ra0'
}

stop() {
    :
}

restart() {
    :
}

And that appears to be sufficient.

Exherbo has finally got a logo thanks to Ida Jensen. Here it is:

You can find the svg file in the exherbo git repository.

Update: I completely forgot to thank the person who actually draw the logo. It was Christel from the Exherbo team who draw the initial cartoon and Ida then later turned it into a digital version! Sorry Christel, I owe you a cookie

Paludis 0.28.0 has been released:

• Compilers without tr1 memory, type traits and functional support are no longer supported. This means you, gcc-3.
• Support for the Exherbo distribution.
• Better detection of SCM versions.
• Bug fixes: pkg_pretend now behaves correctly with resumes (ticket:604). Better support for Portage config files (ticket:597). Package dep spec parsing no longer has trailing hyphen issues (ticket:590).
• New Selection + Filter + Generator interface using Environment, replacing the old PackageDatabase Query (ticket:559).
• importare can now change file ownership to root (ticket:599).

• Logitech. You helpfully claim to be able to provide replacement feet for keyboards free of charge — presumably, breakages are not exactly uncommon. Then you claim not to keep track of whether or not you have any spares for my year-old G15 keyboard, so you can’t give me any. And no, you won’t go and have a look to find out whether you do have any.
• Whoever makes the push pins found on LGA 775 CPUs. Especially the kind that require so much force to lock that it’s a race between the motherboard and my finger to see which will break first.
• Abit. You should be ashamed of the F-I90HD motherboard for all sorts of reasons, but today’s gripe is about the heatsink on the integrated graphics chip. I shouldn’t have to install a cooling fan onto the heatsink to stop the screen from going black when the CPU is under load.

• An interview with Bjarne Stroustrup
• Google C++ Style Guide
• Sustainable Energy - Without the Hot Air, David J.C. MacKay

Some of you might have read my previous blog post about the attack on Exherbo’s quote database that I wrote about yesterday. As promised I am going to write a follow up post about how this was handled and what the result of the cooperation with Gentoo’s infrastructure team was.

First of all I would like to thank Gentoo’s infrastructure team for acting quickly and very professionally. They reacted immediately after I posted the bug and asked for the information they needed (log files) to compare with their own auditing results.

I would especially like to thank robbat2, KingTaco, Ramereth, solar and fox2mike who worked on the issue, reviewed log files and discussed how they were going to handle this incident.

I would like to clarify that Exherbo harbours no ill will towards any of the companies (GNi, Bytemark and OSUOSL). They are providing much needed hardware and infrastructure for many open source projects but have no role beyond as a sponsor.

No company is able to prevent their users from carrying out malicious actions from their network.

I also promised to give you an update based on the information that we received after the blog post and the bug was created:

• Alex Howells (Astinus) has been removed from all Gentoo infrastructure and as you can read on bug #229895, Gentoo’s infrastructure team has informed all the developers via the gentoo-core mailing list about what happened.

  The developers received the following message (taken from the bug):

  As reported on bug #229895, on 27 Jun 2008 at 19:04 UTC, there was unwarranted traffic sent to http://quotes.exherbo.org/ and some of the traffic involved originated from various Gentoo Linux infrastructure machines. The Gentoo infrastructure team conducted an audit based on logs from the machines as well as those provided by Exherbo’s infrastructure team. Based on results of our audit and the user’s admission via email, we have concluded that the said actions were taken by one of our developers named astinus.

  We have taken appropriate action and have removed the developer in question from Gentoo Linux infrastructure. We sincerely apologize to the administrator of quotes.exherbo.org for the involvement of Gentoo infrastructure machines.

• Alex Howells also decided to “remove” his LinkedIn profile so the link from my former blog post is not working anymore.
• Since Gentoo’s infrastructure team handled this so well I see no reason to contact their abuse contact from the respective networks where the machines are located.
• I would also like to point out that Astinus was not kicked out of Gentoo. He left on his own after the bug was filed. Astinus and I talked on IRC and while we still disagree about whether it was an attack or a prank we agreed that this is now a closed chapter and Astinus apologized to me and I accepted.

Once again I would like to thank the Gentoo infrastructure team on behalf of the Exherbo development team for their quick response and their very professional way of handling this issue.

Alex.

Yesterday Exherbo’s Quote database experienced a very primitive attack. The purpose was to turn all the rank values on our quotes into negative numbers.

In this post we are going to analyse the log files from the attack and see “who did this”.

By simply reading the log files I can see that the attack was done via the program called wget which is a fairly shiny little Unix tool for fetching websites and wget is ideal for this kind of attacks.

The first thing we did when Ciaran discovered that all of our quotes had negative values was to fetch our access log file for further analysis.

Anyways, lets get started:

First thing we want to do is to remove all the legal traffic and only get the wget clients. And then, the only really interesting stuff for us is their IP-addresses. We fire off a simple Unix command:

grep -i 'wget' access_log | cut -s -d' ' -f1 > wget_attack

The wget_attack file is available here.

Now we have a file called wget_attack that contains only the IP-address of these requests to our webserver.

I wanted to get an idea about who was behind this attack so it would be fairly nice to do a simple reverse DNS lookup of these IP’s to get an idea about where these attacks were launched from:

for host in $(cat wget_attack | sort -u) ; do
    host_info=$(host "${host}")
    amount=$(grep "${host}" access_log | wc -l)

    echo -e ">>> ${host}:   \t${amount}   \t${host_info} ...";
done

Running this script really gave me a shock:

>>> 140.211.166.168:  18    168.166.211.140.in-addr.arpa domain name pointer osprey.gentoo.osuosl.org. ...
>>> 140.211.166.183:  322   183.166.211.140.in-addr.arpa domain name pointer smtp.gentoo.org. ...
>>> 64.127.104.142:   27    142.104.127.64.in-addr.arpa domain name pointer miranda.amd64.dev.gentoo.org. ...
>>> 86.10.230.114:    554   114.230.10.86.in-addr.arpa domain name pointer cpc3-york1-0-0-cust625.leed.cable.ntl.com. ...
>>> 89.16.163.100:    348   Host 100.163.16.89.in-addr.arpa not found: 3(NXDOMAIN) ...
>>> 89.16.163.108:    345   Host 108.163.16.89.in-addr.arpa not found: 3(NXDOMAIN) ...
>>> 89.16.163.110:    349   Host 110.163.16.89.in-addr.arpa not found: 3(NXDOMAIN) ...
>>> 89.16.176.11:     338   11.176.16.89.in-addr.arpa domain name pointer albatross.gentoo.org. ...

(the integer after the IP-address is the amount of requests in the access_log. The attack started at 18:53:26 UTC and ended at 19:39:46 UTC.)

Yup, that is right. 4 of these machines are Gentoo controlled.

If I recall correctly then smtp.gentoo.org is the main developer machines where Gentoo developers are able to run their IRC clients from, fetch e-mail, etc. so every Gentoo developer has access to this machine. So basically, we now know that a Gentoo developer is behind this.

If we look at Gentoo Infrastructure Server Specifications page, we will see that miranda.amd64.dev.gentoo.org is not listed, but that is probably because it is a machine available to the AMD64 developers in Gentoo (you can see the full list of available developer machines here), but osprey.gentoo.osuosl.org is listed on the infrastructure page about non-developer machines, but albatross.gentoo.org isn’t.

The funny thing about the osprey machine is that it is listed as being Gentoo’s master mirror which probably means that not many people have access to this machine.

Doing a quick whois on the IP-addresses reveals that both osprey.gentoo.osuosl.org and smtp.gentoo.org are hosted by OSUOSL which is a very nice company that hosts various hardware for FOSS projects.

Doing a whois on the miranda.amd64.dev.gentoo.org machines shows that it is hosted on Gentoo’s own netrange at a company called Global Netoptex — a company also knowns as GNi which is one of Gentoo’s sponsors and a company that has two Gentoo developers as employees.

The last interesting machine is albatross.gentoo.org one. Doing a whois on that reveals that it is located on the network of the company called Bytemark Computer Consulting Ltd which among other things also hosts one of Freenode’s IRC servers.

But, uh, the 3 IP-addresses without a reverse DNS entry seems to appear on the same network as the albatross.gentoo.org machine, which is also in Bytemarks IP-range.

Oh, but the fun doesn’t end here. The last IP which looks like a normal “personal” IP-address has something interesting in it. If we just look at it we will see that: cpc3-york1-0-0-cust625.leed.cable.ntl.com. Apparently this IP looks like coming from the city of York in England (I am not sure about that though — and it is not really important either).

Normally I would just fire up Google now, but I thought that the network service called LinkedIn might be useful for this.

Spending two minutes on the site shows that the only person in my network that works for Bytemark is Alex Howells who lists himself as being System Administrator at Bytemark Hosting and Developer at Gentoo Linux. According to his LinkedIn profile, Alex Howells lives in York, United Kingdom.

This begins to look like it is one person who is behind all of this, but a person in the secret IRC channel on Freenode known as #gentoo-infra posted this after we discovered the attack:

[ ... ]
18:55:51 * astinus has a funny idea
18:56:56 astinus: for i in $(seq 1 25); do   wget -O /dev/null http://quotes.exherbo.org/?ratingminus\&id=$i;   sleep 6; done
18:58:55 astinus: additional brownie points if you use -q with wget and CFengine it for a one-time run on 500+ machines
18:59:13 astinus: "Hey guys! All your quotes suck!"
18:59:21 * astinus sighs
[ ... ]

(Timestamps are in UTC.)

In case you wonder: astinus is Alex Howells’s IRC nickname. Doing a simple /whois on IRC shows that as well:

hostmask : astinus!n=alex@gentoo/developer/astinus
ircname  : Alex Howells <alex.howells@0wn3d.us>
channels : @#gentoo-dev #gentoo-userrel #gentoo-server @#gentoo-ops @#gentoo
info     : is identified to services
info     : is signed on as account astinus
idle     : 0d 1h 13m 22s [signon: Sun Jun  8 23:57:52 2008]

From the integer value after the IP-address in the scripts output, we can see that 1380 requsts came from the network of Bytemark and 1934 requests if what we believe is his home machine is included. The other 367 other requests came from either OSUOSL or GNi.

I must admit I am very disappointed with Gentoo these days and this just tops the list. I used to like working for the Gentoo project, but these days this kind of stuff appears to be accepted which is really silly. Gentoo should rather spend some time kicking people like Alex Howells out.

Conclusion
Gentoo should avoid giving people like Alex Howells access to their infrastructure to avoid these very embarrassing issues.

What now?
I have filed a bug to Gentoo’s Bugzilla about this (bug #229895) plus I am going to write an e-mail to the abuse contacts from the various networks where these machines are located.

I will keep this post updated with information when I get it and maybe even write a follow up post.

I am currently using Xen for my virtualization needs and it works pretty well. I am using it on my laptop, my workstation and on my IA64 server.

On my workstation and my server, not having framebuffer support with Xen is okay, given that I never use anything but X on my workstation and I don’t have any console attached to my IA64 server; it is SSH access only.

On my laptop it is a completely different story though. I mainly use my laptop when I am out and I tend not to start X on it, because I don’t have much need of it, so it would be really nice to have working framebuffer there.

Anyways, I hate long blogposts. Here is my solution to get working framebuffer with Xen:

title Gentoo Linux
root (hd0,0)
kernel /xen.gz vga=gfx-1024x768x32
module /kernel-2.6.21-dom0-selinux root=/dev/sda2 vga=0x317 selinux=1

Notice the vga statement on the kernel line.

The vga statement on the module line is probably redundant, but I haven’t removed it yet to see.

Previously, I explained how to implement something like the Active Object thread pattern using smart pointers. Next we’ll use the same trick to implement on-demand, lazy construction.

There’s nothing difficult here, once we realise we can reuse the ‘return a different pointer’ trick. We make use of std::tr1::function rather than a raw function pointer so that parameter values can be bound at pointer-construction time. Again, we parameterise on pointer type, not raw type.

template <typename T_>
class DeferredConstructionPtr
{
    private:
        mutable T_ _ptr;
        std::tr1::function<T_ ()> _f;
        mutable bool _done;

    public:
        DeferredConstructionPtr(const std::tr1::function<T_ ()> & f) :
            _ptr(),
            _f(f),
            _done(false)
        {
        }

        DeferredConstructionPtr(const DeferredConstructionPtr & other) :
            _ptr(other._ptr),
            _f(other._f),
            _done(other._done)
        {
        }

        DeferredConstructionPtr &
        operator= (const DeferredConstructionPtr & other)
        {
            if (this != &other)
            {
                _ptr = other._ptr;
                _f = other._f;
                _done = other._done;
            }
            return *this;
        }

        T_ operator-> () const
        {
            if (! _done)
            {
                _ptr = _f();
                _done = true;
            }

            return _ptr;
        }
};

Again, some caveats:

• Dealing with const is left to the reader.
• Dealing with thread safety is left to the next article.
• If the constructor in question can throw exceptions, the exception will be thrown at what could be a rather unobvious place. This may or may not be a problem.

http://www.boingboing.net/2008/06/11/sabotage-manual-from.html

The Gentoo council, anyone?

With the explosion of overlays used by Gentoo, finding a package can get quite messy. Most users won’t want to set up lots and lots of repositories, so they won’t necessarily know when a package (or an ebuild for an scm or beta version of a package whose stable versions are in the main tree) is available.

Exherbo has similar issues. It’s likely that not-widely-used applications will remain permanently in individual developers’ personal repositories, with only reasonably important packages making it into Arbor. This has further implications for dependencies — for example, if X11 remains in its own repository, but core Arbor packages have optional dependencies upon X, how will that work?

Enter yesterday’s Paludis project: UnavailableRepository.

The idea is simple:

• Have a repository that contains all the packages in all the repositories you don’t have.
• Make it know enough about those packages to show you that they’re available, but not enough to let you do the install. In Paludis terms, this means making the packages be masked with a non-overridable mask, but still support InstallAction.
• Make it less important than any ‘proper’ repository.
• Do something clever to skip doing unavailable IDs for any repository you have configured. (Not strictly speaking necessary, but a lot nicer.)

Configuration is simple. For Exherbo:

format = unavailable
location = /var/db/paludis/repositories/unavailable
sync = tar+http://git.exherbo.org/exherbo_repositories.tar.bz2
importance = -100

And for Gentoo:

format = unavailable
name = layman
location = /var/paludis/repositories/layman
sync = tar+http://git.exherbo.org/layman_repositories.tar.bz2
importance = -100

(Both sync URLs will probably change soon.)

Then you can do this:

$ paludis -q firefox
* net-www/firefox
    unavailable:             (3.0_rc1 (in ::mozilla))X* {:0}
    Description:             The firefox web browser
    Owning repository:       mozilla
    Repository homepage:     http://git.exherbo.org/?p=mozilla.git
    Masked by unavailable:   In a repository which is unavailable

And this:

$ paludis -pi firefox
Building target list...
Building dependency list...
Query error:
  * In program paludis -pi firefox:
  * When performing install action from command line:
  * When executing install task:
  * When building dependency list:
  * When adding PackageDepSpec 'net-www/firefox':
  * All versions of 'net-www/firefox' are masked. Candidates are:
    * net-www/firefox-3.0_rc1:0::unavailable (in ::mozilla): Masked by unavailable (In a repository which is unavailable)

You can even search (on description, at least — searching on other metadata keys won’t find anything):

$ sudo inquisitio browser
* net-www/firefox
    unavailable:             (3.0_rc1 (in ::mozilla))X* {:0}
    Description:             The firefox web browser
    Owning repository:       mozilla
    Repository homepage:     http://git.exherbo.org/?p=mozilla.git
    Masked by unavailable:   In a repository which is unavailable

* net-www/w3m
    unavailable:             (0.5.2 (in ::haskell))X* {:0}
    Description:             Text based WWW browser, supports tables and frames
    Owning repository:       haskell
    Repository homepage:     http://git.exherbo.org/?p=haskell.git
    Masked by unavailable:   In a repository which is unavailable

So what’s in those magic sync tarballs?

$ ll /var/db/paludis/repositories/unavailable/
total 76K
-rw-r--r-- 1 root root  369 2008-06-12 04:47 alsa.repository
-rw-r--r-- 1 root root 3.2K 2008-06-12 04:47 haskell.repository
-rw-r--r-- 1 root root  195 2008-06-12 04:47 kde.repository
-rw-r--r-- 1 root root  666 2008-06-12 04:47 media.repository
-rw-r--r-- 1 root root  238 2008-06-12 04:47 mozilla.repository
-rw-r--r-- 1 root root  334 2008-06-12 04:47 perl.repository
-rw-r--r-- 1 root root 1.2K 2008-06-12 04:47 rbrown.repository
-rw-r--r-- 1 root root  338 2008-06-12 04:47 scientific.repository
-rw-r--r-- 1 root root  20K 2008-06-12 04:47 x11.repository
-rw-r--r-- 1 root root 1.5K 2008-06-12 04:47 xfce.repository

One file per repository, fairly simple. And the files themselves are nice and clean too:

$ cat /var/db/paludis/repositories/unavailable/mozilla.repository
format = unavailable-1
repo_name = mozilla
homepage = http://git.exherbo.org/?p=mozilla.git

dev-libs/
    libIDL/
        :0 0.8.10 ; IDL parsing and compilation library
net-www/
    firefox/
        :0 3.0_rc1 ; The firefox web browser

There’s a bit of metadata about the repository in question (not very much — repositories don’t currently have descriptions or anything like that, and even the homepage is a bit of a hack in a lot of cases), and then data about all the versions.

For each package name, we store each version, its slot, and the description of the best version in each slot (all descriptions is probably a waste of space, considering how little descriptions vary between versions). There aren’t any packages with multiple versions or slots in the above example, but when there are they look like:

sys-devel/
    automake/
        :1.4 1.4_p6 ; Tool used to automatically generate Makefile.in files
        :1.5 1.5 ; Tool used to automatically generate Makefile.in files
        :1.6 1.6.3 ; Tool used to automatically generate Makefile.in files
        :1.7 1.7.9 ; Tool used to automatically generate Makefile.in files
        :1.8 1.8.5 ; Tool used to automatically generate Makefile.in files
        :1.9 1.9.6 ; Tool used to automatically generate Makefile.in files
        :1.10 1.10 1.10.1 ; Tool used to automatically generate Makefile.in files

And that’s all there is to it.

(Well, not entirely. There still has to be a tool to generate the repository content files. Fortunately, dleverton wrote a simple Ruby script using the Paludis bindings that generates them all automatically from the layman master file.)