Archive for December, 2008


Customizing the XO image

Bryan asked to write something about the XO-customization build process for OLE Nepal. See this post as the second installment of the OLE Nepal deployment series.

So here we go:

First: why do we need to customize a standard XO build? What’s wrong with the one supplied by OLPC. Isn’t that one tested through and through, and shaved to perfection?

Absolutely. To perfection… Well, almost… But that’s besides the point. In a software package as big as a total XO install, it’s just about inconceivable that the local requirements overlap a hundred percent with OLPC’s. General stuff, like proper support for Unicode keyboard layouts, is… general enough for OLPC to integrate in the standard software. But between us wanting it, and them integrating it in a stable build lies a time-gap to great to wait for. And since we are progressing with local stuff over time, we will have new requirements, when the old ones are integrated. Also we have, or might have requirements that OLPC just isn’t interested in or for which it just doesn’t have the manpower. And then then we have local configurations like our own update server for Activities or System software, which are scattered around the filesystem.

It’s easiest to keep track of all these changes in one place, and we made our own version-controlled Pilgrim build to do take on that burden. For the uninitiated: Pilgrim is the XO filesystem- and image-builder that the folks at OLPC use themselves to build their images.

The biggest benefits of Pilgrim over for example cloning (which we did before) are:
– All the customizations are contained in scripts, so you don’t have to document them, so you can’t forget certain steps
– It forces you to structure the files needed for customization, and while you’re at it, you might just as well version-control them
– You get a customization factory which builds lots of different useful files, like image files, with which you can flash XO’s, the .usb files which you can use for machine-local olpc-update-s, and an fs tree which you can use for olpc-update over the network. (note btw that olpc-update preserves user-data).

A rival of Pilgrim is Puritan by the way, written in Python, mostly by Michael Stone, which tries to be more elegant about things like caching, debugging and aborting the build process, and building and customizing the image (and has a cool way to use Git for changes). The need for something like Puritan is there, because Pilgrim IS a big daisy-chain of hacks.

Pilgrim nitty-gritty

So what kind of stuff did we change at OLE Nepal? Perhaps a bit more overview first. Say one would want to get a clean 8.2 image. One checks out Pilgrim, sets it to the right build (be it 8.2 or 703 or whatever), installs some dependencies, make-installs it and enters an incantation on the command-line to let it poop out the .img file and its brothers and sisters. And voila!

But one wants and needs to hack pilgrim a bit. The relevant big ones are the ‘pilgrim’ file in the root and the one in the streams.d dir, with some auxillery config files sourced in. A few thousand lines of bash scripting, so it’s a little dynamically scoped monster.

The first step was to cache all the rpms needed for building the XO local, since downloading them from outside Nepal takes hours and out of the box Pilgrim deletes them. We put them in a local Yum repo, and edit the yum directives  in olpc-development-yum-install.conf accordingly.

Also we revived the automatic installation of activities. Which needed a little bit of love, since last it was used it still put all of them in /usr/share/sugar. Also the installer expects an html file with links to the activities you might want to install, which it wgets to a local directory, after which it takes the latest version of the ones you specified in So you might want to have a script to generate that file from the contents of your activities directory, much like what one needs for the software-updater, but a bit less sophisticated.

Related to the last paragraph, you’ll find that the fake mounted image in your build-system will fill up pretty quickly with all that activity content. A bit to quick actually, since the XO uses jffs2 which will compress your fs pretty tight, while pilgrim configures the image on your build system to only one gig. So you might want to set PILGRIM_FS_SIZE_jffs2 and/or PILGRIM_FS_SIZE_devel_jffs2 in to double that amount, which is about right I believe.

Then some customizations on what output would be created was necessary. First of all we want to use olpc-update for updating the XO’s from time to time. olpc-update expects a filesystem tree, and a contents file pooped out by pilgrim. With the standard build, pulled from the interwebs, the files aren’t put in the right order and place (or, at all really) to be consumed by the olpc-update script present on the XO’s.

Also be ware if you want to olpc-update from your own server with a 703 build. Its on-board olpc-update version doesn’t have the option to specify another server than the standard. And the development version of olpc-update doesn’t contain all the bitfrost files needed by 703, since it doesn’t have them itself yet. You can just supplant the bitfrost dir in the development version with the bitfrost dir from a recent XO build ( /usr/lib/python2.5/site-packages/bitfrost  works with the one from 8.2 in any case. Should still send a patch). AANNDDDDD, olpc-update pulls in files from the server through rsync://…, so set up a correct rsync daemon conf file in /etc, and start the rsync daemon at startup.

And also the standard pilgrim will make tars out of the fs that you’ll rarely need but will take forever to build and take up useless space. So we commented them out.

XO customizations

Then we have to deal with the customizations to the XO itself. The whole reason for this operation.

1) We had issues with xkb and the traditional Nepali keyboard layout. SCIM solves these issues, so that’s what we installed. Everything, except for Write, seems to work for us. Since we’re expecting to work that kink out of the cable, we’re switching to SCIM. Adding extra packages in Pilgrim is just a matter of adding them to the yum list in and making sure you point to a repo that holds them and their dependencies.

2) The same holds true for auxillery packages needed by certain activities. We want flash and gnuchess for example, and xpdf, which comes with its own set of dependencies.

3) Then we have config files that we want to hardwire. So the xo’s should update from our schoolservers, not from the wiki or from a location specified by the activities themselves. So we put a file in /etc/olpc-update. Of course you can do that through the gui, but you don’t want to do that for every XO that goes through your hands.

The correct way for incorporating these configurations is perhaps making rpm packages, but to me that seems like to much of a hassle for no real gain, so we’ve got our own for that. Makes it sound nice and official.

Other victims are the scim config dir, the list of default activity favorites, Nepali espeak data and the wpa2 password for the schoolserver if utilized (connecting to aps with wpa2 set is very flaky atm).

Right now we can’t just put our images on non-unlocked (so locked) XO’s, because we can’t sign them (olpc-updating locked XOs to unsigned builds should perhaps not be that hard though, if all one needs to do is tweak the olcp-update script), but apparently OLPC is working on a solution for deployments.

Of course we tested the builds, and they work as expected. So now we’ve got a nice updater and a nice custom image generator. Pretty sweet!

Oh! And happy New Year, to the ones who celebrate! And also for the ones who don’t. No need to be cheap.


Shielding against fate

As written before, at the moment we’ve got two pilot schools, fairly close to our office. We, them and the Department of Education are connected to eachother through a simple network. If something goes wrong software- or hardware-wise, or we need to upgrade XO’s, it’s fairly easy for us to come over and fix stuff on the same day.

But actually that already is a hassle. Whoever goes to the schools looses a day, and putting new activities on 135 XO’s, repeating the same mundane commands, is time consuming. In spring we’re gonna deploy a couple of thousands XO’s to 15 schools around Nepal. We need automation! And not only that; we also need compartimetalization. Actually we are trying to strike a balance between the easiest way to put content on laptops and the most robust way to put content on laptops.

The problem

That is to say, in a country like Nepal you can’t count on anything. Take power for example. Our office is located in the center of Kathmandu, the capital. For two days running now we hardly had any power (so I’m filling my time writing blogposts on my XO). For one we have scheduled power outages which total 10 hours a day (‘loadshedding’). Rumour has it this will go up to 18 hours a day. Our battery-backup system can’t handle that many hours. Also our UPS has problems. Something that’s gonna happen in the field as well, but then there’s no power-engineer present on-site to address the problem.

A related issue is the fickleness of internet connections. Often we at the office don’t have internet because of problems at our ISP or with the connection from Nepal to the outside world. And of course no power automatically means no internet. And this can happen on a number of levels. Our schools are connected to the internet through the computer-lab at the Department of Education (DoE). And DoE is equally affected by loadshedding. Last week we discovered that the capacitators there can’t handle the power-requirements of the battery-rig we set up in the lab. No power at DoE, no internet for the schools.

The solution

And it’s not just about internet access. We’re gathering more and more content for the kids to use during their classes. Way more than we can put on their XO’s. So they need a way to access that information, and we need a way to easily update this information. An ideal scenario would be putting that content on a central server and to let all kids access that content on-demand. An additional benifit is that kids from non-pilot-schools can also benefit from our efforts. But because of network and power issues we can’t rely on a central server alone. And this is where things get messy. So software-wise, what we’re sort of trying to create a little three-layered system.

1) At the top there’s a central server, capable of upgrading the schoolservers and able to serve all our content to everybody that has access to the web.

2) The schoolservers have a local copy of all the content, since we can’t count on the central server. The schoolservers should be able to upgrade the XO’s. Both the system software and the content, which both have different requirements.

3) The XO’s themselves of course.

Upgrading the system-software is the least of our problems. Right now we use olpc-update from local servers who get custom builds, made by Pilgrim, from the central server. Upgrades will be pushed only once in a while. Content is a different problem. Since we can’t count on the schoolserver being up all the time we need to have content on the XO’s themselves. But since we’ve got too much content, some should stay on the server.

Our plan is to have the content needed for specific lessons of the current week, the last week, and the next week on the XO. If the kids want or need to access more info, they should be able to get it on demand. And all of this should be transparent and automatic. We’re gonna use Moodle to specify the content for all the school-weeks on the schoolservers. We’re gonna use google gears to cache three weeks of content on the XO’s. If for a specific week the content is not to be found locally, the XO’s should go to the schoolserver and get it there.

Most of the stuff cited above is being worked on. We’ve got a lot of content. We’ve got a central server for that content. We’re putting the last touches on our customized version of the OLPC schoolserver software (0.4 atm). Thanks to XO version 8.2 we can update our own set of activities from our local server. We’ve got our own Pilgrim XO baking factory. Olpc-update:ing from a local schoolserver works fine.

What doesn’t yet works is the whole Google Gears/Moodle setup. Moodle is set up properly but Google Gears hasn’t had enough love yet. Our in-house developed content also isn’t yet integrated yet, but our individual activities made in both Squeak/Etoys and Flash, can easily be split up and accessed through the web individually.

Also we don’t yet have a mechanism in place to update individual schoolservers from a central server. And again we need a separate mechanism for system-software and content. This is the only aspect we haven’t worked on at all so far, but we still have a few months till the deadline in mid-April. Should be time enough… we hope…


Deployment preparation

So this summer we’re gonna deploy a few thousand XO-s in around 15 schools, in around 5 districts in Nepal. How are we preparing for it?

The first question might be: what do you need?

software infrastructure
a well functioning, customized XO image
– a good plan to set up networks for these schools
– money
– people to go to these schools and have the knowledge to set up the schoolservers, the network, train the teachers and pour their knowledge into a local person, who can troubleshoot and will be our contact when we upgrade or something goes wrong beyond their control

These five items might be a good starting point for a blog series, and darn it, they are. In the next weeks I’ll write a bit on all of them, except for the last one, because there’s already enough written on them over on the OLE Nepal blog. See ‘Deploying Volunteers’ and ‘The Intern Experience’. I’ll link to the items in this post when they are written.


Moving along

The path from the ring-road to my house is littered with speed bumps. It’s ridiculous. There are nine of them all about 20 metres apart. And that’s on a road that’s really some sort of shopping street for the most part, and the road is to small to be able to drive fast in anyways.

I thought it was just my street but a few weeks ago I actually came across an article in the paper which complained about this practice, which apparently is being practiced all around Kathmandu. It seems to me that the bump-layers get a quota per district and they just lay them all at once just to be done with it.

Have you noticed that fate seems to follow exactly the same pattern: Lay lots of speed bumps all at once just to be done with it.

‘Fate spins it’s little web’. Yeah right. Like it’s a delicate operation. ‘Fate bulldozes your house’ would be more apt a lot of the time.

Point in case:

I’m gonna move in the next couple of days. Not once, but thrice. Yes thrice!

1) Tomorrow I’m gonna move in with a couple of Australian girls and an American one. Here, in Kathmandu, Nepal.


Moving-to-Rotterdam gift from '95. Part of the '08 Rotterdam moving preparation photo series. Curtosy of Marja.

2) After having occupied the better half of a floor for four years the stuff I’ve got laying around in my old apartment in Rotterdam, Holland has gotta move because my old roommate is gonna move in with her boyfriend (She’s so selfish. Where am I supposed to live when I get back!).

3) I’m still on the contract of the house I shared with my ex in Eskilstuna, Sweden. She wants to move asap, but she can’t because she has to get my consent to leave the place. I hope she takes some of my stuff with her.

So my stuff all over the world is moving. Perhaps it’s a good thing. Perhaps all the stuff, and the people herding it, will be a bit closer to each other after all of this is done. But I am getting a bit tired of the logistics of it all.

The contract thing is the easiest of course. Filling in some stuff and faxing it. Then after a while there will be some stuff sorting issues but that will be of later concern. The other two sites are a bit messier at the moment.

Here in Kathmandu the situation is a bit weird in a way because I’m not just bringing my own stuff along. Just about all ex-roommates and ex-colleagues left in a hurry for one reason or another. I’m like a little left luggage department. Except for the fact that the luggage isn’t lost but the respective owners. Well, it’s always a matter of perspective of course. It’s just that for once I’m more sympathetic with the luggage.

In Feb I came here with two backpacks and some money in my pocket. Now the money is gone, it seems to have solidified in lots of junk. Which is not true btw. The money went into other things. The junk I inherited from the same people that I got the luggage from. So now all of a sudden I own a rice-cooker, lots of reed furniture (bed, couch, chairs, tables, cupboards, baskets, benches), loads of sci-fi books, lots of clothes that don’t fit me, garden chairs, two of those Italian stove-coffee-brewers (not actual people but those metal things with a beak), roti-creation-utensils, an electric heater, an army of plastic cleaning vessels, a unicycle and lots and lots more. Yesterday evening, when I started packing, I still went into this with the pack-half-an-hour-before-lift-off mindset of a backpacker: “passport, check! money, check! bag seems heavy enough to contain some clothes, check! Let’s get outta here.” I now see the error of my ways.

For the Rotterdam one I am not on-site. That’s a bit crappy. I’d like to know which trigger-objects of my precious memories will be preserved and which ones will be cast into oblivion by ignorant hands, operated by ignorant people.

I asked my roommate Marja to jolt my memory with some piccies. She complied and embarked on an odyssey of photo-shooting. She mailed me around 15 mails which all had attached to them around 6 photo’s. In the end I got 89 pictures worth of info on the state of all my Dutch belongings. Before I looked at the session I thought, for the sake of logistics, to salvage about two boxes. One for important papers and one for mesmerising purposes.


Grumpy old desktops. Part of the '08 Rotterdam moving preparation photo series. Curtosy of Marja.

I should never have looked at those pictures. The greedy little reptile part of my brain took charge quite vigorously when my head tried to democratically decide on the individual pieces. It issued veto after veto and in the end I wanted just about everything except for my old clothes and the array of desktops in the hallway that over the course of four years seemed like ancient dinosaurs, left in the dust by Moore’s law.

Here in Kathmandu I just finished the BIG SORTING. I’m a bit anal about that. All and every item has found it’s brethren. The heap of batteries is lighting up the night. Lots of flirting, lots of rubbing skin, in an electric frenzy that can’t last till morning. Candles ignite with joy seeing loved ones long gone, weeping for those that burned up to quick. But nothing is in boxes yet. And it’s getting late. In my optimism that all will sort itself out I didn’t press my friends hard enough to come and help me, and so I’ve got no sure confirmation. Well, we’ll see about that in the morning.


Cheers to relief-aid

Since I’m living here all by myself in my beautiful three-storey Bagdol abode, I’ve been looking around for a new place lately. Preferably with some other people, cause I don’t really like to live alone. Yesterday I went to check out an apartment very close to my work. I would/will (I dunno yet) be living with two Australian girls from the Hash and so we talked a little the evening I was there. One of them is involved with the flooding that happened some months ago in the east of Nepal. She told one of those classic stories of social horrors that’s hard to comprehend for us mere worker bees.

Just a quick recap: last August the Koshi river broke through it’s eastern embankment (through the collapse of a dam I believe) and flooded a big part of the flat, fertile part of Nepal, the Tarai. Then the water streamed on into the Indian province of Bihar. About 60.000 Nepali’s had to move and about 3 million… let me repeat that… THREE MILLION Bihari’s had to high-tail it. Leaving flooded farmland, dead cattle and destroyed crops which would have been eaten by the locals of course, and which is vital for the rest of the country.

I learned from the room-girl that the water still hasn’t been sealed up yet (the breach grew from 300 meter in August 18 to over 2 km due to erosion). We in Kathmandu probably only have felt this directly in that the loadshedding (scheduled power cuts) hours have gone up; a/the dam(s) alongside the river supplied power. But there’s still running water over farmland, villages and such. I don’t know how much land, and how much people is/are affected, but that isn’t the biggest problem here.

In a couple of months it’s gonna be monsoon season again. Room-girl was quite sure that they couldn’t seal the hole before that. So enormous amounts of water in river, breach not sealed yet… Not good. Embankment will erode once more. Again massive floods.

Now that’s a natural disaster. Force of nature, bad planning, perhaps… no, probably not enough resources allocated. But there’s quite a vicious social horror story attached to this. Which makes this whole thing damn right evil, if not purposefully then through negligence to investigate the issue by the responsible policy-makers:

All those 60.000 people had to go somewhere, right? Most of them went to the big cities, and so a lot went to Kathmandu. As a LOT of rural people went to Kathmandu over the past years. Policy-makers would rather want them out, so now they offer these people the sum of 150.000 rupies a person (could be wrong by a couple of zeroes) in relief-aid IF they sign this piece of paper that they march right back to their ruined farmland and homes and dead buffaloes and wait for the next flood to displace them. Just what the neglected, poor, ethnic-tension ridden Terai needs… Ain’t that evil?

I wouldn’t even mind if someone out there on the interwebz could harshly point out to me where in my post I tried to rape reality.