At various times in Linux’s history, various things about Linux have really sucked:

• Getting hardware to work used to really suck, and you used to have to patch the kernel and recompile your own kernel… and then the kernel got a lot better, and the hardware support got a lot better, and I haven’t had to recompiled a kernel in years, and I’m happier because it largely “just works”.

• Setting up printers used to really suck, with stuffing around with printcap files, and printer configurations, and desperately trying to get it to work … and then cups and printer detection improved, and now it’s generally all painless and point-and-click to install a printer, and I’m happier because it largely “just works”.

• Setting up X-windows used to really suck, with editing X config files and mucking around with modelines… and then monitor detection got better, and I haven’t had to do anything with a X config file in years, and I’m happier because it largely “just works”.

• Getting your Microsoft Office documents migrated from Windows to Linux used to really suck … and then Open Office came along, and it does a fairly good job of importing Office’s documents, and I’m happier because it largely “just works”.

This week I came to realize that there’s one last frontier remaining, where Linux still really sucks. And that frontier is migrating from a Windows graphical email client (Outlook in this case) to a Linux graphical email client (Evolution in this case). It does not “just work” … not at all. This blog entry will now explain why I say this.

Earlier this week, my main hard disk on my main desktop machine (a Windows machine) died, with a horrible repetitive clicking and grinding sound. By sheer random good luck, I had backed up all my data onto an external hard disk about one hour before this happened, plus I had a brand new machine ready to go on which I was considering trying Linux anyway. It looked like the stars were in alignment: Linux on the desktop of my main machine, here we come!

Installing and configuring Ubuntu was totally painless. (If you care, the exact steps followed are here: http://nickj.org/Ubuntu_8.04.1_desktop_setup_steps ). And there are a lot of things that I’m really liking with this new operating system, including the following:

• Complete hardware support: My old Windows install could not detect & use all the cores of my CPU (Windows 2000 Pro does not detect / use a Quad core CPU, whereas Ubuntu 8.04.1 does). In Windows, I needed special drivers for my sound card and my mouse and my keyboard and video card, all of which had to be manually installed or downloaded. In Ubuntu 8.04.1, all my hardware “just works”, or it offers to install restricted drivers (for the video card), and then it “just works”. Nice.
• The ease of installing and uninstalling software (synaptic + aptitude). Having a well-integrated package manager for installing and uninstalling everything is most pleasant.
• The way when I play a video, and don’t have the right codec installed, it will offer to download and install the needed codecs for me, and once it’s finished (which typically takes all of 20 seconds), it will play the video. That’s really nice, and sure beats having to manually work out the right video codec to install. Nice one.

But the migration of data from Outlook 2000 to Evolution was shockingly, appallingly bad. There was a lot of misinformation on the web about approaches that should work, but which actually had data loss, and in the end, it took me four full frustrating days to get most of my personal data moved across. The whole migration was so painful that it has left me quite annoyed. When I hear people bandy around phrases like “the year of the Linux desktop”, I simply makes me think: Dream on! This is not the year of the Linux desktop. This is not even the fucking decade of the Linux desktop. Come back in 2011, at the earliest. And when I hear Evolution described as “an Outlook killer”, I can only laugh. To be an “XYZ killer”, you have to do everything that XYZ did, but better, AND you have to be able to import XYZ’s data. Microsoft Word was a WordPerfect killer, because it did what WordPerfect did, but it did it better (in a WYSIWYG way), and it imported WordPerfect’s data. Same for Excel versus Lotus 1-2-3. Same for Firefox versus Internet Explorer. But Evolution does NOT import data from Outlook (for any meaningful definition of the word “import”), so BY DEFINITION it simply cannot be an Outlook killer. And that’s before even getting to the fact that Evolution is not as feature-rich, nor as mature, nor as user-friendly, nor as bug-free as Outlook.

Understand that I’m no Microsoft apologist – I will happily use FOSS, if it’s as good or better.

However, if you are a masochist, if you enjoy pushing hot needles under your fingernails, then it is possible to make the transition. I evidently fall into this category, because I was too stupid or too stubborn to just give up. So here’s how you too can do the same, but I’m warning you straight up, it is not pretty, and it is not easy, and it is not quick.

And for anyone who says it is easy, allow me to enumerate some of the relevant facts:

• I have/had 4 PST files, not one, like most people (i.e. the data is spread across 4 files, because Outlook barfs when a PST’s size approaches 2 Gb, so I needed to spread it out across multiple files to prevent this).
• I have/had around 10 years worth of data. All of that data, every single bit of it, needs to come with me. This point is non-negotiable.
• I use/used all the features of Outlook, apart from Journaling. That’s Email, Calendar, Contacts, Notes, and Tasks. Five categories of data, every single one of which I need. More details about each:

1. Email. A total of between half a million and 600,000 emails, spread across a 692 nested email folders. These folders are categorized in a hierarchy to keep like mail grouped together. That’s 10 years of work/personal/hobby emails, sent and received and drafts, and this includes a collection of 40,000 spam emails received, kept to help with Bayesian training of spam versus ham.
2. Contacts (roughly 550 contacts, some of which are just an email address, and some of which have complete details, spread across 9 nested folders).
3. Notes (being used to store check-lists, passwords, etc., with 400 notes spread across 4 nested folders).
4. To-do lists (being used to store information, check-lists, and list of bugs or wish-list items in various bits of software that I maintain, with 410 tasks spread across 26 nested folders).
5. Calendar items (10 years of past events, and additional future events, recording both what happened, and predicted dates and deadlines for things that are going to happen).

So that’s the background. Here’s what does not work for migrating this data from Outlook to Evolution:

1. Does not work: Export from Outlook, or getting Outlook to export it’s own data into some industry-standard file format. Ha! Have you ever looked at the File -> Import and Export section of Outlook? Remember, Microsoft are arrogant monopolist pricks, who have no vested interest in helping you move to anything else. So, we get just two export options, both of which are basically useless. Next option please.
2. Does not work: Getting Evolution to read the PST files and import the data directly. It just doesn’t. Various feature requests for this has been open for the past 6 or 7 years, without any visible sign of progress. Move along, nothing to see here.
3. Does not work: Readpst, which is an Ubuntu package, and which is apparently derived from libpst: On the very first PST I tried this with, it gave a series of warning about NULL pointers, gave a series of messages indicating that it wasn’t going to transfer everything anyway, and then proceeded to segfault. Clearly that’s not going to work.
4. Does not work: Moving data from Outlook to Outlook Express, and then moving from Outlook Express into something else. Outlook Express only imports data from the main PST (ignoring the other files), and it loses data (converts all Contacts to mail items, converts all Tasks to mail items, losing many or most emails in the process). That’s right folks, even two Microsoft teams, who presumably work in the same building, can’t get their own email products to import data correctly from each other. Forget this.
5. Does not work: Import into Thunderbird, which uses MBOX format, and then move the MBOX files onto the Linux box, and point Evolution at those. To do start this, you Install Thunderbird, and when you run it for the first time, choose “import from Outlook”, which will import the address book and your mail. I had high hopes for this option, it was very easy to use, and it seemed to work great … at first. However, it has a major problem: severe data-loss. Here’s an example: I have a folder that contains every bit of spam email I was ever sent. It’s useful for training spam detectors, and I found out, it’s also incredibly useful for testing migration tools for data integrity (because spammers send all kinds of weird formats, weird attachments, they ignore standards with impunity, etc.). In short, spam makes the perfect test case. This spam folder has 40,877 pieces of spam mail. How may bits of email do you think Thunderbird imported? The answer is 494 mail messages. That is a 99% data loss rate. Amazing. And there was not a single warning, not a single error – just completely silent 99% data loss. Now, I don’t care about losing my spam, but I do care very much about losing real data, and I had zero faith in Thunderbird at this point to migrate my data without data loss. So, ditch Thunderbird.
6. Does not work: Migrate from Outlook PST to IMAP. Migrate using IMAP. Connect Outlook to an IMAP server on your LAN, copy everything there, connect Evolution to the same server, and copy or move everything from the IMAP server into Evolution. In theory, this should work great, and with a few test folders, it does. But the issue here is one of scalability: it seems to work fine on the simple stuff, but falls apart on the bigger stuff. Moving a single email would work fine. A single folder would work usually fine. But moving a hierarchy of folders with half a gigabyte of email would cause Outlook to start copying data, and then after about 10 minutes it would usually just get stuck, and then about 20 minutes later it would give a dialog box saying that the copy operation had failed. As a result, this option is unusable if you have substantial data, due (I suspect) to an Outlook IMAP bug. Other versions may work fine, but Outlook 2000 was buggy in this regard – for me, it kept hanging and could not completely transfer all of my data – and therefore it was, unfortunately, unsuitable for migrating data.

So what does this leave? At this stage, I thought I was out of options, and was tempted to just give up on Linux, and stick with Windows. Outport would move some of my data, but it would not move email, which is the largest and most complicated single component that I needed to move. Eventually I found the answer: O2M (which is a US$10 commercial product) for moving email + calendar + contact data, and Outport for tasks + notes, and 2 custom scripts I had to write to massage the O2M and Outport data into the correct format. Disclaimer: I don’t have any financial interest in O2M, I don’t know the people involved, and so forth – it simply the best tool that I could find for the job, and O2M does have problems too, but it’s problems are far less severe than the problems with the other methods.

Here is the link for the step-by-step details of how to migrate from Outlook 2000 to Evolution.

It’s that time of year to renew my car registration, and buy the accompanying compulsory third-party insurance. So I tried the NRMA, and price-wise their quote was okay, but I wanted it mailed to me in the post so that I can pay it closer to when I actually need it. Trying to tell the NRMA this proved to be impossible:

… and then clicking the submit button gives this:

… forbidding all English punctuation – that’s a really nice touch! So I removed all commas, full stops, apostrophes, and question marks, leaving one continuous string of text, and clicked submit again. The result is this:

Wow, that’s impressively crap. Being that bad at listening to people’s feedback doesn’t just happen, it takes serious dedication and practise and commitment.

Ok, I’ll bite in response to this “Where did all the PHP programmers go?” blog post:

What I cannot understand is why people with more than one Bachelor Degree in Computer Science recommend using bubble sort.

Sounds wrong but harmless, as you don’t write a sort implementation from scratch in PHP. You write the comparators used for the sort order, but the actual sort implementation is provided for you by language. I presume it uses qsort internally, but don’t know for sure. I have a degree in CS, and I can scarcely even recall the bubble sort algorithm (or even most of the sort algorithms for that matter), for the simple reason that it doesn’t matter in the real world (in 99% of cases) for web developers using scripting languages. That may sound (gasp) shocking, but it’s true – PHP is not a performance-orientated language, and it’s a fairly high-level language with a decent library of native functions, so you don’t generally write sort algorithms (rather you use the library ones that are provided for you, unless you have an overwhelmingly good reason not to).

The question you need to ask is: are you running a Computer Science class on sorting algorithms, or are you looking for people who know PHP and can get your thing built?

“What is the difference between the stack (also known as FILO) and the queue (also known as pipe, also known as FIFO)?”

Maybe rephrase the question to “you want to store multiple bits of information in a data structure or an array or a collection of some sort. How would you add data to the beginning of that data structure, and how would you remove data from the end?”

I.e. focus less on the Computer Science theory, and more on the application of it.

“Using PHP programming language, create a list to store information about people. For each person you’ll need to store name, age, and gender. Populate the list with three sample records. Then, print out an alphabetically sorted list of names of all males in that list. Bonus points for not using the database.”

Here’s a trivial implementation just using arrays, I don’t claim it’s remotely pretty or elegant, and I wrote just to see what’s involved in the above task:

<?php
error_reporting( E_STRICT | E_ALL );

function sort_by_name( $a, $b ) {
       if( $a['name'] === $b['name'] ) return 0;
       return $a['name'] > $b['name'];
}

function printMales( $array ) {
        foreach( $array as $person ) {
                if( $person['gender'] != 'male') continue;
                print "Name: " . $person['name'] . "\n";
        }
}

$people = array( array( 'name' => 'Bob'      , 'age' => 36, 'gender' => 'male'   ),
                 array( 'name' => 'Alice'    , 'age' => 23, 'gender' => 'female' ),
                 array( 'name' => 'Doug'     , 'age' => 63, 'gender' => 'male'   ),
                );

print "Before:\n";
print_r( $people );
usort( $people, 'sort_by_name' );
print "After:\n";
print_r( $people );
print "\n";
printMales( $people );

?>

But you know what? I had to look up the PHP manual for usort because I couldn’t recall off the top of my head whether it was “u_sort” or “usort”, and I couldn’t recall the parameters and their order. Also I had 3 trivial syntax errors that I fixed in 15 seconds. Now, I really hope for this pen-and-paper test that you are giving people access to the PHP manual, or if you are not that you are being very tolerant of minor syntactical errors or people who can’t recall whether the function name has an underscore, or who can’t recall the exact order of the parameters, and so forth. Because the question is: Is this a test of whether someone has memorized the entire PHP manual, or is this a test of whether people who can do what you want? Because when they are working, then you will give them access to the PHP manual – right?! If you want to distress people in the interview, then sure, treat it as a rote memory test of the PHP manual and Computer Science theory, and make it awkward if they get anything wrong – but if you’ve want to solve the problem of finding people then there has to be some leeway for recollection of technical trivia that you can find through Google in a few seconds.

Look, I’ve been in a similar situation of looking for PHP people to hire (the candidates were from China in this case), and the approach we used was to give them a test beforehand that they could do (in 24 hours of their own time), and then if they looked okay then they could get called in for an interview. This allowed culling people who were very bad, or who gave code that didn’t run – as there really is very little excuse for code that’s invalid or that doesn’t work if you’ve got 24 hours and access to the internet and your own computer. Most of the people weren’t great, some were very bad, and some were okay. If it helps, that PHP test is here, and it’s only intended to be a very simple test.