Migrating email from Outlook to Evolution: Linux’s final frontier

Posted on by

At various times in Linux’s history, various things about Linux have really sucked:

  • Getting hardware to work used to really suck, and you used to have to patch the kernel and recompile your own kernel… and then the kernel got a lot better, and the hardware support got a lot better, and I haven’t had to recompiled a kernel in years, and I’m happier because it largely “just works”.

  • Setting up printers used to really suck, with stuffing around with printcap files, and printer configurations, and desperately trying to get it to work … and then cups and printer detection improved, and now it’s generally all painless and point-and-click to install a printer, and I’m happier because it largely “just works”.

  • Setting up X-windows used to really suck, with editing X config files and mucking around with modelines… and then monitor detection got better, and I haven’t had to do anything with a X config file in years, and I’m happier because it largely “just works”.

  • Getting your Microsoft Office documents migrated from Windows to Linux used to really suck … and then Open Office came along, and it does a fairly good job of importing Office’s documents, and I’m happier because it largely “just works”.

This week I came to realize that there’s one last frontier remaining, where Linux still really sucks. And that frontier is migrating from a Windows graphical email client (Outlook in this case) to a Linux graphical email client (Evolution in this case). It does not “just work” … not at all. This blog entry will now explain why I say this.

Earlier this week, my main hard disk on my main desktop machine (a Windows machine) died, with a horrible repetitive clicking and grinding sound. By sheer random good luck, I had backed up all my data onto an external hard disk about one hour before this happened, plus I had a brand new machine ready to go on which I was considering trying Linux anyway. It looked like the stars were in alignment: Linux on the desktop of my main machine, here we come!

Installing and configuring Ubuntu was totally painless. (If you care, the exact steps followed are here: http://nickj.org/Ubuntu_8.04.1_desktop_setup_steps ). And there are a lot of things that I’m really liking with this new operating system, including the following:

  • Complete hardware support: My old Windows install could not detect & use all the cores of my CPU (Windows 2000 Pro does not detect / use a Quad core CPU, whereas Ubuntu 8.04.1 does). In Windows, I needed special drivers for my sound card and my mouse and my keyboard and video card, all of which had to be manually installed or downloaded. In Ubuntu 8.04.1, all my hardware “just works”, or it offers to install restricted drivers (for the video card), and then it “just works”. Nice.
  • The ease of installing and uninstalling software (synaptic + aptitude). Having a well-integrated package manager for installing and uninstalling everything is most pleasant.
  • The way when I play a video, and don’t have the right codec installed, it will offer to download and install the needed codecs for me, and once it’s finished (which typically takes all of 20 seconds), it will play the video. That’s really nice, and sure beats having to manually work out the right video codec to install. Nice one.

But the migration of data from Outlook 2000 to Evolution was shockingly, appallingly bad. There was a lot of misinformation on the web about approaches that should work, but which actually had data loss, and in the end, it took me four full frustrating days to get most of my personal data moved across. The whole migration was so painful that it has left me quite annoyed. When I hear people bandy around phrases like “the year of the Linux desktop”, I simply makes me think: Dream on! This is not the year of the Linux desktop. This is not even the fucking decade of the Linux desktop. Come back in 2011, at the earliest. And when I hear Evolution described as “an Outlook killer”, I can only laugh. To be an “XYZ killer”, you have to do everything that XYZ did, but better, AND you have to be able to import XYZ’s data. Microsoft Word was a WordPerfect killer, because it did what WordPerfect did, but it did it better (in a WYSIWYG way), and it imported WordPerfect’s data. Same for Excel versus Lotus 1-2-3. Same for Firefox versus Internet Explorer. But Evolution does NOT import data from Outlook (for any meaningful definition of the word “import”), so BY DEFINITION it simply cannot be an Outlook killer. And that’s before even getting to the fact that Evolution is not as feature-rich, nor as mature, nor as user-friendly, nor as bug-free as Outlook.

Understand that I’m no Microsoft apologist – I will happily use FOSS, if it’s as good or better.

However, if you are a masochist, if you enjoy pushing hot needles under your fingernails, then it is possible to make the transition. I evidently fall into this category, because I was too stupid or too stubborn to just give up. So here’s how you too can do the same, but I’m warning you straight up, it is not pretty, and it is not easy, and it is not quick.

And for anyone who says it is easy, allow me to enumerate some of the relevant facts:

  • I have/had 4 PST files, not one, like most people (i.e. the data is spread across 4 files, because Outlook barfs when a PST’s size approaches 2 Gb, so I needed to spread it out across multiple files to prevent this).
  • I have/had around 10 years worth of data. All of that data, every single bit of it, needs to come with me. This point is non-negotiable.
  • I use/used all the features of Outlook, apart from Journaling. That’s Email, Calendar, Contacts, Notes, and Tasks. Five categories of data, every single one of which I need. More details about each:
  1. Email. A total of between half a million and 600,000 emails, spread across a 692 nested email folders. These folders are categorized in a hierarchy to keep like mail grouped together. That’s 10 years of work/personal/hobby emails, sent and received and drafts, and this includes a collection of 40,000 spam emails received, kept to help with Bayesian training of spam versus ham.
  2. Contacts (roughly 550 contacts, some of which are just an email address, and some of which have complete details, spread across 9 nested folders).
  3. Notes (being used to store check-lists, passwords, etc., with 400 notes spread across 4 nested folders).
  4. To-do lists (being used to store information, check-lists, and list of bugs or wish-list items in various bits of software that I maintain, with 410 tasks spread across 26 nested folders).
  5. Calendar items (10 years of past events, and additional future events, recording both what happened, and predicted dates and deadlines for things that are going to happen).

So that’s the background. Here’s what does not work for migrating this data from Outlook to Evolution:

  1. Does not work: Export from Outlook, or getting Outlook to export it’s own data into some industry-standard file format. Ha! Have you ever looked at the File -> Import and Export section of Outlook? Remember, Microsoft are arrogant monopolist pricks, who have no vested interest in helping you move to anything else. So, we get just two export options, both of which are basically useless. Next option please.
  2. Does not work: Getting Evolution to read the PST files and import the data directly. It just doesn’t. Various feature requests for this has been open for the past 6 or 7 years, without any visible sign of progress. Move along, nothing to see here.
  3. Does not work: Readpst, which is an Ubuntu package, and which is apparently derived from libpst: On the very first PST I tried this with, it gave a series of warning about NULL pointers, gave a series of messages indicating that it wasn’t going to transfer everything anyway, and then proceeded to segfault. Clearly that’s not going to work.
  4. Does not work: Moving data from Outlook to Outlook Express, and then moving from Outlook Express into something else. Outlook Express only imports data from the main PST (ignoring the other files), and it loses data (converts all Contacts to mail items, converts all Tasks to mail items, losing many or most emails in the process). That’s right folks, even two Microsoft teams, who presumably work in the same building, can’t get their own email products to import data correctly from each other. Forget this.
  5. Does not work: Import into Thunderbird, which uses MBOX format, and then move the MBOX files onto the Linux box, and point Evolution at those. To do start this, you Install Thunderbird, and when you run it for the first time, choose “import from Outlook”, which will import the address book and your mail. I had high hopes for this option, it was very easy to use, and it seemed to work great … at first. However, it has a major problem: severe data-loss. Here’s an example: I have a folder that contains every bit of spam email I was ever sent. It’s useful for training spam detectors, and I found out, it’s also incredibly useful for testing migration tools for data integrity (because spammers send all kinds of weird formats, weird attachments, they ignore standards with impunity, etc.). In short, spam makes the perfect test case. This spam folder has 40,877 pieces of spam mail. How may bits of email do you think Thunderbird imported? The answer is 494 mail messages. That is a 99% data loss rate. Amazing. And there was not a single warning, not a single error – just completely silent 99% data loss. Now, I don’t care about losing my spam, but I do care very much about losing real data, and I had zero faith in Thunderbird at this point to migrate my data without data loss. So, ditch Thunderbird.
  6. Does not work: Migrate from Outlook PST to IMAP. Migrate using IMAP. Connect Outlook to an IMAP server on your LAN, copy everything there, connect Evolution to the same server, and copy or move everything from the IMAP server into Evolution. In theory, this should work great, and with a few test folders, it does. But the issue here is one of scalability: it seems to work fine on the simple stuff, but falls apart on the bigger stuff. Moving a single email would work fine. A single folder would work usually fine. But moving a hierarchy of folders with half a gigabyte of email would cause Outlook to start copying data, and then after about 10 minutes it would usually just get stuck, and then about 20 minutes later it would give a dialog box saying that the copy operation had failed. As a result, this option is unusable if you have substantial data, due (I suspect) to an Outlook IMAP bug. Other versions may work fine, but Outlook 2000 was buggy in this regard – for me, it kept hanging and could not completely transfer all of my data – and therefore it was, unfortunately, unsuitable for migrating data.

So what does this leave? At this stage, I thought I was out of options, and was tempted to just give up on Linux, and stick with Windows. Outport would move some of my data, but it would not move email, which is the largest and most complicated single component that I needed to move. Eventually I found the answer: O2M (which is a US$10 commercial product) for moving email + calendar + contact data, and Outport for tasks + notes, and 2 custom scripts I had to write to massage the O2M and Outport data into the correct format. Disclaimer: I don’t have any financial interest in O2M, I don’t know the people involved, and so forth – it simply the best tool that I could find for the job, and O2M does have problems too, but it’s problems are far less severe than the problems with the other methods.

Here is the link for the step-by-step details of how to migrate from Outlook 2000 to Evolution.

13 thoughts on “Migrating email from Outlook to Evolution: Linux’s final frontier”

  1. I’ve spent a lot of time in trying to find the best possible email client under GNU/Linux. And, Evolution was for a long time my first choice. It is not so full of features, but for a long time, it was the only graphical email client which was able to take more than 10.000 emails in my inbox. Also, it has some Outlook-like features.

    BUT, after the 2.0 version of Thunderbird, with assuming the fact that TB has add-ons, TB became my first choice. It is robust like Evolution now; but it has much more features.

    Tip: Try to analyze KMail features for importing email. KMail is the richest free software email client. Unfortunately, it was never stable. However, it is very useful as a transitional method for moving to Evolution or TB.

  2. That sounds very painful. Kudos to you for persevering.

    Another thing that used to suck and may still suck: laptop wifi.

    Did you consider Thunderbird, or did you want to keep using all those things (notes, tasks etc) in the same application?

  3. Hi Milos,

    I’ll give Evolution a go for a week or two – the Ubuntu folks seem to be pushing it as the default choice, and there’s a lot benefit to running what everyone else is running, in terms of support and fixes and updates.

    I will have a look at Thunderbird on Linux though if Evolution doesn’t work out. I had a look at Kontact / Kmail, and was leaning towards it as my choice because I really liked the look of kmail, it had some annoying issues that turned me off of it:
    1) kontact: Only one set of tasks, when I have a nested hierarchy.
    2) kontact: Only one set of memos.
    3) Kmail: If you reply to an HTML email, it converts it all to plain text, and you cannot turn this off. [http://bugs.kde.org/show_bug.cgi?id=86423 bug report], and [http://www.computerbob.com/guides/why_i_wont_use_kmail_or_kontact.php someone else’s whinge about this].

    — All the best,
    Nick.

  4. Hey Brianna,

    Well I’ll give Evolution a go – I’m kind of used to the idea of one big app that does all your mail/notes/tasks/calendar/contacts, but maybe it’s not a good model. I’m thinking I’ll try it for a few weeks, and then see how I feel. I may just give up and go back to Windows, or may try another graphical mail client on Linux (that’s two comments in favor of Thunderbird, so it looks like that’s the one that I should try first). Thankfully once you’ve got your data into one open source mail client, it _looks_ like it should be easier to move it around between them – hopefully that’s the case in practice! So, will try Evo for a while, and then try Thunderbird if it’s not to my liking.

    Oh, and Wifi on Linux laptops may still sort of suck – I remember having all sorts of problems moving between rooms at conferences, and the wifi would sometimes stop working if you tried to switch from one access point to another, and the solution was to reboot … that may or may not be fixed now though.

    — All the best,
    Nick.

  5. Hi,

    Thanks for the detailed article. I got to know what are my options with Linux. I am on Fedora Core 9 and Evolution is the last thing I need to configure to go totally with Linux. I also intend to use evolution and if I am not able to configure or use it, would go with Thunderbird.

    Destination Infinity

  6. Hi, Nick,

    Did you try moving any other email besides you junk with the Thunderbird method? The reason I ask is that I used this method and successfully migrated 20,000 emails without any loss. I suspect that Evolution treats junk mail differently and may have been deleting most of it upon import (perhaps there’s a rule to delete junk over a certain age?). I also have a simpler email hierarchy than you so that may be part of the problem as well.

    This is how I did it:
    – I installed Thunderbird on my Windows partition and had it import email from Outlook.
    – I located Thunderbird’s mbox files stored in my account Application Data folder which was something like C:/Documents and Setting/My account/ Application Data/Thunderbird/Profiles/random letters/Mail/. From there my pst’s each had their own folder containing all original subfolders
    – I rebooted into my linux partition and opened Evolution, selected Import file from the File menu, and was able to point it at the mbox files on my window’s partition and import them one by one without a loss

    The web site with the closest method to what I wound up using is http://www.softwareinreview.com/migration_guides/moving_email_from_windows_to_linux.html

    Now I just need to figure out how to get Evolution to retain the color coding for my calendar.

    Regards,
    Christian

  7. Hi Christian,

    Yes, I tried migrating all my Outlook mail using Thunderbird. The method I used was the same as the one you used, apart from the last of the 3 steps, where I instead moved the tbird mbox files into evolution’s local mail directory (which I think is ~/.evolution/mail/local/Inbox.sbd/ ) as this saved doing the one-by-one file import for each mbox file / outlook folder. After doing this I noticed that some mail that was not brought over, and I first noticed that mail was missing in several non-spam folders (which I knew for sure contained mail in Outlook, although only a few items), but used the spam email folder as the example because it’s a good way of illustrating the problem using large numbers. Also the problem was with Thunderbird, not with evolution (i.e. the folders with missing mail were missing in both Tbird & evolution, and 99% of the spam mail was missing in both tbird & evolution – i.e. the import from tbird to evolution seemed to work okay, but the mail was lost going from outlook to Thunderbird). I’m not sure of the cause of the mail being lost – could be folder hierarchy, or some other reason, I honestly don’t know.

    — All the best,
    Nick.

  8. Hello,

    very interesting article. Since about 2 months I forced myself into a similar action. Mainly to get rid of the painful experience with Vista, but also to see if I could survive in a business environment (mainly MS-focused) using Ubuntu. In my case I used the 8.10 which improved over 8.04 on network config (especially for wlan and umts) and config for multiple monitors.

    Apart from some small annoyances (like no support for Windows DFS) I must say I’ll stick with Ubuntu as it is a pleasant, stable and very fast OS to work with.

    I agree with your article for me the only reason to go back to Windows is because of Visio and mainly Outlook. I often review visio files and haven’t found a way to deal with them in Ubuntu. But the main annoyance in Linux is Evolution. I’m using it in combination with our exchange 2003 farm and like you I have loads of emails to manage and to archive. Last time I looked my mailbox was about 4 Gb in space, which never caused any problem with Outlook, but it does with Evolution.

    A few times a day I have to force-shutdown Evolution because it stop responding or it looses connection with the backend of exchange. I heard that Apple is planning to use true MAPI connection in the mailclient of their next OS x release. I truly hope that someone develops a plugin or addon that does the same thing for Evolution as I believe this is where the problem is… To my knowledge Evolution uses the dll’s that are used in the OWA environment of exchange. I’m 100% sure that people are willing to pay for a decent mailclient that can be used in combination with Exchange. I know I am…

    best regards,
    Tim

    Bene – Webdesign

  9. Hi Nick,

    Very usefull article!

    In short: my question
    After you have used evolution now for some months, did you change to another mail client, or even change back to windows outlook? Or did you stick with evolution?

    some background information and a story about microsoft licenses:

    I also consider to migrate from windows to ubuntu.
    I’ll already use ubuntu for creating meeting minutes on an acer onenote.
    I”l synchronise these files with a tool ‘beyond compare’ (shareware 30$ – but very fast, stable and usefull).

    The reason I still used Windows was Outlook, having also a mailing archive since 1998 organised in a pst file per year, and a hierarchy of mail folders, tasks and calender events (I don’t use notes)

    The main reason I decided to migrate now also my outlook to the ubuntu environment is the fact that microsoft has found a new way to irritate users. Since a couple of weeks, the automatic update mechanism (wich works great, although you don’t know what it is doing with your system) has installed a feature to check for illegal software.
    I have no problem checkking my computer because everything I install is legal (I tought)…
    After checking, the system now always prompts me with a message that all my licenses of microsoft software are illegal.

    What happened? Last year I bought a new computer, with an OEM version of VISTA installed on it. After reading the legal stuff before installing the license, I had a choice between : don’t install the OEM licence and ask for a refund in the store where I bought te computer (a lot of work and time for only a couple of $$), or install the software and use the prepaid license. I decided to use the prepaid licence.
    I have a subscription to all the microsoft software as a MSDN developer.
    Now it seems that the combination between the OEM Vista version and the licences from my MSDN package is illegal, so I can coose to reinstall the vista version from my developper kit (and then reinstall al the other software again), by all the licenses I use again (double cost) or make a move to ubuntu (including the mail client).
    So I am a little bit angry to Microsoft, because I paid twice for the OS to conclude that by doing that I have illegal software on my comuter… and a lot of extra work or cost.

    I like the OS environment of ubuntu and the possibilities of installing new software packages.

    So I want you to ask, after you have used evolution now for some months, if you still stick with it or did you change to another mail client, or even change back to windows outlook? I’ll appriciate your feedback as I want to use it as an argument to my decision I’ll have to take right now.

    Sincerly,
    Dirk

  10. @Dirk: Yes, I stuck with it, and I’m still using it for all my email, calendar, notes, tasks, and contacts. Evolution is definitely not perfect though (e.g. coming from Outlook, which has certainly had more developers, more time, more testers and more money spent on it, in Evolution you are more likely to encounter bugs in your everyday use and find that it’s less polished), but I have been logging bugs for any features I’d like to see or bugs that I encounter (up to around 60 at last count, of which 18 have already been fixed by the Evolution developers or were duplicates). But, having said that it ties to be a complete PIM, which seeing as you’re using tasks and contacts from Outlook you probably want (otherwise if you just wanted email + calendaring then Thunderbird may be a good option), and the developers are responsive (although quite busy) and contactable which is something you’d never get from Outlook. It’s also nice to get an upgrade every six months, with decent packaging, and to be free of the licensing hassle that you describe (which sounds terrible by the way, great way to annoy people that have paid twice for software by making them reinstall everything). Also, something that may be of interest to you: There’s a new PST importer plugin in Evolution 2.26 (which will be in Ubuntu 9.04) which _should_ allow importing your PST data and save you the migration grief I encountered (I haven’t tested the PST importer though, but if I do I may do a write-up of it, if I get the time). Anyway, I sincerely hope you get the software situation happily sorted out!

    — All the best,
    Nick.

  11. I followed your method. It is an excellent method and it solved most of my problems. Thank you very much! I had about the same problem as you (10+ years of email, complex nested structure on email, contacts, etc.) and this was the one thing that kept me tied to Outlook. Thank you again, so much.

    Only limitation worth noting with the software used for migrating, however, is that using non-ASCII standard text such as accented letters or otherwise in your Outlook will cause some unexpected behaviour when importing Contacts, Tasks and Calendar into Evolution (I have appointments, tasks and contacts which have English, French, Spanish, Chinese and Russian).

    Contacts are troublesome, but you only have to replace all the affected text.

    Calendar completely crashed Evolution and I had to delete the actual calendar files from ~/.evolution/calendar/local/system

    Tasks were the strangest, though. Besides text not displaying correctly (I got this on all three, but it’s the same issue, I guess), Evolution would crash. But only when the cursor passed over the “strange” tasks. I assume this triggers the event that causes the crash.

    Since I had a limited amount of tasks with these problems, I was able to delete them and just type them back in, but this is not really the case for my contacts and calendar. I have plenty of addresses in Cyrillic or Mandarin I’d like to keep and are not too keen on retyping.

    If you’re interested, will let you know if I figure out another way around this.

  12. It’s worse than I though. All non-standard characters are dropped from email as well.
    Back to the drawing board, I guess.

  13. I’ve been using Ubuntu for less than a week. I had avoided making the move because of an inability to get my Outlook email in. I had two fairly large PST files that I wanted to get into Evolution. I was using Outlook 2007 on XP SP3.

    I followed the process included here to get the PST file information into TBird. This was where I got stuck. I could not get Evolution to easily import the TBird email using the Import functionality. On a hunch, I tried moving the TBird email through the file system. Assuming my PST folders were named Foo and Bar, I went to the folder /home//.mozilla-thunderbird/7yz8272m.default/Mail/Local Folders/Outlook Mail.sbd. Inside here were two folders:
    -Foo.sbd and
    -Bar.sbd.

    Additionally, there was a plain text file for each of these folders:
    -Foo and
    -Bar

    and a . msf file for each folder:
    -Foo.msf and
    -Bar.msf.

    I copied these to: /home//.evolution/mail/local (two folders and four files).

    The next time I opened Evolution, th folders Foo and Bar were in the list, under “On This Computer,” just below Inbox. The only down side I’ve found thus far is that all the mail is marked as unread.

    Now, anyone have suggestions for getting contacts out of TBird and into Evolution?

    Thanks to all who wrote and commented on this topic. The information was invaluable in getting my email off the death star that is Windoze.

    Allan

Comments are closed.