Archives For Software

The Linux community has done a good job of copying Windows. Open Office does a pretty good job of copying Microsoft Office. Now there are open source versions of YouTube and MySpace. But where is the innovation? Firefox and its multitude of plugins seems to be the one exception. Meanwhile, companies like Apple – a company that is incredible closed, and secretive has created one innovative product after another. Why is this?

  • Do too many cooks spoil the open source stew and lead to a natural conservatism?
  • Do people save their really good ideas for patents, copyrights, and ultimately some plan for profit?
  • Is the open source community not as dedicated (since I’m guessing it is not their full time job to write free software)?
  • Is the open source community not as talented? Has Microsoft, Apple, and Google hired away all the talent?
  • Is it a resource issue?
  • Or is there some other flaw with how open source works that stifle innovation?

I have been looking at how communities like digg, slashdot, wikipedia, and other smaller forums operate. There does seem to be a conservative nature to large crowds. Mainly because the urge to contribute is high. So high, that people want to comment or tweak something even if they have no real expertise. Another issue is that as soon as some one mentions a novel or creative idea, there are 50 people eager to chime in with why its a bad idea. The burden then falls on the innovator to defend their idea. Without sources or previous examples they can cite (we are talking about innovation after all), the idea simply drowns beneath an ocean of criticism. Anyone surveying the discussion will conclude the idea was shot down.

This is all just speculation on my part. It will take more than just a few case studies to uncover any true underlying pattern.

Now that I have all my old email in one place, my next project will be to create some visualizations to start analyzing it. What do I mean? I’m talking about graphs, charts, tables, heatmaps, grids, networks, etc that illustrate patterns in the email. For example – I have roughly 40,000 emails (closer to 20,000 or so were sent by an actual human, and fewer still were sent only to me – but I don’t know for sure and thats part of the problem). My goal is to have a set of Thunderbird plug ins that will allow me see patterns like these. These are 7 visualizations I think Thunderbird (or Gmail) needs. If I get a lot of interest, I might actually build them so let me know what you think in the comments (no registration needed but I do moderate so it might not show up right away). Continue Reading…

one-client2 This post gets a lot of attention, so I rewrote part of it. It will give you step-by-step instructions to take all your old email (in my case over 10 years worth) and upload it into Gmail. Over the years I have used several different email clients: Pine on Unix, Outlook, Lotus Notes, Yahoo Mail, and Gmail. I am currently pretty comfortable with Gmail and have been using it since it was released. Like many people I am a data pack-rat and have kept all my email files (well, almost all – sadly I discovered a 6 months gap). My vision is to put it all into Gmail so I can leverage the great search and tagging features they provide. So here are the steps.

Overall Strategy – get everything converted and loaded into a local server, then sync it to Google.

Why do it this way? There are many advantages. First, if your just moving 100 to a 1000 messages with no attachments, then you can just skip all this and use the IMAP interface found in Google’s FAQ. That method allows you to “see” you gmail account in outlook (or thunderbird or lotus notes) and then just drag or copy messages from one box to the other.  But if your moving more than 1000 messages (or in my case 20,000 messages) with lots of attachments, that will take you days, and you have to sit there are watch it because you can only move about 200 messages at a time. In this method, we consolidate first to a local email server (I’ll show you how to set it up below) running on pc, then let the two accounts (your local one and gmail) sync on their own. Any other way and you are stuck sitting there, dragging and dropping or copying and pasting 200 messages at a time from one folder to another. Second – this keeps all dates in tact. It is really nice to see message you send in 1995 in your Gmail account. Third, once we do this, we can use it as a local backup of our Gmail – just incase the unthinkable happens to Gmail. (I said don’t think of it!)

Ready? Let’s get started

Here is the over all process – you will notice that I installed a email server on my computer called Mercury. This allows us to set up a IMAP and POP account locally that is much faster and will serve as a local backup of all email. 

overall-process

Step 1) First find all your email

I dug through all my old hard drives and consolidate my mail into one location. As you can see, I have outlook (pst) files, lotus notes (nsf) files and files from an old Unix (pine) account I had at school. The zip file (all 2.7GB of it) is the finished product – all my email from 1995 though 2005 (2006 and 2007 are in gmail already).

 
If you are looking at my files, you see a lot of overlap, that’s ok, better safe than sorry…we can delete the duplicates once they are in Thunderbird using an extension.

Step 2. Download the Software

Why Thunderbird – because it is open source with a large developer base. This means there is a good set of tutorials for writing extensions if I to want later on (for those visualizations). Also, I’m making a long term bet that 50 years from now, there will still be tools that import these files – (since they are just text files with no encoding and no database) – plus Thunderbird is really fast and can read 6gb of email quickly (lotus notes and outlook tend to slow down). So Thunderbird will be the final destination on my computer and I will upload from there.

Step 3. Install the software

  • Thunderbird – just default everything
  • Mercury – Make sure you set it up for both POP and IMAP when you get those options in the wizard
  • Lotus Notes – Default
  • Outlook – Default

Step 4. Import Unix files to Thunderbird

These are the easiest because Thunderbird will read these natively. All you have to do is copy the files in to the local folder and reopen Thunderbird.
A) Find out where thunderbird is storing your local files. To do this go to Tools > Account Settings. (note I renamed the local account to Ben Shoemate (all mail) but what your interested in is the "Local Directory" copy that and go there in windows explorer. This is where you need to copy the Unix files (they are files with no extensions). Just copy them to this "local folder", close and restart Thunderbird. Your old mail will be there! That’s it!

Step 5. Set up a local Email account in Mercury.

Open Mercury – if it is all ready running you will see down in the windows toolbar by the clock.

Otherwise , Start > Program Files > etc..

Once Mercury is open, go to Configuration > Manage Local Users > Add User
Because there is so much mail, I am going to set up a different local account per year. Gmail.com can import from 5 accounts at a time, and this will save a lot of time later on. So just make up an account name – I use my initials and the year and a simple password.

It should look like this when your done.

Each of these accounts are real email addresses. When you are local (in lotus notes and outlook) you can map to them with bs1999@localhost
In gmail, you can add them as bs1999@youripaddress (i.e. [email protected]).


Step 6. Lotus Notes NSF files to your local email with IMAP

Once lotus notes is installed, you can simply double click the nsf files to open them, or you can click File > Database > Open > browse

You should see your old mail now. Ahh the memories….But don’t stop to read everything again or I’ll never finish this tutorial! Now, in lotus notes we are going to map to the local email account you set up in mercury. (Note: Gmail now has IMAP so you could just map directly to gmail if you want using the same procedure – but again, it will take a 100 years if you have this much email).

Go to File > Preferences > Client Reconfiguration Wizard

Check Pop or IMAP and click next.

Select IMAP, enter your new email account you set up in mercury, and type localhost, click next.

enter your account name again, and enter the password you gave yourself.
Click next, enter localhost as the server and your email address again – this doesn’t matter since you will not be using this email address for very long and it won’t show up on any of the mail that goes through it.
next, next, next your done.

To open the view of this in notes, click the "Databases" icon on the left panel and open your new account (this might take 2-3 minutes while notes creates a new nsf file etc.)

Now you should have you local account open. Now simply open the old nsf file, go to the "all documents" folder, select all, copy (I think it will only let you copy 2500 at a time) and then switch to you new email account and paste. Once you have all the messages for that year in the local IMAP account, close the nsf and repeat for all nsf files, and all years.

Step 7 – Outlook to your local email with IMAP


The same as in lotus notes – open your pst, then add the imap account and copy and paste.

  • To open the pst go to File > Manage Data files and click Add.
  • To open the local IMAP account go to Tools > Account Settings > New email account

Check the box that says "Manually configure server settings".

  • Select Internet Mail
  • Enter your Name, Email address (bs1999@localhost) and password, select IMAP, the server is localhost for both incoming and outgoing


Once you have mapped to this account, simply copy your old mail from the PST to the new account.

Step 8 – Clean it up in Thunderbird

Connect Thunderbird to you local email server (mercury) and download all your mail. Just like in outlook and lotus notes, we are going to add an account to Thunderbird and let it download all the mail you just put into it from lotus notes, pine and outlook.

  • Open Thunderbird
  • Click Tools > Account Settings > Add Account
  • Enter Account Name (for me its bs1999@localhost), Name, email address (bs1999@localhost)

  • Click Server settings

  • Click OK.
  • The next thing I did was organize copy them down into my local Thunderbird account and used an extension to delete duplicates
  • Then I spent some time looking for gaps…oh well

Step 9 – Upload to gmail


By now your local account has a lot of mail. Let’s start pulling it into gmail. I set up a new gmail account to test it first. From there I can pull it into my main account.

  • Log in to gmail
  • Click Settings > Accounts > Get Mail from other accounts (Pop3) > Add other accounts
  • Add your accounts from your local account. Note: you need you IP address. If you have a router you will need to route ports for pop3 (port 110) to your local computer. Log into your router to do this. While your there, get the IP of your router (ITS NOT the one that starts with 192. or 10. those are always local addresses)


(that’s NOT my IP address btw)

Next – gmail gives you some great options to tag the mail as it comes in, do it. Even if your not a tagger – do it.

You might also want to leave the message on the server (your computer) if this is a trial run.
Click Add account. If you get a problem, make sure your firewall is open, you port is mapped, and your password is right.

Next I’ll show you how to upload all those old chats

If you liked this post, check out my other greatest hits:

Tonight, I discovered what is wrong with the internet, it is very easy to verify something is possible, but almost impossible to validate that it is not possible. Let me give you an example. I was very excited when I read this article on Google’s Blog about IBM Lotus Sametime supporting a new protocol that would allow it to connect with Google. So, i tried it. I tried adding my gmail account to my IBM Sametime client – Presto! – there I was, but permanently offline…. That was strange, I can clearly see that I’m online… So the search begins. I search IBM’s internal website W3 – lots of IBMers asked the question back in January when the news first broke. Back then the answer was “It can be set up by the network admin at the Sametime server, but IBM has not implemented it.” So there is was, *proof* that what I was trying to do was not possible. But that was January, now its October, no one has said its impossible, today yet, or even this week or month. And this is the problem with definitive dis-proof. There is just no easy way to prove that something is not possible. Continue Reading…

I was playing with Google Trends yesterday. With a few keystrokes you can satisfy your need for both useless trivia interesting facts and the data to back it up. Where else can you discover that basset hounds are more popular in Hungary, that Hillary Clinton is finally more popular than her husband, or that in spring of 2007, google became more popular than sex:
Here are some more fasinating trends (at least to me). Continue Reading…