How to upload and archive all old Chats from Yahoo IM, AOL, and others into Gmail

November 29, 2007 by Ben Shoemate

uploadchats I love how Gmail stores all your chats and makes them available for search. But what about all the chats from before? – I have 10 years of IM’s saved – ICQ, Yahoo IM, AOL IM, Jabber, etc.  My goal is to get them all uploaded into GMail. Here is how I did it.

The first thing you need to know is that in GMail, your chats are actually stored as normal emails with a tag "Chat". So all we need to do is convert the chats to email, upload them and tag them as "chat".  Below are the 5 steps to get them uploaded!

Read more » »

Comments 

End User Agreements – Stealing customer rights with a sneak attack in the fine print

November 29, 2007 by Ben Shoemate

Recently I have become more aware of the many “agreements” that companies use to try and take away the rights of users (and protect themselves from any liability). Almost every activity in this increasingly digital and legalistic world comes with attached fine print that tries to absolve the company of potential bad behavior. When you sign your credit-card slip at Best Buy or Fry’s, you waive all kinds of rights you get under consumer protection law. Of course if you actually stop to read these agreements, it is likely that your session will expire if your online or if your in the checkout line at Target the clerk will likely call the manager or even the police after the first hour or so of haggling (everyone should do this at least once a year – then they would stop doing it I’d wager). It would literally take all day to read, and ask questions (none of which the store clerk would be able to answer) about all the agreements in an average customers shopping cart. Since reading them all is not practical, and understanding them not possible, and negotiating them is NEVER possible – what to do? What’s the point of having rights (like fair use) if they can, and are, so easily swindled away? Read more » »

Comments 

7 Email Visualizations for Thunderbird

November 13, 2007 by Ben Shoemate

Now that I have all my old email in one place, my next project will be to create some visualizations to start analyzing it. What do I mean? I’m talking about graphs, charts, tables, heatmaps, grids, networks, etc that illustrate patterns in the email. For example – I have roughly 40,000 emails (closer to 20,000 or so were sent by an actual human, and fewer still were sent only to me – but I don’t know for sure and thats part of the problem). My goal is to have a set of Thunderbird plug ins that will allow me see patterns like these. These are 7 visualizations I think Thunderbird (or Gmail) needs. If I get a lot of interest, I might actually build them so let me know what you think in the comments (no registration needed but I do moderate so it might not show up right away). Read more » »

Comments 

How to organize your hard drive

November 10, 2007 by Ben Shoemate

This is a problem every one struggles with: what is the best file structure for old files? The problem is actually compounded by the need to do routine back-ups. As I was going through all my old hard drives as part of my email project to upload all my email into Gmail. I took note of the various ways I organized my files over the years, the various folder structures I adopted. My digital life is dominated by projects – personal projects (most half-baked and half-finished), client projects (with thousands of files each with multiple versions), and team projects (like personal projects only with multiple people). Then, like everyone, I have downloaded files (in the form of mp3s, images, videos, and saved web pages), personal photos, and miscellaneous documents.

Here is my strategy… Read more » »

Comments 

How to upload all your old archived Email to Gmail from outlook, lotus notes, and Unix (pine)

November 9, 2007 by Ben Shoemate

one-client2 This post gets a lot of attention, so I rewrote part of it. It will give you step-by-step instructions to take all your old email (in my case over 10 years worth) and upload it into Gmail. Over the years I have used several different email clients: Pine on Unix, Outlook, Lotus Notes, Yahoo Mail, and Gmail. I am currently pretty comfortable with Gmail and have been using it since it was released. Like many people I am a data pack-rat and have kept all my email files (well, almost all – sadly I discovered a 6 months gap). My vision is to put it all into Gmail so I can leverage the great search and tagging features they provide. So here are the steps.

Overall Strategy – get everything converted and loaded into a local server, then sync it to Google.

Why do it this way? There are many advantages. First, if your just moving 100 to a 1000 messages with no attachments, then you can just skip all this and use the IMAP interface found in Google’s FAQ. That method allows you to “see” you gmail account in outlook (or thunderbird or lotus notes) and then just drag or copy messages from one box to the other.  But if your moving more than 1000 messages (or in my case 20,000 messages) with lots of attachments, that will take you days, and you have to sit there are watch it because you can only move about 200 messages at a time. In this method, we consolidate first to a local email server (I’ll show you how to set it up below) running on pc, then let the two accounts (your local one and gmail) sync on their own. Any other way and you are stuck sitting there, dragging and dropping or copying and pasting 200 messages at a time from one folder to another. Second – this keeps all dates in tact. It is really nice to see message you send in 1995 in your Gmail account. Third, once we do this, we can use it as a local backup of our Gmail – just incase the unthinkable happens to Gmail. (I said don’t think of it!)

Ready? Let’s get started

Here is the over all process – you will notice that I installed a email server on my computer called Mercury. This allows us to set up a IMAP and POP account locally that is much faster and will serve as a local backup of all email. 

overall-process

Step 1) First find all your email

I dug through all my old hard drives and consolidate my mail into one location. As you can see, I have outlook (pst) files, lotus notes (nsf) files and files from an old Unix (pine) account I had at school. The zip file (all 2.7GB of it) is the finished product – all my email from 1995 though 2005 (2006 and 2007 are in gmail already).

 
If you are looking at my files, you see a lot of overlap, that’s ok, better safe than sorry…we can delete the duplicates once they are in Thunderbird using an extension.

Step 2. Download the Software

Why Thunderbird – because it is open source with a large developer base. This means there is a good set of tutorials for writing extensions if I to want later on (for those visualizations). Also, I’m making a long term bet that 50 years from now, there will still be tools that import these files – (since they are just text files with no encoding and no database) – plus Thunderbird is really fast and can read 6gb of email quickly (lotus notes and outlook tend to slow down). So Thunderbird will be the final destination on my computer and I will upload from there.

Step 3. Install the software

  • Thunderbird – just default everything
  • Mercury – Make sure you set it up for both POP and IMAP when you get those options in the wizard
  • Lotus Notes – Default
  • Outlook – Default

Step 4. Import Unix files to Thunderbird

These are the easiest because Thunderbird will read these natively. All you have to do is copy the files in to the local folder and reopen Thunderbird.
A) Find out where thunderbird is storing your local files. To do this go to Tools > Account Settings. (note I renamed the local account to Ben Shoemate (all mail) but what your interested in is the "Local Directory" copy that and go there in windows explorer. This is where you need to copy the Unix files (they are files with no extensions). Just copy them to this "local folder", close and restart Thunderbird. Your old mail will be there! That’s it!

Step 5. Set up a local Email account in Mercury.

Open Mercury – if it is all ready running you will see down in the windows toolbar by the clock.

Otherwise , Start > Program Files > etc..

Once Mercury is open, go to Configuration > Manage Local Users > Add User
Because there is so much mail, I am going to set up a different local account per year. Gmail.com can import from 5 accounts at a time, and this will save a lot of time later on. So just make up an account name – I use my initials and the year and a simple password.

It should look like this when your done.

Each of these accounts are real email addresses. When you are local (in lotus notes and outlook) you can map to them with bs1999@localhost
In gmail, you can add them as bs1999@youripaddress (i.e. bs1999@111.111.122.11).


Step 6. Lotus Notes NSF files to your local email with IMAP

Once lotus notes is installed, you can simply double click the nsf files to open them, or you can click File > Database > Open > browse

You should see your old mail now. Ahh the memories….But don’t stop to read everything again or I’ll never finish this tutorial! Now, in lotus notes we are going to map to the local email account you set up in mercury. (Note: Gmail now has IMAP so you could just map directly to gmail if you want using the same procedure – but again, it will take a 100 years if you have this much email).

Go to File > Preferences > Client Reconfiguration Wizard

Check Pop or IMAP and click next.

Select IMAP, enter your new email account you set up in mercury, and type localhost, click next.

enter your account name again, and enter the password you gave yourself.
Click next, enter localhost as the server and your email address again – this doesn’t matter since you will not be using this email address for very long and it won’t show up on any of the mail that goes through it.
next, next, next your done.

To open the view of this in notes, click the "Databases" icon on the left panel and open your new account (this might take 2-3 minutes while notes creates a new nsf file etc.)

Now you should have you local account open. Now simply open the old nsf file, go to the "all documents" folder, select all, copy (I think it will only let you copy 2500 at a time) and then switch to you new email account and paste. Once you have all the messages for that year in the local IMAP account, close the nsf and repeat for all nsf files, and all years.

Step 7 – Outlook to your local email with IMAP


The same as in lotus notes – open your pst, then add the imap account and copy and paste.

  • To open the pst go to File > Manage Data files and click Add.
  • To open the local IMAP account go to Tools > Account Settings > New email account

Check the box that says "Manually configure server settings".

  • Select Internet Mail
  • Enter your Name, Email address (bs1999@localhost) and password, select IMAP, the server is localhost for both incoming and outgoing


Once you have mapped to this account, simply copy your old mail from the PST to the new account.

Step 8 – Clean it up in Thunderbird

Connect Thunderbird to you local email server (mercury) and download all your mail. Just like in outlook and lotus notes, we are going to add an account to Thunderbird and let it download all the mail you just put into it from lotus notes, pine and outlook.

  • Open Thunderbird
  • Click Tools > Account Settings > Add Account
  • Enter Account Name (for me its bs1999@localhost), Name, email address (bs1999@localhost)

  • Click Server settings

  • Click OK.
  • The next thing I did was organize copy them down into my local Thunderbird account and used an extension to delete duplicates
  • Then I spent some time looking for gaps…oh well

Step 9 – Upload to gmail


By now your local account has a lot of mail. Let’s start pulling it into gmail. I set up a new gmail account to test it first. From there I can pull it into my main account.

  • Log in to gmail
  • Click Settings > Accounts > Get Mail from other accounts (Pop3) > Add other accounts
  • Add your accounts from your local account. Note: you need you IP address. If you have a router you will need to route ports for pop3 (port 110) to your local computer. Log into your router to do this. While your there, get the IP of your router (ITS NOT the one that starts with 192. or 10. those are always local addresses)


(that’s NOT my IP address btw)

Next – gmail gives you some great options to tag the mail as it comes in, do it. Even if your not a tagger – do it.

You might also want to leave the message on the server (your computer) if this is a trial run.
Click Add account. If you get a problem, make sure your firewall is open, you port is mapped, and your password is right.

Next I’ll show you how to upload all those old chats…

If you liked this post, check out my other greatest hits:

Comments