Apr 13 2007

Flickr Flooding

Published by at 1:20 am under Tech

Short version – Flickr has removed their 2G/month upload restriction.  You can use an extension called FireUploader to upload thousands of files to Flickr.

Long version –

I like to take pictures.  Lots of pictures.  My favorite feature of the Canon Rebel XT is the “b’juuu b’juuu” setting.  That’s the one that goes “b’juuu b’juuu” when you hold down the button, taking up to 9 pictures at once.  Sweet.

Last year, I took about 5500 pictures, all of them 8 megapixel.  That’s about 16G of pixel abuse.  Add to that the 4G of stuff I had from older cameras, and we’re up past 20G.

Last year, I also bought a Flickr Pro account.  Unlimited storage!  But there was a catch…  A 2G upload limit per month.  So, to get all my stuff backed up on Flickr, I need to split all my pictures into 10 groups of 2G each, and commit to keeping track of what I’ve done each month.  Bleh.

So, I avoided the problem, took more pictures, and felt bad every time a month went by and I’d wasted my 2G quota.  Something had to be done.

I knew Flickr had an API, and I knew that I could break the problem down to a simple queue processing problem.  All I needed to do was relearn how to program, something that would probably take another 10 months. 

So I did what I usually do in this situation.  I Google until I find someone else with my same problem.  And just like browsing the shelves of a library, I found something even better!  Flickr had lifted the 2G/month limit!  W00t!

Well, maybe no so W00t…  After many experiments with diminishing sized batches of photos, I learned that the Flickr Uploadr is not a solid piece of software, at least for more than 100 pictures at once.  Even then, it would fail once in a while, leaving a mess of pictures that never made it into a set.  With 100 pictures at once, I’d have to do this more than 50 times, checking each upload a picture at a time.  Very tedious. 

If only there were an upload tool that could hook to my Flickr account, let me point at thousands of local files, and upload them one at a time unattended – kind of like a smart FTP client.

Another methodology I use to solve these kinds of problems is to whine to all of my friends until they get so sick of hearing about it that they think of something.  My buddy Tad mentioned that he was able to use Amazon’s S3 file storage service to save significant costs on some popular files on his company’s web site.  I asked, “How did you get the files onto S3?”  “I found a Firefox extension that just let me drag and drop.”  Neato, I thought.

Although I didn’t apply Tad’s situation to my Flickr problem, another buddy, Robert, found the S3 extension and IM’d me that the same author had written a Firefox transfer extension for Flickr!

The extension is called FireUploader and it solved all my problems!  It is written by rjonna@rjonna.com.  All I had to do was install the extension, dump all of my thousands of photos onto a machine, fire up the tool in Firefox, select all of my photos, select the set I wanted them to live in, then walk away.  Three days later, and I’ve got gigs and gigs of photos all backed up.

There’s a nice Progress Bar area so I can find any files that failed and reload them.  Actually, once I diagnosed a latency issue with my external drive, and subsequently moved all the files to the local drive, I had no errors with thousands of files.

I love the intarweb!

10 responses so far

10 Responses to “Flickr Flooding”

  1. Jimon 13 Apr 2007 at 12:48 pm

    I’m starting to wonder if I’m one of the tinfoil hat wearing, Luddite crowd. Am I the only one who feels uncomfortable entrusting my content to a corporate enterprise with no contract delineating my rights and setting expectations? I’d love to dump my photos to someone else’s fat-piped server, but I can’t bring myself to do it, but I just can’t bring myself to push the button!

  2. stlsnydeon 19 Sep 2007 at 8:30 pm

    Help! I downloaded FireUploader and successfully uploaded 60 photos (approximately) and then all the pics after number 60 failed to upload. The error message I was receiving said I had “exceeded my user limit”. Should I uninstall and reinstall?
    Many thanks for any advice you can provide!

  3. declanon 19 Sep 2007 at 9:54 pm

    Hi – could it be that your Flickr is full? If you don’t have the Pro version, there are upload limits, and total picture limits.

  4. Anandon 13 Jun 2008 at 8:37 am

    Hi

    you may have solved my biggest headache!! i have 47000 pics that i want to upload for ‘safe keeping’. this software may just be the thing for me. i have a few questions:

    Have you found it to be stable?
    How long does it take you to upload – say 100 pics?
    Do you have to then do sets etc once the pics are in flickr? that could be time consuming

  5. declanon 13 Jun 2008 at 9:15 am

    Hi Anand! So glad to help.

    It is very stable and I get updates once in a while.

    The length of time to update is totally dependent on your network connection. When I was uploading my 30k pictures, I brought them to work to use their fast connection 🙂

    I was able to do sets as I went, but after a while I just lumped things into sets by year and let it rip.

    The developer is very communicative and would love to hear any suggestions.

    D

  6. Darrenon 13 Apr 2009 at 1:58 am

    Hi. I am so glad I found your blog/web site. I can’t wait to get home and try uploading my 50gb of photos.

    Can I just ask. If you have been storing all your photos into lots of nice neat folders, aptly named and dated etc will any of this information be conferred over to flickr? I suspect that it will just be dumped into the photostream which will then need to be organised into sets and batches.

    I have to say, I started using flickr as a backup process. I have always found the batches and sets tedious to to use, whehter it was my slow connection or I just didn’t “get it”. My main problem was sharing the photos with friends. I can’t remember the number of times people would reply to the invite and say they couldn’t open any of the photos, even after I had made them public, which I never intended to do in the first place but felt forced to do so.

    The other point ad nauseum as you say was that the flickr uploader is just not stable. For months I uploaded my sisters wedding 700 wedding pictures. every time i got home from work, I would start again. like banging my head on the wall for months.

    thank you for sharing

    Darren

  7. declanon 26 Apr 2009 at 5:38 pm

    Hi Darren – sorry to take so long to reply! I’m a bad blogger lately. Twitter is ruining me for longer form writing 🙂

    When I did the big dumps to Flickr, I did them in batches based on the organization I had on my local setup. I could create new sets on the fly this way, and add them to the queue.

    Have you started? How is it going?

    I have to say, I’ve recently backed off of using Flickr as a backup solution. I do love that it’s unlimited storage for $29/month, but as I’ve become a more experienced photographer, I’ve switched to shooting almost exclusively in RAW. Flickr just takes jpgs, which are a derivative of my RAWs, so it really isn’t a proper archival of my original work. So, I’ve recently modified my workflow to rating my shoots from 1-5 and only uploading the pictures that I deem 3 or better (how/why I decide on a rating is a whole ‘nuther blog post 🙂 ).

    I’m agonizing over a proper way to backup my 300G, and growing, base of old jpgs and new RAWs… Backblaze is often mentioned, but it has a 5G file limit which doesn’t help one of my other projects – the 40G+ files produced by digitizing my old Hi8 and DV8 tapes. Mozy also looks interesting. I’ll learn more about these over time. Hopefully before the house burns down. 😉

    D

  8. Darrenon 27 Apr 2009 at 9:53 am

    Hey Declan,

    I did start, and I have to say, I think it’s brilliant. My only small gripe was that folders with subfolders are not recognized. And as a minor OCD trait I had a lot of my photo album folders organized and then further subdivided. not a huge loss, just a bit tedious to get the first big dump into flickr going.

    I think someone closed firefox by accident the other day, so i have to find out exactly what I have dumped into flickr and what is still to upload. A shame that it can not be a stand alone programme which also remembers where the queue is, because that queue got deleted when firefox shut down.

    I was wondering if after I have uploaded all my photos and I discontinue pro service, will i still be able to access all my pictures but just with upload restrictions applied. If that is the case then I may not spend the $29/month but rather a la carte when I need to do a big upload.

    I found your blog very useful

    thanks

  9. declanon 04 May 2009 at 7:28 am

    Hi Darren – glad it’s mostly working out!

    If you go back to non-pro on Flickr, you’ll only be able to see your last 200 photos. All of your photos will continue to exist, but just the last 200 will show up. Flickr is just $29/year, not month, so it’s not that bad, I think 😉

    D

  10. Steveon 19 Jan 2011 at 4:05 pm

    Big thanks for this recommendation – the flaky flickr uploadr was really beginning to drive me nuts.

Trackback URI | Comments RSS

Leave a Reply