Posts tagged “flickr”

Allspaw's playbook 2008/10/23

I just finished the proper book part (appendices tomorrow) of Allspaw's The Art of Capacity Planning. His playbook is now my playbook. I've got more than my fair share of charts and graphs (as all good engineers do) but am to some degree at a loss in using them. On one hand, the book presents nothing but common sense and on the other, it has saved me endless trial-and-error. I've been reading it on the morning bus and have put something into practice each day.

Most of his advice on growing a service applies equally to launching a service, albeit with a nasty chicken-and-egg problem that needs solving before your boss will let you buy hardware. By piping production traffic into my dev environment I've been able, in stages at least, to put every bit of my architecture under real production load (empirical, as per the Allspaw doctrine), piling on more and more until my work queues go hockey stick. (That'd be the bad kind of hockey stick.) The result speaks for itself.

Thank you, John Allspaw, for giving me a script to follow for the past week and a half.

Comments (0)

Dopploadr at Yahoo! Open Hack Day 2008/09/14

My hat's off to Yahoo! for putting on a great show this weekend. More than one person told me how much more excited they are about Yahoo! after experiencing Hack Day. It really does bring out the best in everyone. Plus, the Girl Talk set Friday night was killer.

MikeD and I showed up proudly wearing our OpenDNS t-shirts and were taken aback by the number of excited OpenDNS users we encountered — thanks, guys. Mike did the vast majority of the work on our main hack, a reworking of the OpenDNS Guide to use Yahoo!'s BOSS search API. We're excited about the prospects of integrating our user's preferences directly into search results. My part in all of this was to wrangle the markup and CSS. We really could roll this out next week and we wanted it to look that way. In the demo, Mike mentioned (casually) that we could throw a few million queries per day at BOSS. We aren't kidding.

Enough OpenDNS, tell me about the cheating

Before I sat down to crank out the eye-candy for the OpenDNS Guide, I built an extension for the Flickr Uploadr. But I wrote the Flickr Uploadr. This is totally cheating. I'd like to think that the Flickr judges felt the same way and that's why I didn't win any prizes but it doesn't matter because some people I really admire graciously complimented my work and I expect it will see quite a bit of day-to-day use. I hope it can also serve as a good starting point for future extensions. Without further ado:

Dopploadr

Dopploadr is an extension for the open-source, Mozilla-based Flickr Uploadr. As I said in the demo, it works just like a Firefox extension because it is. The Uploadr includes Mozilla's Extension Manager.

Once you install Dopploadr, Uploadr will take you to Dopplr to authenticate. My apologies for the copy & paste step but there's no equivalent of flickr.auth.getToken in the Dopplr API. One other catch that should be mentioned is that Dopplr's API must be accessed over SSL and (at Yahoo! and during my demo, at least) their servers or certificates frequently and randomly fail. Try again.

Once you're authenticated, there's really no UI to it. The status bar will indicate you're signed in, saying "Dopploadr for rcrowley" or some such but the real magic happens behind the scenes. Each time Uploadr thumbnails a photo, it hands off the date-taken field during the after_thumb event and Dopploadr decides where you were at that time, noting the latitude/longitude and some tags like:

When you upload a batch, Uploadr will again be really secretive but Just Work. The before_one_upload event will combine the user's tags with the geo tags above and then let Uploadr continue with the upload. When Uploadr finishes, it hands off the photo's ID to Dopploadr during the after_one_upload event so Dopploadr can call flickr.photos.geo.setLocation with the stored latitude/longitude.

My official monkeypatch count sits at 3. First was adding the after_logout event that I still haven't actually used (see also version 0.2). Second was a short walk around the DOM to insert Dopploadr's string bundle (strings accessible from JavaScript). Third was adding a counter for outstanding Dopplr API calls and making the buttons.upload.enable() method sensitive to the count to prevent uploading before everything's been geotagged.

Here's the photo I uploaded during my demo: http://flickr.com/photos/rcrowley/2853699765/. (See, it really works!)

You can download Dopploadr 0.1 or check out the code on GitHub.

My sincerest thanks once more to the folks all over Yahoo! that made Hack Day happen.

Comments (9)

Uploadr extension extensions 2008/09/07

Getting re-acquainted with Flickr Uploadr, I discovered a sorely missing piece of the Extension API. There are plenty of hooks to catch higher-level events than clicks and mouseovers — things like logins, uploads and adding photos. However, there is no way to store persistent data and associate it with a certain Flickr user. Here is a way:

var userinfo = {
	set: function(k, v) {

		// Only allow userinfo if they're signed into Flickr
		if (!users.nsid) { return false; }

		// It's just a hash for storing whatever you want
		var u = users.list[users.nsid];
		u.userinfo = u.userinfo || {};
		u.userinfo[k] = v;

		return true;
	},
	get: function(k) {
		if (!users.nsid) { return undefined; }
		if (!users.list[users.nsid].userinfo) { return undefined; }
		return users.list[users.nsid].userinfo[k];
	},
	unset: function(k) {
		if (!users.nsid) { return true; }
		if (!users.list[users.nsid].userinfo) { return true; }
		delete users.list[users.nsid].userinfo[k];
		return true;
	}
};

Another thing I found missing was an after_logout event. So I monkeypatched one:

users.old_logout = users.logout;
users.logout = function(save) {
	users.old_logout(save);
	extension.after_logout.exec();
};
extension.after_logout = new extension.Handler();

extension.after_logout.add(function() {
	Cc['@mozilla.org/consoleservice;1'].getService(Ci.nsIConsoleService)
		.logStringMessage('after_logout!');
});

T-minus five days and I've got my development environment ready to go. I'll be at Yahoo! Open Hack Day September 12-13 in Sunnyvale, hacking on Uploadr. Hopefully others will join me in creating some cool extensions. Stop by and say hello.

Comments (0)

Starting at OpenDNS today 2008/04/08

There are so many things to love about Flickr and especially about working at Flickr that I initially thought myself crazy for even considering leaving. But with time came sanity and I saw the pros for leaving start to outweigh the cons.

The folks at Flickr are without a doubt the most talented group I've ever had the honor of working with and I will miss them all. Flickr as a whole is a dream, a place with passionate staff matching passionate members and no shortage of new ground to break into, but my piece of this puzzle was not exactly right for me. I developed the new Flickr Uploadr desktop client as an open-source, extendable app for getting your photos and metadata onto Flickr. The project had everything I wanted — new technology, technical difficulty, performance concerns, design quirks and the goal of open-source extendability. But for all of that it had a fatal flaw — I am not a desktop software developer.

Thinking more forwardly, "I am not a desktop software developer" started to say what I really meant: "I do not want to be a desktop software developer." Selfish personal preference.

This job at OpenDNS has all the ingredients of a good time in my book — C++, maybe some map/reduce, I/O bottlenecks, oh-my-god-databases-are-slow and a whole lot of networking (the TCP/IP kind). The people there are awesome just like Flickr's and I have no doubt I'll become just as attached to them.

(Housekeeping: it looks like the XULRunner saga on rcrowley.org has come to an end for the time being. I almost guarantee I'll be back to dabbling with it sometime, though, since it'd be a shame for all that practice to go to waste. Don't count on Windows XPCOM, though.)

Comments (16)

Stop doing it yourself and use CPAN — curvr-0.2 2008/02/05

Just like the career of the average software engineer, Curvr has now grown up just a bit and started reusing other people's code.

Dopplr integration in curvrmail

Geotagging is awesome but a bit of a pain to automate with any kind of accuracy. I'm making a compromise here by automatically tagging things using my location according to Dopplr. I had this working with my original PHP curvrmail script but have since switched to using Flickr::Upload::Dopplr just to practice some Perl.

AutoManual image rotation

Since my N73 isn't friendly enough to rotate pictures for me, I've added a way to specify rotation in the email you send to Curvr. Prefixing with "L " or "l " will rotate 90 degrees counterclockwise and prefixing with "R " or "r " will rotate 90 degrees clockwise. Note in both cases that it is a letter followed by a space followed by the normal title/tags syntax.

It is a bit awkward to think about in terms of left and right, since I could either be talking about the direction you rotated the camera or the direction the photo should be rotated (which itself could be rather ambiguous, so we'll say that it is relative to the top). So here's the official word: the rotate syntax wants to know which direction you rotated the camera to take the picture.

curvrconf for getting API tokens

Getting tokens for the Flickr and Dopplr API, while not something you need to do often, is quite a drag. Enter curvrconf, the copy-and-paste simple way to get your tokens out. All it needs is to be setup with the same API key as curvrmail and it'll guide you through grabbing tokens. For the Flickr API, setup your key for web-based auth and give it a garbage URL to return to (I use http://localhost:81/).

As usual, the API key is up to you. I accidentally committed my keys/secrets/tokens to Subversion but have expired all of them so don't even bother trying to steal my accounts.

Anti-blue C code

The main curvr program has for a long time introduced some strange blue artifacts into my photos. I figured this was due to some underflow or overflow issues with my color maps. I finally took the time to do some serious experimentation and it turns out that if I just favor a stronger red channel in these edge cases, the artifacts seem to go away. Strange and beautiful indeed. I can't fully explain the phenomenon but believe it's somehow related to my using the red channel as my indicator of color value (as many photographers do when converting to black & white).

So there it is, curvr-0.2.

Er, here it is: http://svn.rcrowley.org/svn/curvr/tags/curvr-0.2

Comments (0)

Flickr Uploadr 3.0 lives 2007/12/20

Time to take a deep breath. Flickr Uploadr 3.0 is out for all and two minor revisions later is looking pretty good. There are still issues to be resolved but I won't spend any time here whining about desktop software or how hard it is.

Richard's laptop haulInstead, a couple of lessons I definitely learned the hard way from this release.

The next challenge will be turning the GPL'ed Uploadr into something moldable and bendable for other developers to play with. In the meantime, check out the new Flickr Uploadr and its source code at http://flickr.com/tools/uploadr/.

Photo from Cal.

Comments (4)

Launching a localized XULRunner app 2007/12/13

The average XULRunner developer gets to choose at the beginning of any project whether to work with the 1.8 or 1.9 series. 1.8 gets you stability and heartache; 1.9 gets you trunk-y goodness and a sweet thread library. I'm of course simplifying things.

Until yesterday the prospect of localizing XULRunner itself had not even crossed my mind. An error to be sure but one that was actually easily corrected. We at Flickr chose to work with the 1.9 branch to take advantage of the newly enjoyable thread library. Working with these nightly builds was great as I was able to watch bugs die each time I pulled a new nightly.

The trouble began when I started putting together final builds yesterday. Things like the Mozilla upgrade system, file picker dialog and standard OK/Cancel buttons were not localized because nightly builds come only in English. Bummer. LXR came to my rescue, though. I was able to pull together my own localizations of XULRunner by scraping pieces from the Firefox 2 codebase.

I am not sure what the lesson here is but with ample planning and testing, apps built on XULRunner trunk can be localized just like any other. It would be awesome if, in the future, trunk builds included the entire localization tree, which would allow my build process to just pick out the appropriate JAR file for each language (which is essentially what I do now). Look to the Flickr Uploadr source (more on that soon) for an example of how all this works.

Comments (0)

Where did all my XULRunner tips go? 2007/10/20

Despite my blog being stale like the bread in my cabinet, I've still been elbow deep in XULRunner all day every day. Now 3 1/2 months in, the ugly stuff has taken shape and it's time to lock down the user interface. I still haven't found an elegant way to keep clicks on the scrollbar from firing my sequence of mousedown, mousemove and mouseup events, but gimme a second chance.

As expected, I'm learning a lot about releasing software in the real world, where it has to stand up to real revisions, new features and public scrutiny. I like to think I've always built software that's up to the task but a UI worthy of the Flickr name is a lot of work. And... Surprise! My engineer's touch is not good enough. Gino and George, with a healthy bit of Stewart, have made it look awesome. And my role in the design-y stuff? Implementing some of the ideas so we can play with them before springing them on unsuspecting users.

But stay tuned. It turns out that work not only fluctuates between C++ and JavaScript but between engineering (then) and design (now). After round one is out in the wild it will be time for more hacking in the great undocumented. Lather, rinse, repeat.

The XPCOM tips will be back soon!

Comments (0)

JSONRequest amendment 2007/09/23

JSONRequest was born from Crockford's head more than a year ago, but only this month has it really gotten interesting. Collin Jackson has released a Firefox extension implementing the JSONRequest spec. I was naturally compelled to play with it.

JSONRequest is a proposed new standard that could replace XMLHttpRequest and shed the crippling "same domain policy" that makes it a pain to consume outside APIs in Javascript. It's secure enough for cross-site-scripting for a couple of reasons. I'm lazy, so here are Crockford's words:

  1. JSONRequest does not send or receive cookies or passwords in HTTP headers. This avoids false authorization situations. Knowing the name of a site does not grant the ability to use its browser credentials.
  2. JSONRequest works only with JSON text. The JSONRequest cannot be used to access legacy data or documents or scripts. This avoids attacks on internal websites which assume that access is sufficient authorization. A request will fail if the response is not perfectly UTF-8 encoded. Suboptimal aliases and surrogates will fail. A request will fail if the response is not strictly in JSON format. A request will fail if the server does not respond to POST with a JSON payload.
  3. Reponses will be rejected unless they contain a JSONRequest content type. This makes it impossible to use JSONRequest to obtain data from insecure legacy servers.
  4. JSONRequest reveals very little error information. In some cases, the goal of a miscreant is to access the information that can be obtained from an error message. JSONRequest does not return this information to the requesting script. It may provide the information to the user through a log or other mechanism, but not in a form that the script can ordinarily access.
  5. JSONRequest accumulates random delays before acting on new requests when previous requests have failed. This is to frustrate timing analysis attacks and denial of service attacks.
From this list, #1 and #3 make the most difference. Without sending cookies, it will be very difficult to send requests impersonating a logged-in user. By requiring the application/jsonrequest content type on the response, the standard requires that the API provider take action to allow JSONRequests. This leaves no excuse for lazy API providers who don't make their API secure.

As for other security holes, I've been poking around a bit against the Flickr API. I had to steal an API key, the shared secret for that key and a user's token that came from that API key before I could do anything nasty. Now, since getting these three pieces of information gives you free reign even without JSONRequest, I don't see much of a threat here.

As a bit of an aside, OAuth seems to work very much like the Flickr API and so JSONRequest will be equally innocuous for OAuth users.

OK, finally the amendment part

Requiring that all POST requests be sent as well-formed JSON data severely limits JSONRequest's ability to take web apps to the next level. The inability to send traditional POST payloads isn't such a big deal, but the inability to send multipart POSTs for file uploads is tragic.

Of course, it isn't as easy as just lifting the JSON-only restriction on POST payloads. As Mike pointed out to me last night, old sites using GET variables to handle authentication (PHPSESSID anyone?) are pretty much sitting ducks. But there's always another way.

Allowing traditional or multipart POSTs without some prior setup can be hazardous. Although these will always return an error to the caller, on the server side they may be successful which could really muck up your databases. The solution is simply to require a successful JSON POST immediately prior to accepting the multipart request as a way for the server to say "yes, I want that multipart POST." In pseudocode, something like this: 

JSONRequest.post_multipart = function(url, send, done, timeout) {
  JSONRequest.post(url, {'multipart': 1}, function(requestNumber, value, exception) {
    if (value) {
      // Here we would take params passed to post_multipart and actually
      // send the multipart request.  The result of that would be passed
      // to the function the called passed in the done param.  The
      // intermediate JSON POST is essentially invisible.
    } else {
      console.log(exception.message);
    }
  }
};
With this we will finally have hack-free file uploads in addition to safe cross site scripting. Even Microsoft can't say no to implementing that.

Comments (0)

Help me decide what C code to write 2007/09/16

The majority of my (awesomest ever) day job at Flickr is written in Javascript. I do write some C++ code for some of the trickier parts, but wc tells me more that only about a tenth of my code is C++.

That tenth doesn't really do it for me, though. It isn't dirty and dangerous. Because of the Netscape Portable Runtime, it's actually rather tame. See, I spent college SEGFAULTing at least once a day. In my nearly three months at Flickr, I've managed to do so only twice.

Fear not -- I have a way to get my fix. I'm intrigued by FUSE and started to think about the design of a Flickr filesystem. About 14 seconds after that I discovered Manish Rai Jain's flickrfs project. It is more complicated than I would have designed, but because of that it's incredibly powerful. Flickr isn't out as a possible project for a FUSE filesystem but I'd rather not step on his toes. Onward.

The problem is, I'm short on imagination right now. I think the most powerful aspect of making non-filesystem data available as a filesystem is the huge set of possibilities that open up when you start chaining standard UNIX commands.

Do any C-shy webdev types want anything else out of their filesystem?

Comments (2)

It's been two weeks? 2007/07/05

Time flies at Flickr.  I am sure this is not due to my frequent absence for things like getting parking passes, meeting the Comcast guy, and meeting the (very late) movers.  It's because the place is just awesome.  Never have I encountered a smarter group of people -- it is downright intimidating at times.  But intimidation never lasts because of how welcoming they have been and how frequently "real work" is interrupted by a finger-rocket war.

Though I just got my MacBook Pro today and still don't have access to my work email, I've been quite busy, learning the ins and outs of XULRunner and Apollo.  Apollo is actually made up entirely of new and exciting contradictions that prevent any real web-like development of desktop apps.  XULRunner, not to be outdone, provides the developer an exercise in divining best-practices from partial examples, partial documentation, and partial Firefox builds.  (Tomorrow will be fun.)

Back when Apollo still seemed promising (yesterday) I was going to write an entire entry about how to work around a major shortcoming in the platform, but since it seems likely I won't be doing much Apollo work, I thought I'd share this little tidbit here.  You cannot embed a Flash movie in an HTML Apollo app as you would in a normal webpage, meaning that hybrid HTML/Flash apps aren't possible.  The workaround, if you really need Flash, is to build the entire app using Flex.  I know, it'll feel like developing Javascript in 1998 when there weren't good libraries like jQuery, what with all the click="foo();" attributes scattered about.  It's OK, though, you'll make it just like I did.  More to come on XULRunner as I learn.

Comments (1)

Other tags: