This post is about two things. Most proudly it is about me finally taking the time to setup all of the necessary gears and levers for my phone to post processed photos through to Flickr. Bonus-ly, it is about rolling what seems like your own email server from a few config files and Yahoo! Mail (or GMail, if your twisted brain is so inclined).

First, the fun stuff. I posted in November about curvr, my automated and assumption-filled command-line Photoshop. Refresher course here: http://rcrowley.org/2007/11/08/introducing-curvr/. The day after I wrote curvr I started to experience what can only be described as “crunch time” on Flickr Uploadr, so a quick shell script called curvall was born so I could at least use curvr after Bluetoothing photos from my phone.

Almost two months later and this is just silly. A different kind of crunch time, if you will. From a distance the solution isn’t nearly as ugly as the syntax of .procmailrc but I won’t be mean. I setup an MX record for my domain to send mail to the box in my apartment. From there, postfix rejects the riff-raff and passes the good stuff on to procmail which either works photos over with curvr or forwards to Yahoo! Mail for my consumption.

Postfix, there’s more to life than UNIX accounts

We need to teach postfix what email addresses to accept.

/etc/postfix/virtual:

r@rcrowley.org rcrowley@localhost
SPECIAL_CURVRMAIL_ADDRESS@rcrowley.org rcrowley@localhost

/etc/postfix/main.cf (available in SVN):

# See /usr/share/postfix/main.cf.dist for a commented, more complete version

# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${queue_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${queue_directory}/smtp_scache

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

myhostname = banzai
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = banzai, localhost.localdomain, localhost
relayhost =
mynetworks = 127.0.0.0/8
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all

# Nasty homegrown config
virtual_alias_domains = rcrowley.org
virtual_alias_maps = hash:/etc/postfix/virtual
mailbox_command = /usr/bin/procmail

It really is important not to put the same domain in mydestination and virtual_alias_domains. Fecal matter and oscillating device and all that. Then run these commands:

$ sudo postmap /etc/postfix/virtual
$ sudo postfix reload

Procmail, your syntactic colon is full of shit

If photo, Flickr it. If not, throw it out into the cold.

~/.procmailrc (also available in SVN):

:0 bfW
* ^TO_<SECRET_CURVRMAIL_ADDRESS>@rcrowley.org
| /home/rcrowley/bin/curvrmail

:0 E :
! <normal_forwarding_destination>@yahoo.com

Now, Stephen R. van den Berg willing, email to your special address will be sent through curvr and onto Flickr (assuming you updated the curvrmail script with your secret Flickr email address) and all of your normal email will be forwarded somewhere else.

Yahoo-oooo!

I admit it, I like Yahoo! Mail. So I have my .procmailrc forwarding to Yahoo! Mail. Over at Yahoo! Mail I have a default mailbox setup as the-first-letter-of-my-first-name@rcrowley.org so that I can also send mail as my better email address.

Anyway, it’s all in SVN: http://svn.rcrowley.org/svn/curvr/

I’m interested to know how similar this all is to Aaron’s filtr workflow.

Before I forget, some rather scattered mental notes from last week in Ireland.

1. Airbus’ 330 is the nicest plane I’ve ever been on.
2. The Irish are much less savagely capitalistic than Americans. Case in point, everything was closed on December 26th for St. Steven’s Day. Me, I’ve never heard of it.
3. The food is amazing. Stews, gravies, lamb and all came with the typical “Irish flag” plate of carrots, multiple varieties of potatoes and green beans or broccoli.
4. Ford Motor Company, for all its shortcomings, is doing surprisingly well in Ireland. Nearly every delivery van, all of the police cars (Garda!) and a healthy portion of the citizen cars were Fords.
5. Continuing the automotive train of thought, Ford, Mazda, Toyota and Honda all had almost completely different product lines than in the US. I can understand skipping out on the big trucks but they all make some reasonable cars that I can’t understand reinventing for Ireland/UK drivers.
6. The roads are laid out terribly. They use a white line for both the center divider and lane dividers. Sometimes they’re all a single dotted line, meaning you need God, the Pope, Jesus, your navigator and your mother all working together to stay in the right place.
7. The speed limit on barely-wide-enough-to-fit-2-cars roads is 100 km/hr. WTF?
8. Irish cigarettes bear warnings that do not mess around. The best one: “Smoking kills.” I know, pics-or-it-didn’t-happen — coming soon to a Flickr near you.
9. The Irish accent is substantially different than the English accent. (Of course I can’t provide an example in text.)
10. Roundabouts are far more efficient than stoplights.
11. It is odd and comforting at the same time to see whole families together in the pub.
12. You can always find live music in Temple Bar, Dublin.

Time to take a deep breath. Flickr Uploadr 3.0 is out for all and two minor revisions later is looking pretty good. There are still issues to be resolved but I won’t spend any time here whining about desktop software or how hard it is.

Instead, a couple of lessons I definitely learned the hard way from this release.

• Desktop software, despite my best efforts, just can’t be deployed as rapidly as a website can. I’m still trying to find the sweet spot between pushing fixes as soon as they’re fixes and not bothering users with a constant stream of incremental updates.
• Some bugs are not as bad as they seem. And specifically not as bad as the one you might introduce hurrying out a fix. The transparency the Flickr Forum gives us in diagnosing bugs and communicating workarounds is invaluable here. Keeping members in the loop is just about as important as fixing the bugs.
• Other people’s computers are more different than your’s than you can possibly imagine (or replicate with 6 computers). Statically link like your life depends on it! (But that’s only half the battle.) Diagnosing problems when you can’t just fire up cmd.exe or Terminal is a daunting task.

The next challenge will be turning the GPL’ed Uploadr into something moldable and bendable for other developers to play with. In the meantime, check out the new Flickr Uploadr and its source code at http://flickr.com/tools/uploadr/.

Photo from Cal.

The average XULRunner developer gets to choose at the beginning of any project whether to work with the 1.8 or 1.9 series. 1.8 gets you stability and heartache; 1.9 gets you trunk-y goodness and a sweet thread library. I’m of course simplifying things.

Until yesterday the prospect of localizing XULRunner itself had not even crossed my mind. An error to be sure but one that was actually easily corrected. We at Flickr chose to work with the 1.9 branch to take advantage of the newly enjoyable thread library. Working with these nightly builds was great as I was able to watch bugs die each time I pulled a new nightly.

The trouble began when I started putting together final builds yesterday. Things like the Mozilla upgrade system, file picker dialog and standard OK/Cancel buttons were not localized because nightly builds come only in English. Bummer. LXR came to my rescue, though. I was able to pull together my own localizations of XULRunner by scraping pieces from the Firefox 2 codebase.

I am not sure what the lesson here is but with ample planning and testing, apps built on XULRunner trunk can be localized just like any other. It would be awesome if, in the future, trunk builds included the entire localization tree, which would allow my build process to just pick out the appropriate JAR file for each language (which is essentially what I do now). Look to the Flickr Uploadr source (more on that soon) for an example of how all this works.

Why can’t XULRunner do this already?

Without really saying what “this” is, Mozilla wants XULRunner to be the standard play for cross-platform GUI applications and yet it has a bit of trouble interacting with 3rd party libraries. Most mature libraries (say, GraphicsMagick) still use char * or std::string to represent file paths. To interact with them, then, you have to be able to get to a char * representation of files you’re trying to open.

Windows and Mac character encodings 101

As is becoming tradition, Windows makes my life difficult by using UTF-16 as its Unicode of choice. UTF-16 uses 2 bytes (usually) to represent each character, meaning it isn’t binary compatible with ASCII, which uses 1 byte per character. These 2-byte characters are stored in wchar_ts which are twice as wide as chars. It isn’t possible to pass a wchar_t * through normal C libraries that expect strings as char *.

Mac OS X uses UTF-8 internally and is my hero. UTF-8 is used internally to represent strings and has the advantage of using single bytes (that’s char for C programmers playing along) as its base unit. While a single character in UTF-8 can span up to 8 bytes, the in-memory representation of these characters will pass muster with a C compiler. When the UTF-8 bytes are passed through some C library they can arrive intact to the fopen system call, which understands the UTF-8 bytes and acts like you’d expect.

There’s an easy way?

There could be. Using the @mozilla.org/file/local;1 it might be possible to get the path as an nsACString, Mozilla’s portable representation of single-byte ASCII strings. But, the reference warns that the native path available from nsILocalFile is not for passing to C libraries and isn’t guaranteed to be correct. So it’s up to me.

No pain, no gain

Bear with me, this gets ugly. The solution that lets me take paths from JavaScript-land to C-land, open Unicode paths with a normal char * and do so on Windows and Mac OS X is stupefying (and stupid). The iinitial path is stored as a JavaScript string, so the boundary between JavaScript and C will be an nsAString, which represents strings as UTF-16.

The string takes quite a journey here, so let’s cover the easy side first. Macs understand UTF-8, and I’m making a transformation to UTF-8 in JavaScript. This means that the only transform needed in C is a type change, not an encoding change. The UTF-8 string enters C-land as wide characters and is simply cast character-by-character down to an array of bytes.

Windows is quite a bit more work, since a path in Windows that needs Unicode characters can’t easily be represented as a char *. Enter the Windows API function GetShortPathName. This can convert any path to an old-school Windows path that fits in ASCII and uses that familiar “8.3″ naming convention. I got the Mac half working first, so the Windows implementation is actually more complicated than it needs to be — more on that later. Without further ado:

string * conv_path(const nsAString & fake) {

	// Fun with Windows paths
#ifdef XP_WIN

	// UTF-16 but really UTF-8 nsAString to really UTF-8 nsCString
	nsCString utf8 = NS_LossyConvertUTF16toASCII(fake);

	// UTF-8 nsCString to UTF-16 nsEmbedString
	nsEmbedString & utf16 = NS_ConvertUTF8toUTF16(utf8);

	// UTF-16 nsEmbedString to wchar_t[]
	wchar_t * w_arr = new wchar_t[utf16.Length() + 1];
	if (0 == w_arr) return 0;
	wchar_t * w_arr_p = w_arr;
	PRUnichar * w_start = (PRUnichar *)utf16.BeginReading();
	const PRUnichar * w_end = (PRUnichar *)utf16.EndReading();
	while (w_start != w_end) {
		*w_arr_p++ = (wchar_t)*w_start++;
	}
	*w_arr_p = 0;

	// GetShortPathName to get guaranteed ASCII
	wchar_t s_arr[4096];
	if (0 == GetShortPathNameW(w_arr, s_arr, 4096)) {
		delete [] w_arr;
		return 0;
	}
	delete [] w_arr;

	// wchar_t[] to ASCII nsEmbedString
	nsEmbedString ascii;
	wchar_t * s_arr_p = s_arr;
	while (*s_arr_p) {
		ascii.Append((char)*s_arr_p++);
	}

	// Macs don’t need any help since they understand UTF-8
#else
	nsEmbedString ascii;
	ascii.Assign(fake);
#endif

	// Convert the nsEmbedString into a std::string
	char * c_arr = c_arr = new char[ascii.Length() + 1];
	if (0 == c_arr) return 0;
	char * c_arr_p = c_arr;
	PRUnichar * c_start = (PRUnichar *)ascii.BeginReading();
	const PRUnichar * c_end = (PRUnichar *)ascii.EndReading();
	while (c_start != c_end) {
		*c_arr_p++ = (char)*c_start++;
	}
	*c_arr_p = 0;
	string * str = new string(c_arr);
	delete [] c_arr;
	return str;

}

This works simply because, on either platform, the result is a char * that actually uniquely represents the desired file.

My code sucks, let me count the ways

There are of course things that could be better. First of all, after reading the code again (I wrote it Thursday and haven’t looked back), I see that I should refactor a bit. By leaving the string in UTF-16 representation for passing to C-land, I can add a transform from UTF-16 to UTF-8 to the Mac version and greatly simplify the Windows version. I’ll put that on my to-do list. The other place I hope to improve is in the event that GetShortPathName fails. In this case, copying the file under a new ASCII filename to a temporary directory (found using GetTempPath) would still let the file be accessed with a char * path.

Why did I have to do this?

It would be sweeeeeet if Mozilla integrated this code (after a healthy dose of optimization) into the toolkit as something like NS_ConvertToNativePath(nsAString) or some such function. Having this will make integrating with third-party libraries much easier.