Manifest (verb): To embed XML describing libraries which are expected to be available at runtime, done in the course of developing software using Microsoft Visual Studio.

The wisdom of the collective T-Rex (#xulrunner on irc.mozilla.org) says that statically linking against the Microsoft C Runtime (msvcr80.dll and friends) is the way to go. However I’ve run into a case where I cannot do this (deciding whether this is due to technical limitations or lack of competence is for readers to help me decide).

The image processing component of the Flickr Uploadr links against GraphicsMagick and Exiv2, libraries that do image processing and EXIF/IPTC metadata processing, respectively. Without them, it doesn’t fly. Problem is, they both link the C Runtime themselves by using std::string and such. Both of these libraries will build fine given the /MT flag, which instructs them to statically link the C Runtime. When they themselves are statically linked into the XPCOM component later, all hell breaks loose. It seems that the static linker can’t resolve the duplicate definitions caused by overzealous static linking.

The solution, as it stands, is simply to include the C Runtime and its manifest in the distribution. Placing the following files in Uploadr’s components/ directory has allowed life to go on for the moment.

• msvcr80.dll
• msvcp80.dll
• msvcm80.dll
• Microsoft.VC80.CRT.manifest

Lastly, make sure your XPCOM project in Visual Studio is set to embed its manifest from Project Properties » Manifest Tool » Input and Output » Embed Manifest.

A more optimal solution might be to combine the Visual Studio projects for GraphicsMagick (and therefore libjpeg, etc), Exiv2 and the XPCOM component into one large tree that Visual Studio might be able to better manage. Has anyone out there done such a thing with Microsoft’s static linker (even if not with XULRunner)?

The average XULRunner developer gets to choose at the beginning of any project whether to work with the 1.8 or 1.9 series. 1.8 gets you stability and heartache; 1.9 gets you trunk-y goodness and a sweet thread library. I’m of course simplifying things.

Until yesterday the prospect of localizing XULRunner itself had not even crossed my mind. An error to be sure but one that was actually easily corrected. We at Flickr chose to work with the 1.9 branch to take advantage of the newly enjoyable thread library. Working with these nightly builds was great as I was able to watch bugs die each time I pulled a new nightly.

The trouble began when I started putting together final builds yesterday. Things like the Mozilla upgrade system, file picker dialog and standard OK/Cancel buttons were not localized because nightly builds come only in English. Bummer. LXR came to my rescue, though. I was able to pull together my own localizations of XULRunner by scraping pieces from the Firefox 2 codebase.

I am not sure what the lesson here is but with ample planning and testing, apps built on XULRunner trunk can be localized just like any other. It would be awesome if, in the future, trunk builds included the entire localization tree, which would allow my build process to just pick out the appropriate JAR file for each language (which is essentially what I do now). Look to the Flickr Uploadr source (more on that soon) for an example of how all this works.

Why can’t XULRunner do this already?

Without really saying what “this” is, Mozilla wants XULRunner to be the standard play for cross-platform GUI applications and yet it has a bit of trouble interacting with 3rd party libraries. Most mature libraries (say, GraphicsMagick) still use char * or std::string to represent file paths. To interact with them, then, you have to be able to get to a char * representation of files you’re trying to open.

Windows and Mac character encodings 101

As is becoming tradition, Windows makes my life difficult by using UTF-16 as its Unicode of choice. UTF-16 uses 2 bytes (usually) to represent each character, meaning it isn’t binary compatible with ASCII, which uses 1 byte per character. These 2-byte characters are stored in wchar_ts which are twice as wide as chars. It isn’t possible to pass a wchar_t * through normal C libraries that expect strings as char *.

Mac OS X uses UTF-8 internally and is my hero. UTF-8 is used internally to represent strings and has the advantage of using single bytes (that’s char for C programmers playing along) as its base unit. While a single character in UTF-8 can span up to 8 bytes, the in-memory representation of these characters will pass muster with a C compiler. When the UTF-8 bytes are passed through some C library they can arrive intact to the fopen system call, which understands the UTF-8 bytes and acts like you’d expect.

There’s an easy way?

There could be. Using the @mozilla.org/file/local;1 it might be possible to get the path as an nsACString, Mozilla’s portable representation of single-byte ASCII strings. But, the reference warns that the native path available from nsILocalFile is not for passing to C libraries and isn’t guaranteed to be correct. So it’s up to me.

No pain, no gain

Bear with me, this gets ugly. The solution that lets me take paths from JavaScript-land to C-land, open Unicode paths with a normal char * and do so on Windows and Mac OS X is stupefying (and stupid). The iinitial path is stored as a JavaScript string, so the boundary between JavaScript and C will be an nsAString, which represents strings as UTF-16.

The string takes quite a journey here, so let’s cover the easy side first. Macs understand UTF-8, and I’m making a transformation to UTF-8 in JavaScript. This means that the only transform needed in C is a type change, not an encoding change. The UTF-8 string enters C-land as wide characters and is simply cast character-by-character down to an array of bytes.

Windows is quite a bit more work, since a path in Windows that needs Unicode characters can’t easily be represented as a char *. Enter the Windows API function GetShortPathName. This can convert any path to an old-school Windows path that fits in ASCII and uses that familiar “8.3″ naming convention. I got the Mac half working first, so the Windows implementation is actually more complicated than it needs to be — more on that later. Without further ado:

string * conv_path(const nsAString & fake) {

	// Fun with Windows paths
#ifdef XP_WIN

	// UTF-16 but really UTF-8 nsAString to really UTF-8 nsCString
	nsCString utf8 = NS_LossyConvertUTF16toASCII(fake);

	// UTF-8 nsCString to UTF-16 nsEmbedString
	nsEmbedString & utf16 = NS_ConvertUTF8toUTF16(utf8);

	// UTF-16 nsEmbedString to wchar_t[]
	wchar_t * w_arr = new wchar_t[utf16.Length() + 1];
	if (0 == w_arr) return 0;
	wchar_t * w_arr_p = w_arr;
	PRUnichar * w_start = (PRUnichar *)utf16.BeginReading();
	const PRUnichar * w_end = (PRUnichar *)utf16.EndReading();
	while (w_start != w_end) {
		*w_arr_p++ = (wchar_t)*w_start++;
	}
	*w_arr_p = 0;

	// GetShortPathName to get guaranteed ASCII
	wchar_t s_arr[4096];
	if (0 == GetShortPathNameW(w_arr, s_arr, 4096)) {
		delete [] w_arr;
		return 0;
	}
	delete [] w_arr;

	// wchar_t[] to ASCII nsEmbedString
	nsEmbedString ascii;
	wchar_t * s_arr_p = s_arr;
	while (*s_arr_p) {
		ascii.Append((char)*s_arr_p++);
	}

	// Macs don’t need any help since they understand UTF-8
#else
	nsEmbedString ascii;
	ascii.Assign(fake);
#endif

	// Convert the nsEmbedString into a std::string
	char * c_arr = c_arr = new char[ascii.Length() + 1];
	if (0 == c_arr) return 0;
	char * c_arr_p = c_arr;
	PRUnichar * c_start = (PRUnichar *)ascii.BeginReading();
	const PRUnichar * c_end = (PRUnichar *)ascii.EndReading();
	while (c_start != c_end) {
		*c_arr_p++ = (char)*c_start++;
	}
	*c_arr_p = 0;
	string * str = new string(c_arr);
	delete [] c_arr;
	return str;

}

This works simply because, on either platform, the result is a char * that actually uniquely represents the desired file.

My code sucks, let me count the ways

There are of course things that could be better. First of all, after reading the code again (I wrote it Thursday and haven’t looked back), I see that I should refactor a bit. By leaving the string in UTF-16 representation for passing to C-land, I can add a transform from UTF-16 to UTF-8 to the Mac version and greatly simplify the Windows version. I’ll put that on my to-do list. The other place I hope to improve is in the event that GetShortPathName fails. In this case, copying the file under a new ASCII filename to a temporary directory (found using GetTempPath) would still let the file be accessed with a char * path.

Why did I have to do this?

It would be sweeeeeet if Mozilla integrated this code (after a healthy dose of optimization) into the toolkit as something like NS_ConvertToNativePath(nsAString) or some such function. Having this will make integrating with third-party libraries much easier.

Rather than including an outside JavaScript library just to do MD5 sums, how about using Mozilla’s nsICryptoHash interface? For the Flickr API, this took slightly more effort than expected because it only returns data in base64-encoded ASCII or binary. Fortunately for me, the binary format was packed as a string, so making this usable was a matter of a little bin2hex function. Behold:

var _md5 = null;
try {
	_md5 = Cc['@mozilla.org/security/hash;1'].createInstance(Ci.nsICryptoHash);
} catch (err) {
	Components.utils.reportError(err);
}
var md5 = function(str) {
	if (null == _md5) {
		return ”;
	}

	// Build array of character codes to MD5
	var arr = [];
	var ii = str.length;
	for (var i = 0; i < ii; ++i) {
		arr.push(str.charCodeAt(i));
	}
	_md5.init(Ci.nsICryptoHash.MD5);
	_md5.update(arr, arr.length);
	var hash = _md5.finish(false);

	// Unpack the binary data bin2hex style
	var ascii = [];
	ii = hash.length;
	for (var i = 0; i < ii; ++i) {
		var c = hash.charCodeAt(i);
		var ones = c % 16;
		var tens = c >> 4;
		ascii.push(String.fromCharCode(tens + (tens > 9 ? 87 : 48)) +
			String.fromCharCode(ones + (ones > 9 ? 87 : 48)));
	}

	return ascii.join(”);
};

Today Om is reporting that Meebo is launching its own platform for third-party Meebo widgets. I could care less about the widgets, but could this be a REST API for sending IMs? No (dammit).

The Meebo API is actually an ingenious bit of Flash with a JavaScript wrapper that talks to Meebo. Once your hosted app gets some params from Meebo, you can do exciting things like listen for new messages. Through the same JavaScript-wrapped Flash object, you can also send messages. Praise be! For more, check out the Meebo JavaScript docs, which are minimal to say the least. Oh well, there’s sample code.

A couple weeks back, Instantbird did a lap around the Internet. I must call failure here, because when I tried to sign into Yahoo! Messenger it crashed immediately, spectacularly and consistently (tried four times). I know what you’re thinking: but Yahoo! Messenger is a closed system, of course there will be problems. Yeah, but Adium is built on the same library and pretty much always signs in happily (but never, ever shows me when Cal is online).

Seeing as Instantbird is built on XULRunner (my new life partner), a nice JavaScript way to send IMs would be just sweet. From my limited research, it seems embedding Flash in XULRunner is possible, as is opening a browser window within Instantbird to snag the authentication params from Meebo. Then it’s smooth sailing. Maybe.

The glaring hole in all of this is that while you can sendMsg to your heart’s content, thus far you have no one to send to.  This isn’t a full-blown API, folks.  It’s a widget platform.  There’s no way to get at someone’s buddy list.  Back to libpurple, I suppose.  But if Meebo made the rest of their service available as a REST API (to complement the far superior Flash method of sending/receiving messages), I’d be sold.

Despite my blog being stale like the bread in my cabinet, I’ve still been elbow deep in XULRunner all day every day. Now 3 1/2 months in, the ugly stuff has taken shape and it’s time to lock down the user interface. I still haven’t found an elegant way to keep clicks on the scrollbar from firing my sequence of mousedown, mousemove and mouseup events, but gimme a second chance.

As expected, I’m learning a lot about releasing software in the real world, where it has to stand up to real revisions, new features and public scrutiny. I like to think I’ve always built software that’s up to the task but a UI worthy of the Flickr name is a lot of work. And… Surprise! My engineer’s touch is not good enough. Gino and George, with a healthy bit of Stewart, have made it look awesome. And my role in the design-y stuff? Implementing some of the ideas so we can play with them before springing them on unsuspecting users.

But stay tuned. It turns out that work not only fluctuates between C++ and JavaScript but between engineering (then) and design (now). After round one is out in the wild it will be time for more hacking in the great undocumented. Lather, rinse, repeat.

The XPCOM tips will be back soon!

I have been spending an inordinate amount of time these days with XULRunner, Mozilla’s awesome cross-platform SDK used to build Firefox and Thunderbird. XULRunner is bit of an oddity, more akin to a web browser than the Java Virtual Machine or the Ruby interpreter. While it is most definitely focused on the development of desktop applications, it feels too similar to developing websites in many cases.

XUL is a specific subset of XML that is understood by XULRunner and is used to visually layout desktop applications. Interaction with a XUL app is handled by Javascript in much the same way that Javascript handles interactions with websites today. You can even include the XHTML namespace in your XUL app to really start to blur the lines between the desktop and the browser:

<window id="main" title="&title;" width="750" height="500"
xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
xmlns:html="http://www.w3.org/1999/xhtml">

But let’s stop right there. XUL is not XHTML. They do not work the same way and any attempts to make them do so will result in only frustration. This is where my biggest criticism of XUL comes. Visually, the code looks like a website but functionally the code works much more like Java Swing/AWT or Visual C++ resources. Websites work in blocks while XUL works in boxes.

Here’s a bit of a crash course in boxes. There are two basic types of boxes, vertical and horizontal. A vertical box will stack its child elements vertically, expanding them to fill the entire width available. A horizontal box will do just the opposite. Vertical boxes will by default only use as much vertical space as necessary, but with the flex="1" attribute they can be made to fill all available vertical space, too. Horizontal boxes similarly can be made to take all available horizontal space.

Throw all your old CSS techniques out the window now. What I found to be most helpful to me while laying out a XUL app was actually to turn CSS off. In contrast to the semantically pristine XHTML 1.1 we’ve all been writing lately, XUL layouts very much depend on the markup. This is both a blessing and a curse — our typical CSS kung fu doesn’t really matter but now we don’t have to worry about graceful degradation since we control the runtime used with our app.

The takeaway here is to throw semantics out the window for XUL apps. Focusing on building what you want and keeping your code maintainable are two big enough challenges.

XULRunner provides a nice way to localize apps through the use of custom document-type declarations (*.dtd files) and *.properties files used from within your Javascripts (more info on ). The markup below brings one of these “string bundles” into the app:

<stringbundleset>
	<stringbundle id="locale" src="chrome://app/locale/main.properties" />
</stringbundleset>

Then just one line of Javascript brings it from the XUL world into the Javascript world. The string bundle element itself contains methods getString and getFormattedString that will bring localized content into your Javascript.

var locale = document.getElementById('locale');
locale.getString('foo');
locale.getFormattedString('bar', [asdf, qwerty]);

But getFormattedString doesn’t work right in XULRunner 1.9a7. It substitutes only the first character of the string (in my experience). Here is a transparent Javascript-only fix:

var locale = document.getElementById('locale');

// Now hack locale.getFormattedString to work like it should
locale.getFormattedString = function(id, args) {
	var str = locale.getString(id);
	var ii = args.length;
	for (var i = 0; i < ii; ++i) {
		var regex = new RegExp('%' + (i + 1) + '\$[ds]‘);
		str = str.replace(regex, args[i]);
	}
	return str;
};

This properly takes care of the replacement of each element in args. In the example above where the variables asdf and qwerty were passed, the contents of asdf would replace %1$s and the contents of qwerty would replace %2$s. Note the numbering of the placeholders differing from normal array indexing. Also note that for semantic reasons, a d can be used in the placeholder to denote a number, but no special case is necessary in Javascript. Happy localizing!

Things change quickly in Internet-land, and between versions 1.8 and 1.9, the Gecko SDK received a major upgrade to its thread library. What once took multiple XPCOM objects, more than few hacks and a prayer for good measure can now be done in a single Javascript file with minimal interference from the world of C++ beneath.

Here is a sample of how one might spin an event off onto a background thread acting as a job queue. The execution is fairly straightforward: a background thread is created; the UI dispatches events onto this background thread; the job on the thread dispatches an event back to the UI thread when it is finished.

Before we start, let’s define the objects used to cross the thread boundary. They must be XPCOM objects (but can be implemented in pure Javascript and don’t even need to live in the components/ directory) and must implement the nsIRunnable interface. This interface requires that QueryInterface respond positively to nsIRunnable and that a run() function be defined.

var Foo = function(id) {
	this.id = id;
};
Foo.prototype = {
	run: function() {
		try {

			// Do some work, for now just print something
			Components.utils.reportError('This is where the background work happens.');

			// Callback to the main thread
			main.dispatch(new FooCallback(this.id, 'SUCCESS'),
				background.DISPATCH_NORMAL);

		} catch (err) {
			Components.utils.reportError(err);
		}
	},
	QueryInterface: function(iid) {
		if (iid.equals(Ci.nsIRunnable) || iid.equals(Ci.nsISupports)) {
			return this;
		}
		throw Components.results.NS_ERROR_NO_INTERFACE;
	}
};
var FooCallback = function(id, result) {
	this.id = id;
	this.result = result;
};
FooCallback.prototype = {
	run: function() {
		try {

			// This is where we can work with the main thread after the job
			alert('id: ' + id + ', result: ' + result);

		} catch (err) {
			Components.utils.reportError(err);
		}
	},
	QueryInterface: function(iid) {
		if (iid.equals(Ci.nsIRunnable) || iid.equals(Ci.nsISupports)) {
			return this;
		}
		throw Components.results.NS_ERROR_NO_INTERFACE;
	}
};

Here is the code we need to actually use these objects to spawn a background thread and dispatch a job to it. The callback will end up on the main thread’s event queue.

background = Cc['@mozilla.org/thread-manager;1'].getService().newThread(0);
main = Cc['@mozilla.org/thread-manager;1'].getService().mainThread;
background.dispatch(new Foo(id), background.DISPATCH_NORMAL);

The first two lines here should be executed on application startup and will leave thread references in main and background. The third line actually puts a job on the background queue. This can be called anywhere and will likely be used inside a loop to push a lot of compute-intensive work onto the background thread, allowing the UI thread to remain responsive.

A note about the id passed to the thread objects: this is not strictly necessary, but I find it very convenient to have the option of tracking individual jobs through the event queue. The id is especially useful when you’re using the callback to update a certain element of the UI with the result of the job. Then in the callback it’s as simple as document.getElementById(this.id).

Apologies for the bits-and-pieces examples. I’m ripping this out of unreleased code so I don’t have time to create a demo app for each one. I recommend starting with Mark Finkle’s hello world.

This assumes just a little knowledge of how to get a basic Hello World XULRunner app going. If you need help, I swear by Mark Finkle.

Every would-be Firefox extension developer (myself certainly included) has at some point sparred with overlays and been vanquished. They are not the friendliest things, as they are actually quite powerful and can handle a wide range of tasks. When building a Firefox extension, you build overlays to hook into the menus, create sidebars and toolbars, add status icons and do most anything else you can dream up.

To understand overlays, it helps to first get a look at the main application <window>. Here is a simple application window:

<?xml version="1.0"?>
<?xml-stylesheet href="chrome://global/skin/" type="text/css"?>
<?xml-stylesheet href="chrome://foo/skin/main.css" type="text/css"?>
<?xul-overlay href="chrome://foo/content/overlay_asdf.xul"?>
<?xul-overlay href="chrome://foo/content/overlay_qwerty.xul"?>
<!DOCTYPE window SYSTEM "chrome://uploadr/locale/foo.dtd">
<window id="main" title="&title;" width="750" height="500"
xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
	<script src="chrome://uploadr/content/main.js" />
	<toolbox>
		<menubar id="menu">
			<menu id="menu_file" label="&menu.file;">
				<menupopup>
					<menuitem id="menu_go" label="&menu.file.go;"
					oncommand="go();" />
					<menuitem id="menu_FileQuitItem" label="&menu.file.exit;"
					oncommand="exit();" />
				</menupopup>
			</menu>
		</menubar>
	</toolbox>
	<vbox flex="1">
		<hbox id="view_asdf" flex="1" />
		<vbox id="view_qwerty" flex="1" />
	</vbox>
</window>

This application does a couple of things of note. First, it sets up a XULRunner application window with a menu and a couple of boxes, but more importantly, at the top, it includes the two overlays we’re about to write.

Before we write the overlays, though, let’s add to the main.css file we’re including to make this interesting. The two overlays are going to be two different views in our application so we won’t want to see them both at the same time. For starters we’ll just hide #view_qwerty.

#view_qwerty {
	display: none;
}

Now come the overlays. An <overlay> is contructed very much like a <window> and can include all of the same types of content. The trick is in how they are inserted into the DOM. When an overlay is being processed, elements whose ID is shared by a similarly typed element in the main window, the contents of the overlay element are appended to the contents of the same element in the main window.

Knowing that, let’s write our two overlays. First overlay_asdf.xul and then overlay_qwerty.xul.

<?xml version="1.0"?>
<?xml-stylesheet href="chrome://global/skin/" type="text/css"?>
<?xml-stylesheet href="chrome://foo/skin/main.css" type="text/css"?>
<!DOCTYPE overlay SYSTEM "chrome://foo/locale/main.dtd">
<overlay id="overlay_asdf"
xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
xmlns:html="http://www.w3.org/1999/xhtml">
	<hbox id="view_asdf" flex="1">
		<html:p>This is view asdf.</html:p>
	</hbox>
</overlay>

<?xml version="1.0"?>
<?xml-stylesheet href="chrome://global/skin/" type="text/css"?>
<?xml-stylesheet href="chrome://foo/skin/main.css" type="text/css"?>
<!DOCTYPE overlay SYSTEM "chrome://foo/locale/main.dtd">
<overlay id="overlay_qwerty"
xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
xmlns:html="http://www.w3.org/1999/xhtml">
	<vbox id="view_qwerty" flex="1">
		<html:p>This is view qwerty.</html:p>
	</vbox>
</overlay>

The structure of the overlays is very similar to the structure of the window, so we’ll just take that as is and move on. The important thing to notice here is the way that #view_asdf is a <hbox> just as it is in the main window, while #view_qwerty is a <vbox>, again as it is in the main window. This is not strictly required, as the elements must only be similar, but because boxes want to be horizontal, you can introduce weirdness into your app if you’re careless about matching these tags up. So the moral of the story is to match your overlay tags with the tag being overlaid in the main window.

So now we have an app that, when run, will show us a window with a menu and a paragraph proclaiming “This is view asdf.” Too exciting. One little bit of Javascript in main.js will bring it all together.

var go = function() {
	var asdf = document.getElementById('view_asdf');
	var qwerty = document.getElementById('view_qwerty');
	if ('none' == asdf.style.display) {
		asdf.style.display = '-moz-box';
		qwerty.style.display = 'none';
	} else {
		asdf.style.display = 'none';
		qwerty.style.display = '-moz-box';
	}
};

With any luck, running your app will be able to toggle between its two views by selecting the “Go” option from the “File” menu. Here’s a peek under the hood. When XULRunner brings in your overlays, it’s updating the main DOM. The end result is a DOM that looks like this:

<?xml version="1.0"?>
<?xml-stylesheet href="chrome://global/skin/" type="text/css"?>
<?xml-stylesheet href="chrome://foo/skin/main.css" type="text/css"?>
<?xul-overlay href="chrome://foo/content/overlay_asdf.xul"?>
<?xul-overlay href="chrome://foo/content/overlay_qwerty.xul"?>
<!DOCTYPE window SYSTEM "chrome://uploadr/locale/foo.dtd">
<window id="main" title="&title;" width="750" height="500"
xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
	<script src="chrome://uploadr/content/main.js" />
	<toolbox>
		<menubar id="menu">
			<menu id="menu_file" label="&menu.file;">
				<menupopup>
					<menuitem id="menu_go" label="&menu.file.go;"
					oncommand="go();" />
					<menuitem id="menu_FileQuitItem" label="&menu.file.exit;"
					oncommand="exit();" />
				</menupopup>
			</menu>
		</menubar>
	</toolbox>
	<vbox flex="1">
		<hbox id="view_asdf" flex="1">
			<html:p>This is view asdf.</html:p>
		</hbox>
		<vbox id=”view_qwerty” flex=”1″>
			<html:p>This is view qwerty.</html:p>
		</vbox>
	</vbox>
</window>

The overlays buy us code separation, shorter files, and more sanity. Three cheers for overlays!