rant

NOTE: Before I start, I should mention that most of the pain I’m about to describe, along the method of finding and working around the problem, will soon be a thing of the past. I’ve spoken to some of the Nexus team members, and they assure me they’re preparing to move Nexus off of Plexus, and onto Guice for dependency injection. So, once Nexus 1.6.0 comes out [I hope] it’ll be time to scrap these instructions.

First, a little background.

I’ve developed two Nexus plugins now, both small, and in each case I’ve run into strange behavior where my custom REST resources would go missing with nary an error to the console or logs. After specifying a particular resource URI, I’d build and deploy the plugin to my local Nexus instance, then try to hit it using curl with something like the following:

curl --include --basic -H 'Accept: application/json' http://admin:admin123@localhost:8081/nexus/service/local/echo

If all was well, the resource should respond with some result, throw an error, or something. But all I received was a 404 Not Found for my trouble. I attempted to dial up the log-level in sonatype-work/nexus/conf/log4j.properties, and scanned both nexus-webapp-1.5.0/logs/wrapper.log and sonatype-work/nexus/logs/nexus.log (the location I told log4j to use via a FileAppender). Nada.

When I developed my first plugin, this was the point where I started to bang my head against the wall in earnest. I couldn’t see any problems, it appeared from the logs that my plugin was loaded (it was listed in the plugin-manager output for my plugin bundle), and yet curl doesn’t lie. After about ten hours of remote debugging, I found a breakpoint that let me see what was going on. Eventually, I found a way to code around the problem, commented the hack liberally in my own code, filed a JIRA (NEXUS-3308), and moved on. Unfortunately, when it came time to test the second plugin, I’d already dumped all the old debug breakpoints from the first go-round. So, this time I’m going to document the setup I used to find and fix the problem before I forget it again.

Prepare your Nexus instance.

Before we even talk about remote debugging from Eclipse, we need to setup Nexus to listen on a JPDA (java debugger) port. To do this, I modified the stock Java Service Wrapper wrapper.conf file that comes with Nexus as follows:

You’ll notice that I actually have two variants of this JPDA configuration: one with suspend=y, and one with suspend=n. In most of my previous experience with remote debugging, I’ve been starting the application with the debugger attached so I can catch boot errors, things like that. But in the case of Nexus, I almost never use the suspend=y approach, since this makes the whole startup process excruciatingly slow. It’s not Nexus’ fault, it’s just what happens when your remote debugger is watching Every. Single. Line. If you’re even halfway paying attention, you should have no problem watching the logs and attaching your remote debugger in plenty of time to catch the breakpoint you’re interested in. Particularly if it’s the breakpoint I’ll discuss below.

Now that you have Nexus rigged for debugging, let’s talk about the remote debugger before we restart. Once we restart Nexus, we’ll need to be on the ball to attach the debugger and intercept our breakpoint of choice at boot.

Setup your remote debugger.

What follows is described in terms of Eclipse. It’s the IDE I use, for a number of reasons. I get endless shit about this from the IDEA fanatics I run into here and there. For those people: I understand you’re probably of above-average intelligence. You will probably have no problem at all translating my instructions into IDEA-ese.

To start, get the project setup so you can actually navigate something meaningful with your debugger. This means attaching the sources for the plexus-container-default dependency in your project. I’m using m2eclipse, which makes it pretty easy. The following is a view of my Package Explorer, poised to go download and attach the Plexus source code:

Next, setup a new Debug Configuration for your remote Nexus instance. This is pretty straightforward; the main points to notice is that your plugin project is selected as the main entry point, the host is set to localhost (or, wherever your Nexus instance is running), and the port is set to 8000 (this must correspond with the wrapper.conf modifications above). Your configuration should look something like this:

Now we’re ready to configure that critical breakpoint, and setup our debugging environment so we can see the output. To set the breakpoint, I navigated through the plexus-container-default artifact until I found org.codehaus.plexus.component.collections.AbstractComponentCollection, scrolled to line 159, and set the breakpoint, like so:

(You can set this breakpoint easily by double-clicking in the left gutter on line 159.) Also, notice that the breakpoint shows up in my Breakpoints View, in the upper-right corner. If you read NEXUS-3308, you’ll understand why this particular breakpoint is so critical…

Once you’ve set the breakpoint, also setup a watched expression to output the full exception stacktrace. To do this, right-click in the Watched Expressions View, and select Add Watched Expression. Then, enter the following:

Finally, since the result of that watched expression will be quite verbose, it’s important to extend the maximum buffer for the output pane in the Watched Expression View. To avoid having to do this twice, I’ve set mine to 0 (no maximum). Just select Max Length… from the following menu, and set it appropriately:

Debug!

Now, we’re ready to dig in and find that elusive bug that’s been making life hell. To do this, you’re going to be paying attention!

Ready? Okay, first stop Nexus and then re-run it in console mode. This allows you to stop it with CTL-C after you’re done debugging. The commands look something like this:


~/apps/nexus/current/bin/jsw/macosx-universal-32/nexus stop
~/apps/nexus/current/bin/jsw/macosx-universal-32/nexus console


As soon as you see output like the following, attach your remote debugger:

INFO   | jvm 1    | 2010/03/16 17:22:30 | Initializing Nexus (OSS), Version 1.5.0

Now, simply wait. If your Nexus component is failing when it initializes, the breakpoint will halt the system and the watched expression above will contain the full stacktrace. If you click on that watched expression, then click on the output pane, you can use CTL-A (CMD-A), CTL-C (CMD-C) to Select All of the stacktrace, then Copy it to your clipboard. From here, I’ve had a lot of luck opening Any Old Text Editor and pasting the stacktrace there for perusal. You can now detach the debugger (the button that looks like an electrical ‘N’) and let Nexus finish booting, or hit the continue button (looks like a pause/play button on a DVD player) to set a course for the next failing component.

One word of warning: if you take your sweet time here, you’ll miss your chance. This is because the default startup timeout for the Java Service Wrapper in Nexus is 90 seconds. If you take longer than about 30-45 seconds poking through running code, Java Service Wrapper will kill the process. Of course, if you really have to have extra time, that’s configurable too.

After all, I’m one of the people responsible for Apache Maven 2.x, including its plugin framework which seems to confuse the hell out of Maven newbies. I used to chalk most of that up to a lack of documentation, and I’m still sure that is part of the problem. But now, I’ve got a somewhat different perspective on plugin development in a dependency-injected context, and I think there’s probably more making life difficult for third-party plugin developers than simply being too lazy to read a book.

Recently, I’ve been working on a few custom plugins to work with Sonatype Nexus for Red Hat (whom I joined back in January). It may sound surprising that I’m having some trouble with this since I used to work at Sonatype, after all. But alas, the closest I ever got to working on Nexus was writing a client API that I could wrap in a Maven plugin, and a few tools and configurations for building Nexus itself. Anyway, while the Nexus documentation for plugin developers is a little scant and shallow, I don’t really think it’s the main problem. Rather, it seems like developers have a tendency to include a plugin framework for their application without thinking much about the fact that people – people who don’t have a deep understanding of the application’s architecture – will actually try to use that framework. I haven’t understood this problem as well as I should with Maven, since I do have such a deep architectural understanding of it. Now I see that by providing a plugin framework for your application, you’re combining the hardest parts of standalone application development with the hardest parts of library development. In building a customer-facing application, you’re concerned with providing clean UI and an unbreakable user experience. These things require quite a bit of focused testing, and hiding a lot of the debug-level information from the user, so the output (console and UI) is clean and understandable. On the other hand, when building a good library, you need to worry about maintaining a strict respect of the library’s API contract over time, good developer documentation, the accessibility (and tunability) of debug-level information, and an absolute minimum of unexpected or magical behavior. Again, this requires a ton of focused testing, and some deep thinking about how to provide useful feedback to the developer in the event of an error. When the application is deployed, the developer should be able to hide this troubleshooting information (see application concerns, above).

Where many applications with plugin frameworks seem to really fall down is in supporting the plugin developer. The plugin framework itself seems to be a concession to make the lives of internal developers simpler, and this “simplicity” is usually passed on to third-party developers as well – as an afterthought, without considering that third-party developers don’t work with this stuff all day, every day. The result is an application that taunts the outside developer with the promise of extensibility, only to make life hell for those who succumb to that particular siren song. Finding themselves knee-deep without a shovel, such developers often have no other choice but to hook up a debugger, set some optimistic breakpoints, and continue down the rabbit hole into an endless iteration of debug session after debug session. “OK, I think I’ve finally isolated the code I need…Whoops! Just missed it.” CTL-C, set another breakpoint (or refine an existing one because unqualified, it seems to be the usage equivalent of Object.hashCode()) and run it again…and again….and again. Welcome back to the world of trial-and-error programming. If you’re a decent programmer, you haven’t been this helpless to fix a coding error since you learned to write code. Often, after eight hours of debugging or more, our poor developer finds out that he has an extra %s in his String.format() call, and the resulting RuntimeException is being swallowed up by the plugin framework. Or, he realizes that he’s been trying to retrieve an attribute named ‘repository-id’ from the Context attributes, when he should have been retrieving it from the Request attributes. Isn’t it obvious that the context has a different lifecycle than the request? Well, they’re both passed in next to one another in the same method signature, so, well…No, it’s not obvious.

And maybe that’s the real problem with plugin development: the pitfalls, the potential for errors, even the philosophy that went into the application’s architecture is not obvious. Since most third-party developers aren’t psychic, confusion and torture ensues. And, while most developers these days – myself included – have a big ole crush on dependency-injection frameworks like Guice, in many ways they only exacerbate the confusion by adding a new dimension of spooky action at a distance. Plugins have always been collections of hook implementations, event listeners, and callbacks that tie into an existing application…that’s sort of the definition. But with DI frameworks, these listeners and callbacks have never looked more disembodied. Making matters worse, DI frameworks themselves often over-promise and under-deliver. Like the rest, they are pieces of software, and may contain their own quirks or bugs. At times, the application developers may even place strange and seemingly arbitrary restrictions on how the underlying DI system is used (like forcing a component interface and its implementation to come from the same jar) that undermine the promise of a componentized architecture. In the end, the unflinching observer is all too often challenged to say that using a DI container to support a plugin framework saves anyone any pain at all.

To some extent, the fragmentation issues are unavoidable in plugin development. Fundamentally, you’re adding a small slice of functionality onto carefully selected extension points in an existing application; when you don’t understand how that application functions internally, finding the right leverage points for your new feature will always be hard. But how much harder is your application actually making the life of your third-party developers? Providing and advertising a plugin framework for your application implies the desire to create an ecosystem around that application. But providing a plugin framework and API that doesn’t behave as expected, doesn’t provide much help to third-party developers, isn’t well documented, and doesn’t have an comprehensive suite of tests to continually verify the plugin contract…all of this is counter-productive at best. At worst, it can alienate some of your application’s biggest fans.

I started out by looking at the Pebble2Wordpress importer code, which is written in Ruby. (I first found a reference to this project from The Spritle Blog blog post, which has some more information about how it’s meant to be used.) I selected this project in particular because it’s written in a language I can fine tune as needed.

I’m not sure whether this project ever really worked as-is out of the source repository, but I can say for sure that it didn’t work for me using the instructions provided on the blog post. In an attempt to understand where the developer was coming from, and determine whether I could cobble together something to work for my own particular migration, I dug into the source. What I found seemed pretty overwrought and confusing, particularly for something that amounts to a simple migration script to morph XML documents into DB calls (not something that will ever attract a huge user base). After refactoring the code to provide minimal error handling, and to inline all class definitions into one ruby script alongside the code that uses these classes, I thought I had something understandable to start testing.

The general approach of the code is to create ActiveRecord representations for terms (read: categories), term-post relationships, posts, and comments. Once the database is connected, the script iterates through all *.xml files in the inputXML directory, and parses the Pebble XML format into Post/Comment/Category instances which are then saved into the WP database. During the parsing process, certain things like image URLs are massaged from the old Pebble layout to something more compatible with WP. Since the default WP uploads directory has the date baked into the file path, I opted to simplify things and put all migrated files into the uploads/ base directory. This has an added benefit of giving me one-stop shopping for anything that breaks during the migration process, instead of picking over a nested directory structure.

Before you can start using the code, you’ll need to install some RubyGems and OS hooks:

$  sudo apt-get -y install rubygems mysql-dev
...
$  gem install activerecord activerecord-jdbcmysql-adapter
...

Next, you need to create a new directory alongside the convert.rb script, called inputXML/. Into this you should copy all of the XML files from the pebble data directory. My data directory was in /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/, so I used the following commands to achieve this:

$ find /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/2009 -type f -name '*.xml' -exec cp '{}' inputXML/ \;
...
$ find /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/2008 -type f -name '*.xml' -exec cp '{}' inputXML/ \;
...
$ find /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/2007 -type f -name '*.xml' -exec cp '{}' inputXML/ \;
...
$ find /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/2006 -type f -name '*.xml' -exec cp '{}' inputXML/ \;

I know, using some more arcane shell-fu I could have combined the individual commands into one. However, rather than take a chance of fouling something up with an overly-complex command line, I chose to use the Base arrow-up feature to retrieve and modify my previous command. Simple, effective, and nearly fool-proof.

Luckily, in my case I had a bare WP install that I could abuse a bit while I fine tuned the migration code. Along with a decent database client (I love the free version of DBVisualizer for this), rolling back failed migrations are a breeze. After a failed migration attempt, I simply issued the following SQL commands to get back near enough to the base install:

DELETE FROM wp_posts;

DELETE FROM wp_comments;

DELETE FROM wp_term_relationships WHERE object_id $  999

NOTE: I chose to delete term relationships where the object ID is 1000 or greater, since it seems like the auto-increment field for posts starts at around 1000. This may not be the case in all environments.

I’m not going to pollute this blog post by giving a full listing of the migration script, but you can download it here: convert.rb. It has an accompanying database configuration YAML file that looks like this: database.yml. The script expects the inputXML/ directory and the database.yml file to be in the same directory as the script itself. Use the script if you want to, enjoy it, but above all, DON’T BLAME ME IF YOU BLOW SOMETHING UP.

I then tested, rinsed, and repeated until I was pleased with the result. As a final step, I copied the contents of the files/ and images/ directories from the Pebble data directory into my WP install:

$  cp -rf /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/images/* /opt/web/sites/johnofalltrades.name/htdocs/wp-content/uploads
...
$  cp -rf /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/files/* /opt/web/sites/johnofalltrades.name/htdocs/wp-content/uploads
...

That’s about all there is to it. Hackish? Maybe. Effective? Definitely. I’m very happy in my new WordPress digs!

UPDATE: I’ve submitted my changes back to the project in this issue.