rant

NOTE: Before I start, I should mention that most of the pain I’m about to describe, along the method of finding and working around the problem, will soon be a thing of the past. I’ve spoken to some of the Nexus team members, and they assure me they’re preparing to move Nexus off of Plexus, and onto Guice for dependency injection. So, once Nexus 1.6.0 comes out [I hope] it’ll be time to scrap these instructions.

First, a little background.

I’ve developed two Nexus plugins now, both small, and in each case I’ve run into strange behavior where my custom REST resources would go missing with nary an error to the console or logs. After specifying a particular resource URI, I’d build and deploy the plugin to my local Nexus instance, then try to hit it using curl with something like the following:

curl --include --basic -H 'Accept: application/json' http://admin:admin123@localhost:8081/nexus/service/local/echo

If all was well, the resource should respond with some result, throw an error, or something. But all I received was a 404 Not Found for my trouble. I attempted to dial up the log-level in sonatype-work/nexus/conf/log4j.properties, and scanned both nexus-webapp-1.5.0/logs/wrapper.log and sonatype-work/nexus/logs/nexus.log (the location I told log4j to use via a FileAppender). Nada.

When I developed my first plugin, this was the point where I started to bang my head against the wall in earnest. I couldn’t see any problems, it appeared from the logs that my plugin was loaded (it was listed in the plugin-manager output for my plugin bundle), and yet curl doesn’t lie. After about ten hours of remote debugging, I found a breakpoint that let me see what was going on. Eventually, I found a way to code around the problem, commented the hack liberally in my own code, filed a JIRA (NEXUS-3308), and moved on. Unfortunately, when it came time to test the second plugin, I’d already dumped all the old debug breakpoints from the first go-round. So, this time I’m going to document the setup I used to find and fix the problem before I forget it again.

Prepare your Nexus instance.

Before we even talk about remote debugging from Eclipse, we need to setup Nexus to listen on a JPDA (java debugger) port. To do this, I modified the stock Java Service Wrapper wrapper.conf file that comes with Nexus as follows:

You’ll notice that I actually have two variants of this JPDA configuration: one with suspend=y, and one with suspend=n. In most of my previous experience with remote debugging, I’ve been starting the application with the debugger attached so I can catch boot errors, things like that. But in the case of Nexus, I almost never use the suspend=y approach, since this makes the whole startup process excruciatingly slow. It’s not Nexus’ fault, it’s just what happens when your remote debugger is watching Every. Single. Line. If you’re even halfway paying attention, you should have no problem watching the logs and attaching your remote debugger in plenty of time to catch the breakpoint you’re interested in. Particularly if it’s the breakpoint I’ll discuss below.

Now that you have Nexus rigged for debugging, let’s talk about the remote debugger before we restart. Once we restart Nexus, we’ll need to be on the ball to attach the debugger and intercept our breakpoint of choice at boot.

Setup your remote debugger.

What follows is described in terms of Eclipse. It’s the IDE I use, for a number of reasons. I get endless shit about this from the IDEA fanatics I run into here and there. For those people: I understand you’re probably of above-average intelligence. You will probably have no problem at all translating my instructions into IDEA-ese.

To start, get the project setup so you can actually navigate something meaningful with your debugger. This means attaching the sources for the plexus-container-default dependency in your project. I’m using m2eclipse, which makes it pretty easy. The following is a view of my Package Explorer, poised to go download and attach the Plexus source code:

Next, setup a new Debug Configuration for your remote Nexus instance. This is pretty straightforward; the main points to notice is that your plugin project is selected as the main entry point, the host is set to localhost (or, wherever your Nexus instance is running), and the port is set to 8000 (this must correspond with the wrapper.conf modifications above). Your configuration should look something like this:

Now we’re ready to configure that critical breakpoint, and setup our debugging environment so we can see the output. To set the breakpoint, I navigated through the plexus-container-default artifact until I found org.codehaus.plexus.component.collections.AbstractComponentCollection, scrolled to line 159, and set the breakpoint, like so:

(You can set this breakpoint easily by double-clicking in the left gutter on line 159.) Also, notice that the breakpoint shows up in my Breakpoints View, in the upper-right corner. If you read NEXUS-3308, you’ll understand why this particular breakpoint is so critical…

Once you’ve set the breakpoint, also setup a watched expression to output the full exception stacktrace. To do this, right-click in the Watched Expressions View, and select Add Watched Expression. Then, enter the following:

Finally, since the result of that watched expression will be quite verbose, it’s important to extend the maximum buffer for the output pane in the Watched Expression View. To avoid having to do this twice, I’ve set mine to 0 (no maximum). Just select Max Length… from the following menu, and set it appropriately:

Debug!

Now, we’re ready to dig in and find that elusive bug that’s been making life hell. To do this, you’re going to be paying attention!

Ready? Okay, first stop Nexus and then re-run it in console mode. This allows you to stop it with CTL-C after you’re done debugging. The commands look something like this:


~/apps/nexus/current/bin/jsw/macosx-universal-32/nexus stop
~/apps/nexus/current/bin/jsw/macosx-universal-32/nexus console


As soon as you see output like the following, attach your remote debugger:

INFO   | jvm 1    | 2010/03/16 17:22:30 | Initializing Nexus (OSS), Version 1.5.0

Now, simply wait. If your Nexus component is failing when it initializes, the breakpoint will halt the system and the watched expression above will contain the full stacktrace. If you click on that watched expression, then click on the output pane, you can use CTL-A (CMD-A), CTL-C (CMD-C) to Select All of the stacktrace, then Copy it to your clipboard. From here, I’ve had a lot of luck opening Any Old Text Editor and pasting the stacktrace there for perusal. You can now detach the debugger (the button that looks like an electrical ‘N’) and let Nexus finish booting, or hit the continue button (looks like a pause/play button on a DVD player) to set a course for the next failing component.

One word of warning: if you take your sweet time here, you’ll miss your chance. This is because the default startup timeout for the Java Service Wrapper in Nexus is 90 seconds. If you take longer than about 30-45 seconds poking through running code, Java Service Wrapper will kill the process. Of course, if you really have to have extra time, that’s configurable too.

I started out by looking at the Pebble2Wordpress importer code, which is written in Ruby. (I first found a reference to this project from The Spritle Blog blog post, which has some more information about how it’s meant to be used.) I selected this project in particular because it’s written in a language I can fine tune as needed.

I’m not sure whether this project ever really worked as-is out of the source repository, but I can say for sure that it didn’t work for me using the instructions provided on the blog post. In an attempt to understand where the developer was coming from, and determine whether I could cobble together something to work for my own particular migration, I dug into the source. What I found seemed pretty overwrought and confusing, particularly for something that amounts to a simple migration script to morph XML documents into DB calls (not something that will ever attract a huge user base). After refactoring the code to provide minimal error handling, and to inline all class definitions into one ruby script alongside the code that uses these classes, I thought I had something understandable to start testing.

The general approach of the code is to create ActiveRecord representations for terms (read: categories), term-post relationships, posts, and comments. Once the database is connected, the script iterates through all *.xml files in the inputXML directory, and parses the Pebble XML format into Post/Comment/Category instances which are then saved into the WP database. During the parsing process, certain things like image URLs are massaged from the old Pebble layout to something more compatible with WP. Since the default WP uploads directory has the date baked into the file path, I opted to simplify things and put all migrated files into the uploads/ base directory. This has an added benefit of giving me one-stop shopping for anything that breaks during the migration process, instead of picking over a nested directory structure.

Before you can start using the code, you’ll need to install some RubyGems and OS hooks:

$  sudo apt-get -y install rubygems mysql-dev
...
$  gem install activerecord activerecord-jdbcmysql-adapter
...

Next, you need to create a new directory alongside the convert.rb script, called inputXML/. Into this you should copy all of the XML files from the pebble data directory. My data directory was in /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/, so I used the following commands to achieve this:

$ find /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/2009 -type f -name '*.xml' -exec cp '{}' inputXML/ \;
...
$ find /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/2008 -type f -name '*.xml' -exec cp '{}' inputXML/ \;
...
$ find /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/2007 -type f -name '*.xml' -exec cp '{}' inputXML/ \;
...
$ find /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/2006 -type f -name '*.xml' -exec cp '{}' inputXML/ \;

I know, using some more arcane shell-fu I could have combined the individual commands into one. However, rather than take a chance of fouling something up with an overly-complex command line, I chose to use the Base arrow-up feature to retrieve and modify my previous command. Simple, effective, and nearly fool-proof.

Luckily, in my case I had a bare WP install that I could abuse a bit while I fine tuned the migration code. Along with a decent database client (I love the free version of DBVisualizer for this), rolling back failed migrations are a breeze. After a failed migration attempt, I simply issued the following SQL commands to get back near enough to the base install:

DELETE FROM wp_posts;

DELETE FROM wp_comments;

DELETE FROM wp_term_relationships WHERE object_id $  999

NOTE: I chose to delete term relationships where the object ID is 1000 or greater, since it seems like the auto-increment field for posts starts at around 1000. This may not be the case in all environments.

I’m not going to pollute this blog post by giving a full listing of the migration script, but you can download it here: convert.rb. It has an accompanying database configuration YAML file that looks like this: database.yml. The script expects the inputXML/ directory and the database.yml file to be in the same directory as the script itself. Use the script if you want to, enjoy it, but above all, DON’T BLAME ME IF YOU BLOW SOMETHING UP.

I then tested, rinsed, and repeated until I was pleased with the result. As a final step, I copied the contents of the files/ and images/ directories from the Pebble data directory into my WP install:

$  cp -rf /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/images/* /opt/web/sites/johnofalltrades.name/htdocs/wp-content/uploads
...
$  cp -rf /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/files/* /opt/web/sites/johnofalltrades.name/htdocs/wp-content/uploads
...

That’s about all there is to it. Hackish? Maybe. Effective? Definitely. I’m very happy in my new WordPress digs!

UPDATE: I’ve submitted my changes back to the project in this issue.