Posts Tagged Nexus

Nexus Development Hint: Debug a Missing Plugin Resource

Posted by on Thursday, 18 March, 2010

To counter-balance my rant the other day about plugin development (specifically in Nexus), I thought I’d offer a method I developed for coping with the hidden errors I ran into.

NOTE: Before I start, I should mention that most of the pain I’m about to describe, along the method of finding and working around the problem, will soon be a thing of the past. I’ve spoken to some of the Nexus team members, and they assure me they’re preparing to move Nexus off of Plexus, and onto Guice for dependency injection. So, once Nexus 1.6.0 comes out [I hope] it’ll be time to scrap these instructions.

First, a little background.

I’ve developed two Nexus plugins now, both small, and in each case I’ve run into strange behavior where my custom REST resources would go missing with nary an error to the console or logs. After specifying a particular resource URI, I’d build and deploy the plugin to my local Nexus instance, then try to hit it using curl with something like the following:

curl --include --basic -H 'Accept: application/json' http://admin:admin123@localhost:8081/nexus/service/local/echo

If all was well, the resource should respond with some result, throw an error, or something. But all I received was a 404 Not Found for my trouble. I attempted to dial up the log-level in sonatype-work/nexus/conf/log4j.properties, and scanned both nexus-webapp-1.5.0/logs/wrapper.log and sonatype-work/nexus/logs/nexus.log (the location I told log4j to use via a FileAppender). Nada.

When I developed my first plugin, this was the point where I started to bang my head against the wall in earnest. I couldn’t see any problems, it appeared from the logs that my plugin was loaded (it was listed in the plugin-manager output for my plugin bundle), and yet curl doesn’t lie. After about ten hours of remote debugging, I found a breakpoint that let me see what was going on. Eventually, I found a way to code around the problem, commented the hack liberally in my own code, filed a JIRA (NEXUS-3308), and moved on. Unfortunately, when it came time to test the second plugin, I’d already dumped all the old debug breakpoints from the first go-round. So, this time I’m going to document the setup I used to find and fix the problem before I forget it again.

Prepare your Nexus instance.

Before we even talk about remote debugging from Eclipse, we need to setup Nexus to listen on a JPDA (java debugger) port. To do this, I modified the stock Java Service Wrapper wrapper.conf file that comes with Nexus as follows:

You’ll notice that I actually have two variants of this JPDA configuration: one with suspend=y, and one with suspend=n. In most of my previous experience with remote debugging, I’ve been starting the application with the debugger attached so I can catch boot errors, things like that. But in the case of Nexus, I almost never use the suspend=y approach, since this makes the whole startup process excruciatingly slow. It’s not Nexus’ fault, it’s just what happens when your remote debugger is watching Every. Single. Line. If you’re even halfway paying attention, you should have no problem watching the logs and attaching your remote debugger in plenty of time to catch the breakpoint you’re interested in. Particularly if it’s the breakpoint I’ll discuss below.

Now that you have Nexus rigged for debugging, let’s talk about the remote debugger before we restart. Once we restart Nexus, we’ll need to be on the ball to attach the debugger and intercept our breakpoint of choice at boot.

Setup your remote debugger.

What follows is described in terms of Eclipse. It’s the IDE I use, for a number of reasons. I get endless shit about this from the IDEA fanatics I run into here and there. For those people: I understand you’re probably of above-average intelligence. You will probably have no problem at all translating my instructions into IDEA-ese.

To start, get the project setup so you can actually navigate something meaningful with your debugger. This means attaching the sources for the plexus-container-default dependency in your project. I’m using m2eclipse, which makes it pretty easy. The following is a view of my Package Explorer, poised to go download and attach the Plexus source code:

Next, setup a new Debug Configuration for your remote Nexus instance. This is pretty straightforward; the main points to notice is that your plugin project is selected as the main entry point, the host is set to localhost (or, wherever your Nexus instance is running), and the port is set to 8000 (this must correspond with the wrapper.conf modifications above). Your configuration should look something like this:

Now we’re ready to configure that critical breakpoint, and setup our debugging environment so we can see the output. To set the breakpoint, I navigated through the plexus-container-default artifact until I found org.codehaus.plexus.component.collections.AbstractComponentCollection, scrolled to line 159, and set the breakpoint, like so:

(You can set this breakpoint easily by double-clicking in the left gutter on line 159.) Also, notice that the breakpoint shows up in my Breakpoints View, in the upper-right corner. If you read NEXUS-3308, you’ll understand why this particular breakpoint is so critical…

Once you’ve set the breakpoint, also setup a watched expression to output the full exception stacktrace. To do this, right-click in the Watched Expressions View, and select Add Watched Expression. Then, enter the following:

Finally, since the result of that watched expression will be quite verbose, it’s important to extend the maximum buffer for the output pane in the Watched Expression View. To avoid having to do this twice, I’ve set mine to 0 (no maximum). Just select Max Length… from the following menu, and set it appropriately:

Debug!

Now, we’re ready to dig in and find that elusive bug that’s been making life hell. To do this, you’re going to be paying attention!

Ready? Okay, first stop Nexus and then re-run it in console mode. This allows you to stop it with CTL-C after you’re done debugging. The commands look something like this:


~/apps/nexus/current/bin/jsw/macosx-universal-32/nexus stop
~/apps/nexus/current/bin/jsw/macosx-universal-32/nexus console

As soon as you see output like the following, attach your remote debugger:

INFO   | jvm 1    | 2010/03/16 17:22:30 | Initializing Nexus (OSS), Version 1.5.0

Now, simply wait. If your Nexus component is failing when it initializes, the breakpoint will halt the system and the watched expression above will contain the full stacktrace. If you click on that watched expression, then click on the output pane, you can use CTL-A (CMD-A), CTL-C (CMD-C) to Select All of the stacktrace, then Copy it to your clipboard. From here, I’ve had a lot of luck opening Any Old Text Editor and pasting the stacktrace there for perusal. You can now detach the debugger (the button that looks like an electrical ‘N’) and let Nexus finish booting, or hit the continue button (looks like a pause/play button on a DVD player) to set a course for the next failing component.

One word of warning: if you take your sweet time here, you’ll miss your chance. This is because the default startup timeout for the Java Service Wrapper in Nexus is 90 seconds. If you take longer than about 30-45 seconds poking through running code, Java Service Wrapper will kill the process. Of course, if you really have to have extra time, that’s configurable too.