I’ve spent the past several days working on the presentation, and one thing that strikes me is how much work goes into so very little content. The idea and general outline are the easy part! After this comes hours of concocting graphics that illustrate your point while de-cluttering the slides, then hours of dealing with the most ruthless of editors: the stopwatch. Long before you’ve satisfied this critic, you’re forced to give up on some of the ideas you previously regarded as absolute essentials. In the end, you come to realize that you’ve initially had enough content to fill three hours, not fifty minutes.

I’ve opted this year to do a LOT more slides, many of which are variations on the same message that each highlight a different aspect. I’m hoping this will have the effect of presenting the same amount of content in a much clearer way, but who knows? I’m going on the theory that a picture is worth a thousand words, so we’ll see if that’s true…I should have some results for you in a couple weeks. I’ve chosen to stay away from live demos, mainly because most presentations I’ve seen that include them don’t work well (especially when the presentation is only an hour long). Usually, there are enough things that go wrong to derail the point of the presentation. Besides, I’m still relatively new to this, and I’m not willing to play fast and loose with the time management just yet, thanks very much.

Beyond struggling with the work involved in getting the slides just right, I’m also struck by the fact that each message has its own optimum medium. People in the business world are fond of saying that some things just need to be solved face-to-face, or that sometimes there’s simply no substitute for a phone call. Mostly I tend to think of these people as just not trying hard enough, or giving in to the habits bred by long years spent in co-located offices and cubicle farms. But I think that giving a technical presentation makes this point in a completely backward way…if you’ve ever tried to explain a technical concept in mere words and hand gestures, you know what I mean. As you attempt to manipulate imaginary solid shapes in mid-air, all the while relating the concept using a bunch of very technical words that have extremely closely-related definitions in their everyday use, you begin to understand the absurdity of it all. This is probably where a good set of diagrams – or a live demo – can save the day. I’m hoping that’s true. Otherwise, people are going to be in for one hell of an imaginary puppet show!

Wish me luck! And, if you’re at JBossWorld this year, stop by Wednesday afternoon and say hi. I’ll be presenting Maven Best Practices.

rant

NOTE: Before I start, I should mention that most of the pain I’m about to describe, along the method of finding and working around the problem, will soon be a thing of the past. I’ve spoken to some of the Nexus team members, and they assure me they’re preparing to move Nexus off of Plexus, and onto Guice for dependency injection. So, once Nexus 1.6.0 comes out [I hope] it’ll be time to scrap these instructions.

First, a little background.

I’ve developed two Nexus plugins now, both small, and in each case I’ve run into strange behavior where my custom REST resources would go missing with nary an error to the console or logs. After specifying a particular resource URI, I’d build and deploy the plugin to my local Nexus instance, then try to hit it using curl with something like the following:

curl --include --basic -H 'Accept: application/json' http://admin:admin123@localhost:8081/nexus/service/local/echo

If all was well, the resource should respond with some result, throw an error, or something. But all I received was a 404 Not Found for my trouble. I attempted to dial up the log-level in sonatype-work/nexus/conf/log4j.properties, and scanned both nexus-webapp-1.5.0/logs/wrapper.log and sonatype-work/nexus/logs/nexus.log (the location I told log4j to use via a FileAppender). Nada.

When I developed my first plugin, this was the point where I started to bang my head against the wall in earnest. I couldn’t see any problems, it appeared from the logs that my plugin was loaded (it was listed in the plugin-manager output for my plugin bundle), and yet curl doesn’t lie. After about ten hours of remote debugging, I found a breakpoint that let me see what was going on. Eventually, I found a way to code around the problem, commented the hack liberally in my own code, filed a JIRA (NEXUS-3308), and moved on. Unfortunately, when it came time to test the second plugin, I’d already dumped all the old debug breakpoints from the first go-round. So, this time I’m going to document the setup I used to find and fix the problem before I forget it again.

Prepare your Nexus instance.

Before we even talk about remote debugging from Eclipse, we need to setup Nexus to listen on a JPDA (java debugger) port. To do this, I modified the stock Java Service Wrapper wrapper.conf file that comes with Nexus as follows:

You’ll notice that I actually have two variants of this JPDA configuration: one with suspend=y, and one with suspend=n. In most of my previous experience with remote debugging, I’ve been starting the application with the debugger attached so I can catch boot errors, things like that. But in the case of Nexus, I almost never use the suspend=y approach, since this makes the whole startup process excruciatingly slow. It’s not Nexus’ fault, it’s just what happens when your remote debugger is watching Every. Single. Line. If you’re even halfway paying attention, you should have no problem watching the logs and attaching your remote debugger in plenty of time to catch the breakpoint you’re interested in. Particularly if it’s the breakpoint I’ll discuss below.

Now that you have Nexus rigged for debugging, let’s talk about the remote debugger before we restart. Once we restart Nexus, we’ll need to be on the ball to attach the debugger and intercept our breakpoint of choice at boot.

Setup your remote debugger.

What follows is described in terms of Eclipse. It’s the IDE I use, for a number of reasons. I get endless shit about this from the IDEA fanatics I run into here and there. For those people: I understand you’re probably of above-average intelligence. You will probably have no problem at all translating my instructions into IDEA-ese.

To start, get the project setup so you can actually navigate something meaningful with your debugger. This means attaching the sources for the plexus-container-default dependency in your project. I’m using m2eclipse, which makes it pretty easy. The following is a view of my Package Explorer, poised to go download and attach the Plexus source code:

Next, setup a new Debug Configuration for your remote Nexus instance. This is pretty straightforward; the main points to notice is that your plugin project is selected as the main entry point, the host is set to localhost (or, wherever your Nexus instance is running), and the port is set to 8000 (this must correspond with the wrapper.conf modifications above). Your configuration should look something like this:

Now we’re ready to configure that critical breakpoint, and setup our debugging environment so we can see the output. To set the breakpoint, I navigated through the plexus-container-default artifact until I found org.codehaus.plexus.component.collections.AbstractComponentCollection, scrolled to line 159, and set the breakpoint, like so:

(You can set this breakpoint easily by double-clicking in the left gutter on line 159.) Also, notice that the breakpoint shows up in my Breakpoints View, in the upper-right corner. If you read NEXUS-3308, you’ll understand why this particular breakpoint is so critical…

Once you’ve set the breakpoint, also setup a watched expression to output the full exception stacktrace. To do this, right-click in the Watched Expressions View, and select Add Watched Expression. Then, enter the following:

Finally, since the result of that watched expression will be quite verbose, it’s important to extend the maximum buffer for the output pane in the Watched Expression View. To avoid having to do this twice, I’ve set mine to 0 (no maximum). Just select Max Length… from the following menu, and set it appropriately:

Debug!

Now, we’re ready to dig in and find that elusive bug that’s been making life hell. To do this, you’re going to be paying attention!

Ready? Okay, first stop Nexus and then re-run it in console mode. This allows you to stop it with CTL-C after you’re done debugging. The commands look something like this:


~/apps/nexus/current/bin/jsw/macosx-universal-32/nexus stop
~/apps/nexus/current/bin/jsw/macosx-universal-32/nexus console


As soon as you see output like the following, attach your remote debugger:

INFO   | jvm 1    | 2010/03/16 17:22:30 | Initializing Nexus (OSS), Version 1.5.0

Now, simply wait. If your Nexus component is failing when it initializes, the breakpoint will halt the system and the watched expression above will contain the full stacktrace. If you click on that watched expression, then click on the output pane, you can use CTL-A (CMD-A), CTL-C (CMD-C) to Select All of the stacktrace, then Copy it to your clipboard. From here, I’ve had a lot of luck opening Any Old Text Editor and pasting the stacktrace there for perusal. You can now detach the debugger (the button that looks like an electrical ‘N’) and let Nexus finish booting, or hit the continue button (looks like a pause/play button on a DVD player) to set a course for the next failing component.

One word of warning: if you take your sweet time here, you’ll miss your chance. This is because the default startup timeout for the Java Service Wrapper in Nexus is 90 seconds. If you take longer than about 30-45 seconds poking through running code, Java Service Wrapper will kill the process. Of course, if you really have to have extra time, that’s configurable too.

After all, I’m one of the people responsible for Apache Maven 2.x, including its plugin framework which seems to confuse the hell out of Maven newbies. I used to chalk most of that up to a lack of documentation, and I’m still sure that is part of the problem. But now, I’ve got a somewhat different perspective on plugin development in a dependency-injected context, and I think there’s probably more making life difficult for third-party plugin developers than simply being too lazy to read a book.

Recently, I’ve been working on a few custom plugins to work with Sonatype Nexus for Red Hat (whom I joined back in January). It may sound surprising that I’m having some trouble with this since I used to work at Sonatype, after all. But alas, the closest I ever got to working on Nexus was writing a client API that I could wrap in a Maven plugin, and a few tools and configurations for building Nexus itself. Anyway, while the Nexus documentation for plugin developers is a little scant and shallow, I don’t really think it’s the main problem. Rather, it seems like developers have a tendency to include a plugin framework for their application without thinking much about the fact that people – people who don’t have a deep understanding of the application’s architecture – will actually try to use that framework. I haven’t understood this problem as well as I should with Maven, since I do have such a deep architectural understanding of it. Now I see that by providing a plugin framework for your application, you’re combining the hardest parts of standalone application development with the hardest parts of library development. In building a customer-facing application, you’re concerned with providing clean UI and an unbreakable user experience. These things require quite a bit of focused testing, and hiding a lot of the debug-level information from the user, so the output (console and UI) is clean and understandable. On the other hand, when building a good library, you need to worry about maintaining a strict respect of the library’s API contract over time, good developer documentation, the accessibility (and tunability) of debug-level information, and an absolute minimum of unexpected or magical behavior. Again, this requires a ton of focused testing, and some deep thinking about how to provide useful feedback to the developer in the event of an error. When the application is deployed, the developer should be able to hide this troubleshooting information (see application concerns, above).

Where many applications with plugin frameworks seem to really fall down is in supporting the plugin developer. The plugin framework itself seems to be a concession to make the lives of internal developers simpler, and this “simplicity” is usually passed on to third-party developers as well – as an afterthought, without considering that third-party developers don’t work with this stuff all day, every day. The result is an application that taunts the outside developer with the promise of extensibility, only to make life hell for those who succumb to that particular siren song. Finding themselves knee-deep without a shovel, such developers often have no other choice but to hook up a debugger, set some optimistic breakpoints, and continue down the rabbit hole into an endless iteration of debug session after debug session. “OK, I think I’ve finally isolated the code I need…Whoops! Just missed it.” CTL-C, set another breakpoint (or refine an existing one because unqualified, it seems to be the usage equivalent of Object.hashCode()) and run it again…and again….and again. Welcome back to the world of trial-and-error programming. If you’re a decent programmer, you haven’t been this helpless to fix a coding error since you learned to write code. Often, after eight hours of debugging or more, our poor developer finds out that he has an extra %s in his String.format() call, and the resulting RuntimeException is being swallowed up by the plugin framework. Or, he realizes that he’s been trying to retrieve an attribute named ‘repository-id’ from the Context attributes, when he should have been retrieving it from the Request attributes. Isn’t it obvious that the context has a different lifecycle than the request? Well, they’re both passed in next to one another in the same method signature, so, well…No, it’s not obvious.

And maybe that’s the real problem with plugin development: the pitfalls, the potential for errors, even the philosophy that went into the application’s architecture is not obvious. Since most third-party developers aren’t psychic, confusion and torture ensues. And, while most developers these days – myself included – have a big ole crush on dependency-injection frameworks like Guice, in many ways they only exacerbate the confusion by adding a new dimension of spooky action at a distance. Plugins have always been collections of hook implementations, event listeners, and callbacks that tie into an existing application…that’s sort of the definition. But with DI frameworks, these listeners and callbacks have never looked more disembodied. Making matters worse, DI frameworks themselves often over-promise and under-deliver. Like the rest, they are pieces of software, and may contain their own quirks or bugs. At times, the application developers may even place strange and seemingly arbitrary restrictions on how the underlying DI system is used (like forcing a component interface and its implementation to come from the same jar) that undermine the promise of a componentized architecture. In the end, the unflinching observer is all too often challenged to say that using a DI container to support a plugin framework saves anyone any pain at all.

To some extent, the fragmentation issues are unavoidable in plugin development. Fundamentally, you’re adding a small slice of functionality onto carefully selected extension points in an existing application; when you don’t understand how that application functions internally, finding the right leverage points for your new feature will always be hard. But how much harder is your application actually making the life of your third-party developers? Providing and advertising a plugin framework for your application implies the desire to create an ecosystem around that application. But providing a plugin framework and API that doesn’t behave as expected, doesn’t provide much help to third-party developers, isn’t well documented, and doesn’t have an comprehensive suite of tests to continually verify the plugin contract…all of this is counter-productive at best. At worst, it can alienate some of your application’s biggest fans.

I started out by looking at the Pebble2Wordpress importer code, which is written in Ruby. (I first found a reference to this project from The Spritle Blog blog post, which has some more information about how it’s meant to be used.) I selected this project in particular because it’s written in a language I can fine tune as needed.

I’m not sure whether this project ever really worked as-is out of the source repository, but I can say for sure that it didn’t work for me using the instructions provided on the blog post. In an attempt to understand where the developer was coming from, and determine whether I could cobble together something to work for my own particular migration, I dug into the source. What I found seemed pretty overwrought and confusing, particularly for something that amounts to a simple migration script to morph XML documents into DB calls (not something that will ever attract a huge user base). After refactoring the code to provide minimal error handling, and to inline all class definitions into one ruby script alongside the code that uses these classes, I thought I had something understandable to start testing.

The general approach of the code is to create ActiveRecord representations for terms (read: categories), term-post relationships, posts, and comments. Once the database is connected, the script iterates through all *.xml files in the inputXML directory, and parses the Pebble XML format into Post/Comment/Category instances which are then saved into the WP database. During the parsing process, certain things like image URLs are massaged from the old Pebble layout to something more compatible with WP. Since the default WP uploads directory has the date baked into the file path, I opted to simplify things and put all migrated files into the uploads/ base directory. This has an added benefit of giving me one-stop shopping for anything that breaks during the migration process, instead of picking over a nested directory structure.

Before you can start using the code, you’ll need to install some RubyGems and OS hooks:

$  sudo apt-get -y install rubygems mysql-dev
...
$  gem install activerecord activerecord-jdbcmysql-adapter
...

Next, you need to create a new directory alongside the convert.rb script, called inputXML/. Into this you should copy all of the XML files from the pebble data directory. My data directory was in /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/, so I used the following commands to achieve this:

$ find /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/2009 -type f -name '*.xml' -exec cp '{}' inputXML/ \;
...
$ find /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/2008 -type f -name '*.xml' -exec cp '{}' inputXML/ \;
...
$ find /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/2007 -type f -name '*.xml' -exec cp '{}' inputXML/ \;
...
$ find /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/2006 -type f -name '*.xml' -exec cp '{}' inputXML/ \;

I know, using some more arcane shell-fu I could have combined the individual commands into one. However, rather than take a chance of fouling something up with an overly-complex command line, I chose to use the Base arrow-up feature to retrieve and modify my previous command. Simple, effective, and nearly fool-proof.

Luckily, in my case I had a bare WP install that I could abuse a bit while I fine tuned the migration code. Along with a decent database client (I love the free version of DBVisualizer for this), rolling back failed migrations are a breeze. After a failed migration attempt, I simply issued the following SQL commands to get back near enough to the base install:

DELETE FROM wp_posts;

DELETE FROM wp_comments;

DELETE FROM wp_term_relationships WHERE object_id $  999

NOTE: I chose to delete term relationships where the object ID is 1000 or greater, since it seems like the auto-increment field for posts starts at around 1000. This may not be the case in all environments.

I’m not going to pollute this blog post by giving a full listing of the migration script, but you can download it here: convert.rb. It has an accompanying database configuration YAML file that looks like this: database.yml. The script expects the inputXML/ directory and the database.yml file to be in the same directory as the script itself. Use the script if you want to, enjoy it, but above all, DON’T BLAME ME IF YOU BLOW SOMETHING UP.

I then tested, rinsed, and repeated until I was pleased with the result. As a final step, I copied the contents of the files/ and images/ directories from the Pebble data directory into my WP install:

$  cp -rf /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/images/* /opt/web/sites/johnofalltrades.name/htdocs/wp-content/uploads
...
$  cp -rf /opt/web/sites/ejlife.net/var/pebble/blogs/buildchimp/files/* /opt/web/sites/johnofalltrades.name/htdocs/wp-content/uploads
...

That’s about all there is to it. Hackish? Maybe. Effective? Definitely. I’m very happy in my new WordPress digs!

UPDATE: I’ve submitted my changes back to the project in this issue.

Our first focus is on user training, starting with a Maven user training session scheduled the Monday before JavaOne 2007 kicks off. We’ll be in the area throughout JavaOne, so don’t hesitate to get in touch. Training is far from the final word for Sonatype, but you’ll have to stay tuned for the details!

Our approach is to grow this company from the ground up, with nothing but top-quality members, partners, products, and information. We’re not relying on hype or analysts here; we understand what it takes to put together a first-rate development infrastructure, and we’re interested in building one for you. We want to start by helping you un-break your build process, using Maven.

<shameless-plug>
Oh, and if I haven’t already mentioned it, take a gander at our website!
</shameless-plug>

BTW, this isn’t my normal programming blog any more, so I’ll be cross-posting this to the BuildChimp.

The reasons are manifold, but mainly that I think it’d be better to have a clean separation between the content I write for my profession, community involvement, and industry, from that of my personal life. I’m aware that I could tweak Pebble a little and provide a code-only RSS feed for JavaBlogs to use, but I prefer a clean separation that will allow me to manage the two bodies of content separately.

So, if you read this blog for technical content, please update your Google Reader (or whatever) to: http://www.ejlife.net/blogs/buildchimp.

In the end, I found that my concept of a business that centered its existence on open source was quite different from theirs. Aside from that, I felt that I was treading dangerously close to being put in a position where I’d have to compromise my own personal credibility and integrity on behalf of the company. Finally, I didn’t really feel that Mergere’s approach to the Maven community was a healthy one…and this is a community which means a great deal to me.

So, I’ve taken some time to regain my sense of pride in the work that I do, and re-explore the open source side of the Maven community. I’m finally back to the point where I enjoy coding again. I’m not quite ready to dive into coffee-shop ownership yet, so I’m going to be easing back into the life of a software engineer.

Oh, I also spent a bit of time turning some perfectly good, green coffee beans to this dingy brown color in my home-built roasting drum, but that’s a topic for another day…

I just thought you should know.

Technorati Tags: code, maven

Unfortunately for me, I work on several development teams, and consistent code formatting really is critical, to help ensure SVN diffs are readable. This is one place where it seems that NetBeans is sorely lacking. I poked and prodded for hours, but have been unable to configure anything more than indentation level for wrapped lines. No way to specify where I want wrapped method arguments to go (aligned with the start of the arguments on the previous line, thank you very much). No way to tell it where I want throws and extends clauses to go.

I even did some searching around via Google for third-party plugins that might make this possible, but the results are disappointing. After using NetBeans for a couple of days, I’m very motivated to use it, but this sort of thing is a deal-breaker. *sigh* Guess I’ll have to take another look at it in the coming years, to see whether such a basic function has been added. Unfortunately, I really don’t have the time to write one myself…

Technorati Tags: code, IDEs

I noticed yesterday that I cannot right-click within the Package Explorer view anymore. Hmm, what’s changed? Well, I did update Subclipse to 0.9.34 the other day…

Just for good measure, I installed the newest milestone of Eclipse (hey, I’ve been meaning to do this anyway). Then, I added in each of the add-on plugins I use one by one. After each one, I restarted and verified that the right-click worked. I got to the second plugin: Subclipse. Sure enough, bye-bye right-click. WTF?? So, I’m uninstalling and seeing if I can install an older version (like 0.9.32, which was working almost just fine before)…sure, I still have to use the svn command-line client, because actual server interaction via Subclipse makes you want to stab yourself in the eyes repeatedly with a newly sharpened pencil. But I’ve grown accustomed to this, and it works. At least I have (a) Diff vs. base revision, and (b) a right-click in Eclipse.

My question is: What the hell are these guys doing releasing a product which is apparently untested in any sane way??

sigh

There are two very interesting things about this project, IMO.

1. There are endless possibilities for spin-off projects, like using similar concepts to address different phases of the SDLC, for instance.
2. The implementation of Maven 2.0 has generated a ton of high-quality APIs for things like artifact (read: jar, etc.) resolution, SCM access, document translation, and object modelling. These APIs are easy to reuse outside of Maven, so they can be used to fuel [1] above.

For example, I’ve recently begun work on a new application launcher that will work similarly to Java WebStart. Except that it’ll use the Maven artifact repository to build up the classpath, and is primarily focused on command-line-driven applications. It has an advanced CLI parser that allows multiple usage profiles, with separate validation per profile. It can be booted in a variety of ways, including via Spring. There is more to come.

Originally, I’d hoped to make this code embeddable in any jar, such that it could turn that jar into an executable jar with all these bells and whistles. However, it looks like these features are a little too heavy to do that. So, I’ll have to settle for a small installed binary version, which can load applications from an XML application description that may live anywhere (access will be provided via maven-wagon, a transport layer).

More to come later.