NTLMv2 authentication from Java: A developer’s odyssey

A project in the works here at Vodori involves making a set of SOAP web service calls to an external system. This is generally a pretty routine exercise: set up the correct client code from the WSDL, integrate some kind of delivery mechanism, and then make the calls. At least, it’s routine when it isn’t hitting one roadblock after another.

Getting started: Axis to Axis2 to Spring-WS

For our purposes, we started out using Apache Axis since it was a pre-existing piece of our platform. We used the existing WSDL2JAVA process to generate the client and connectivity code.

However, when we tried to actually connect to the system, we received a 401 response—our credentials were denied for failing an NTLM authentication. Digging around, I determined that the most likely culprit was a version mismatch between the NTLM support in Axis and the latest NTLMv2 support by the web service server.

We tried switching out to the newer Axis2 project, which boasted better NTLM support. Unfortunately, a change in the WSDL2JAVA process consolidated all of the code into a massive 11 MB, 226k-line Java source file. Including that file in our project ground IntelliJ to a halt, and it started throwing OutOfMemoryException errors whenever it would try to compile.

At this point, we switched to JAXB2’s xjc jar to recreate the client files and Spring’s Spring-WS package to handle the transport. This provided a simpler code setup and much greater visibility into how the calls were being made. Since we’re living in 2012 and not 2006, we probably should have started here in the first place.

Programmer vs. programming, stage 1: The die is cast.

More problems

With the simpler setup in place, we were able to isolate the source of our troubles: Apache’s HTTPClient. This library is pretty rock-solid and has been around for a while, and everyone uses it. It is the default (i.e., only) option for Axis, Axis2, and Spring-WS’s latest release.

One drawback: the library doesn’t support the latest authentication schemes, and Apache has since replaced it with the HTTPComponents project and its own HTTPClient class. A major side effect of this change was a wholesale break from the old 3.X HTTPClient codebase and package structure, precluding its use as a drop-in replacement.

Of course, we aren’t the only people encountering such snags. The ubiquity of these tools within the Java and Spring ecosystems has generated plenty of discussion and advice to draw upon. Looking at the Spring-WS site and its JIRA, I came across issue SWS-563, which is a request for exactly what I needed.

The bad news is, it won’t see the light of day until 2.1 (or 2.1M1 for now), and Spring hasn’t yet published that version’s release calendar. The good news is, attached to SWS-563 are the three files updated by the Spring team to solve this issue. Downloading them and plugging them into my local codebase overwrote them in the Classpath, thus enabling me (for now) to use the most recent HTTPClient code.

Believing I had solved the issue, I ran my unit tests once more to validate the connection.

They failed.

Programmer vs. programming, stage 2: This time, it’s personal.

Fourth Eighth time’s the charm

Being stubborn—and a geek—I decided to take a deeper look at the authentication messages. I logged out the actual NTLM handshake messages, ran them through a Base64Decoder, and analyzed the structure.

I found that part of the NTLMv2 handshake, our Type-3 message, was failing due to incorrect formatting. I chalked this up to the HTTPClient still not working correctly, so I went back to the Apache site and noticed this message in their NTLM guide:

“HttpClient as of version 4.1 supports NTLMv1 and NTLMv2 authentication protocols out of the box using a custom authentication engine. However, there are still known compatibility issues with newer Microsoft products as the default NTLM engine implementation is still relatively new.”

Luckily, they include some sample code that uses the Samba team’s JCIFS implementation as their NTLM engine. Pulling down that code, creating the relevant classes, and wiring it in was easier than I expected. With all of these components in place—JAXB2 2.X generated files, Spring-WS transport code, the Spring 2.1M1 files, HTTPComponents, and JCIFS—the web services connected successfully.