Every SOAP integration is a snowflake

48:56 reading time


While the recipe for JSON-SOAP interfaces with CXF, Jackson, and Spark seems great on paper, it’s important to keep in mind that the majority of time spent working on such a SOAP integration is likely to be in configuration. Consider the scenario where you already have base classes to handle run-time configuration, client set-up, and service routes. Now it’s just a matter of setting up data structures to accept JSON inputs, and passing the data along to the SOAP port, right? Wrong.

Here we list some issues we’ve run into with SOAP integrations, and barebones solutions.

An endpoint with a self-signed certificate

Consider a development endpoint provided by a vendor that uses HTTPS but has a self-signed SSL certificate. Access to the endpoint is only available through a VPN between you and the vendor. In this circumstance, you probably don’t care too much about the validity of the development endpoint’s HTTPS certificate.

There are multiple solutions to this problem, but one is to simply import the certificate into an independent Java keystore. This way you don’t have to “ignore” SSL certificate verification: you’ve explicitly trusted it in your application.

First, create a JKS-formatted keystore from the PEM data retrieved from the openssl s_client command:

echo | \
openssl s_client -connect dev.my.random.server:443 2>&1 | \
sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | \
keytool -importcert \
  -noprompt -trustcacerts -storepass changeme \
  -keystore ./keystore -alias dev_https

Next, instruct your Maven project to use that keystore. In your pom.xml, you might have an exec-maven-plugin as a build plugin dependency. (If you don’t, it’s awesome; use it.)

Be sure to include the configuration:

<configuration>
  <mainClass>com.me.myapp.ClassWithMainMethod</mainClass>
  <systemProperties>
    <systemProperty>
      <key>javax.net.ssl.trustStore</key>
      <value>${env.KEYSTORE}</value>
    </systemProperty>
    <systemProperty>
      <key>javax.net.ssl.trustStorePassword</key>
      <value>changeme</value>
    </systemProperty>
  </systemProperties>
</configuration>

This will instruct the maven exec:java target to set the respective system properties that dictate the default Java keystore location and password. Here, we’re using the shell environment’s KEYSTORE variable to set the path of the keystore we generated above. Additionally, in a very insecure way, we’ve hardcoded our keystore password as “changeme”.

You could also use the -D flags to Java to set Java system properties, but, in my opinion, the pom-based configuration is more convenient.

HTTPS client authentication with “Apache-style” keys

Consider the situation where you’ve already integrated with a vendor through some other means, and you have the “traditional” pem and key files stored somewhere. Maybe other items in your stack still depend on those files. Let’s find a way to use our existing public and private keys without generating a new project-specific keystore.

There’s a great article on this already, but, a somewhat repeated code snippet can’t hurt anyone, right?

In this case, we use the great BouncyCastle library to help out.

The dependencies for our maven pom.xml resemble:

<dependency>
  <groupId>org.bouncycastle</groupId>
  <artifactId>bcprov-jdk15on</artifactId>
  <version>1.50</version>
</dependency>
<dependency>
  <groupId>org.bouncycastle</groupId>
  <artifactId>bcpkix-jdk15on</artifactId>
  <version>1.50</version>
</dependency>

With this, we have a pretty simple class to handle our public/private keys for SSL auth. Like above, let’s assume our keystore’s password is “changeme”. Import statements are omitted because they’re 30% of the file:

public class KeyReader {
    private static final char[] keystorePass = "changeme".toCharArray();

    public static SSLSocketFactory getSSLSocketFactory() throws Exception {
        return getSSLSocketFactory("/path/to/public-key.pem",
                                   "/path/to/private-key.key");
    }

    public static SSLSocketFactory getSSLSocketFactory(String pemPath, String keyPath) throws Exception {
        return new KeyReader().socketFactory(pemPath, keyPath);
    }

    protected Reader open(String fileName) throws Exception {
        return new InputStreamReader(
            new AutoCloseInputStream(
                Channels.newInputStream(
                    FileChannel.open(Paths.get(fileName)))));
    }

    protected Object loadPem(String path) throws Exception {
        return new PEMParser(open(path)).readObject();
    }

    protected KeyPair authKeyPair(String keyPath) throws Exception {
        KeyFactory kf = KeyFactory.getInstance("RSA");
        PEMKeyPair pem = (PEMKeyPair)loadPem(keyPath);
        PublicKey pub = kf.generatePublic(
            new X509EncodedKeySpec(pem.getPublicKeyInfo().getEncoded()));
        PrivateKey prv = kf.generatePrivate(
            new PKCS8EncodedKeySpec(pem.getPrivateKeyInfo().getEncoded()));
        return new KeyPair(pub, prv);
    }

    protected Certificate myCertificate(String certPath) throws Exception {
        return new JcaX509CertificateConverter()
                    .setProvider("BC")
                    .getCertificate((X509CertificateHolder)loadPem(certPath));
    }

    protected KeyStore getKeyStore(String pemPath, String keyPath) throws Exception {
        Key prv = authKeyPair(keyPath).getPrivate();
        Certificate[] certs = { myCertificate(pemPath) };
        KeyStore ks = KeyStore.getInstance("JKS");
        ks.load(null);
        ks.setKeyEntry("something", prv, keystorePass, certs);
        return ks;
    }

    protected KeyManager[] getKeyManagers(String pemPath, String keyPath) throws Exception {
        KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
        kmf.init(getKeyStore(pemPath, keyPath), keystorePass);
        return kmf.getKeyManagers();
    }

    protected SSLSocketFactory socketFactory(String pemPath, String keyPath) throws Exception {
        Security.addProvider(new BouncyCastleProvider());
        SSLContext ctx = SSLContext.getInstance("TLS");
        ctx.init(getKeyManagers(pemPath, keyPath), null, null);
        return ctx.getSocketFactory();
    }
}

Now, using this certificate loader is pretty simple in your main client class. Consider a method like:

public Client withSSLKeys() {
    HTTPConduit conduit = (HTTPConduit)this.client.getConduit();
    TLSClientParameters tls = conduit.getTlsClientParameters();
    try {
        if (tls == null) {
            tls = new TLSClientParameters();
        }
        tls.setSSLSocketFactory(KeyReader.getSSLSocketFactory());
        conduit.setTlsClientParameters(tls);
    } catch (Exception err) {
        err.printStackTrace();
    }
    return this;
}

When instantiating your Client instance in your Spark application, chain this method and be happy.

WSSE using Username/Password Authentication

This is one of the simpler “gotchas.” If your WSDL contains a WS-Policy that requires authentication using a username and password, CXF’s got you covered. (Sometimes WSDLs lack security policies even though the service expects them to be observed, and then you may have to implement your own interceptor.)

Assuming the best case scenario, maybe you’ll want to add something like this to your client class:

private Map<String, Object> requestContext(MyServicesPort port, String endpoint, String username, String password) {
    Map<String, Object> rc = ((BindingProvider)port).getRequestContext();
    rc.put("ws-security.username", username);
    rc.put("ws-security.password", password);
    rc.put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, endpoint);
    rc.put("com.sun.xml.internal.ws.request.timeout", 180000);  //maybe?
    rc.put("com.sun.xml.ws.request.timeout", 180000);
    rc.put("com.sun.xml.ws.connect.timeout", 60000);
    // ...
    return rc;
}

In this case, it’s assumed that you have the “port” (see a wsdl2java-generated sample client) and endpoint URL available, and that you’re just supplying a username and password from configuration located somewhere else. The “ws-security” values are actually constants that can be referenced explicity through the SecurityConstants class.

The other constants we’ve added to the request context are timeout values; this isn’t the only way or place to set timeouts in milliseconds; also have a look at HTTPClientPolicy.setConnectionTimeout() and setReceiveTimeout().

Unusual Security Policies

Consider the situation where you’ve violated a security policy required by a service because it doesn’t precisely match up with CXF’s bundled policy handlers.

One offender might be a service exposed through a BEA WebLogic server. In this case, you may have to produce your own interceptors.

A way to deal with this follows:

public class BeaWsse extends AbstractPolicyInterceptorProvider {
    private static final long serialVersionUID = (some long value);
    private static final QName QN = new QName(
      "http://www.bea.com/wls90/security/policy", "Identity");
    private static final Collection<QName> ASSERTION_TYPES;

    static {
        ASSERTION_TYPES = new ArrayList<QName>();
        ASSERTION_TYPES.add(QN);
    }

    public BeaWsse() {
        super(ASSERTION_TYPES);
        this.getOutInterceptors().add(new BeaWsseOutInterceptor());
    }
}

class BeaWsseOutInterceptor extends AbstractSoapInterceptor {
    // portions yanked from: org.apache.cxf.ws.security.wss4j.
    // {AbstractTokenInterceptor,UsernameTokenInterceptor}
    public BeaWsseOutInterceptor() {
        super(Phase.PREPARE_SEND);
    }

    protected Header securityHeader() {
        Document doc = DOMUtils.createDocument();
        Element el = doc.createElementNS(WSConstants.WSSE_NS, "wsse:Security");
        el.setAttributeNS(WSConstants.XMLNS_NS, "xmlns:wsse", WSConstants.WSSE_NS);
        SoapHeader sh = new SoapHeader(new QName(WSConstants.WSSE_NS, "Security"), el);
        sh.setMustUnderstand(true);
        return sh;
    }

    protected WSSecUsernameToken usernameToken(String userName, String password) {
        WSSecUsernameToken ut = new WSSecUsernameToken(WSSConfig.getNewInstance());
        ut.setPasswordType(WSConstants.PASSWORD_TEXT);
        ut.setUserInfo(userName, password);
        return ut;
    }

    protected WSSecUsernameToken usernameToken(Message msg) {
        String user = (String)msg.getContextualProperty(SecurityConstants.USERNAME);
        String pass = (String)msg.getContextualProperty(SecurityConstants.PASSWORD);
        return usernameToken(user, pass);
    }

    protected void addTokenToHeader(Header sh, WSSecUsernameToken tok) {
        Element el = (Element)sh.getObject();
        tok.prepare(el.getOwnerDocument());
        el.appendChild(tok.getUsernameTokenElement());
    }

    @Override
    public void handleMessage(SoapMessage msg) throws Fault {
        Header sh = securityHeader();
        addTokenToHeader(sh, usernameToken(msg));
        msg.getHeaders().add(sh);
    }
}

Now, add this interceptor to your cxf.xml: CXF’s Spring Framework-based mechanism for configuration.

First, ensure that you have the dependencies in your pom.xml to support loading the cxf.xml:

<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-context</artifactId>
  <version>4.0.6.RELEASE</version>
</dependency>

Then, in your main/resources/ directory, add a cxf.xml file that looks like:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:cxf="http://cxf.apache.org/core"
       xmlns:p="http://cxf.apache.org/policy"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://cxf.apache.org/core
             http://cxf.apache.org/schemas/core.xsd
             http://cxf.apache.org/policy
             http://cxf.apache.org/schemas/policy.xsd
             http://www.springframework.org/schema/beans
             http://www.springframework.org/schema/beans/spring-beans.xsd">
  <cxf:bus>
    <cxf:features>
      <p:policies ignoreUnknownAssertions="true"/>
    </cxf:features>
  </cxf:bus>
  <bean class="package.for.BeaWsse"/>
</beans>

Now every request will use your special BeaWsse interceptor to handle this particular security policy.

Adding SOAP Headers

Maybe it’s not enough to add your security policy interceptor. Maybe you need to include additional SOAP headers that repeat items that were already submitted elsewhere.

Here’s an example of an “Out” interceptor that adds various header elements:

public class RequestHeader extends AbstractSoapInterceptor {
    public RequestHeader() {
        super(Phase.PREPARE_SEND);
    }

    protected Header generate(String uuid) throws JAXBException, DatatypeConfigurationException {
        JAXBElement<?> je = new ObjectFactory()  // (generated class)
            .createSomeCustomHeader(new SomeCustomHeaderType()
                .withDateTime(new XMLGregorianCalendar())
                .withUniqueId(uuid));
        return new SoapHeader(je.getName(),
                              je.getValue(),
                              new JAXBDataBinding(je.getDeclaredType()));
    }

    @Override
    public void handleMessage(SoapMessage msg) throws Fault {
        String id = (String)msg.getContextualProperty(MyConstants.ID);
        try {
            msg.getHeaders().add(generate(id));
        } catch (Exception err) {
            err.printStackTrace();
        }
    }

Our Client instance on creation would have added the appropriate value to the “requestContext” (see above) using something like:

cx.put(MyConstants.ID, UUID.randomUUID().toString());

You can then add RequestHeader as an “out” interceptor, either via configuration, or in code, using something like:

public Client withOutInterceptor(Interceptor<? extends Message> cept) {
    this.client.getOutInterceptors().add(cept);
    return this;
}
// ...
this.withOutInterceptor(new RequestHeader());

Why do dates have to be so difficult?

Every time I’ve seen a Date type in a WSDL, it’s interpreted as a XMLGregorianCalendar in the generated client. That extra conversion can be an annoyance. There’s probably a way to change that, but right now, I just want to be able to pass a date format and its input JSON string value, and forget about it. Conveniently, we have a snippet for that:

public class DateHelper {
    public static XMLGregorianCalendar newXMLGregorianCalendar(String format, String time) throws DatatypeConfigurationException, ParseException {
        // format examples: http://docs.oracle.com/javase/6/docs/api/java/text/SimpleDateFormat.html
        if (StringUtils.isBlank(time)) {
            return null;
        }

        java.util.Date dt = new SimpleDateFormat(format, Locale.US).parse(time);
        GregorianCalendar cal = new GregorianCalendar();
        cal.setTime(dt);
        return DatatypeFactory.newInstance().newXMLGregorianCalendar(
            cal.get(Calendar.YEAR),
            cal.get(Calendar.MONTH) + 1,  // months in Calendar start with zero
            cal.get(Calendar.DAY_OF_MONTH),
            dt.getHours(),
            dt.getMinutes(),
            dt.getSeconds(),
            DatatypeConstants.FIELD_UNDEFINED,
            DatatypeConstants.FIELD_UNDEFINED);
    }
}

What about X.509 Certificates and WS-Security?

Thanks! I’ve been itching to answer that one. First, let’s assume that you just need to sign each payload, and you have a private and a public key for this purpose. They’ve been delivered to you as separate files in PEM format.

As we’ve seen earlier, not using a “keystore” for containing public and private keys takes some effort. Let’s just build a keystore.

cat private.key public.cer > combined.pem
openssl pkcs12 \
  -export \
  -in combined.pem \
  -out my_special_keystore.pkcs12 \
  -name keystoreAliasName \
  -noiter \
  -nomaciter

Just for kicks, let’s make our keystore’s password “soverysafe”.

Notice we’ve avoided using Java’s keytool and haven’t made a jks-formatted file. We don’t need to!

Before we dig any deeper, be sure you have the appropriate dependency in your pom.xml:

<dependency>
  <groupId>org.apache.cxf</groupId>
  <artifactId>cxf-rt-rs-security-xml</artifactId>
  <version>${cxf.version}</version>
</dependency>

If your WSDL defines a WS-SecurityPolicy that requires this type of authentication, CXF makes your life easy: use the SecurityConstants.SIGNATURE_PROPERTIES constant to locate the appopriate properties file definining Merlin settings (shown below). Then, just add this to your port’s request context. Awesome!

Lucky you, your WSDL lacks any security policy at all, even though the service expects a signed payload adhering to the WS-Security standard. You’ll have to manually initialize another interceptor to inject the relevant parts. Fortunately, it’s not as bad as the BeaWsse example; the tooling is already there for you. Copied from CXF’s own documentation, you’ll need something like this:

Map<String,Object> outProps = new HashMap<String,Object>();
outProps.put(WSHandlerConstants.ACTION, WSHandlerConstants.SIGNATURE);
outProps.put(WSHandlerConstants.USER, "keystoreAliasName");
outProps.put(WSHandlerConstants.PW_CALLBACK_CLASS, KeystorePasswordHandler.class.getName());
outProps.put(WSHandlerConstants.SIG_PROP_FILE, "wss4j.properties");

myClient.withOutInterceptor(new WSS4JOutInterceptor(outProps));

Where keystoreAliasName is the name we specified above when creating our keystore.

Now we need a properties file containing Merlin configurations, and we need a class to handle passwords associated with the keystore. Let’s assume that properties file is at src/main/java/resources/wss4j.properties, and that my_special_keystore.pkcs12 is located in the same place. Keep in mind that this keystore might be entirely separate from the keystore used to house unsigned HTTPS certificates (though it doesn’t need to be), and that in real life you’d likely place this keystore somewhere safe and outside your project.

An example of what might be included in the properties file:

org.apache.ws.security.crypto.provider=org.apache.ws.security.components.crypto.Merlin
org.apache.ws.security.crypto.merlin.keystore.file=my_special_keystore.pkcs12
org.apache.ws.security.crypto.merlin.keystore.type=pkcs12
org.apache.ws.security.crypto.merlin.keystore.alias=keystoreAliasName
org.apache.ws.security.crypto.merlin.keystore.password=soverysafe
org.apache.ws.security.crypto.merlin.keystore.private.password=soverysafe

We also shouldn’t forget to modify our pom.xml to note that special items in the resources folder should be copied to the compiled environment.

Easy way—include a maven plugin:

  <build>
    ...
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-resources-plugin</artifactId>
        <version>2.6</version>
      </plugin>

Careful! The maven-resources-plugin can “filter” resources—that is, it can dynamically modify them, substituting variables, etc. You won’t want your pkcs12 file “filtered”; if maven modifies it, it’ll corrupt it.

The resources section then would resemble:

<resource>
  <directory>${basedir}/src/main/resources</directory>
  <filtering>true</filtering>
  <includes>
    <include>**/*.xml</include>
    <include>**/*.properties</include>
  </includes>
</resource>
<resource>
  <directory>${basedir}/src/main/resources</directory>
  <filtering>false</filtering>
  <includes>
    <include>**/*.pkcs12</include>
  </includes>
</resource>

Finally, that KeystorePasswordHandler class referenced by the configuration. We’ve done something really awesome here: the password for our keystore is “soverysafe”, the password for our private key within that keystore is also “soverysafe”, and we’ve hardcoded it into our handler, like we have with our configuration file.

public class KeystorePasswordHandler implements CallbackHandler {
    public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {
        WSPasswordCallback pc = (WSPasswordCallback) callbacks[0];
        pc.setPassword("soverysafe");
    }
}

The configuration file stores the password for the keystore, and the callback class delivers passphrases on keys stored within the keystore, if they’re also password protected. You could also store a private key passphrase in the configuration file using the org.apache.ws.security.crypto.merlin.keystore.private.password property; the callback class is useful if you have multiple private keys in your keystore that have different passphrases … and it gives you more control over how your passwords are retrieved (i.e., not hard-coded).

In this case, since we only have one private key in our keystore, we can opt for just using the configuration file, and being very, very safe.

And how do I test any of this?

Earlier we touched on testing tools. Sometimes it can be helpful to actually make assertions against the generated XML messages—test that your header-modifying interceptor really modifies the headers in the right way, etc. Here’s a recipe to help with that: create an interceptor that replaces the actual message-sending interceptor, and capture the data for assertions with XmlMatchers (or whatever).

Your interceptor might resemble:

public class FauxOutInterceptor extends AbstractSoapInterceptor {
    private ByteArrayOutputStream writeBack;

    public FauxOutInterceptor(ByteArrayOutputStream writeBack) {
        super(Phase.PREPARE_SEND_ENDING);
        this.writeBack = writeBack;
    }

    private void removeInterceptor(InterceptorChain chain, Class interceptorClass) {
        ListIterator<Interceptor<? extends Message>> iter = chain.getIterator();
        while (iter.hasNext()) {
            PhaseInterceptor<? extends Message> cept = (PhaseInterceptor<? extends Message>)iter.next();
            if (interceptorClass.getName().equals(cept.getId())) {
                chain.remove(cept);
                break;
            }
        }
    }

    @Override
    public void handleMessage(SoapMessage msg) throws Fault {
        removeInterceptor(msg.getInterceptorChain(), MessageSenderEndingInterceptor.class);
        try {
            SOAPMessage soap = msg.getContent(SOAPMessage.class);
            soap.writeTo(writeBack);
        } catch (Exception err) { }
    }
}

In a client-testing class include a ByteArrayOutputStream variable, and pass it to your FauxOutInterceptor. Add the FauxOutInterceptor to your initiated Client’s interceptor chain, and invoke a client operation. Make sure to configure the Client to time out at zero seconds, and be ready to catch a bunch of SOAPFaultExceptions. The nice thing about this strategy, though, is that it runs through the entire chain. Your Hamcrest assertions can then resemble:

// request is the instance's ByteArrayOutputStream, and
// ns is a SimpleNamespaceContext configured for various bindings.

assertThat(the(request.toString()), hasXPath("//x:Foo//y:Bar[text()]", ns));

Simple Object Access Protocol. Have fun, and good luck!


30e9354fde8b14f9d85628775b7c1bd6

Ian Melnick
Senior Software Engineer