Netty Conflict with TwitterServer and Play WS

TL;DR when an app runs from the IDE but not when invoked by another means, such as command line, it could be a dependency conflict where classpath ordering matters. AKA, seemingly inexplicable errors are often explicable, but only after some headachy troubleshooting

I might be the first person to run TwitterServer with akka and Play WS. There's really no reason for this combination of frameworks. I first started working with Play WS (built on AsyncHttpClient) because it's super simple (much lass convoluted than akka-http). See, super easy:

val updateIp = client
 .url(s"https://app.herokuapp.com/ip-address/${ip}")
 .withMethod("POST")
 .execute()

updateIp.onComplete {
 case Success(response) =>
   response.status match {
     case 200 =>
       log.info(s"Heroku ip address update returned ${response.body}")
     case _ =>
       log.warning(s"Heroku update ip address failed with status ${response.status}")
   }
 case Failure(e) =>
   log.warning(s"Heroku update ip address failed with exception ${e}")
}

Then I added akka and later tossed TwitterServer into the mix for the sweet stats. Here's a screenshot of the admin server UI that comes with TwitterServer

Admittedly this chart isn't too exciting but the stats provided by TwitterServer (actually from Finagle) and the included utilities (profiling etc) are exceedingly useful.

After I added TwitterServer, the app was running just fine in Intellij but I could not run from the command line. Every time I saw:

java.net.BindException: Failed to bind to 0.0.0.0/0.0.0.0:9999: Unable to create Channel from class class io.netty.channel.socket.nio.NioServerSocketChanne\

When I see this my first reaction is I left an instance running

netstat -lnp | grep 9990

but nope, so this made absolutely no sense.

Then, if I removed TwitterServer from the code but kept the dependency I got past the bind exception:

WARN  c.r.app.AppActor - Failed to get ip from heroku java.net.ConnectException: Unable to create Channel from class class io.netty.channel.socket.nio.NioSocketChannel

In this case there were conflicting dependencies with Netty. Specifically the Netty version required by TwitterServer and Play WS were not in agreement. When I was running from Intellij the Netty version that loaded first happened to work for both (Play WS and TwitterServer). But, the SBT JavaServerAppPackaging loaded the problematic/conflicting Netty classes first, which resulted in the weird and misleading Netty errors. I was able to solve the problem by simply shuffling the order of the dependencies in build.sbt.

Works:

libraryDependencies += "com.twitter" %% "twitter-server" % "1.25.0"

libraryDependencies += "com.twitter" % "finagle-stats_2.11" % "6.40.0"

libraryDependencies += "com.typesafe.play" %% "play-ws" % "2.5.4"

libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.4.16"

libraryDependencies += "com.typesafe.akka" %% "akka-http-core" % "10.0.0"

libraryDependencies += "com.typesafe.akka" %% "akka-http" % "10.0.0"

libraryDependencies += "com.typesafe.akka" %% "akka-http-testkit" % "10.0.0"

libraryDependencies += "com.typesafe.akka" %% "akka-http-spray-json" % "10.0.0"

libraryDependencies += "com.typesafe.akka" %% "akka-http-jackson" % "10.0.0"

libraryDependencies += "com.typesafe.akka" %% "akka-http-xml" % "10.0.0"

libraryDependencies += "com.typesafe.akka" %% "akka-slf4j" % "2.4.16"

libraryDependencies += "ch.qos.logback"    %  "logback-classic" % "1.1.3"

libraryDependencies += "com.typesafe.scala-logging" %% "scala-logging" % "3.1.0"

Fails:

libraryDependencies += "com.typesafe.play" %% "play-ws" % "2.5.4"

libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.4.16"

libraryDependencies += "com.typesafe.akka" %% "akka-http-core" % "10.0.0"

libraryDependencies += "com.typesafe.akka" %% "akka-http" % "10.0.0"

libraryDependencies += "com.typesafe.akka" %% "akka-http-testkit" % "10.0.0"

libraryDependencies += "com.typesafe.akka" %% "akka-http-spray-json" % "10.0.0"

libraryDependencies += "com.typesafe.akka" %% "akka-http-jackson" % "10.0.0"

libraryDependencies += "com.typesafe.akka" %% "akka-http-xml" % "10.0.0"

libraryDependencies += "com.typesafe.akka" %% "akka-slf4j" % "2.4.16"

libraryDependencies += "ch.qos.logback"    %  "logback-classic" % "1.1.3"

libraryDependencies += "com.typesafe.scala-logging" %% "scala-logging" % "3.1.0"

libraryDependencies += "com.twitter" %% "twitter-server" % "1.25.0"

libraryDependencies += "com.twitter" % "finagle-stats_2.11" % "6.40.0"

Another solution would be to ditch Play WS and use finagle-http instead. Or in the case where you want to use multiple versions of the same library you can load them in their on Classload or use something like OSGI which excels at solving this sort of issue.

A tool that is useful debugging these sorts of dependency errors is the sbt dependencyTree plugin. Example output:

| | | | +-io.netty:netty-codec-http:4.1.6.Final[[0m
| | | | | +-io.netty:netty-codec:4.1.6.Final[[0m
| | | | |   +-io.netty:netty-transport:4.1.6.Final[[0m
| | | | |     +-io.netty:netty-buffer:4.1.6.Final[[0m
| | | | |     | +-io.netty:netty-common:4.1.6.Final[[0m
| | | | |     | [[0m
| | | | |     +-io.netty:netty-resolver:4.1.6.Final[[0m
| | | | |       +-io.netty:netty-common:4.1.6.Final[[0m
| | | | |       [[0m
| | | | +-io.netty:netty-handler-proxy:4.1.6.Final[[0m
| | | | | +-io.netty:netty-codec-http:4.1.6.Final[[0m
| | | | | | +-io.netty:netty-codec:4.1.6.Final[[0m
| | | | | |   +-io.netty:netty-transport:4.1.6.Final[[0m
| | | | | |     +-io.netty:netty-buffer:4.1.6.Final[[0m
| | | | | |     | +-io.netty:netty-common:4.1.6.Final[[0m
| | | | | |     | [[0m
| | | | | |     +-io.netty:netty-resolver:4.1.6.Final[[0m
| | | | | |       +-io.netty:netty-common:4.1.6.Final[[0m
| | | | | |       [[0m

And finally a diff of the classpath produced by JavaServerAppPackaging (working version top) shows the netty 3.10 from TwitterServer picked up first while the non-working script picks up several play dependencies, ex $lib_dir/com.typesafe.play.play-netty-utils-2.5.4.jar.

Git Commit Maven Plugin

I'm not sure how I survived so long without this. As a developer, almost everyone has run into a situation where a bug is reported but the question is what version. For releases there's generally a tag that identifies the release and makes it easy to find the source code to begin the investigation. But, often, especially with snapshot builds, it may be less clear. This simple addition to the Maven pom.xml (below) creates a git.properties file in your build artifact. There are many properties you can include and I've excluded some that were not particularly useful for my purposes. The most important, in my opinion include:

git.dirty=true

git.commit.time=24.01.2017 @ 07\:08\:22 MST

git.branch=master

git.commit.id=79b91f49b53eab5fe96efd2a2d9438ea3e3a0d00

git.tags=

The git.dirty indicates there are uncommitted changes in git. This build should never be dirty since if it is the commit hash does not reflect what was built. I believe you can fail the build if dirty. With web apps it's easy to expose this information via an endpoint. Also a good practice is to print the contents of the file to a log on start-up. Big tip of the hat goes to Konrad Malawski (akka core developer) for developing this plugin.

<plugin>
   <groupId>pl.project13.maven</groupId>
   <artifactId>git-commit-id-plugin</artifactId>
   <version>2.2.1</version>
   <executions>
      <execution>
         <id>get-the-git-infos</id>
         <goals>
            <goal>revision</goal>
         </goals>
      </execution>
      <execution>
         <id>validate-the-git-infos</id>
         <goals>
            <goal>revision</goal>
         </goals>
         <phase>package</phase>
      </execution>
   </executions>

   <configuration>
      <generateGitPropertiesFile>true</generateGitPropertiesFile>
      <generateGitPropertiesFilename>${project.build.outputDirectory}/git.properties</generateGitPropertiesFilename>
      <excludeProperties>
         <excludeProperty>git.build.*</excludeProperty>
         <excludeProperty>git.commit.user.email</excludeProperty>
         <excludeProperty>git.commit.message.*</excludeProperty>
         <excludeProperty>git.commit.id.describe</excludeProperty>
         <excludeProperty>git.commit.user.name</excludeProperty>
         <excludeProperty>git.remote.origin.url</excludeProperty>
         <excludeProperty>git.commit.id.abbrev</excludeProperty>
         <excludeProperty>git.closest.tag.name</excludeProperty>
         <excludeProperty>git.closest.tag.commit.count</excludeProperty>
         <excludeProperty>git.remote.origin.url</excludeProperty>
      </excludeProperties>
   </configuration>
</plugin>

Google Oauth2 Redux

I'm just going to leave this here. It's nothing particularly interesting but there was some googling and what not in arriving at the proper configuration and usage.

I had been using google oauth in one of my projects and that worked until I added a dependency that conflicted with google's oauth dependencies and gave me the following error:

java.lang.NoSuchMethodError: com.google.common.primitives.UnsignedInteger.asUnsigned(I)Lcom/google/common/primitives/UnsignedInteger;at com.google.api.client.util.Data.<clinit>(Data.java:81)at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:502)at com.google.api.client.json.JsonParser.parse(JsonParser.java:289)

I upgraded to the latest version of the google oauth libraries (1.22.0), however some aspects of the api had changed and the proper usage wasn't exactly clear. After some digging around I finally I got it working

First the maven dependencies:

<dependency>
   <groupId>com.google.api-client</groupId>
   <artifactId>google-api-client</artifactId>
   <version>1.22.0</version>
</dependency>
<dependency>
   <groupId>com.google.apis</groupId>
   <artifactId>google-api-services-oauth2</artifactId>
   <version>v2-rev124-1.22.0</version>
</dependency>
<dependency>
   <groupId>com.google.oauth-client</groupId>
   <artifactId>google-oauth-client</artifactId>
   <version>1.22.0</version>
</dependency>
<dependency>
   <groupId>com.google.oauth-client</groupId>
   <artifactId>google-oauth-client-servlet</artifactId>
   <version>1.22.0</version>
   <exclusions>
      <exclusion>
         <groupId>javax.servlet</groupId>
         <artifactId>servlet-api</artifactId>
      </exclusion>
   </exclusions>
</dependency>
<dependency>
   <groupId>com.google.oauth-client</groupId>
   <artifactId>google-oauth-client-jetty</artifactId>
   <version>1.22.0</version>
   <exclusions>
      <exclusion>  <!-- I'm explicitly including jetty so don't need this -->         <groupId>org.mortbay.jetty</groupId>
         <artifactId>jetty</artifactId>
      </exclusion>
   </exclusions>
</dependency>

And my Jetty deps. Any other servlet container should work too

<dependency>
   <groupId>org.eclipse.jetty</groupId>
   <artifactId>jetty-server</artifactId>
   <version>8.1.8.v20121106</version>
</dependency>
<dependency>
   <groupId>org.eclipse.jetty</groupId>
   <artifactId>jetty-servlet</artifactId>
   <version>8.1.8.v20121106</version>
</dependency>
<dependency>
   <groupId>org.eclipse.jetty</groupId>
   <artifactId>jetty-util</artifactId>
   <version>8.1.8.v20121106</version>
</dependency>

The first time I started the app I got the following error:

java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s signer information does not match signer information of other classes in the same package

The occurs because the servlet-api conflicted with the Jetty8 servlet api. To fix I added an exclusion to the  com.google.oauth-client dependency.

There are two servlets needed. One initiates the oauth process by redirecting to google, while the other receives the redirect back from google with auth tokens. Since they both share code I created a separate class to avoid duplicating code

import java.io.IOException;
import java.util.Arrays;
import java.util.List;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import com.google.api.client.extensions.servlet.auth.oauth2.AbstractAuthorizationCodeServlet;
import org.apache.log4j.Logger;

import com.google.api.client.auth.oauth2.AuthorizationCodeFlow;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;


// some doc https://developers.google.com/api-client-library/java/google-api-java-client/oauth2public class Oauth2Servlet extends AbstractAuthorizationCodeServlet {

   private final Logger log = Logger.getLogger(this.getClass());
   
   public final static HttpTransport HTTP_TRANSPORT = new NetHttpTransport();

   public final static JsonFactory JSON_FACTORY = new JacksonFactory();

   public final static List<String> SCOPES = Arrays.asList("https://www.googleapis.com/auth/userinfo.profile",
         "https://www.googleapis.com/auth/userinfo.email");

   @Override   protected void doGet(HttpServletRequest request, HttpServletResponse response)
         throws IOException {
      log.debug("Auth successful");
      response.sendRedirect(OauthCommon.SERVLET_CONTEXT + "/");
   }

   @Override   protected String getRedirectUri(HttpServletRequest req) throws ServletException, IOException {
      return OAUTH_COMMON.getRedirectUri(req);
   }

   @Override   protected AuthorizationCodeFlow initializeFlow() throws IOException {
      return OAUTH_COMMON.initializeFlow();
   }

   @Override   protected String getUserId(HttpServletRequest req) throws ServletException, IOException {
      return OAUTH_COMMON.getUserId(req);
   }
}

Here's the callback servlet. When onSuccess is called we have a oauth token and can get make calls to oauth endpoints to get user info. In this case I've requested the userinfo and profile scopes so I can get basic user info simply by calling oauth2.userinfo().get().execute()

import java.io.IOException;
import java.util.Date;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import com.google.api.services.oauth2.model.Userinfoplus;
import org.apache.log4j.Logger;

import com.google.api.client.auth.oauth2.AuthorizationCodeFlow;
import com.google.api.client.auth.oauth2.AuthorizationCodeResponseUrl;
import com.google.api.client.auth.oauth2.Credential;
import com.google.api.client.extensions.servlet.auth.oauth2.AbstractAuthorizationCodeCallbackServlet;


import com.google.api.services.oauth2.Oauth2;

public class Oauth2ServletCallback extends AbstractAuthorizationCodeCallbackServlet {

   private final Logger log = Logger.getLogger(this.getClass());

   @Override   protected void onSuccess(HttpServletRequest req, HttpServletResponse resp, Credential credential) throws ServletException, IOException {
      log.debug("Callback success:  token: " + credential.getAccessToken() + ", refresh token " + credential.getRefreshToken() + ", expires " + new Date(credential.getExpirationTimeMilliseconds()));

      Oauth2 oauth2 = new Oauth2.Builder(Oauth2Servlet.HTTP_TRANSPORT, Oauth2Servlet.JSON_FACTORY, credential).setApplicationName("").build();

      Userinfoplus userinfo = null;
      
      try {
         userinfo = oauth2.userinfo().get().execute();

         log.debug("User email is " + userinfo.getEmail() + ", name is " + userinfo.getName());
         
          req.getSession().setAttribute("AUTH", credential);
          req.getSession().setAttribute("USER", userinfo.getName());
          req.getSession().setAttribute("EMAIL", userinfo.getEmail());

          // redirect back to our app          resp.sendRedirect(OauthCommon.SERVLET_CONTEXT + "/");
          return;
      } catch (Exception e) {
         log.warn("Unable to get userinfo", e);
      }
   }

   @Override   protected void onError(HttpServletRequest req, HttpServletResponse resp, AuthorizationCodeResponseUrl errorResponse) throws ServletException, IOException {
      throw new RuntimeException("grrrrrrrrrr: " + errorResponse);
   }

   @Override   protected String getRedirectUri(HttpServletRequest req) throws ServletException, IOException {
      return OAUTH_COMMON.getRedirectUri(req);
   }

   @Override   protected AuthorizationCodeFlow initializeFlow() throws IOException {
      return OAUTH_COMMON.initializeFlow();
   }

   @Override   protected String getUserId(HttpServletRequest req) throws ServletException, IOException {
      return OAUTH_COMMON.getUserId(req);
   }
}

And lastly here's the shared code

import java.io.IOException;
import java.util.UUID;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;

import com.google.api.client.auth.oauth2.AuthorizationCodeFlow;
import com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.http.GenericUrl;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.client.util.store.MemoryDataStoreFactory;
import com.google.api.services.oauth2.Oauth2;
import com.google.api.services.oauth2.model.Userinfoplus;
import org.apache.log4j.Logger;

public class OauthCommon {

   private final Logger log = Logger.getLogger(this.getClass());
public final static OauthCommon OAUTH_COMMON = new OauthCommon();

   public final static String SERVLET_CONTEXT = "/app";
   public final static String OAUTH_CALLBACK_PATH = "/oauthcallback";
   public final static String OAUTH_PATH = "/oauth";
   public final static String CALL_BACK_URI = SERVLET_CONTEXT + OAUTH_CALLBACK_PATH;
   public final static String OAUTH_URI = SERVLET_CONTEXT + OAUTH_PATH;

   protected String getRedirectUri(HttpServletRequest req) throws ServletException, IOException {
      GenericUrl url = new GenericUrl(req.getRequestURL().toString());
      url.setRawPath(CALL_BACK_URI);
      return url.build();
   }

   public AuthorizationCodeFlow initializeFlow() throws IOException {
      return new GoogleAuthorizationCodeFlow.Builder(
            new NetHttpTransport(), JacksonFactory.getDefaultInstance(),
            "YOUR CLIENT ID GOES HERE",
            "YOUR SECRET GOES HERE",
            Oauth2Servlet.SCOPES      ).setDataStoreFactory(new MemoryDataStoreFactory()).setAccessType("offline").build();
   }
   
   public String getUserId(HttpServletRequest req) throws ServletException, IOException {
      String id = req.getSession().getId();
      log.debug("Session id is " + id);
      return id;
   }
}

If you acquired a auth token by another means and wanted to use the google oauth library to retrieve data, you could do like so

      GoogleCredential credential = new GoogleCredential().setAccessToken(accessToken);
      Oauth2 oauth2 = new Oauth2.Builder(new NetHttpTransport(), new JacksonFactory(), credential).setApplicationName(
            "Oauth2").build();
      Userinfoplus userinfo = oauth2.userinfo().get().execute();

I'm using embedded Jetty with a self-signed cert and configuring the servlets as follows:

SslContextFactory sslContextFactory = new SslContextFactory("../keystore");
sslContextFactory.setKeyStorePassword("supersecret");
SslSelectChannelConnector selectChannelConnector = new SslSelectChannelConnector(sslContextFactory);
selectChannelConnector.setPort(8443);
server.setConnectors(new Connector[] { selectChannelConnector });

ServletContextHandler context = new ServletContextHandler(ServletContextHandler.SESSIONS);
context.setContextPath(OauthCommon.SERVLET_CONTEXT);
server.setHandler(context);

context.addServlet(new ServletHolder(new MyAppServlet()),"/");
context.addServlet(new ServletHolder(new Oauth2Servlet()), OauthCommon.OAUTH_PATH);
context.addServlet(new ServletHolder(new Oauth2ServletCallback()), OauthCommon.OAUTH_CALLBACK_PATH);
context.addServlet(new ServletHolder(new LogoutServlet()), "/logout");

log.debug("Started Jetty");

server.start();
server.join();