Don't Await when using AsyncFlatSpec in ScalaTest!

There is a subtle behavior to be aware of if you use the scala.concurrent Await in an AsyncFlatSpec. The Scaladoc of Future.apply shows that it requires an implicit ExecutionContext

def apply[T](body: =>T)(implicit execctx: ExecutionContext)

The problem is ScalaTest will provide one for you and it is single-threaded, so it will block on Await.

For example the following never gets past Await

val f = Future(1)
Await.result(f, Duration.Inf)

This can be solved with

val f = Future(1)(aDifferentExecutionContext)
Await.result(f, Duration.Inf)

But the better approach is not to use Await at all in a AsyncFlatSpec because when using
AsyncFlatSpec you can return Future[Assertion], allowing ScalaTest will resolve the Future.

Google Drive API Quickstart in Scala

I'm working on a Scala app that needs to upload files to google drive. While google as a company hasn't exactly embraced Scala, it does use Java extensively and it's generally easy to interop with Scala to Java. So, in this post I sketch out a simple Scala wrapper that works with the Java google drive client library. I did find a Scala google drive library on github but it was not in maven central, so I was not able to use it directly, however I did borrow some of its concepts. Also, the gradle quickstart commands failed on my machine. Fortunately gradle is not required and we can just as easily use maven or in the case of scala, sbt.

To get started, first you need to create an api project https://console.cloud.google.com/apis/dashboard

Then click the enable apis and enable drive api. To be clear, the google account you are using here is your developer account, which may not be the same google account that owns the google drive to be accessed.

https://console.cloud.google.com/apis/api/drive.googleapis.com/overview

Then go to credentials -> create credentials and select oauth client id

I selected "other" since I'm building a daemon app

Download the file it generates: client_secret_<somenumber>.json

Create a new Scala/sbt project in Intellij. Edit build.sbt

name := "name"version := "1.0"scalaVersion := "2.11.8"

libraryDependencies += "ch.qos.logback"    %  "logback-classic" % "1.1.3"
libraryDependencies += "com.google.api-client" % "google-api-client" % "1.22.0"

libraryDependencies += "com.google.oauth-client" % "google-oauth-client-jetty" % "1.22.0"

libraryDependencies += "com.google.apis" % "google-api-services-drive" % "v3-rev83-1.22.0"

Here's my Scala client, adapted from the Java Quickstart At some point I'll get this on github, once I clean it up.

import java.io.{File, InputStreamReader}
import java.util

import com.google.api.client.auth.oauth2.Credential
import com.google.api.client.extensions.java6.auth.oauth2.AuthorizationCodeInstalledApp
import com.google.api.client.extensions.jetty.auth.oauth2.LocalServerReceiver
import com.google.api.client.googleapis.auth.oauth2.{GoogleAuthorizationCodeFlow, GoogleClientSecrets}
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport
import com.google.api.client.http.FileContent
import com.google.api.client.json.jackson2.JacksonFactory
import com.google.api.client.util.store.FileDataStoreFactory
import com.google.api.services.drive.model.{File => DriveFile}
import com.google.api.services.drive.{Drive, DriveScopes}
import org.slf4j.LoggerFactory

import scala.collection.JavaConverters._

object GoogleDriveClient {

  def main(args: Array[String]): Unit = {
    // first time setup    

GoogleDriveClient("my drive app", "drive-oauth-credentials", authorizationFlow = true)
  }

  def apply(appName: String, credentialsPath: String, authorizationFlow: Boolean = false) = {
    val credentialsFile = new File(credentialsPath)

    if (!authorizationFlow) {
      // on server we obviously won't be able to do the oauth auth with browser -- app needs to be deployed with creds      

if (!credentialsFile.exists) {
        throw new RuntimeException(s"Credentials does not exist ${credentialsFile.getAbsolutePath}")
      }
    }

    new GoogleDriveClient(appName, credentialsFile)
  }
}

class GoogleDriveClient(appName: String, credentialsFile: File) {

  val log = LoggerFactory.getLogger(GoogleDriveClient.getClass)

  /** Global instance of the JSON factory. */  

private val JSON_FACTORY = JacksonFactory.getDefaultInstance  

/** Global instance of the HTTP transport. */  

val HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport()
  val DATA_STORE_FACTORY = new FileDataStoreFactory(credentialsFile)

  private val SCOPES = util.Arrays.asList(DriveScopes.DRIVE)

  private[this] def authorize: Credential = {
val in = GoogleDriveClient.getClass.getResourceAsStream("/client_secret.json")
    val clientSecrets = GoogleClientSecrets.load(JSON_FACTORY, new InputStreamReader(in))

    // Build flow and trigger user authorization request.    val flow = new GoogleAuthorizationCodeFlow.Builder(
      HTTP_TRANSPORT,
      JSON_FACTORY,
      clientSecrets,
      SCOPES)
      .setDataStoreFactory(
        DATA_STORE_FACTORY      ).setAccessType("offline").build // must set offline of you wont get an refreshToken and the token will only work for a few hours or so
    val credential: Credential = new AuthorizationCodeInstalledApp(flow, new LocalServerReceiver()).authorize("user")
    log.info("Credentials saved to " + credentialsFile.getAbsolutePath)
    credential
  }

  val drive = {
    val credential = authorize

    new Drive.Builder(
      HTTP_TRANSPORT, JSON_FACTORY, credential)
      .setApplicationName(appName)
      .build()
  }

  val root = drive.files.get("root").execute

  def pregenerateIds(numOfIds: Int): Seq[String] = {
    drive.files().generateIds().setSpace("drive").setCount(numOfIds).execute().getIds.asScala.toSeq
  }

  def createFolder(name: String, parent: Option[String]): DriveFile = {
    val fileMetadata = new DriveFile()
    fileMetadata.setName(name)

    parent.foreach(p => fileMetadata.setParents(Seq(p).asJava))

    fileMetadata.setMimeType("application/vnd.google-apps.folder")

    drive.files().create(fileMetadata)
      .setFields("id")
      .execute()
  }

  def uploadFile(localFile: File, parentDriveOption: Option[DriveFile], idOption: Option[String], checkIfExists: Boolean = false): DriveFile = {
    val fileType = "image/jpeg"
    def upload(): DriveFile = {
      val file = new DriveFile()
      file.setName(localFile.getName)

      parentDriveOption.foreach(p => file.setParents(List(p.getId).asJava))
      idOption.foreach(id => file.setId(id))

      val mediaContent = new FileContent(fileType, localFile)

      val result: DriveFile = drive.files.create(file, mediaContent).setFields("id").execute()

      log.debug("upload result is " + result.getId)

      result
    }

    if (checkIfExists) {
      val existing = getFileByParentAndName(parentDriveOption.get, localFile.getName)

      if (existing.isDefined) {
        log.info("File already exists in google drive")
        existing.get
      } else {
        upload()
      }
    } else {
      upload()
    }
  }

  def getFilesByParent(parent: DriveFile): List[DriveFile] = {
    val request: Drive#Files#List = drive.files.list.setQ("'%s' in parents and trashed = false".format(parent.getId))
    request.execute.getFiles.asScala.toList
  }

  def getFileByParentAndName(parent: DriveFile, name: String): Option[DriveFile] = {
    val request = drive.files.list.setQ("'%s' in parents and name = '%s' and trashed = false".format(parent.getId, name))
    request.execute.getFiles.asScala.toList.headOption
  }

  // unwind the path, and find each part by parent, starting with the root folder  // e.g. /raspi/camera1/files will return the [files], if it exists  

def findFileByPath(path: String): Option[DriveFile] = {
    def findFileByPath(parts: List[String], parent: DriveFile): Option[DriveFile] = {
      parts match {
        case Nil =>
          Some(parent)
        case head :: Nil =>
          getFileByParentAndName(parent, head)
        case head :: tail =>
          getFileByParentAndName(parent, head).map { child =>
              findFileByPath(parts = tail, parent = child)
          }.getOrElse(None)

      }
    }

    findFileByPath(path.split("/").toList, root)
  }
}

Create a src/main/resources folder and drop in your client_secret.json, downloaded from the api console. Make sure it is named client_secret.json, as that is what the app expects:

val in = GoogleDriveClient.getClass.getResourceAsStream("/client_secret.json")

Run the class and it will open a browser and send you to Google to select an account. This account does not need to be the same account has account that was used to generate the api key. This is the account to access google drive. Once this completes it will create a directory containing a single file "StoredCredential. From not on, as long as it finds this file you will not be prompted for authorization. Notice I asked for scope  private val SCOPES = util.Arrays.asList(DriveScopes.DRIVE)
which is basically sudo. You may want to ask for a more restrictive scope, e.g read-only. This should be a arg to the class. Secondly I asked for offline access

setAccessType("offline")

If I did not then this app would only work for 4 hours or however the access token last before expiration. With offline access, the api can ask for a new token with the refresh token.

Now we're in business and can make api calls. First it's important to note that google drive is not exactly like your filesystem. A folder may have multiple files of the same name, and a file can have multiple parents (folders). For example, if you want to create a file if it does not already exist, you'll need to check first; otherwise you'll end up with duplicates. I provided getFileByParentAndName
to check for an existing file. You can also provide an file id, which is a good idea because you could end up with duplicates if you use retry logic. The upload implementation I provided takes a parameter to check if the file exists first.

Simple operations on a filesystem, such as listing the contents of a folder aren't so simple with drive. To do this you perform a query: '%s' in parents and name = '%s' and trashed = false
providing the parent, which in this case is the google drive folder. Folders are just files but they have a special mime type: "application/vnd.google-apps.folder"
The api reference https://developers.google.com/drive/v3/reference/files/list mentions the query parameter but does not indicate query language. You have to skip to https://developers.google.com/drive/v3/web/search-parameters to learn how to contruct a query.

The v3 reference provides some examples in different languages, but only for some operations https://developers.google.com/drive/v3/web/manage-uploads

It was somewhat frustrating at first to perform some basic operations because many of the examples were using version 2 of the api and I had selected version 3, which hopefully will have some more runway. Some of the api parameters and usage had changed.

Finally, I should note that I'm really interested in server to server auth. I'm only using oauth because that is what available documentation lead me to. In this case I'm requesting oauth access to my own account and only my account, which is not exactly the intention of oauth, but is seems it should work as long as I don't need to reauthorize.

Publishing to Maven Central

I couldn't find any coherent guides for publishing to Maven Central, so here's my somewhat incoherent guide.

Account Setup

The first step is to request an account from Sonatype, via Jira. This is described under initial setup

http://central.sonatype.org/pages/ossrh-guide.html

This can take a few days since a human needs to evaluate the request, for example to verify you own the domain of the groupId.

Save jira username/password credentials

Configuration

Install gpg

brew install gpg

brew install gpg-agent

I got the following error, apparently related to upgrading to mac sierra

Error: The following formula:

gpg-agent

cannot be installed as a binary package and must be built from source.

Create a key with gpg gen-key

Save your passphrase somewhere safe, like keypass

gpg --list-keys

/Users/andrew/.gnupg/pubring.gpg

--------------------------------

pub   4096R/1BC87E06 2017-01-07

uid                  Andrew Rapp

sub   4096R/C654CD46 2017-01-07

Send your pub key to key servers

$ gpg --keyserver hkp://pool.sks-keyservers.net --send-keys 1BC87E06

Now add the maven publishing configuration to your pom.xml

This is somewhat documented here

http://central.sonatype.org/pages/apache-maven.html#deploying-to-ossrh-with-apache-maven-introduction

I've included my pom.xml at the bottom of this post

Update maven settings ( ~/.m2/settings.xml):

<settings>

<servers>

<server>

  <id>ossrh</id>

  <username></username><!-- sonatype jira account -->

  <password></password><!-- jira password -->

</server>

</servers>

<profiles>

<profile>

<id>gpg-profile</id>

<properties>

  <gpg.useagent>true</gpg.useagent>

</properties>

</profile>

</settings>

Publishing

Run gpg-agent so we don’t need to put the passphrase in mvn config

eval $(gpg-agent --daemon --no-grab --write-env-file $HOME/.gpg-agent-info)

export GPG_TTY=$(tty)

export GPG_AGENT_INFO

Release a Snapshot

Set the Maven version in the pom.xml to -SNAPSHOT. You can use maven for this:

mvn versions:set -DnewVersion=0.9.1-SNAPSHOT

export JAVA_HOME=`/usr/libexec/java_home -v 1.6` && mvn clean deploy

If you’re not concerned with the version of Java then skp the JAVA_HOME export

You should get prompted for your gpg passphrase

Login to https://oss.sonatype.org/#view-repositories and search for your groupid

Snapshots

https://oss.sonatype.org/content/repositories/snapshots/com/rapplogic/

https://oss.sonatype.org/content/groups/public/com/rapplogic/

For Releases

Release a Final Version

Update version in pom.xml

mvn versions:set -DnewVersion=0.9.1

NOTE: If in pom.xml autoReleaseAfterClose is true it will release after deploy. Otherwise will go to staging area and will need to be released manually

export JAVA_HOME=`/usr/libexec/java_home -v 1.6` &&  mvn clean deploy -P release

Alternatively you can release via the sonatype Nexus Repository Manager https://oss.sonatype.org/#welcome

It can take hours for the artifact to land on maven central.

Be sure to remove the artifacts from your local repo ~/.m2 if you want to be sure to pull from maven central version

Tag the release and push to git

git tag with release, e.g. git tag v0.9.1

git push origin master --tags

Now rev the pom version to the next SNAPSHOT version, e.g.

mvn versions:set -DnewVersion=0.9.2-SNAPSHOT

And if I misled you, this guide looks pretty good

http://www.ryanchapin.com/fv-b-4-783/How-To-Distributed-Artifacts-in-the-Maven-Central-Repository.html

pom.xml

<groupId>com.rapplogic</groupId>
<artifactId>xbee-api</artifactId>
<packaging>jar</packaging>
<version>0.9.4-SNAPSHOT</version>

<name>${project.groupId}:${project.artifactId}</name>
<description>A java library for communicating with XBee radios</description>
<url>https://github.com/andrewrapp/xbee-api/</url>

<licenses>
   <license>
      <name>GPL license, Version 3.0</name>
      <url>https://www.gnu.org/licenses/gpl-3.0.html</url>
   </license>
</licenses>

<developers>
   <developer>
      <name>Andrew Rapp</name>
      <email></email>
   </developer>
</developers>

<scm>
   <connection>scm:git:git@github.com:andrewrapp/xbee-api.git</connection>
   <developerConnection>scm:git:git@github.com:andrewrapp/xbee-api.git</developerConnection>
   <url>https://github.com/andrewrapp/xbee-api/</url>
</scm>

<properties>
   <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
   <maven.compiler.source>1.6</maven.compiler.source>
   <maven.compiler.target>1.6</maven.compiler.target>
</properties>

<distributionManagement>
   <snapshotRepository>
      <id>ossrh</id>
      <url>https://oss.sonatype.org/content/repositories/snapshots</url>
   </snapshotRepository>
   <repository>
      <id>ossrh</id>
      <url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
   </repository>
</distributionManagement>

<build>
   <plugins>
      <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-compiler-plugin</artifactId>
         <configuration>
            <source>1.6</source>
            <target>1.6</target>
         </configuration>
      </plugin>

      <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-gpg-plugin</artifactId>
         <executions>
            <execution>
               <id>sign-artifacts</id>
               <phase>deploy</phase>
               <goals>
                  <goal>sign</goal>
               </goals>
            </execution>
         </executions>
      </plugin>

      <plugin>
         <groupId>org.sonatype.plugins</groupId>
         <artifactId>nexus-staging-maven-plugin</artifactId>
         <version>1.6.7</version>
         <extensions>true</extensions>
         <configuration>
            <serverId>ossrh</serverId>
            <nexusUrl>https://oss.sonatype.org/</nexusUrl>
            <autoReleaseAfterClose>true</autoReleaseAfterClose> <!-- if true does what you think; otherwise need to release via sonatype UI -->         </configuration>
      </plugin>

      <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-source-plugin</artifactId>
         <executions>
            <execution>
               <id>attach-sources</id>
               <goals>
                  <goal>jar</goal>
               </goals>
            </execution>
         </executions>
      </plugin>

      <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-javadoc-plugin</artifactId>
         <executions>
            <execution>
               <id>attach-javadocs</id>
               <goals>
                  <goal>jar</goal>
               </goals>
            </execution>
         </executions>
         <configuration>
            <!--mvn 3.3.9: Exit code: 1 - javadoc: error - invalid flag: -Xdoclint:none-->            <!-- need the following or fails on: get self-closing element not allowed -->            <additionalparam>-Xdoclint:none</additionalparam>
         </configuration>
      </plugin>
   </plugins>
</build>

Restarting a Server From a Web Service

Often times you may want a simple web service call to restart your server. There is a problem however if the web service is running in the application to be restarted. You can't simply call a restart script from your application, at least in Java, since Java needs to die to complete the call, and this would kill the restart process along with it. To overcome this limitation you can wrap the restart script in a nohup and background it.

Create nohup-restart.sh, containing the following:

nohup ./bin/restart.sh $1 > restart.out 2> restart.err &

Then create restart.sh, containing:

sleep 3

kill $1

app.sh

The sleep is to give the web service time to respond before it shuts down.

You can pass the pid from Java into the script. The pid is accessed by

ManagementFactory.getRuntimeMXBean().getName()

But strip out the @machine part).

Now call Runtime.exec with fully qualified path to script. Depending on how it was started you might be able to get away with a path relative to user.dir.

process = Runtime.getRuntime().exec(new String[] {scriptPath, "nohup-restart.sh", ManagementFactory.getRuntimeMXBean().getName().split("@")[0]});

If the script is daemon it's much simpler since it can be called without the nohup nonsense, however it should be called in a Thread to give it time to respond.

When String.length() Lies

With most languages developers are accustomed to dealing with a string object that has a length size method, which returns the size of the string. But this fails in Java when dealing with certain characters in the unicode range, including some emoji. The reason for this is characters are internally represented with UTF-16, allowing for up to 65536 characters, but some characters requires three bytes or more and so therefore cannot be expressed in a single String character.

Here's what the cactus emoji 🌵 looks like in the debugger. The emoji character occupies two characters of the String, index 6, and 7, yet of course is a single character. If you wrote code that iterated over the String and printed each character you would most definitely not see a cactus emoji in the output, yet if you print the string then you would see it, assuming the console or app doing the printing is capable of rendering emoji.

In general this hasn't been much of an issue since the vast majority of characters exist within 0xFFFF, or the Basic Multilingual Plane (BMP). But now, with the emergence of emoji some of those new characters are being placed above 0xFFFF, simply because we've run out of code points in the BMP. The range > 0xFFFF is known as the Supplementary Multilingual Plane. That range goes from 10000–​1FFFF and there's even additional ranges, up to F0000–​10FFFF.

Characters that fall into the SMP require four bytes or two Java characters. The first character falls in the high range and the second in the low range:

high range 0xD800..0xDBFF.

low range  0xDC00..0xDFFF.

Now, when you see a character that is >= 0xD800 and <= 0xDBFF, you know that it is greater than two bytes and is a surrogate pair.

The rules for decoding the pair into the unicode code point is as follows (from wikipedia)

Consider the encoding of U+10437 (𐐷):

• Subtract 0x10000 from 0x10437. The result is 0x00437, 0000 0000 0100 0011 0111.
• Split this into the high 10-bit value and the low 10-bit value: 0000000001 and 0000110111.
• Add 0xD800 to the high value to form the high surrogate: 0xD800 + 0x0001 = 0xD801.
• Add 0xDC00 to the low value to form the low surrogate: 0xDC00 + 0x0037 = 0xDC37.

What this means for the developer is that string.length may report a size that is larger than the visible character size of the String. Also, when using the substring method you must be careful not to chop in the middle of a surrogate pair. For example,

if (string.charAt(splitPosition) >= 0xD800 && string.charAt(splitPosition) <= 0xDBFF)

Then the substring position must be adjusted to splitPosition + 1 or splitPosition - 1.

You can easily determine if an emoji will occupy more than one character in a string by looking at the unicode code point. As an example, hot beverage is U+2615 (hex). That's just two bytes so no issue there. But, cactus is U+1F335 and requires three bytes, so it would need two characters in a String.

Emoji reference: http://apps.timwhitlock.info/emoji/tables/unicode