Thursday, December 13, 2012

AES Encryption Bouncy Castle Example

Here's a little example, showing how to perform AES encryption in CTR mode using Bouncy Castle. My example is in Scala, but the idea is the same for Java. I've placed all of the code on Git: https://github.com/travisdazell/AES-CTR-BOUNCYCASTLE

To start with, here's an "encrypt" method that takes a key, IV (initialization vector), and message and returns the encryption of the message under the specified key and IV. All of the method parameters are expected to be hex-encoded. Note that I'm using a utility method I created to convert the hex strings to byte arrays, which is what the crypto library expects as input. This utility method is included in the code on Git.

def encrypt(hexEncodedIv : String, hexEncodedKey : String, hexEncodedMessage : String) = {
    // we're using Bouncy Castle
    Security.addProvider(new org.bouncycastle.jce.provider.BouncyCastleProvider())

    // create our key specification
    val secretKeySpec = new SecretKeySpec(hexStringToByteArray(hexEncodedKey), "AES")
     
    // create an AES engine in CTR mode (no padding)
    val aes = Cipher.getInstance("AES/CTR/NoPadding", BouncyCastleProvider.PROVIDER_NAME)
     
    // initialize the AES engine in encrypt mode with the key and IV
    aes.init(Cipher.ENCRYPT_MODE, secretKeySpec, new IvParameterSpec(hexStringToByteArray(hexEncodedIv)))
     
    // encrypt the message and return the encrypted byte array
    aes.doFinal(hexStringToByteArray(hexEncodedMessage))
}


Our "decrypt" method will be the same as the "encrypt" method; however, instead of Cipher.ENCRYPT_MODE, we'll use Cipher.DECRYPT_MODE. Of course, we'll be passing the ciphertext to the method instead of the plaintext message.

def decrypt(hexEncodedIv : String, hexEncodedKey : String, hexEncodedCipherText : String) = {
    // we're using Bouncy Castle
    Security.addProvider(new org.bouncycastle.jce.provider.BouncyCastleProvider())

    // create our key specification
    val secretKeySpec = new SecretKeySpec(hexStringToByteArray(hexEncodedKey), "AES")
     
    // create an AES engine in CTR mode (no padding)
    val aes = Cipher.getInstance("AES/CTR/NoPadding", BouncyCastleProvider.PROVIDER_NAME)
     
    // initialize the AES engine in decrypt mode with the key and IV
    aes.init(Cipher.DECRYPT_MODE, secretKeySpec, new IvParameterSpec(hexStringToByteArray(hexEncodedIv)))
     
    // decrypt the ciphertext and return the plaintext as a byte array
    aes.doFinal(hexStringToByteArray(hexEncodedCipherText))
}

Friday, November 30, 2012

Converting a Hex String to a Byte Array in Scala

Here's a Scala method that converts a hex string to an array of bytes. This is useful when needing to encrypt a hex string, because most of the crypto libraries operate on byte arrays.


def hexStringToByteArray(s : String) = {
    val len = s.length();
    var data = new Array[Byte](len / 2)
    var i = 0

    while (i < len) {
        val b = (Character.digit(s.charAt(i), 16) << 4) +
                (Character.digit(s.charAt(i+1), 16))

        data(i / 2) = b.asInstanceOf[Byte]

        i+=2
    }

    data
}

Friday, November 23, 2012

Many Time Pad Attack - Crib Drag

The one time pad (OTP) is a type of stream cipher that is a perfectly secure method of encryption. It's very simple to implement and is perfectly secure as long as the length of the key is greater than or equal to the length of the message. That's it's major downfall. However, it also requires that the key never be used more than once. This tutorial shows what happens when you re-use a key to encrypt more than one message. I also show how to uncover the plain-text of two messages that have been encrypted with the same key, without even knowing the key. I use a method called crib dragging.

Let's begin with a brief description of OTP and how it works. Let's take the following message and key:

message = "Hello World"
key = "supersecret"

If we convert both the message and key to hex strings, we get the following:

message = "48656c6c6f20576f726c64"
key = "7375706572736563726574"

If we do a simple XOR of the two hex strings we get the following cipher-text:

cipher-text = "3b101c091d53320c000910"


If we XOR the cipher-text with the key, we can recover the plain-text. That's how OTP works. Without the key, you have no way of uncovering the plain-text.

Let's consider what happens when you have two messages encrypted with the same key. Take the following two messages and key:

message1 = "Hello World"
message2 = "the program"
key = "supersecret"

If we convert each message and the key to hex strings, and then encrypt each message using a simple XOR with the key, we'll get the following cipher-texts:

cipher-text1: "3b101c091d53320c000910"
cipher-text2: "071d154502010a04000419"


Let's say that all we have is the two cipher-texts and the knowledge that they were encrypted with a supposed OTP; however, they were both encrypted with the same key. To attack this encryption and uncover the plain-text, follow the steps below.

  1. Guess a word that might appear in one of the messages
  2. Encode the word from step 1 to a hex string
  3. XOR the two cipher-text messages
  4. XOR the hex string from step 2 at each position of the XOR of the two cipher-texts (from step 3)
  5. When the result from step 4 is readable text, we guess the English word and expand our crib search.
    1. If the result is not readable text, we try an XOR of the crib word at the next position.

Step 1 seems difficult (guessing a word that might appear in one of the messages), but when you think about it, the word "the" is the most commonly used English word. So, we'll start with assuming "the" is in one of the messages. After encoding "the" as a hex string, we'll get "746865". That takes care of steps 1 and 2. If we XOR the two cipher-texts, we'll get the following result:

cipher-text1 XOR cipher-text2 = "3c0d094c1f523808000d09"

The next step is to XOR our crib word "746865" at each position of the XOR of the cipher-texts. What we'll do is slide "746865" along each position of "3c0d094c1f523808000d09" and analyze the result. After the first XOR, we get the following result:

         3c0d094c1f523808000d09
XOR  746865
----------------------------------
          48656c

When we convert the hex string "48656c" to ASCII, we get the following text, "Hel". This takes us to step 5 from above. Because this looks like readable text, we can assume that the word "the" is in the first position of one message. If we didn't get readable text, we would slide "746865 (the)" one position to the right and try again (and keep repeating until the end of 3c0d094c1f523808000d09).

Note that we don't know which message contains the word "the". It could be in either message1 or message2. Next, we need to guess what the word "Hel" is when fully expanded. It could be "Help", "Hello", etc. If we guess "Hello", we can convert "Hello" to a hex string, we get "". We then XOR it with the XOR of the two cipher-texts (just like we did with "the"). Here's the result:

         3c0d094c1f523808000d09
XOR  48656c6c6f
----------------------------------
          7468652070

"7468652070"  when converted to ASCII, is "the p". We then repeat the process, guessing what "the p" might be when expanded and then XOR that result with the XOR of the cipher-texts. Granted, guessing what "the p" might expand to is not super easy, but you get the idea. If we were to guess "the program", convert it to a hex string, and XOR it with the XOR of the cipher-texts, we'll get "Hello World".

This is called crib dragging. My suggestion is to first try " the " (note the spaces before and after). Most cipher-texts that you'll try cracking will contain that word somewhere in the text. If the result of your crib drag yields gibberish, then you can be sure " the " isn't in either of the plain-text messages. So, try another commonly used English word or phrase and keep trying until the result yields something that looks like readable text. Then you can just expand your guess and keep XORing until you uncover the plain-text messages.

In a future blog post, I'll demonstrate an implementation of a crib drag in Scala.

Monday, November 19, 2012

Dependency Management - Avoid Cyclic Dependencies

We're always striving to design software that's easy to maintain. With every code change, we try to avoid code bloat and develop a solution that's simple, while still meeting the project requirements. As a result, we measure our success by whether or not somebody else can look at our code and understand it clearly. Code clarity is a very important measure of maintainability, but we also need to look at our project from a 1,000 foot view and take a moment to look at the dependencies in our project.

To start with, I recommend modeling each component and the compile-time dependencies amongst those components. As an example, think of each component as a Java package. Take, for example, the following high-level view of a system with just three components. Here, component A depends on component C, component B depends on component A, and component C depends on component A.




A red flag should go up if you see this kind of compile-time cyclic dependency in your projects. There are a number of problems with this type of cyclic dependency. First, you can't reuse just one component without bringing along the others. Second, if you modify one component, you indirectly affect the others. This creates a nightmare to test. Also, when a new requirement arrives on your desk, it's difficult at first glance how wide or deep the change will ripple through the system. In some cases, you're forced to follow the affected code through all of the components in order to determine the impact that a change requires. Architecture 101 teaches us that any change that goes wide or deep is a high-risk change. Cyclic dependencies often result in the smallest of changes affecting multiple components.

Avoid compile-time cyclic dependencies! We should always strive for acyclic dependencies. Take the following graph as an example. All I've done for this example is reversed the dependency from A => C to   C => A. This is just to illustrate the example. The result, however, is zero circular dependencies in our system. The issues discussed in the cyclic example go away (or at least aren't as serious).



This example is very trivial. In reality, we're usually working on very large applications with many, many components. So, it's clearly not as easy a task to avoid cyclic dependencies in a real world project. However, it's very important to manage your dependencies and avoid compile-time cyclic dependencies by keeping it at the forefront of your designs.


Saturday, November 17, 2012

Camel File Poller and JMS Consumer in Scala

I put the code on  GitHub: https://github.com/travisdazell/Camel-Scala-FilePoller

In this tutorial, I'm going to demonstrate using Apache Camel to write a Scala file poller. When a file arrives in the "incoming" directory, I'm placing a JMS message onto a queue. Lastly, I'll have a JMS consumer written in Scala that will receive the message from the queue. I'll be using ActiveMQ as my JMS message broker. The main focal point in the tutorial is using Apache Camel for creating a file poller. I'm not doing anything special with ActiveMQ and the JMS consumer could just as easily have been written in Java. I just chose to use Scala, because I like it for its expressiveness.

I won't go through the steps of setting up the architecture. I'll cover that in a future tutorial. For now, I assume you have ActiveMQ running, the Camel JARs downloaded, and your Scala development environment ready to go. Now on to the code.

There are only two Scala classes needed. The first component is the Scala class that defines our Camel routes. We have a single route that routes incoming files to a JMS consumer. Here's our Camel route definition in Scala.

import org.apache.camel.scala.dsl.builder.RouteBuilder
import org.apache.camel.model.dataformat.BindyType

class JmsRoutes extends RouteBuilder {
    "file://incoming" --> "activemq:queue:test.queue"
}


The second component in the application is the JMS consumer. This consumer simply blocks, waiting for a file to arrive in a directory named "incoming". Here's the code for the JMS message consumer.

import javax.jms.ConnectionFactory
import org.apache.camel.CamelContext
import org.apache.camel.ConsumerTemplate
import org.apache.camel.impl.DefaultCamelContext
import org.apache.camel.component.jms.JmsComponent
import org.apache.activemq.ActiveMQConnectionFactory

object JmsConsumer {
    def main(args : Array[String]) {
        val connectionFactory = new ActiveMQConnectionFactory("vm://localhost?broker.persistent=false")
        val context = new DefaultCamelContext()

        context.addComponent("activemq", JmsComponent.jmsComponentAutoAcknowledge(connectionFactory))

        context.addRoutes(new JmsRoutes())

        val consumerTemplate = context.createConsumerTemplate()

        context.start()
       
        consumerTemplate.receive("activemq:queue:test.queue")

        println("received message")
       
        context.stop()
    }
}



This shows how simple it is to use Camel for routing messages in a Scala application. I like the expressiveness of the routing syntax. Camel is a very powerful integration framework, and yet it's extremely easy to use.

XOR Hex-Encoded Strings in Scala

It's been said that all cryptographers do is XOR things together. Although that's not entirely true, we do find ourselves XORing things quite a bit... maybe there is some truth to that after all. Anyway, the other day, while I was writing a two-time pad attack on stream ciphers, I needed a few utility methods to make the string operations a bit easier. One of the methods was performing XOR operations on two hex-encoded strings. Here's the utility method in Scala. I posted it here in case anybody else might find it useful.

def xorHexStrings(hexString1 : String, hexString2 : String) = {
    val iterator1 = hexString1.sliding(2, 2)
    val iterator2 = hexString2.sliding(2, 2)
    val result =  new StringBuilder
        
    if (hexString2.length > hexString1.length) {
        while (iterator1.hasNext) {
            val i = Integer.toString(Integer.parseInt(iterator1.next, 16) ^
                    Integer.parseInt(iterator2.next, 16), 16)

            if (i.length == 1) result.append("0")
            result.append(i)

        }
            
        while (iterator2.hasNext) result.append(iterator2.next)
    }
    else {
        while (iterator2.hasNext) {
            val i = Integer.toString(Integer.parseInt(iterator1.next, 16) ^
                    Integer.parseInt(iterator2.next, 16), 16)

            if (i.length == 1) result.append("0")
            result.append(i)

        }
          
        while (iterator1.hasNext) result.append(iterator1.next)
    }
        
    result.toString()
}

If we call this method within our main function, we can see the results.

def main(args : Array[String]) {
    val hexString1 = "1274560603"
    val hexString2 = "876429"

    //prints 95107f0603
    println(xorHexStrings(hexString1, hexString2))
}

Wednesday, November 7, 2012

Exception Shielding Your Services

The most basic of all security mechanisms in SOA is shielding or hiding exceptions to the consumer of your services. Don't overlook this key feature when designing your services. Let's say you're constructing a service that inserts a record to an underlying database. Most likely, you'll have a try-catch somewhere in your service that handles SQLException(s). In a library routine that will be used by your internal applications, it's not much of a concern to expose the details of that exception to the caller. However, in a service that will potentially be consumed by outside clients, exposing the message within the exception could give the client too much information, revealing how your service is designed and built. This could be a security risk, allowing somebody to exploit the service.

To mitigate that risk, it's best to sanitize the exception and return a generic error message to the caller of the service. This can be done in a couple of ways:

  1. You can have a utility service that resides within your local network (not accessible to the outside world) that takes an exception and the service method name as input, and returns a sanitized message that's safe to be returned to the client.
  2. You can write the sanitation logic within each, individual service.

I prefer to use option 2, but that's just my personal preference. I find that every service has its own unique nuances that make it easier to deal with exceptions in the service that caught the exception. Often times, the message you return to the client will contain some verbiage regarding the service method/capability itself. For example, if you have a service method named "CreateCustomer" and a SQLException is caught within the service, you might return a message to the client that says something like, "An error occurred while creating the customer record". Having a single utility service that manages all of the possible mappings from exception to sanitized message can get bloated very quickly.

After you've decided on your strategy for sanitizing exceptions, you need to make sure you're persisting these exceptions with the full exception message (inner message, stack trace, etc.) somewhere for debugging purposes. A good approach is to have a database table where you log the exception message, along with a unique identifier, such as a GUID. When you return the sanitized message to the client, return the GUID along with it. That way, when a client reports an error message, you can lookup the raw exception message and debug the issue much easier.

Lastly, define a standard means for returning exceptions to the consumers of your services. If you're developing a SOAP service, you can make use of javax.xml.soap.SOAPException to wrap your exceptions that you return to the client. However you decide to do it, keep it consistent throughout your services, so that your consumers don't have to write custom exception handling code for every service they consume.

Tuesday, November 6, 2012

Scala Loops

Just like in Java, Scala has while and do loops. For example, you can write a while loop like this.

while (x > 0) {
    y = y + 1
}


However, there is no for loop like you're familiar with in Java with a structure of for (initialize; test; update). However, once you get comfortable with Scala, you'll see that the for loop in Scala is much better. Here's a for loop that loops from 1 to 10.

for (x <- 1 to 10) {
}


To me, the use of the to syntax is very expressive. The call 1 to 10 returns a Range of those values from 1 to 10. It's inclusive, so 10 is part of that Range. What's even cooler, is that the type of the variable x is auto-magically the same element type of the collection.

Let's say you need the range to exclude the last element. This is often the case if you're iterating over the characters in a string or array. You can do that by changing the to to until. The following code will iterate over the string "Travis" one character at a time.

val name = "Travis"
     for (x <- 0 until name.length) {
}


Of course, you can be even more concise in Scala and write this instead.

val name = "Travis"
    for (x <- name) {
}


When you look at the syntax, it makes sense, because in the context of the code, we already know that name is a string, so we'd expect "c <- name" to iterate over that string and make "c" a character type.

With every new Scala feature that I learn, I get even more excited to find the next great feature in the language that builds on the previous feature. I don't know about you, but whether I like it or not, the need for nested loops is always there. Scala gives us a really clean syntax for writing nested loops that de-clutters the code and makes it much easier to read. Take the following Java code, for example.

//Java code
for (int i=1; i < 5; i++) {
   for (int j=1; j < 5; j++)    {
   }
}



In Scala, we can write this instead.

// prints 1 2 2 4 3 6 4 8
for (i <- 1 until 5; j <- 1 until 3) {
   print (i * j + " ")


These are called generators and you can have as many as you like. Each generator can have a guard, which is just a Boolean condition that follows an if.

// prints 2 2 3 6 4 8
for (i <- 1 until 5; j <- 1 until 3 if i != j) {
   print (i * j + " ")
}


If you need to, you can define variables inside your for loop definition and use them in the expression to evaluation the loop. For example, the following loop defines a new variable named "y" that is used inside the loop.

// prints 9 10 11 12 13 14 15 16 18 20 22 24 26 28
for (i <- 1 until 3; y = 10 - i; j <- y until 15) {
   print (i * j + " ")
}


If you need to build a collection of values from a loop, you can do this very concisely as well. Here's a Scala loop that constructs a collection of integer values. In Scala, we call this a for comprehension.

// Builds a Vector(2, 4, 6, 8, 10)
for (i <- 1 to 5) yield i * 2


If you haven't picked up Scala yet, I encourage you to take it for a test drive and I think you'll really like its expressiveness. It's free of clutter, easy to read, and quick to write (i.e. less code). Loops are a great example of this.

Wednesday, October 31, 2012

If, Else, and Match in Scala

Scala improves upon basic conditional statements in a number of ways. Here are a few of the main points in understanding how to write conditional statements in Scala.

If Statements

If expressions have values. This is actually a very useful mechanism. Similar to the last statement in a function, an if-else statement returns the value of the expression that follows the if-else. For example, the following if-statement has a value of true or false depending on the value of x.

if (x < 10) true else false

You can actually initialize a variable with the result of this expression if you want.

val result = if (x < 10) {
   true
}
else {
   false
}


Match Statements


We don't have traditional switch-case statements in Scala, but to no surprise, there's something even better. Scala's "match" expressions are an improved switch statement. Here's an example of a "match" expression that sets the value of a variable "n" based on the value of a variable named "number". If number is 0, then n becomes "zero". If number is 1, then n becomes "one". If number is any other value, then n becomes "some other value".

val n = number match {
   case 0 => "zero"
   case 1 => "one"
   case _ => "some other value"
}

There's no ability for fall-through to occur, like can often happen in a traditional switch statement. Rarely, do you want fall-through, but if you do need to have a range of values evaluate to the same result, you can do that with a "guard". For example, we can modify our match expression to match values of 0, 1, 10-20, and 30-50. Anything else is caught with our "case _" and will result in a value of "some other value". The guard can be any Boolean condition.

val n = number match {
  case 0 => "zero"
  case 1 => "one"
  case num if 10 until 20 contains num => "between ten and twenty"
  case num if 30 until 50 contains num => "between thirty and fifty"
  case _ => "some other value"

}

Objects in Scala

The more I develop in Scala, the more I love it. One of the many things that I love about what Martin did with Scala is to create the "object" construct to easily manage singletons in our applications. Many people frown at singletons and they certainly can be misused. However, there are times when they're the right design for the task at hand, and Scala gives us a way to create singletons in a very clean way.

In Java, you'd most likely implement the singleton design pattern by making the default constructor private and then providing a public static method that returns the sole instance of the class. Well, Scala has no static methods or fields. Instead, we have the "object" construct. With "object", Scala takes care of instantiating a single instance of our object with the features that are defined within the object. Take the following object defined in Scala.

object TemperatureConverter {
    def celsiusToFahrenheit(celsius : Double) = {
        celsius * 9 / 5 + 32
    }
  
    def fahrenheitToCelsius(fahrenheit : Double) = {
        (fahrenheit - 32) * 5 / 9
    }

}

When you need to convert fahrenheit to celsius, you can write this.

TemperatureConverter.fahrenheitToCelsius(82)



Scala takes care of creating the object the first time it's used, so we don't have to worry about it. Scala objects make short work of implementing singletons. This can be very handy when writing utility methods or the case when you need to guarantee that only a single instance of an object exists in your application.

What about when you need to add "static" methods to your class? Since Scala doesn't have "static" methods, we can create a companion object for our class. For example, we might have the following Customer class with a companion object for creating a unique customer number.

class Customer {
    val customerNumber = Customer.getNextCustomerNumber()
    ....
}

object Customer {
    private var uniqueNumber = 0
    private def getNextCustomerNumber() = {
        uniqueNumber += 1
        uniqueNumber
    }
}

Saturday, October 27, 2012

UI Mediator Design Patten

This blog post is an introduction to the UI Mediator SOA design pattern designed by [Utschig, Maier, Trops, Normann, Winterberg] and described in the book "SOA Patterns" by Thomas Erl. The book is a great read and I recommend it to any SOA implementor.

One of the main design goals of a Service Oriented Architecture is to promote flexibility and de-coupling of systems and applications via services. This often comes in the form of asynchronous messaging at the cost of super-fast performance. We choose a message-oriented middleware because it provides flexibility, but often times these messages are asynchronous and aren't guaranteed to return timely responses. The problem manifests itself in the form of a UI that is constantly waiting for a message response. That's where the UI Mediator pattern saves us.

The UI Mediator pattern solves this problem by placing a new "mediator" service between the caller and the service endpoint that does two things:

  1. Invokes the destination service we need to consume
  2. Returns status updates to the UI

Conceptually, the UI Mediator pattern looks like this. The Mediator Service returns updates to the caller while the background services are processing the request.




More often than not, it's acceptable to let the user continue working in another area of the application while a large operation is being processed. We don't want to block the user from working while we wait for a message response, but we need to notify them of progress and alert them when the process has completed successfully (or potentially returned an error at some point in the processing). The UI Mediator pattern does just that and it's a useful implementation when the use case allows background processing, but requires progress updates to the caller.

Thursday, October 25, 2012

Writing an Annotation Based Processor in Java

The code for this tutorial is on GitHub: https://github.com/travisdazell/Annotation-Based-Processor

Annotations have been around since Java 5. They provide a way to mark (i.e. annotate) Java code in a clean and concise manner. With Java 6 and 7, we can do even more with annotations. The most exciting of which, in my opinion, is the ability to write our own annotations and even process our own annotations to detect code issues or even generate source code at compile time.

In this example, we'll create an annotation named @Metrics. We'll be able to mark Java classes with @Metrics and at compile-time, generate a report of all methods defined in the class. To create your own annotation and its corresponding processor, follow these steps.

  1. Define the annotation. This is done by defining the annotation with @interface.
     public @interface Metrics {
     }

  
     2. We can now annotate any class in our project with our new annotation.

     import net.travisdazell.annotations.Metrics;

     @Metrics
     public class CustomerDaoImpl implements CustomerDao {
        public Customer getCustomer(int customerId) {
           // return Customer record
        }
     }


     3. The next step is to write a processor that, at compile time, will report all of the methods defined in CustomerDaoImpl.java. Here's our processor in its entirety.


package net.travisdazell.annotations.processors;

import java.lang.reflect.Method;
import java.util.Set;

import javax.annotation.processing.AbstractProcessor;
import javax.annotation.processing.RoundEnvironment;
import javax.annotation.processing.SupportedAnnotationTypes;
import javax.annotation.processing.SupportedSourceVersion;
import javax.lang.model.SourceVersion;
import javax.lang.model.element.Element;
import javax.lang.model.element.TypeElement;
import javax.tools.Diagnostic;

import net.travisdazell.annotations.Metrics;

@SupportedAnnotationTypes("net.travisdazell.annotations.Metrics")
@SupportedSourceVersion(SourceVersion.RELEASE_6)
public class MetricsProcessor extends AbstractProcessor {
   
    @Override
    public boolean process(Set<? extends TypeElement> arg0,
            RoundEnvironment roundEnv) {

        StringBuilder message = new StringBuilder();

        for (Element elem : roundEnv.getElementsAnnotatedWith(Metrics.class)) {
            Metrics implementation = elem.getAnnotation(Metrics.class);

            message.append("Methods found in " + elem.getSimpleName().toString() + ":\n");

            for (Method method : implementation.getClass().getMethods()) {
                message.append(method.getName() + "\n");
            }

            processingEnv.getMessager().printMessage(Diagnostic.Kind.NOTE, message.toString());
        }

        return false; // allow others to process this annotation type
    }
}


    4. Next, we need to package our annotation processor and annotation in a JAR file. We must also include a META-INF\services directory in our JAR file. Within the services directory, include a file named javax.annotation.processing.Processor. In that file, type the fully qualified name of the annotation processor.



Here's what my JAR file looks like when exploded in Eclipse.



5. Next, to test our annotation processor, create a class and annotate it with @Metrics.

package net.travisdazell.projects.testproject.dao.impl;

import net.travisdazell.annotations.Metrics;

@Metrics
public class CustomerDaoImpl {
}



6.When we compile this class, we'll get a message displaying the list of methods in the class. As expected, we see what's been inherited from java.lang.Object.




Writing Basic Language Constructs in an ANTLR Grammar

When designing a new general purpose programming language, or anything beyond a simple DSL, you'll encounter the need to write grammar rules for basic language constructs such as branch operations and loops. Here are a few grammar snippets that I re-use just about every time I design a DSL in ANTLR. Of course, the syntax is up to you to define how you want, but this helps as a general template for frequently occurring language constructs.

Credit is due entirely to Scott Stanchfield. I'd recommend checking out his videos as they are excellent tutorials on writing ANTLR grammars. Most of these constructs are taken directly from his tutorials. I usually find myself copying and pasting these basic grammar rules and then modifying the syntax as I need for the language I'm designing.

Expressions

term
    :   
        IDENT
        | '(' expression ')'
        | INTEGER
    ;
   
negation
    :    'not'* term
    ;
   
unary
    :    ('+' | '-')* negation
    ;
   
mult
    :    unary (('*' | '/' | 'mod') unary)*
    ;
   
add
    :    mult (('+' | '-') mult)*
    ;
   
relation
    :    add (('<' | '<=' | '>' | '>=' | '=' | '!=') add)*
    ;
   
expression
    :    relation (('and' | 'or') relation)*
    ;



Assignment Statements


assignmentStatement
    :    IDENT ':=' expression ';'
    ;



If-Else Statements

ifStatement
    :
        'if' '(' expression ')' statement+
        ('else if' '(' expression ')' statement+)*
        ('else' statement+)*
        'endif'
    ;



Loops

exitStatement
    :
        'exit' 'when' expression ';'
    ;

loopStatement
    :
        'loop'
        (statement|exitStatement)*
        'endloop'
    ;
   
whileStatement
    :
        'while' '(' expression
        (statement|exitStatement)*
        'endloop'
    ;


Fragments

fragment LETTER
    :
        ('a'..'z' | 'A'..'Z')
    ;
   
fragment DIGIT
    :
        '0'..'9'
    ;
   
STRING_LITERAL
    :
        '"' ~('\r' | '\n' | '"')* '"'
    ;
   
INTEGER
    :    ('1'..'9')DIGIT*
    ;

IDENT: LETTER (LETTER | DIGIT)*;

WS: (' ' | '\t' | '\r' | '\n' | '\f')+ {$channel = HIDDEN;};

COMMENT
    :    '//' .* ('\r' | '\n') {$channel = HIDDEN;};
   
MULTILINE_COMMENT
    :    '/*' .* '*/' {$channel = HIDDEN;};



Thursday, October 18, 2012

Implementing a DSL in Java Using the Builder Pattern

DSLs are really just expressive APIs. There are a variety of ways to implement DSLs. The simplest DSL, although limited in its capabilities, is an internal DSL (i.e. a DSL written in an existing programming language). In Java, we can use a builder pattern to design an internal DSL very easily.

Let's say, for example, we want a DSL that we can use to create customer records. In addition to a customer POJO, creating a customer instance would probably look something like what's shown below.

Customer customerOne = new Customer();

customerOne.setFirstName("John");
customerOne.setLastName("Doe");

DateFormat formatter = new SimpleDateFormat("yyyy-MM-dd");

try {
    Date dateOfBirth = (Date)(formatter.parse("1971-01-26"));
    customerOne.setDateOfBirth(dateOfBirth);
} catch (ParseException e) {
    System.err.println(e.getMessage());
}


This code doesn't look too bad to a Java developer, but for somebody who isn't overly familiar with Java, this looks cryptic. Not only that, but there is quite a bit of code here that doesn't have anything to do with the customer domain. It's just there because that's what Java requires. We can make this quite a bit better by building a fluent DSL using a builder pattern.

Here's what the code looks like for our builder pattern. Remember, this is the code we developers would write to provide a fluent API for creating customer records. This is the DSL. So, don't be overwhelmed by the amount of code. Our goal here is to provide an easy to use API for creating customer records that hides some of the inherent complexity of the Java code we saw above.

public class Customer {

   static class Builder {

      private String firstName;
      private String lastName;
      private Date dateOfBirth;

      public Builder() {}
     
      public Builder withFirstName(String firstName) {
         this.firstName = firstName;
         return this;
      }

      public Builder withLastName(String lastName) {
         this.lastName = lastName;
         return this;
      }

      public Builder bornOn(String dateOfBirthValue) {
         DateFormat formatter = new SimpleDateFormat("yyyy-MM-dd");
         try {
            Date dateOfBirth = (Date)(formatter.parse(dateOfBirthValue));
            this.dateOfBirth = dateOfBirth;
         } catch (ParseException e) {
            System.err.println(e.getMessage());
         }

         return this;
      }

      public Customer build() {
         return new Customer(this);
      }
   }

   private final String firstName;
   private final String lastName;
   private final Date dateOfBirth;

   private Customer(Builder b) {
      firstName = b.firstName;
      lastName = b.lastName;
      dateOfBirth = b.dateOfBirth;
   }

   public String toString() {
      return "First Name = " + this.firstName +
             ",Last Name = " + this.lastName +
             ",Date of Birth = " + this.dateOfBirth.toString();
   }

}


Now to create a customer record, we can use the builder pattern we constructed. The result is an expressive API that focuses on the domain (customer records) and hides the innate complexities of Java.


Customer c = new Customer.Builder()
   .withFirstName("John")
   .withLastName("Doe")
   .bornOn("1971-01-26")
   .build();


This type of implementation still comes with some of the nuances of its implementation language. Languages like Groovy and Scala do a better job of creating internal DSLs because of their support for higher-order functions (functions that take other functions as parameters). JDK 8 promises to do that for us and I look forward to being able to take advantage of them. In the meantime, Groovy and Scala are great languages for internal DSL development. I'll cover those examples in a later post. But, there's still alot you can do with internal DSLs in Java 7 and a builder pattern is an easy way to do it.

Monday, October 15, 2012

Calling C++ from Java Using JNI

JNI is part of the Java SDK and we can use it to call native code (e.g. C++) from within our Java applications. This can be very useful when you have existing C++ code that contains business logic that you don't have the time or resources to migrate into your Java application. Let me start by saying that I don't think this should be your first course of action in this scenario. If possible, I think a cleaner approach would be to wrap the C++ implementation in a web service and then call the web service from within your Java application. However, if your Java app will be deployed to the same server as your native code and/or your native code is running on some legacy mainframe that makes it difficult to expose as a web service, JNI can be a very good choice.

Our first step is to code-up our Java application and compile it. Our Java class will look something like this.

public class TestApplication {
    public native int getNumber();

    public static void main(String args[]) {
        System.loadLibrary("TestApplication");
        TestApplication test = new TestApplication();

        System.out.println(test.getNumber());
    }
}

The native keyword in the method declaration simply tells the Java compiler that the getNumber() method is implemented in some native library outside of this Java class. The code inside the main method simply loads the external library (which we'll create in a later step) and calls the native getNumber() method that is implemented in our C++ code.

Next, we need to create a C++ header file that will define our native functions. This header file is not the same header file you may already have in your C++ code. This header file will be created against our TestApplication.class file. To generate this header file, run the following command at the command line from within the same directory as your TestApplication.class file.


>  javah TestApplication

The auto-generated .h file will look like this.

/* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class TestApplication */

#ifndef _Included_TestApplication
#define _Included_TestApplication
#ifdef __cplusplus
extern "C" {
#endif
/*
 * Class:     TestApplication
 * Method:    getNumber
 * Signature: ()I
 */
JNIEXPORT jint JNICALL Java_TestApplication_getNumber
  (JNIEnv *, jobject);

#ifdef __cplusplus
}
#endif
#endif


The next step is to create the .cpp file that will contain the implementation of our native getNumber() method. Typcially, with JNI development, this C++ code is a wrapper around existing C++ code that we just expose using JNI. In other words, we're not editing the existing C++ library, but instead creating an adapter that will then call our legacy C++ code. Here's the .cpp source file in its entirety.

#include "TestApplication.h"

JNIEXPORT jint JNICALL Java_TestApplication_getNumber
 (JNIEnv *env, jobject obj) {
  return 10;
}


All I'm doing is including the header file that we automatically generated in the previous step and our native method is returning an integer value of 10. This is a very trivial example, but I want to keep it simple for this tutorial. In a later post, I'll talk about the details of JNIEnv, how to pass C++ object instances around, why the JNI methods are named the way they are, etc.. But that deserves a discussion of its own.

Next, we need to create the shared library that we referenced in our Java application. Depending on which C++ compiler you're using, this may vary a little, but the following command will automatically create the shared library if you're using the cl compiler on Windows.

Here are a few things to keep in mind:

  • The -I flag is just telling the C++ compiler to include those directories as include directories.You'll need to modify those to your Java install directory.
  • Make sure you have write permission to the directory where you're trying to create the DLL. If you don't, you'll get an error. Here, I'm just creating it in the local directory
  • Make sure that the C++ compiler you're using is the same architecture as your JVM (i.e. 32-bit or 64-bit).

cl -I"C:\jdk1.6.0_24\include" -I"C:\jdk1.6.0_24\include\win32" -LD TestApplication.cpp -FeTestApplication.dll


Now, if we run our Java program, we'll see the value from our native method being returned to our Java application and the value of 10 being printed to the console. In a later post, I'll talk more about the details of JNI and look at some examples that involve passing objects to our C++ application.