Throttling Service Requests Using Camel

This tutorial shows how to use Camel to throttle the sending of messages to a service. This is necessary when you don't want to overload the service. For example, the service might have a published service level agreement (SLA) requiring you to only send a certain number of requests per second or minute. If you're not familiar with Camel, read my blog post on Implementing Pipes and Filters Using Camel.

The source code for this example is available on GitHub: https://github.com/travisdazell/camel-throttler

Let's consider an integration scenario where we receive orders via an incoming file drop. We then pick up the file, process it with some custom Java code, and then place a message on a JMS queue. Here's the Camel route, with a throttler that only allows one message to be sent to the JMS queue every five seconds.

<camelcontext xmlns="http://camel.apache.org/schema/spring">
    <route>
        <from uri="file://incoming" />
  
        <to uri="bean:orderProcessor?method=processIncomingOrder" />
  
        <throttle timePeriodMillis="5000">
            <constant>1</constant>
            <to uri="activemq:queue:ORDERS.INCOMING" />
        </throttle>
    </route>
</camelcontext>

Implementing a Content-Based Router Using Camel

Often times, we're faced with the problem of routing a message at run-time based on the content of the message. Consider the following example. Let's say we receive all incoming orders in a single queue. However, we have two order processing services. The first service processes our normal customer orders, but we have a separate service that processes orders placed internally by employees. How do we solve this problem? A content-based router pattern solves this problem of determining where to route a message at run-time. Here's an example of a Content-Based Router implementing the example I described (the image is credited to http://www.enterpriseintegrationpatterns.com/ContentBasedRouter.html).

Thankfully, it's extremely easy to implement a content-based router pattern in Camel. If you're not familiar with using Camel, refer to my tutorial on Implementing Pipes and Filters Using Camel. It's a different integration pattern, but it walks through the basics of using Camel.

Now that you're familiar with Camel, let's look at how you would define a content-based router in your Camel route.

    <!-- content based router -->
    <route>
        <!-- get message from JMS ORDERS.INCOMING queue -->
        <from uri="activemq:queue:ORDERS.INCOMING">

            <!-- route the order to the correct queue -->
            <choice>
                <when>
                    <simple>${header.type} == 'customer'</simple>
                    <to uri="activemq:queue:ORDERS.CUSTOMER"></to>
                </when>
                <when>
                    <simple>${header.type} == 'employee'</simple>
                    <to uri="activemq:queue:ORDERS.EMPLOYEE"></to>
                </when>
            </choice>
        </from>
    </route>

Web Service Client in Java Using JAX-WS

This tutorial describes how to consume a SOAP web service in Java using JAX-WS. For this tutorial, we'll create a client application that makes a call to a SOAP service hosted on wsf.cdyne.com.

1. The first step is to import the WSDL of the web service to create the client proxy code. From the command prompt, navigate to the directory where you want to generate the client code and type: "wsimport -s . http://wsf.cdyne.com/WeatherWS/Weather.asmx?WSDL". After running this command, you'll find a \com\cdyne\ws\weatherws directory has been created with a number of .java and .class files.




 

2.  Next, we'll write a simple Java class and main method to invoke the web service.
  public class WeatherTest {
  
    public static void main(String args[]) {
        Weather weather = new Weather();
      
        WeatherSoap weatherSoap = weather.getWeatherSoap();
      
        ForecastReturn forecastReturn = weatherSoap.getCityForecastByZIP("56701");
      
        ArrayOfForecast arrayOfForecast = forecastReturn.getForecastResult();
        List<Forecast> forecasts = arrayOfForecast.getForecast();
      
        for (Forecast forecast : forecasts) {
            System.out.println(forecast.getDate().toString() + " " + forecast.getDesciption());
        }
    }

}



3. The output from this program is:

    2013-03-14T00:00:00 Mostly Cloudy
    2013-03-15T00:00:00 Snow
    2013-03-16T00:00:00 Partly Cloudy
    2013-03-17T00:00:00 Partly Cloudy
    2013-03-18T00:00:00 Snow
    2013-03-19T00:00:00 Partly Cloudy
    2013-03-20T00:00:00 Partly Cloudy

Implementing Pipes and Filters Using Camel

Pipes and Filters is an integration pattern that solves the problem of having to process a job or task in a series of sub-tasks. If you're not familiar with pipes and filters, read my post on describing the design: Pipes and Filters - Integration Pattern.

This tutorial describes how to implement the pipes and filters integration pattern using Apache Camel. The source code is on GitHub: https://github.com/travisdazell/camel-pipes-and-filters

The key component of this application is the Camel route that's defined in spring-context.xml:

<camelContext xmlns="http://camel.apache.org/schema/spring">

  <!-- pipes and filters example -->
  <route>
 <!-- get message from JMS FOO.BAR queue -->
 <from uri="activemq:queue:FOO.BAR" />

 <!-- use the simple language to transform the input to upper case -->
 <transform>
  <simple>${body.toUpperCase()}</simple>
 </transform>
 
 <!-- route the input to the messageReceiver bean, which will modify the message slightly -->
 <to uri="bean:messageReceiver?method=processMessage"/>
 
 <!-- route the output from the messageReceiver bean to the output console -->
 <to uri="stream:out"/>
  </route>

</camelContext>

This route has four steps/filters. The pipeline begins when a JMS message is posted on the FOO.BAR queue. Next, I'm using Camel to convert the message body to upper-case. The third step in the pipeline sends the result of the upper-case transformation to a Spring bean with an ID of "messageReceiver". Notice the use of "?method=processMessage". This allows us to route the message to the processMessage() method on the messageReceiver bean. The last step of the pipeline is to output the results of the processMessage() method to the console.

In summary, the following items are the main components when implementing the pipes and filters integration pattern in Camel:

1. Defining the Camel route. In this example, I'm using Spring to define the route.
2. The result of the first step is the input to the second step. The result of the second step is the input to the third step. And so on...
3. Use the following Camel URI to route a specific message in a Java class, where {beanId} is substituted with the ID of the Spring bean and {methodName} is the name of the method in the bean.

uri="bean:{beanId}?method={methodName}"

Using ActiveMQ as an XMPP (Jabber) Messaging Server

This tutorial demonstrates how to use ActiveMQ as a message broker for XMPP (Jabber). In this tutorial, we'll install ActiveMQ and use it for our message broker (i.e. server). We'll then use the Spark instant messaging client to create a chat room so that we can send and receive messages between IM clients.

The first step is installing ActiveMQ. For this example, we'll install everything on a Windows PC. Follow these steps to install ActiveMQ:

1. Go to http://activemq.apache.org and click the download link on the right-hand side of the page.
2. Click the link to download most recent stable version (5.7.0 at the time of this writing).
3. Click on the download link for the Windows distribution.

4. Unzip the download to a directory on your PC. I have it unzipped on the root of my C: drive.
5. Open the \apache-activemq-5.7.0\conf\activemq.xml file in an editor.
6. Edit the "transportConnectors" section of the configuration file by adding the XMPP transport.

        <transportConnectors>
               ...
               <transportConnector name="xmpp" uri="xmpp://0.0.0.0:61222"/>
        </transportConnectors>

7. From the command prompt, run \apache-activemq-5.7.0\bin\activemq

Now that you have ActiveMQ installed and the XMPP transport running on port 61222, open the following URL in your browser to view the ActiveMQ web console: http://localhost:8161/admin/

At this point, ActiveMQ is running and waiting for XMPP messages to arrive. We just need an IM client installed to start sending messages. Download and install Spark (http://www.igniterealtime.org/downloads/download-landing.jsp?file=spark/spark_2_0_0.exe). Note that I'm using v2.0.0.

After installing Spark, open it and enter the following at the login screen. Of course, you can use your own name, but put @localhost after your username and put "localhost" for the server.

Next, click on the "Advanced" link at the bottom of the login dialog. Within this screen, un-check the box for "Automatically discover host and port" and enter "localhost" for the Host and "61222" for the Port. Click OK.

Now click the "Login" button. After logging in, you'll see the following screen.

Next, we need to create a chat room. The chat room will auto-magically become a JMS Topic on the ActiveMQ server. From within Spark, there is a button underneath the "Online" indicator to "Join Conference Room". After clicking that button, you'll see the following dialog.

Next, click on the button for "Create or Join Room". You'll see the following dialog where you'll enter the name of the JMS Topic. I chose to name it CHAT.ROOM.

After clicking the "Create" button, click the "Join" button on the next dialog.

The next dialog will display the chat room window and show that you've joined the room.

The next step is to publish an IM message and watch it appear in our Spark client. To test this, go back to your browser where you have your ActiveMQ web console (http://localhost:8161/admin/). Click on the menu link that says "Topics". You'll see a list of JMS Topics on your ActiveMQ server and one of them will be the CHAT.ROOM Topic that you just created. Click on the link for that Topic.

After clicking on that link, you can post a message to the Topic by entering a message in the text area and clicking the "Send" button.

Now, if you go back to your Spark IM client, you'll see the message has been received by the client. That's it!

Pipes and Filters - Integration Pattern

Arguably one the simplest integration problems we encounter is having to process a complex job in a sequence of steps. Most of these problems are part of a workflow requirement. Examples include processing an order, approving a purchase order, and applying a credit to an account. Each of these business problems usually requires a series of steps to be performed in a certain sequence. This integration problem can be easily solved using Pipes and Filters. Here's an example of Pipes and Filters being applied to the processing of an incoming order (image is credited to http://www.eaipatterns.com/PipesAndFilters.html).





Essentially, Pipes and Filters is just a breakdown of a requirement into a series of connected steps. Each step is called a "filter" and the transitions between two filters is called a "pipe". When faced with having to process a large job in a series of processing steps, you should consider implementing the Pipes and Filters pattern.

It's important to implement a Pipes and Filters pattern in such a way that makes it easy to add and remove filters at any location in the process. To achieve this flexibility, I recommend using a framework such as Camel. Camel makes it very easy to add, remove, replace, or rearrange the processing steps with very little work and risk to your project. Read my tutorial on Implementing Pipes and Filters Using Camel to learn how Apache Camel makes this easy to implement and maintain.

Dependency Management - Avoid Cyclic Dependencies

We're always striving to design software that's easy to maintain. With every code change, we try to avoid code bloat and develop a solution that's simple, while still meeting the project requirements. As a result, we measure our success by whether or not somebody else can look at our code and understand it clearly. Code clarity is a very important measure of maintainability, but we also need to look at our project from a 1,000 foot view and take a moment to look at the dependencies in our project.

To start with, I recommend modeling each component and the compile-time dependencies amongst those components. As an example, think of each component as a Java package. Take, for example, the following high-level view of a system with just three components. Here, component A depends on component C, component B depends on component A, and component C depends on component A.

A red flag should go up if you see this kind of compile-time cyclic dependency in your projects. There are a number of problems with this type of cyclic dependency. First, you can't reuse just one component without bringing along the others. Second, if you modify one component, you indirectly affect the others. This creates a nightmare to test. Also, when a new requirement arrives on your desk, it's difficult at first glance how wide or deep the change will ripple through the system. In some cases, you're forced to follow the affected code through all of the components in order to determine the impact that a change requires. Architecture 101 teaches us that any change that goes wide or deep is a high-risk change. Cyclic dependencies often result in the smallest of changes affecting multiple components.

Avoid compile-time cyclic dependencies! We should always strive for acyclic dependencies. Take the following graph as an example. All I've done for this example is reversed the dependency from A => C to   C => A. This is just to illustrate the example. The result, however, is zero circular dependencies in our system. The issues discussed in the cyclic example go away (or at least aren't as serious).

This example is very trivial. In reality, we're usually working on very large applications with many, many components. So, it's clearly not as easy a task to avoid cyclic dependencies in a real world project. However, it's very important to manage your dependencies and avoid compile-time cyclic dependencies by keeping it at the forefront of your designs.

Camel File Poller and JMS Consumer in Scala

I put the code on  GitHub: https://github.com/travisdazell/Camel-Scala-FilePoller

In this tutorial, I'm going to demonstrate using Apache Camel to write a Scala file poller. When a file arrives in the "incoming" directory, I'm placing a JMS message onto a queue. Lastly, I'll have a JMS consumer written in Scala that will receive the message from the queue. I'll be using ActiveMQ as my JMS message broker. The main focal point in the tutorial is using Apache Camel for creating a file poller. I'm not doing anything special with ActiveMQ and the JMS consumer could just as easily have been written in Java. I just chose to use Scala, because I like it for its expressiveness.

I won't go through the steps of setting up the architecture. I'll cover that in a future tutorial. For now, I assume you have ActiveMQ running, the Camel JARs downloaded, and your Scala development environment ready to go. Now on to the code.

There are only two Scala classes needed. The first component is the Scala class that defines our Camel routes. We have a single route that routes incoming files to a JMS consumer. Here's our Camel route definition in Scala.

import org.apache.camel.scala.dsl.builder.RouteBuilder
import org.apache.camel.model.dataformat.BindyType

class JmsRoutes extends RouteBuilder {
    "file://incoming" --> "activemq:queue:test.queue"
}

The second component in the application is the JMS consumer. This consumer simply blocks, waiting for a file to arrive in a directory named "incoming". Here's the code for the JMS message consumer.

import javax.jms.ConnectionFactory
import org.apache.camel.CamelContext
import org.apache.camel.ConsumerTemplate
import org.apache.camel.impl.DefaultCamelContext
import org.apache.camel.component.jms.JmsComponent
import org.apache.activemq.ActiveMQConnectionFactory

object JmsConsumer {
    def main(args : Array[String]) {
        val connectionFactory = new ActiveMQConnectionFactory("vm://localhost?broker.persistent=false")
        val context = new DefaultCamelContext()

        context.addComponent("activemq", JmsComponent.jmsComponentAutoAcknowledge(connectionFactory))

        context.addRoutes(new JmsRoutes())

        val consumerTemplate = context.createConsumerTemplate()

        context.start()
       
        consumerTemplate.receive("activemq:queue:test.queue")

        println("received message")
       
        context.stop()
    }
}

This shows how simple it is to use Camel for routing messages in a Scala application. I like the expressiveness of the routing syntax. Camel is a very powerful integration framework, and yet it's extremely easy to use.

Exception Shielding Your Services

The most basic of all security mechanisms in SOA is shielding or hiding exceptions to the consumer of your services. Don't overlook this key feature when designing your services. Let's say you're constructing a service that inserts a record to an underlying database. Most likely, you'll have a try-catch somewhere in your service that handles SQLException(s). In a library routine that will be used by your internal applications, it's not much of a concern to expose the details of that exception to the caller. However, in a service that will potentially be consumed by outside clients, exposing the message within the exception could give the client too much information, revealing how your service is designed and built. This could be a security risk, allowing somebody to exploit the service.

To mitigate that risk, it's best to sanitize the exception and return a generic error message to the caller of the service. This can be done in a couple of ways:

1. You can have a utility service that resides within your local network (not accessible to the outside world) that takes an exception and the service method name as input, and returns a sanitized message that's safe to be returned to the client.
2. You can write the sanitation logic within each, individual service.

I prefer to use option 2, but that's just my personal preference. I find that every service has its own unique nuances that make it easier to deal with exceptions in the service that caught the exception. Often times, the message you return to the client will contain some verbiage regarding the service method/capability itself. For example, if you have a service method named "CreateCustomer" and a SQLException is caught within the service, you might return a message to the client that says something like, "An error occurred while creating the customer record". Having a single utility service that manages all of the possible mappings from exception to sanitized message can get bloated very quickly.

After you've decided on your strategy for sanitizing exceptions, you need to make sure you're persisting these exceptions with the full exception message (inner message, stack trace, etc.) somewhere for debugging purposes. A good approach is to have a database table where you log the exception message, along with a unique identifier, such as a GUID. When you return the sanitized message to the client, return the GUID along with it. That way, when a client reports an error message, you can lookup the raw exception message and debug the issue much easier.

Lastly, define a standard means for returning exceptions to the consumers of your services. If you're developing a SOAP service, you can make use of javax.xml.soap.SOAPException to wrap your exceptions that you return to the client. However you decide to do it, keep it consistent throughout your services, so that your consumers don't have to write custom exception handling code for every service they consume.

UI Mediator Design Patten

This blog post is an introduction to the UI Mediator SOA design pattern designed by [Utschig, Maier, Trops, Normann, Winterberg] and described in the book "SOA Patterns" by Thomas Erl. The book is a great read and I recommend it to any SOA implementor.

One of the main design goals of a Service Oriented Architecture is to promote flexibility and de-coupling of systems and applications via services. This often comes in the form of asynchronous messaging at the cost of super-fast performance. We choose a message-oriented middleware because it provides flexibility, but often times these messages are asynchronous and aren't guaranteed to return timely responses. The problem manifests itself in the form of a UI that is constantly waiting for a message response. That's where the UI Mediator pattern saves us.

The UI Mediator pattern solves this problem by placing a new "mediator" service between the caller and the service endpoint that does two things:

1. Invokes the destination service we need to consume
2. Returns status updates to the UI

Conceptually, the UI Mediator pattern looks like this. The Mediator Service returns updates to the caller while the background services are processing the request.




More often than not, it's acceptable to let the user continue working in another area of the application while a large operation is being processed. We don't want to block the user from working while we wait for a message response, but we need to notify them of progress and alert them when the process has completed successfully (or potentially returned an error at some point in the processing). The UI Mediator pattern does just that and it's a useful implementation when the use case allows background processing, but requires progress updates to the caller.