Iago, A Load Generator README

• Iago Quick Start
  • Iago Prerequisites
  • Preparing Your Test
  • Executing Your Test
• Iago Overview
  • Supported Services
  • Transaction Requirements
  • Sources of Transactions
• Iago Architecture Overview
• Implementing Your Test
  • Scala Example
  • Scala Thrift Example
  • Java Example
  • Java Thrift Example
  • Code Annotations for the Examples
• Configuring Your Test
• Contributing to Iago

Iago Quick Start

NOTE: This repo has only recently been made public and our velocity is high at the moment. Please join [email protected] for updates and to ask questions.

If you are already familiar with the Iago Load Generation tool, follow these steps to get started; otherwise, start with the Iago Overview and perhaps Iago Philosophy, also known as "Why Iago?". For questions, please contact [email protected].

Iago Prerequisites

1. Download and unpack the Iago distribution. We support Scala 2.9 and recommend you clone the latest master: master.

2. Read the documentation.

Preparing Your Test

1. Identify your transaction source; see Transaction Requirements and Sources of Transactions for more information.
2. In Scala, extend the Iago server's RecordProcessor or ThriftRecordProcessor class, or in Java, extend LoadTest or ThriftLoadTest; see Implementing Your Test for more information.
3. Create a launcher.scala file in your Iago config directory with the appropriate settings; see Configuring Your Test for more information.

Executing Your Test

Launch Iago from the distribution with java -jar iago_jar -f your_config. This will create the Iago processes for you and configure it to use your transactions. To kill a running job, add -k to your launch parameters: java -jar iago_jar -f your_config -k.

If you launch your Iago job on your local machine and an old Iago job is still running, it probably won't get far: it will attempt to re-use a port and fail. You want to kill the running job, as described above.

If you build via Maven, then you might wonder "How do I launch Iago 'from the distribution'?" The steps are:

% mvn package -DskipTests
% mkdir tmp; cd tmp
% unzip ../target/iago-version-package-dist.zip
% java -jar iago-version.jar -f config/my_config.scala

Don't assume that you can skip the package/unzip steps if you're just changing a config file. You need to re-package and unzip again.

If you are using Iago as a library, for example, in the case of testing over the Thrift protocol or building more complex tests with HTTP or Memcached/Kestrel, you should instead add a task to your project's configuration. See Configuring Your Test for more information.

Top

Iago Overview

Iago is a load generation tool that replays production or synthetic traffic against a given target. Among other things, it differs from other load generation tools in that it attempts to hold constant the transaction rate. For example, if you want to test your service at 100K requests per minute, Iago attempts to achieve that rate.

Because Iago replays traffic, you must specify the source of the traffic. You use a transaction log as the source of traffic, in which each transaction generates a request to your service that your service processes.

Replaying transactions at a fixed rate enables you to study the behavior of your service under an anticipated load. Iago also allows you to identify bottlenecks or other issues that may not be easily observable in a production environment in which your maximum anticipated load occurs only rarely.

Top

Supported Services

Iago can generate service requests that travel the net in different ways and are in different formats. The code that does this is in a Transport, a class that extends ParrotTransport. Iago comes with several Transports already defined. When you configure your test, you will need to set some parameters; to understand which of those parameters are used and how they are used, you probably want to look at the source code for your test's Transport class.

• HTTP: Use FinagleTransport
• Thrift: Use ThriftTransport
• Memcached: Use MemcacheTransport
• Kestrel: Use KestrelTransport
• UDP: Use ParrotUdpTransport

Your service is typically an HTTP or Thrift service written in either Scala or Java.

Top

Transaction Requirements

For replay, Iago recommends you scrub your logs to only include requests which meet the following requirements:

• Idempotent, meaning that re-execution of a transaction any number of times yields the same result as the initial execution.
• Commutative, meaning that transaction order is not important. Although transactions are initiated in replay order, Iago's internal behavior may change the actual execution order to guarantee the transaction rate. Also, transactions that implement Future responses are executed asynchronously. You can achieve ordering, if required, by using Iago as a library and initiating new requests in response to previous ones. Examples of this are available.

Unless you change your configuration's reuseFile parameter, make sure that your sample log has at least 1000 items.

Top

Sources of Transactions

Transactions typically come from logs, such as the following:

• Web server logs capture HTTP transactions.
• Proxy server logs can capture transactions coming through a server. You can place a proxy server in your stack to capture either HTTP or Thrift transactions.
• Network sniffers can capture transactions as they come across a physical wire. You can program the sniffer to create a log of transactions you identify for capture.

In some cases, transactions do not exist. For example, transactions for your service may not yet exist because they are part of a new service, or you are obligated not to use transactions that contain sensitive information. In such cases, you can provide synthetic transactions, which are transactions that you create to model the operating environment for your service. When you create synthetic transactions, you must statistically distribute your transactions to match the distribution you expect when your service goes live.

Top

Iago Architecture Overview

Iago consists of one or more Iago feeders, which reads your transaction source, and one or more Iago servers, which format and deliver requests to the service you want to test. The feeder contains a Poller object, which is responsible for guaranteeing one minute's worth of transactions in the pipeline to the Iago servers. Metrics are available in logs, and we expect future enhancements to support parsing and visualizing this data.

The Iago servers generate requests to your service. Together, all Iago servers generate the specified number of requests per minute. A Iago server's RecordProcessor object executes your service and maps the transaction to the format required by your service.

Top

Implementing Your Test

The following sections show examples of implementing your test in both Scala and Java. See Code Annotations for the Examples for information about either example.

Top

Scala Example

To implement a load test in Scala, you must extend the Iago server's RecordProcessor class to specify how to map transactions into the requests that the Iago server delivers to your service. The following example shows a RecordProcessor subclass that implements a load test on an EchoService HTTP service:

package com.twitter.example

import org.apache.thrift.protocol.TBinaryProtocol

import com.twitter.parrot.processor.RecordProcessor                                     // 1
import com.twitter.parrot.thrift.ParrotJob                                              // 2
import com.twitter.parrot.server.{ParrotRequest,ParrotService}                          // 3
import com.twitter.logging.Logger
import org.jboss.netty.handler.codec.http.HttpResponse

import thrift.EchoService

class EchoLoadTest(parrotService: ParrotService[ParrotRequest, HttpResponse]) extends RecordProcessor {
  val client = new EchoService.ServiceToClient(service, new TBinaryProtocol.Factory())  // 4
  val log = Logger.get(getClass)

  def processLines(job: ParrotJob, lines: Seq[String]) {                                // 5
    lines map { line =>
      client.echo(line) respond { rep =>
        if (rep == "hello") {
          client.echo("IT'S TALKING TO US")                                             // 6
        }
        log.info("response: " + rep)                                                    // 7
      }
    }
  }
}

Top

Scala Thrift Example

To implement a Thrift load test in Scala, you must extend the Iago server's Thrift RecordProcessor class to specify how to map transactions into the requests that the Iago server delivers to your service. The following example shows a ThriftRecordProcessor subclass that implements a load test on an EchoService Thrift service:

package com.twitter.example

import org.apache.thrift.protocol.TBinaryProtocol

import com.twitter.parrot.processor.ThriftRecordProcessor                               // 1
import com.twitter.parrot.thrift.ParrotJob                                              // 2
import com.twitter.parrot.server.{ParrotRequest,ParrotService}                          // 3
import com.twitter.logging.Logger

import thrift.EchoService

class EchoLoadTest(parrotService: ParrotService[ParrotRequest, Array[Byte]]) extends ThriftRecordProcessor(parrotService) {
  val client = new EchoService.ServiceToClient(service, new TBinaryProtocol.Factory())  // 4
  val log = Logger.get(getClass)

  def processLines(job: ParrotJob, lines: Seq[String]) {                                // 5
    lines map { line =>
      client.echo(line) respond { rep =>
        if (rep == "hello") {
          client.echo("IT'S TALKING TO US")                                             // 6
        }
        log.info("response: " + rep)                                                    // 7
      }
    }
  }
}

Top

Java Example

To implement a load test in Java, you must extend the Iago server's LoadTest class to specify how to map transactions into the requests that the Iago server delivers to your service. The LoadTest class provides Java-friendly type mappings for the underlying Scala internals. The following example shows a LoadTest subclass that implements a load test on an EchoService HTTP service:

package com.twitter.jexample;

import com.twitter.example.thrift.EchoService;
import com.twitter.parrot.processor.LoadTest;                                           // 1
import com.twitter.parrot.thrift.ParrotJob;                                             // 2
import com.twitter.parrot.server.ParrotRequest;                                         // 3

import com.twitter.parrot.server.ParrotService;                                         // 3
import com.twitter.util.Future;
import com.twitter.util.FutureEventListener;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.jboss.netty.handler.codec.http.HttpResponse

import java.util.List;

public class EchoLoadTest extends LoadTest {
  EchoService.ServiceToClient client = null;

  public EchoLoadTest(ParrotService<ParrotRequest, HttpResponse> parrotService) {
    super(parrotService);
    client = new EchoService.ServiceToClient(service(), new TBinaryProtocol.Factory()); // 4
  }

  public void processLines(ParrotJob job, List<String> lines) {                         // 5
    for(String line: lines) {
      Future<String> future = client.echo(line);
      future.addEventListener(new FutureEventListener<String>() {
        public void onSuccess(String msg) {
          System.out.println("response: " + msg);
        }

      public void onFailure(Throwable cause) {
        System.out.println("Error: " + cause);
      }
     });
    }
  }
}

Top

Java Thrift Example

To implement a Thrift load test in Java, you must extend the Iago server's ThriftLoadTest class to specify how to map transactions into the requests that the Iago server delivers to your service. The ThriftLoadTest class provides Java-friendly type mappings for the underlying Scala internals. The following example shows a ThriftLoadTest subclass that implements a load test on an EchoService Thrift service:

package com.twitter.jexample;

import com.twitter.example.thrift.EchoService;
import com.twitter.parrot.processor.ThriftLoadTest;                                     // 1
import com.twitter.parrot.thrift.ParrotJob;                                             // 2
import com.twitter.parrot.server.ParrotRequest;                                         // 3
import com.twitter.parrot.server.ParrotService;                                         // 3
import com.twitter.util.Future;
import com.twitter.util.FutureEventListener;
import org.apache.thrift.protocol.TBinaryProtocol;

import java.util.List;

public class EchoLoadTest extends ThriftLoadTest {
  EchoService.ServiceToClient client = null;

  public EchoLoadTest(ParrotService<ParrotRequest, byte[]> parrotService) {
    super(parrotService);
    client = new EchoService.ServiceToClient(service(), new TBinaryProtocol.Factory()); // 4
  }

  public void processLines(ParrotJob job, List<String> lines) {                         // 5
    for(String line: lines) {
      Future<String> future = client.echo(line);
      future.addEventListener(new FutureEventListener<String>() {
        public void onSuccess(String msg) {
          System.out.println("response: " + msg);
        }

      public void onFailure(Throwable cause) {
        System.out.println("Error: " + cause);
      }
     });
    }
  }
}

Top

Code Annotations for the Examples

You define your Iago subclass to execute your service and map transactions to requests for your service:

1. Import com.twitter.parrot.processor.RecordProcessor (Scala) or LoadTest (Java), whose instance will be executed by a Iago server.
2. Import com.twitter.parrot.thrift.ParrotJob, which contains the Iago server class.
3. Import com.twitter.parrot.server.ParrotService and com.twitter.parrot.server.ParrotRequest
4. Create an instance of your service to be placed under test. Your service is a client of the Iago service.
5. Define a processLines method to format the request and and execute your service.
6. Optionally, you can initiate a new request based on the response to a previous one.
7. Optionally, do something with the response. In this example, the response is logged.

Top

Configuring Your Test

To configure your test, create a launcher.scala file that that creates a ParrotLauncherConfig instance with the configuration parameters you want to set.

There are several parameters to set. A good one to figure out early is transport; that will in turn help you to find out what, e.g., responseType you need.

The following example shows parameters for testing a Thrift service:

import com.twitter.parrot.config.ParrotLauncherConfig

new ParrotLauncherConfig {
  distDir = "."
  jobName = "load_echo"
  port = 8080
  victims = "localhost"
  log = "logs/yesterday.log"
  requestRate = 1
  numInstances = 1
  duration = 5
  timeUnit = "MINUTES" // affects duration; does not affect requestRate

  imports = "import com.twitter.example.EchoLoadTest"
  responseType = "Array[Byte]"
  transport = "ThriftTransport"
  loadTest = "new EchoLoadTest(service.get)"
  parser = "thrift"
}

Note: For a sample configuration file, see config/launcher.scala within the Iago distribution.

You can specify any of the following parameters:

Parameter	Description	Required or Default Value
jobName	A string value that specifies the the name of your test. Example: jobName = "testing_tasty_new_feature"	Required
log	A string value that specifies the complete path to the log you want Iago to replay. The log should be on your local file system. The log should have at least 1000 items or you should change the reuseFile parameter. Example: log = "logs/yesterday.log"	Required
victims	A string of comma-separated values that specify the hosts on which to execute the load test. Example: victims = "www1,www2"	Required
port	An integer value that specifies the port on which to deliver requests to the victims. Example: port = 9000	Required
distDir	The subdirectory of your project you're running from, if any. Example: distDir = "target"	"."
customLogSource	A string with Scala code that will be put into the Feeder config Example: customLogSource = "FILL IN HERE"	""
scheme	A string value that specifies the scheme portion of a URI. Example: scheme = "http"	http
header	A string value that specifies the HTTP Host header. Example: header = "api.yourdomain.com"	""
duration	An integer value that specifies the time to run the test in timeUnit units. Example: duration = 5
timeUnit	A string value that specifies time unit of the duration. It contains one of the following values: "MINUTES" "HOURS" "DAYS" Example: timeUnit = "MINUTES"
maxRequests	An integer value that specifies the total number of requests to submit to your service. Example: maxRequests = 10000	1000
reuseFile	A boolean value that specifies whether or not to stop the test when the input log has been read through. Setting this value to true will result in Iago starting back at the beginning of the log when it exhausts the contents. If this is true, your log file should at least be 1,000 lines or more. Example: reuseFile = false	true
requestRate	An integer value that specifies the number of requests per second to submit to your service. Example: requestRate = 10 Note: if using multiple server instances, requestRate is per-instance, not aggregate.	1
loggers	A List of LoggerFactories; allows you to define the type and level of logging you want Example: import com.twitter.logging.LoggerFactory import com.twitter.logging.config._ new ParrotLauncherConfig { ... loggers = new LoggerFactory( level = Level.DEBUG, handlers = new ConsoleHandlerConfig() ) }	Nil
numFeederInstances	Will bring up the specified number of feeder instances Example: numFeederInstances = 2	1
numInstances	An integer value that specifies the number of Iago servers concurrently making requests to your service. Example: numInstances = 2	1
parser	A string value that specifies how the request is to be interpreted. It is one of the following values: "http" "thrift" Example: parser = "thrift"	http
verboseCmd	A boolean value that specifies the level of feedback from Iago. A value of true specifies maximum feedback. Example: verboseCmd = true	false
reuseConnections	A boolean value that specifies whether connections to your service's hosts can be reused. A value of true enables reuse. Setting this to false greatly increases your use of ephemeral ports and can result in port exhaustion, causing you to achieve a lower rate than requested Example: reuseConnections = false	true
doConfirm	If set to false, you will not be asked to confirm the run. Example: doConfirm = false	true
hostConnectionCoresize	Number of connections per host that will be kept open, once established, until they hit max idle time or max lifetime Example: hostConnectionCoresize = 1	1
hostConnectionLimit	Limit on the number of connections per host Example: hostConnectionLimit = 4	Integer.MAX_VALUE
hostConnectionIdleTimeInMs	For any connection > coreSize, maximum amount of time, in milliseconds, between requests we allow before shutting down the connection Example: hostConnectionIdleTimeInMs = 50000	60000
hostConnectionMaxIdleTimeInMs	The maximum time in milliseconds that any connection (including within core size) can stay idle before shutdown Example: hostConnectionMaxIdleTimeInMs = 500000	300000
hostConnectionMaxLifeTimeInMs	The maximum time in milliseconds that a connection will be kept open Example: hostConnectionMaxLifeTimeInMs = 10000	Integer.MAX_VALUE
maxPerHost	Maximum number of parrot_server instances per mesos box Example: maxPerHost = 3	1
serverXmx	Defines heap size. Suggested not to be higher than 8 GB (will cause issues scheduling) Example: serverXmx = 5000	4000
thriftClientId	If you are making Thrift requests, your clientId Example: thriftClientId = "projectname.staging"	""
createDistribution	You can use this field to create your own distribution rate, instead of having a constant flow. You will need to create a subclass of RequestDistribution and import it. Example: createDistribution = """createDistribution = { rate => new MyDistribution(rate) }"""	""

####Extension Point Parameters

Alternative Use: You can specify the following extension point parameters to configure projects in which Iago is used as both a feeder and server. The Iago feeder provides the log lines to your project, which uses these log lines to form requests that the Iago server then handles:

Parameter	Description	Required or Default Value
imports	Imports from this project to Iago Example: If ProjectX includes Iago as a dependency, you would specify: import org.jboss.netty.handler.codec.http.HttpResponse import com.twitter.projectX.util.ProcessorClass	import org.jboss.netty.handler.codec.http.HttpResponse import com.twitter.parrot.util.LoadTestStub
requestType	The request type of requests from Iago. Examples: ParrotRequest for most services (including HTTP and Thrift)	ParrotRequest
responseType	The response type of responses from Iago. Examples: HttpResponse for an HTTP service Array[Byte] for a Thrift service	HttpResponse
transport	The kind of transport to the server, which matches the responseType you want. Example: The Thrift Transport will send your request and give back Future[Array[Byte]].	FinagleTransport
loadTest	Your processor for the Iago feeder's lines, which converts the lines into requests and sends them to the Iago server. Example: new LoadTestStub(service.get)	new LoadTestStub(service.get)

Top

Contributing to Iago

Iago is open source, hosted on Github here. If you have a contribution to make, please fork the repo and submit a pull request.