IwTest: advanced test automation for webMethods™ IntegrationServer

Create and run automated unit tests, system tests and integration tests for SoftwareAG's webMethods™ IntegrationServer. Thanks to its declarative approach, IwTest let's you focus on data rather than on mechanics.

There are two costs associated with test cases:

 The cost to create them
 The cost to maintain them

and IwTest addresses both. Its main advantages are:

  1. IwTest minimizes the effort to create test cases by offering a unique record & generate facility, which supports generating dynamic stubs.
  2. IwTest minimizes the effort to maintain test cases by offering a unique repair facility. This is very useful should the service under test have changed.
  3. IwTest supports complex pub/sub-scenario's and BPM processes in a distributed environment.
  4. IwTest runs server-side and allows for native (FLOW code) validation extensions, fully leveraging Developer skills.

Version 1.8.4 of IwTest is now available!

Exciting new feature: dynamic code coverage


Download Trial Version

Real automation: service recording and test case generation

IwTest's unique feature is the ability to record service invocations server-side and use the recorded inputs and outputs to generate test cases, including stubs. No need to manually save inputs and outputs.

Creating a test suite is as simple as:

  1. Select a service for recording
    • Optionally: select one or more services for which you'd like to create stubs.


  2. Execute the service with different inputs


  3. Generate the test suite
    • ... or append test cases to an existing test suite


  4. Optionally post-edit to:
    • give the test cases meaningful names
    • set the type of the test suite
    • define regular expressions for fields with varying contents, like timestamps.


That's it.

Recording not only works on the IntegrationServer you're developing on. You can also record services in an environment with more realistic data, e.g. in your User Acceptance Test environment and download the recorded data for test suite generation.

Note: Exceptions are recorded and treated as expected error situations. IwTest will include them automatically when generating a test suite.

Running Test Suites

You can incorporate IwTest easily in your CI/CD-pipeline. Each test suite is a regular flow service, so deploying the tests is not any different than deploying a package. Use for example curl or wget to run your test suites. These filters are supported in order to select the test suites to run:

  • Type: unit, system or integration
  • Package name
  • Folder name
  • Service name

Run all available:

curl http://host:5555/invoke/iw.test.execution.pub/run

Run only unit tests:

curl http://host:5555/invoke/iw.test.execution.pub/run?type=unit

Run only the test suites in the packages whose names start with Acme:

curl http://host:5555/invoke/iw.test.execution.pub/run?packageFilter=Acme.*

Download the results in JUnit format:

curl http://host:5555/invoke/iw.test.results.pub/export?format=junit

Global Code Coverage

Get insight in your test case coverage. Discover for which percentage of the services you do, and for which percentage you don't have test cases. Drill down per package in order to discover exactly which service does have test cases, and which they are, and which services do not have any.

Dynamic Code Coverage analysis

Get insight into what part of you code base is actually touched by the test cases you've created. Get a look into the flow code and see which paths were executed.

Use cases

Unit tests

Use IwTest to create unit tests in no time. Let IwTest record the service invocations while you manually execute your flow services

System tests

Use IwTest to create system tests with equally little effort by additionally recording data for the services you would like to stub.

Integration tests

Use IwTest to create end-2-end tests by adding callbacks and additional stubs for asynchronously executed services

webMethods upgrade project: regression tests

Are you facing a webMethods upgrade and you don't have a set of reliable regression tests? Use IwTest to create them quickly. Let IwTest record service invocations in your QA environment and build a regression test suite over night! You're saving incredible amounts of time, effort and tedious work.

Features of IwTest

  • Facility to record service invocations and generate test cases based on the recorded data, including dynamic stubs.
  • Support for remote service recording
  • Facility for validating intermediate results
  • Newly recorded test cases can be appended to an existing test suite
  • Facility to repair all test cases in a suite in case the underlying service has changed.
  • Static validation of service inputs and outputs against their definition
  • Facility to execute a service, against the local or a remote IS, outside the context of a test case
  • Advanced duplication functionality: when you duplicate a test case, IwTest will actually re-run the test case, possibly which altered input data, and leverage the recording facility to create a new test case
  • Support for pub/sub scenario's: assertions against asynchronous events
  • Support for distributed environments with two or more Integration Servers
  • No extra tooling or servers needed: a test suite is just another webMethods™ flow service
  • Detailed error reporting: all validation errors are reported, not just the first one
  • Dynamic test suite execution: no need to add each test suite to a list for execution
  • Global Code Coverage Analysis: graphs showing globally and per package how many test cases there are, covering how many services.
  • Dynamic Code Coverage Analysis: get insight into what percentage of your code base is actually executed by your tests. Inspect in detail which paths of you flow code has actually been visited.
  • Test execution report in JUnit format for easy integration with e.g. Jenkins
  • Flexible assertions:
    • against XPath-like expressions
    • against a saved pipeline
    • with support for regular expressions
  • Very easy installation: it's just two IntegrationServer packages with no external dependencies
  • Completely web-based, so no Eclipse plugin
  • Deploy test suites like any other flow service


back to top

Repairing failed test cases

IwTest minimizes the time necessary to maintain your test cases with it unique ability to repair test cases.

Say you have a service with 25 test cases and suppose you need to change the service. Now you have 25 failing test cases. Rather than update each test case manually, you can use the repair facility and have IwTest update them for you. It will even retain all the regular expressions you might have defined in the expected results.

Dynamic Stubbing with IwTest

When one of the services that you want to stub is executed multiples times when you invoke the top-level service, things get complicated. Of course you don't want it to return the same result every time, and it's cumbersome to write an actual stub service as a replacement for the real service.

This scenario is tackled by IwTest by using dynamic stubs. IwTest will record the different input/output combinations of the sub-service. During test case execution IwTest will replace the calls to the external service with a dynamic stub. The actual input is matched against the recorded inputs and on a match the corresponding output (or exception!) is returned.

Pub/Sub scenario's

IwTest supports pub/sub scenario's via so-called callbacks. In your test case you tell IwTest to capture the pipeline that's present at the beginning or end of a service invocation and run assertions against that data. This service may be located on another IntegrationServer!

IwTest doesn't care how the (asynchronous) execution of that service was triggered. It may have been a Broker message, JMS message, MQTT event, Web Service call etc.

By the way, you're not limited by just one callback: you can define as many in a test case as you like.

Of course on the remote IntegrationServer you can again define stubs so you can prevent that the asynchronously executed service sends data to an external system.


back to top

Demonstration

Watch this video to get an impression of how IwTest works:


back to top

Continuous Integration support

Execute your test cases from your Continuous Integration tool (e.g. Jenkins, or Bamboo) with a single command. With another single command download the results in JUnit-format.

Jenkins has an excellent JUnit plugin that keeps history of the various test runs, highlights the differences between them, produces nice graphs and notifies developers when a build failed or has become unstable.


back to top

Licensing

You can download and evaluate IwTest for free. The first 90 days it's fully functional, only the number of test cases that you can execute is limited to 50. After the evaluation period the recording and test suite generation function is disabled.

You can either purchase a permanent license or take a subscription. The permanent license doesn't expire, but is only valid for a major version. With a subscription on the other hand you're entitled to all upgrades.

Trial Permanent Subscription
# Test cases 50 Unlimited Unlimited
Stubbing
Async
Recording 90 Days
Bug fixes 90 Days
Minor upgrades 90 Days
Major upgrades 90 Days

You can request a quote by sending an email to


back to top

Download IwTest

IwTest consists of two regular IS packages: IwTest & IwTestAgent. A package with samples is also included.

Version Date Distribution MD5 Checksum
1.8.4 February 10 2024 IwTest v1.8.4 c952999cd04365db4cb4707fc8e7fc37
Instructions:
  • Download the distribution & unpack the zip file
  • Copy IwTest.zip & IwTestAgent.zip to the 'replicate/inbound' directory of your IS
  • Optionally also copy IwTestSamples.zip
  • In Administrator: Packages -> Management -> Install Inbound Release. No restart required!
  • IwTest is available at [IS url]/IwTest and under 'Solutions' in the IS Admin page

Upgrades

The latest version of the major release are always downloadable from this website. If you purchased a permanent license and would like to upgrade to the newest new major version, then ask for a quote by sending an email to

Support

You can send bug reports and feature requests to


back to top

Planned features

The following features are planned for upcoming releases:

  • Test suite level stubs
  • Statistics gathering for top-level services


back to top

Frequently Asked Questions

WmTestSuite lets you leverage inputs and outputs you've gathered in Designer. In IwTest however you can record on the server and include sub-services in order to generate stubs for them. Furthermore, IwTest supports asynchronous and distributed scenario's, which WmTestSuite does not. Besides, it lacks a repair function.

CATE provides similar and more advanced functionality, but divided over two different products (CATE Developer Edition (DE) and CATE Enterprise Edition (EE)) and against a much higher license fee. Besides, CATE lacks a repair function.

IwTest does not support (JMS) Messaging directly. However, IwTest does support asynchronous (pub/sub) and distributed scenario's.

In your test case you can wait for a service to execute asynchronously, either on the same IS, or on a remote IS, and validate its inputs, outputs or even exception.

If there is no subscribing service (yet), then you could also create a separate trigger in your test package and subscribe to the message that you want to capture.

Note that you can extend your test case with multiple asynchronous events.

IwTest supports webMethods 9.6 and higher. The sources are compiled targeting Java 6.

Test cases come at a cost. First there is the cost to create them (with IwTest, that cost is relatively small, but still) and later there is the cost to update them as you apply fixes or enhance the functionality.

As a first approach, focus on your transformation services (of course you've separated your transformation logic from the tranport logic, right?).

The best time is when initial development is done and your code is in functional acceptance testing, but before it's released to production.

Later you can shift your focus to end-to-end scenarios and - time and budget permitting - to utility services.

If you have your test data available, then with IwTest's record & generate facility, it's literally a matter of minutes. Also if you want to stub services.

If on the other hand you need to prepare your test data and vary it in order to test different mapping variations or to evoke different responses from the target system, then it will take as long as it takes you to prepare your data.

Yes, but IwTest cannot link asynchronous events to one test case. So the data will be recorded, but in pub/sub scenario's you'll have to manually extend a test case it with a so-called 'callback' to add an asynchronous event.

No, IwTest doesn't produce excessive amounts of data unless you tell it to. There is a parameter that controls the minimum interval between two service invocation recordings. The default value is 2 seconds.

Yes! Just specify the services IwTest should collect data for.

Warning! IntegrationWise does not recommend to use IwTest in a production environment. Do this only in a QA environment.

Obviously there is a performance penalty when recording data that flows through you integration platform. And although IwTest has been tested extensively, IntegrationWise cannot assume responsibility for any negative effects the use of IwTest might have.

IwTest supports the same data types the IDataXMLCoder class supports, so that's everything you can do a 'Save Pipeline to File' for. So IwTest does not support an xml 'node'. If the service you want test takes a node as an input, simply create a wrapper flow service that takes an xml-string as an input, call 'pub.xml:xmlStringToXMLNode' and then invoke your flow service with the 'node'.

Most probably your service produces different results each time it's executed. There might be for example a timestamp in the outputs, or an auto-generated id. This is something IwTest cannot detect.

The solution is to use a regular expression in the expected results. This also works for fields that are numbers or dates. For example replace:

<number type="java.lang.Long">1829382910123</number>

with:

<value>/\d+/</value>

Because CI tools can take care of this. Jenkins for example has a marvelous JUnit plugin. It compares the test results of the current run with those of the previous runs and presents the differences nicely. In addition it draws nice graphs of history of the test runs. There is nothing IwTest could add here.

No, use tools like JMeter to do that. Nevertheless, IwTest can be of great help here. Before you run your performance tests you can tell IwTest to create stubs for those services that interact with an external system. By isolating your integration platform like this, you can much better judge how well it performs. Usually the external systems are the performance bottle necks, not the integration platform.

No, IwTest does not require wM DevOps Edition.

About

IwTest is a product of IntegrationWise B.V., a Netherlands based IT consultancy that specialises in integration projects using SoftwareAG's webMethods™ Digital Business Platform.