I had not heard of this approach before and am sure it will be useful in future, provided you have sufficient access to create the bindings in the first place.

A typical approach by performance testers is to rely on 95th percentiles, which is effectively a Service Level Agreement (SLA) saying that 95 percent of all my samples have a response time below 5 seconds. This is often specified as a Non Functional Requirement (NFR).

But so what? What does this really tell us? More importantly, if the SLA fails, you’re probably left hanging in the wind trying to explain how “bad” it failed to a (now interested) Project Manager. On the flip side, if everything is green, how close were you to failing? Are you an n’th degree away from failing? What about comparing two different application’s response time performance when they have different NFRs to begin with? Getting confused?

Enter the Apdex performance index. Apdex is a numerical measure of user satisfaction that can be built from metrics expressed via more traditional SLAs and/or NFRs. The fundamental objective of Apdex is to simplify the reporting of application response time measurements by making it possible to represent any such measurement using a common metric. This can be reported on by extracting data from Load Runner / JMeter and analysing within Excel as demonstrated below.

Read on to find out more about these scores and how they are calculated.
(more…)

Here is a great analogy of throughput and latency. I don’t pretend to understand queuing theory (just yet), but am working on it

To use FireWatir, you will have to install a Firefox plugin that enables use of a JavaScript Shell (jssh). FireWatir works on Windows, Mac and Linux.
Install FireWatir: [sudo] gem install FireWatir
Install Plugin: http://wiki.openqa.org/display/WTR/FireWatir

I will be using FireWatir on my mac for impromptu testing, since more recently I’ve found it beneficial to write automated test cases to provide consistent input when recording load test scenarios in JMeter or LoadRunner. For those interested in even more x-browser support, check out SafariWatir as well: http://safariwatir.rubyforge.org/

Watir is a simple open-source library for automating web browsers. It allows you to write tests that are easy to read and easy to maintain. It is optimized for simplicity and flexibility. Watir drives browsers the same way people do. It clicks links, fills in forms, presses buttons. Watir also checks results, such as whether expected text appears on the page.

Its main purpose will be to serve as a working repository of examples that answer questions newcomers have when scripting in watir. Its secondary purpose is to act as an extended regression suite of more applied test cases, which with the blessing of the lead developers of watir (Brett Pettichord) will help test future releases of watir. A working code base of unit tests is being maintained at http://code.google.com/p/justaddwatir/

If you’d like to get involved in the project or you have a blog about watir which you’d like to be aggregated, please send an email to tim.koops at gmail.com

However, if it is vital to your script (for example, a login page) then you can use an alternative recording tool called Badboy Software. I frequently use this to record HTTPS then export the test plan to JMeter. Alternatively you can manually create the HTTPS request sample yourself.

JMeter can run HTTPS request samplers, it just can’t record them using the native proxy recorder.

So a quick hack to do it in C is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

random_alpha(char* param_name, int length) {
  char buff[32] = "";
  int r,i;
  char c;
  srand((unsigned int)time(0)); //Seed number for rand()
  for (i = 0; i < length; i++) {
	// A-Z = 65-90 = rand() % 25 + 65
    r = rand() % 25 + 65;
    c = (char)r;
		buff[i] = c;
    printf("%c", c);
  }
  lr_save_string(buff, param_name);
  return 0;
}

I keep that as part of my functions library (referenced in globals.h) and can call on it in a script using the following syntax.
random_alpha(char* param_name, int length)

Example:
random_alpha("advertiser_name_suffix", 4);

Returns:
Notify: Saving Parameter "advertiser_name_suffix = SDNG"

In practice, correlation requirements relate to things like session ids, viewstate, cookies, date time stamps and other transactional information. So more often than not you will see this data changing following a post back or response from the server. With the introduction of AJAX calls, correlation requirements probably increase (per page) as the number of transactional requests probably increases as well.

In LoadRunner, you typically correlate data manually using the web_reg_save_param function. This in effect is a prepared statement, placed before the request that will return the data in the response you are looking for. For example, your function will specify a left and right boundary of the data that it is looking for before you make the request. Then the request is made and if a match is found, then the data is saved in a parameter which you can later call as a variable in subsequent requests. LoadRunner also has a feature called Correlation Studio which takes affect during recording of your script, using a similar approach (left and right boundaries) and then dynamically swapping corresponding matches with the variabilzed data. All this is quite painful though, and whenever I’m specifying primitive left and right boundaries or using the flaky Correlation Studio (I can explain if you like), I’m left wishing I could just use regular expressions …

The trick to correlating data easily with JMeter is to first know what data it is that needs correlating! (duh…) Experience will make this easier as a lot of common web apps have common dynamic data requirements as already mentioned. Another trick is to always record the script twice (and diff to see differences between unique user sessions), then more often than not record 2+ iterations of an action within a single script (and diff to see differences between user transactions/actions). If you do this before anything else, and diff the results, you will save bucketloads of heartache in getting your script harnesses running, no matter which tool you choose.

But back to JMeter, once you do know what data you need to correlate, simply create a Post Processor -> Regular Expression Extractor to ‘variablize’ that data. Huh? Read on for a more detailed explanation of this.
(more…)

If you roll-up your transactions into separate transaction controllers as I demonstrated in a previous post, you will get useful summary views for all your transactions:

Alternatively, drop a stat agg visualizer listener under a single controller to get a graph for a single transaction, useful if clients are only interested in certain transactions.

In the past I’ve relied heavily on trace options within LoadRunner and the like, but often find this a tad cumbersome. LiveHTTPHeaders is a great plugin for Firefox but it only captures header information. WireShark is great if you need to ‘go deep’, but often equates to opening a walnut with a sledgehammer. FireBug and IE Developer Toolbar are great to see what’s happening on a page-by-page basis. TCPdump is great if I’m on a Mac or Linux box, but it can be tricky parsing binary data. In any case most client sites are windows based for me.

So that’s a great list of tools to help you correlate and sniff out dynamic data, but the closest I’ve found for IE to doing what I want, that is, capturing all HTTP data going back and forth with the ability to filter for specific information from a console/command line is HttpWatch. A $295USD price tag / crippled basic (free) edition really forced me down this path…to roll my own HttpWatch using Ruby and the WEBBrick proxy.

Read on for a description of the code.
(more…)

The intent is to keep the data collection process detailed enough but simple. Presentation of the collected data then uses the API to present charts in a sparkline format.

Sparkline is a type of information graphics, that presents in a simple and condensed way trends and variation, associated with a measurement such as average temperature or stock market activity. These are often used as elements of a small multiple with several lines used together.

The Sparkline is proposed by Edward Tufte for “small, high resolution graphics embedded in a context of words, numbers, images” Tufte describes sparklines as “data-intense, design-simple, word-sized graphics”. Sparklines are intended to be succinct, memorable, and located where they are discussed.

The finished product looks like this:

Other functionality includes the ability to automatically ‘band’ expected data ranges for each metric collected, using a 30 day (customizable) sample that shows the 20 - 80th percentile. This should offer some insight into trends, and leverage off the sparkline format in being able to quickly assess a large range of metrics or performance data.

There’s also a SiteScope plugin, or in fact an emulator, so that if you’re still using LoadRunner you will be able to dip into the same metrics collected by NumbrCrunchr in a similar fashion to SiteScope with minimal configuration changes.

I’ve decided to go open source with this project in the interests of improving and continually updating its capabilities. If you’d like to contribute to the source please go to the project for more information. If you’d just like to see a quick demo, please go to NumbrCrunchr.com.

Given a list of n integers and another integer called m, determine (true / false) if there exist 2 numbers in that list which sum up to m.
Example: 2,6,4,9,1,12,7 and m=14 -> 2 and 12 sum up to 14, so the answer is true.
Provide the best algorithm in both manners: performance and memory to solve this puzzle. Don’t forget to mention the complexity of your solution!

Read on for my thoughts in solving this challenge.
(more…)

One of the first things I like to do when performance testing is establish in reasonable detail what the Application Simulation Model (ASM) will entail. You can create an ASM visually using something like UCML. This is a good starting point for performance testers looking to structure their test planning efforts.

UCML™ is a set of symbols that can be used to create visual system usage models and depict associated parameters. When applied to Performance Testing these symbols can serve to represent the workload distributions, operational profiles, pivot tables, matrixes, and Markov chains that performance testers often employ to determine what activities are to be included in a test and with what frequency they’ll occur.

More often than not though, I create an ASM using a spreadsheet format as I find this easiest to communicate with people from the business teams. If people are interested I will share some of the spreadsheet templates. The following screenshot gives you an idea of what I’m talking about:

The pertinent points for an ASM, is that it should:

e.g. “we have 3 user groups with a total of 150 users, 80% of them are helpdesk support, 10% are team leads and 10% are management”

e.g. “we’re aiming for around 240 trans/hour, of which 50% are searches, 25% are updates and 25% are reports”

Read on to discover how to turn ASMs into working JMeter test plans.
(more…)

Specifically I wanted to achieve the following:

To that end I was reasonably successful. The only limitation was that I only had my macbookpro with me and probably would have been shunned even if I attempted to load a windows XP virtual machine at a rails camp full of mac and linux geeks. So I chose FireWatir as my test platform running on my Mac, but am reasonably comfortable you can easily port this framework for use with Watir.

Now on with the framework contents …
(more…)

I’m currently using a wamp box to serve up content for performance metrics. What I needed was a Ruby script which could interrogate data on this installation.

You can download the binary required for this from here. After you install the gem (I do this off-line as my command prompt doesn’t allow me internet access here at work) you may find that you get an error similar to the following:
"NameError: uninitialized constant Mysql"

See what’s happening?
(more…)

In short this is a great (free) opportunity to attend a conference using an open session format. By that I mean the conference is run/organized like a wiki, where the attendees nominate topics they’d like to discuss or facilitate and then a user vote system organizes and makes it happen.

I avoided the performance testing oriented sessions and went for sessions that spoke about exploratory testing, defect management, benefits of test driven development, skills required for test automation and the last one titled ‘do we still need testers?’ …

They all generated a healthy amount of discussion and it was great to hear other people’s experienced opinions on the subject matter. It was also a great opportunity to network, find out who’s working on similar problems in the local industry and from afar. There was a healthy interest in using JMeter as a performance test tool to support CI, so I’ll be posting some more targeted blogs about this in the near future, as well as some consolidated info on performance metrics collection (without SiteScope) which a few people were also interested in.

Anyway, if you haven’t heard about the conference, keep an eye out for next years version. It’s well worth the zero dollar commitment, free beer and t-shirt. All you have to do is ‘use your feet’.
=)

* Works with badly formed HTML that can be found “in the wild”.
* The diffing is more specialized in HTML than XML tree differs. Changing part of a text node will not cause the entire node to be changed.
* In addition to the default visual diff, HTML source can be diffed coherently.
* Provides easy to understand descriptions of the changes.
* Allow easy browsing of the modifications through keyboard shortcuts.

Of late have been working with pesky AJAX controls inside a web-app, and one of my gripes with more traditional tools such as wdiff or diff, is that they don’t tell you *where* in the line the change has occurred. Read on for some hacky ways I’ve implemented daisydiff with LoadRunner and other open source tools in general …
(more…)

RailsCamp is a get together of mostly Rails developers, but also a fairly eclectic mix of Ruby hackers and geeks in general. 3 nights of drinking, camping and coding with NO internet. Possibly the hardest habit to kick is to stop checking for emails every 5 minutes.

For those with harder to kick habits such as Twittering, there was a ‘Twetter’ clone amongst other cool apps such as the ‘Duke’. The Duke was a user vote playlist driver for winamp (I think) and I got to observe just how close the geek community is when it comes to music tastes. Daft Punk was on fairly high rotation. Of mention was the first time we got Rick Rolled, which then led to a flurry of hack attempts at the guy hosting it on his Mac.

Guitar Hero was in force, as was Urban Terror and a touch of BZFlag. Nice.

Accommodation, setup and food was awesome, the organisers really did do a good job in getting this together. Big thanks to those guys! Perhaps the best part was the ability to network with like-minded people and discuss all things related to Ruby or Rails in a collaborative manner. I managed to get a little work done relating to a test framework for use with FireWatir which I’ll post later.

IF you didn’t or couldn’t make it, keep an eye out for the next one to occur. Definitely worth the $100 to nerd on for a full weekend …

I’ve been using this tool mostly from a windows box, and knew of the firefox equivalent, aptly named ‘Firewatir’ which is currently a Google code project. The beauty of this version, is that it allows you as a tester to explore functionality on your favourite platform for development, as I like to leave windows at work, and take OSX with me on the train.

Read on for some simple instructions in getting this running on your Macbook Pro …
(more…)

A work colleague found a knowledge base reference that recommended using an RTE Vuser for secure FTP recording and replay. Document ID 41820 refers. I don’t know, but that seemed like a kludge to me, and I wasn’t keen on a multi-protocol harness, so we went searching on Google for a Java alternative.
(more…)