JMeter Tips and Tricks – Correlating with Regular Expressions

Another post in this current theme of JMeter tips and tricks… You will hear LoadRunner consultants preaching to the converted about the requirement to correlate data accurately. What they’re referring to, is making sure that dynamic data received by the client from the server (typically in a response body, header or url), which changes from request to request, is accurately correlated. If you don’t, you will probably ‘break’ the application logic with most obvious signs during replay being things like HTTP Server 500 errors.

In practice, correlation requirements relate to things like session ids, viewstate, cookies, date time stamps and other transactional information. So more often than not you will see this data changing following a post back or response from the server. With the introduction of AJAX calls, correlation requirements probably increase (per page) as the number of transactional requests probably increases as well.

In LoadRunner, you typically correlate data manually using the web_reg_save_param function. This in effect is a prepared statement, placed before the request that will return the data in the response you are looking for. For example, your function will specify a left and right boundary of the data that it is looking for before you make the request. Then the request is made and if a match is found, then the data is saved in a parameter which you can later call as a variable in subsequent requests. LoadRunner also has a feature called Correlation Studio which takes affect during recording of your script, using a similar approach (left and right boundaries) and then dynamically swapping corresponding matches with the variabilzed data. All this is quite painful though, and whenever I’m specifying primitive left and right boundaries or using the flaky Correlation Studio (I can explain if you like), I’m left wishing I could just use regular expressions …

The trick to correlating data easily with JMeter is to first know what data it is that needs correlating! (duh…) Experience will make this easier as a lot of common web apps have common dynamic data requirements as already mentioned. Another trick is to always record the script twice (and diff to see differences between unique user sessions), then more often than not record 2+ iterations of an action within a single script (and diff to see differences between user transactions/actions). If you do this before anything else, and diff the results, you will save bucketloads of heartache in getting your script harnesses running, no matter which tool you choose.

But back to JMeter, once you do know what data you need to correlate, simply create a Post Processor -> Regular Expression Extractor to ‘variablize’ that data. Huh? Read on for a more detailed explanation of this.

In my following example, I have 4 dummy pages that return unique jsessionid’s for each page as follows:

  • jsessionid=111_CATSNDOGS!50775&
  • jsessionid=222_BATSNCATS!021278&
  • jsessionid=333_SHEEPNGOATS!140679&
  • jsessionid=444_FISHNCHIPS!120880&
  • I’ve done this to simulate the changing session information typically found in any jsp based web app. To look for a jsessionid I simply add a new Post Processor called the Regular Expression Extractor. The following screenshot shows an example:
    regex extractor

    In the example provided, the reference name is what I can later call upon as a variable within the script, for example, ${jsessionid_trans}. The regular expression is the pattern I am looking for. Note in using a parenthesis I’m specifying a subexpression which I can get to with the match result number of n(1). If you’re new to regex, my example is equivalent to saying look for a left boundary of jsessionid= then match any character until you hit an ampersand. Even if you know nothing about regex, this will probably get you out of hot water 90% of the time. The real power in regex though is the ability to manipulate and narrow in on search results, or search for varying patterns (like telephone numbers, alphanumeric sequences and so on). I still find the Perl regular expression tutorials the best:
    http://perldoc.perl.org/perlretut.html

    So that’s it, you’ve found the jsessionid and to use it later on in your script, you simply call on it as a variable i.e. ~koops/test_jmeter/page2.html?value=${jsessionid}
    I often find it is easier to open up your test plan (.jmx) in your favourite text editor and then use the same regular expression to search and replace all instances of that pattern with your newly set variable.

    However there is a twist to all this, which I refer to as scope. You really need to understand the scope of your regular expression extractors and how they are applied. By scope I’m loosely referring to variable scope in a normal programming language (which JMeter is not). I think the best way to understand the scope of your extractors, is to identify which ‘container’ they are placed in. For a JMeter test plan, there are multiple containers you might place these extractors in:

  • The overall Test Plan, which I call ‘global’ in scope
  • An individual Thread Group, which I call ‘thread’ in scope
  • An individual Transaction Controller, which I call ‘local’ in scope
  • An individual Request Sampler, which I call ‘trans’ in scope
  • If you’re familiar with programming languages and scoping, then you’ll probably identify the pecking order of the scope I’ve just mentioned, from global right down to transaction level (I’ve made up a few names there). Global variables will override local variables and so on down the chain. So you need to be careful where you place your regular expression extractors and what reference (variable) names you use, as these can interfere with each other. Sometimes it is useful to limit your extractor in scope to a particular transaction, thread group or test plan.

    In my example listed above, I’ve written a test plan to explore the different scopes identified. Using the debug sampler, you can see that the trans scope is limited to single request sampler, because it is added as a child element. This means the variable jsessionid_trans gets set only once. The local scope is limited to a transaction controller. I placed it before the second request sampler but not as a child to the first, to highlight how it sets the variable jsessionid_local twice (once for each request contained in that transaction controller). The thread scope is limited to the entire thread group, so you will see the jsessionid_thread variable set three times as each page is requested. The same applies to the global scope in which case the jsessionid_global variable is set on each occurrence of the pattern. I’ve also added a second thread group with its own debug sampler to highlight how only global variables are accessible by this thread group. You can run this test plan (regex_test.jmx) yourself and check out the results as shown below:
    regex_scope

    Finally, here are some example regular expressions which have worked well for me for different types of web apps:
    jsessionid=(.+?)&
    jsessionid=(.+?)'
    id="__VIEWSTATE" value="(.+?)"
    jsd=(d+)
    strEntry=(.+?)'
    CdzSession' value='(.+?)'
    WebiSession' value='(.+?)'
    DOCTOKEN VALUE='(.+?)'
    etc ...

    About these ads

    11 thoughts on “JMeter Tips and Tricks – Correlating with Regular Expressions

    1. Thanks for the informative article. VS2008 testing edition also has some good mechanisms for extracting and correlating data (including regexes) but for the price, it is hard to beat JMeter.

    2. Thanks, for the information which U have provided regarding the Correlation.
      I have tried with that, But I couldn’t do it successfully.
      When ever I log in to the application “TokenID ” will be generated.
      I went from the Regular Expression.
      I have filled the fields in RegularExpression Extracter.Please help me out inthis issue.

    3. Thanks for such a detaled description about Jmeter session id concept.

      However, the concept of correlation is not very clear to me.If jsession is changes on every page then how would i use regular expression extracter to correlate the session id.

      Can we use URL Rewritting as well for this session id.

      Please help. Your help would be appreciated.

      Thanks
      Bose

    4. Could you please let me know the process to record web application(http) using Jmeter?

      Thanks in advance,

      Regards,
      –Lalith Kumar Chaganti

    5. Hi,

      I am facing problem with request connection through proxy settings.I am not getting 404 error when i start running the proxy but i do get the nodes added in the left end tree.Basically i am getting through 80 port but not able to connect to webserver port i.e 7001. Once i can establish connection i will be able to dop perf testing on my java struts application. Its a web application so i hope someone helps me out.

      Rgds,
      Ramesh

    6. Hi,

      Thanks for the informative article

      It helped me a lot but i have stucked with the viewstate correlation, Can you please elaborate correlation of Viewstate using Jmeter.

      Thanks is advance

      Regards,
      Suresh

    7. I am still having some query regarding correlation:

      1. Correlation can be achieved by both Pre Processors –> HTTP URL Re-writing Modifier and Post Processors -> Regular Expression Extractor??

      2. If yes, then what is the difference between both the process?

      3. I have gone through many links which says that correlation can be achieved by Regular Expression Extractor, so could you please put some light on this.

      i am dealing with a URL like this:
      /site/website?S=bemxrfs&W=mc51pww2&N=aqrgiea&A=bwvjpm1

      the S=bemxrfs&W=mc51pww2&N=aqrgiea&A=bwvjpm1
      dynamically change every link that is accessed.

      how do we capture and parameterized this if we are going to use
      HTTP URL Re-writing Modifier?

      i am dealing with a URL like this:
      /site/website?S=bemxrfs&W=mc51pww2&N=aqrgiea&A=bwvjpm1

      the S=bemxrfs&W=mc51pww2&N=aqrgiea&A=bwvjpm1
      dynamically change every link that is accessed.

      how do we capture and parameterized this if we are going to use
      HTTP URL Re-writing Modifier?

      thanks

    8. Hi,

      It seemed that the link to screenshot after “The following screenshot shows an example” is not working.

      Cheers,

    9. Amazing, truly excellent information. Your blog is really awesome. I bookmarked this and will come back once again.
      JMeter Tips and Tricks – Correlating with Regular Expressions Altentee » Performance & Test Automation Experts

    10. Hi,

      Can someone help tell me or have similar experience how to capture the dynamic values from the http header and set it as send parameter in the next request using regular expression extractor?

      Suppose the values of Location parameter in the http header request(Method is POST)is unique for each request.
      How to capture this unque value on each request and use it in the next request?

      Thanks,
      Prakash

    11. does anyone have the images from this post??

    Comments are closed.