The solution as it turns [...]]]> Recently a client asked Altentee to generate some load against their subscription to Google Analytics (GA). Calls to GA and other 3rd party monitoring tools are typically requests we blacklist or consider out of scope. So it was unusual to think the other way around, how would you performance test GA?

The solution as it turns out is quite simple. Google Analytics is well documented here in terms of what query parameters constitute the GIF request that registers traffic.

The key parameters which need to be catered for when spoofing traffic are:
utmn => Unique ID generated for each GIF request to prevent caching of the GIF image.
utmhid => Another unique ID not documented but I think is related to AdSense
plus all the GA cookies including utma, utmb, utmc and utmz

The best way to parametize this is to observe real traffic with a tool like FireBug in the Net(work) panel. That way you can make a copy of a real GIF request. Essentially you are going to fake the call to __utm.gif. An example request looks like this:

http://www.google-analytics.com/__utm.gif?utmwv=4.7.2&utmn=123456789&utmhn=www.altentee.com&utmcs=UTF-8&utmsr=1920×1200&utmsc=24-bit&utmul=en-us&utmje=1&utmfl=10.0%20r45&utmdt=AlteneeBlog&utmhid=1697634062&utmr=-&utmp=%2F&utmac=UA-12345678-3&utmcc=__utma%3D199946558.28502083.1280297456.1280637882.1281056979.6%3B%2B__utmz%3D199946558.1280297456.1.1.utmcsr%3D(direct)%7Cutmccn%3D(direct)%7Cutmcmd%3D(none)%3B&gaq=1

All you need to do to spoof traffic or generate page views to your GA account is make repeated calls to this URL after parametizing key fields previously mentioned. An example request in JMeter with corresponding parametized functions in bold (random and time) looks like this:

/__utm.gif?utmwv=4.7.2&utmn=${__Random(10000000,60000000,utmn)}&utmhn=www.altentee.com&utmcs=UTF-8&utmsr=1366×768&utmsc=24-bit&utmul=en-us&utmje=1&utmfl=10.1%20r53&utmdt=PerformanceTest&utmhid=${__Random(100000000,900000000,utmhid)}&utmr=-&utmp=%2F&utmac=UA-12345678-3&utmcc=__utma%3D199946558.1448353202.${__time()}.${__time()}.${__time()}.4%3B%2B__utmz%3D199946558.${__time()}.1.1.utmcsr%3D(direct)%7Cutmccn%3D(direct)%7Cutmcmd%3D(none)%3B&gaq=1

You could also build this URL from scratch although it’s much easier to copy and modify the target URL for the site you’re actually testing. Please don’t use the URLs above as I’ve faked other things like the Google Analytics account code (utmac). It’s also worth changing the page title (utmdt) so you can separate real traffic from the fake traffic generated during performance testing.

Build a simple test plan in JMeter with a HTTP sampler generating requests to the aforementioned URL, put in a constant timer (or similar) so you don’t spam GA’s servers and away you go. You can then measure response time to this service for as many threads as you like. I generated 10K page views in 5 minutes which was more than sufficient to satisfy the client’s requirements.

The problem here is that you cannot auto authenticate to a proxy server using Chrome –proxy-server=my.proxy.com:3128 or using Firefox.

Chrome will always prompt you for a user name and password on an authenticating proxy when it starts up [...]]]> A question came up on stack overflow about how to route Watir through an authenticating proxy within your script.

The problem here is that you cannot auto authenticate to a proxy server using Chrome –proxy-server=my.proxy.com:3128 or using Firefox.

Chrome will always prompt you for a user name and password on an authenticating proxy when it starts up (via Webdriver or manually). Firefox will not prompt (when launched via WebDriver) and will silently fail.

I *almost* got this to work by modifying headers on the fly with a Firefox add on (Modify Headers). Basic authentication is just base-64 encoded so you can force each request to authorize itself by modifying the Proxy-Authorization request header e.g.:
Proxy-Authorization Basic asHJksaKHs87akhkjah7

However whilst this works for manually launched instances of Firefox, Webdriver launched instances would complain that the add on is not compatible with FF3.6.3 and so once again would fail. The other problem with this approach is that I couldn’t find a similar plugin for Chrome.

Eventually I gave in and went with a slightly more complicated workaround. That is, chaining an authenticated proxy (with squid) running on localhost with a cache_peer relationship to the authenticated proxy running out there on the ‘tubes’. I had thought about using Webrick as a proxy (as per previous posts) but lack of SSL support is a gotcha for me. Squid seemed most appropriate in this case:

To install squid from source:
cd /opt/src
sudo curl -O http://www.squid-cache.org/Versions/v2/2.7/squid-2.7.STABLE9.tar.gz
sudo tar xjf squid-2.7.STABLE9.tar.gz
cd squid-2.7.STABLE9
# I wanted SSL support for this
./configure --enable-ssl
make
sudo make install

Edit the squid.conf (location is normally /etc/squid/squid.conf on linux):
sudo vim /usr/local/squid/etc/squid.conf

# Don't want to run squid as root
# TAG: cache_effective_user
cache_effective_user nobody
# TAG: cache_effective_group
cache_effective_group wheel
# Point this proxy to a peer cache, and provide the logon credentials
# TAG: cache_peer
cache_peer proxy.altentee.com parent 3128 0 default no-query login=username:mypassword
never_direct allow all
# Allow access from my local private network
# TAG: cache_peer
acl localnet src 10.0.0.0/0

Save changes and change ownership as appropriate:
sudo chown -R nobody:wheel /usr/local/squid

Initialize the cache and startup squid:
sudo /usr/local/squid/sbin/squid -z
sudo /usr/local/squid/sbin/squid -D

Now you can configure firefox (using -ProfileManager) or chrome (using –proxy-server=127.0.0.1) to point to your local proxy server. This proxy will then authenticate on your behalf and peer all requests via the original parent.

A little long winded but achieves the outcome. Added side benefit is you’ve now got a local proxy (with access logs) that you can start doing interesting things with in your automation / performance tests like modifying headers and so forth.

Hope this helps!

The BrowserMob interface has a handy findRegexMatches method you can call as follows:

It takes a string input and [...]]]> If you’re using an RBU or VU you may need to extract content from the previous response. For example, enumerate a link to a PDF file for subsequent download.

The BrowserMob interface has a handy findRegexMatches method you can call as follows:

var link = browserMob.findRegexMatches(selenium.getHtmlSource(), 'href="(.+?pdf)"');
browserMob.beginStep("16_download_content");
  browser.get(link[0], 200); 
browserMob.endStep();

It takes a string input and regex pattern parameters. For the string input I am simply passing in the HTML source from the previous request based on a browser object:
var browser = browserMob.getActiveHttpClient();

The regex pattern is self explanatory. The link object will be an array of matches; in this case I’m issuing a get request for the first link in the match array. You might want to add some more code to make this more robust e.g. if no links are found etc.

Pretty simple hey!

Think Time Think time, is normally defined as the amount of time a virtual user ‘thinks’ between individual steps within a transaction. This time is usually excluded from response time measurements [...]]]> If you come from the commercial toolset mindset, you might be interested in how we achieve think time and pacing when using alternatives such as BrowserMob.

Think Time
Think time, is normally defined as the amount of time a virtual user ‘thinks’ between individual steps within a transaction. This time is usually excluded from response time measurements and is an important inclusion in terms of script behaviour. Having no think time means the virtual user will race through transactions more quickly than a normal user would which potentially creates unrealistic load. So how do we emulate think time behaviour in a BrowserMob script?

The BrowserMob replay engine drives Selenium scripts, so you will need to play in the JavaScript sandbox so to speak, in order to emulate think time. There’s some well documented ways to implement this on the BrowserMob blog here.

The BrowserMob interface (API documentation here) has a pause method as follows:
browserMob.pause(15000);

This effectively pauses the user for a period of 15 seconds in the previous example.

The Selenium interface (API documentation here) also has a similar setSpeed method which will set the number of milliseconds to pause after each step elminating the need to specify individual pause statements between each step as follows:
selenium.setSpeed(2000);

If you would like to emulate some common think time settings as you might see in a LoadRunner script you can implement the following at the top of your RBU or VU scripts:

// Multiply think time by:
var think_time_multiple = 1;
 
// Use random percentage for think time e.g. 50 - 100:
var think_time_percentage = 50 + (Math.random() * 100);
 
// Limit think time to:
var think_time_limit = 0; 
 
// Set think time to:
var think_time = 3; //seconds
 
if(think_time_limit > 0) {
  selenium.setSpeed(think_time_limit * 1000);
} else {
  selenium.setSpeed(think_time * think_time_multiple * (think_time_percentage/100) * 1000);
}

If you need additional think time after steps then you can add the following statement where required:
browserMob.pause(3000 * think_time_multiple * (think_time_percentage/100));

Pacing
Some people like to control transaction volume/throughput by messing about with think time. I don’t favour this approach because you need to first be cognisant of how long each iteration will take including think time then adjust think time accordingly. Once it’s set (or hardcoded into your script) then the vusers are stuck with that setting. I much prefer the pacing concept, which determines how long a vuser should ‘wait’ before starting the next iteration. This wait time is determined by elapsed time including any server side processing, not just user think time. So how do we emulate pacing in a BrowserMob script?

First I like to set some targets such as the transaction rate (per hour per user) and then calculate the pacing from this target. I also set a variable iterate to true, which comes into play in the main body of the script.

// Run logic
var transaction_rate = 10;  // Target transaction rate per hour per user
var pacing  = 60*60*1000 / transaction_rate;
var iterate = true;

Now we’ve got the target and pacing set, we can get into the main logic. Let’s start an iteration.

while (iterate) {
  var start = new Date();
  browserMob.beginTransaction();
  ...

Notice I’ve used a while loop which checks for the iterate boolean and creates a start date time stamp. I also initiate a BrowserMob transaction.

After we complete the iteration (which is essentially a bunch of BrowserMob steps) we can close the while loop with some pacing logic.

  ...
  // Calculate Pacing
  var finish  = new Date(); 
  var elapsed = finish.getTime() - start.getTime();
 
  if (browserMob.isValidation()) {
    browserMob.log("Adjusted pacing would be "+ (pacing - elapsed) + " msecs");
    browserMob.log("or " + Math.round(3600000 / (pacing - elapsed) ) + " trans/hour");
    iterate = false;
  } else {
    if(pacing > elapsed)
      browserMob.pause(pacing - elapsed);
  }
 
  browserMob.endTransaction();
}

This will calculate the finish date time stamp and then determine the difference between the previously calculated target pacing and the actual time elapsed. If there is a difference then the user will pause, otherwise it will just continue on to the next iteration. This basically allows you to target a set number of transactions per hour. When validating the script it will output these calculations to the log file.

Google AdWords page quality criteria are many, a prominent check being your landing page load time. A slow load time can lead to a lower quality score.

Make sure you’re not wasting effort on advertising campaigns. Altentee uses BrowserMob to help check website performance for clients from multiple locations. You [...]]]> Can you afford a slow landing page?

Google AdWords page quality criteria are many, a prominent check being your landing page load time. A slow load time can lead to a lower quality score.

Make sure you’re not wasting effort on advertising campaigns. Altentee uses BrowserMob to help check website performance for clients from multiple locations. You can run a free check using the form below.

If you need help optimizing your landing page, or help with performance testing in general feel free to contact Altentee today.

I’m using MySQL’s AES which is better than older standards. Basically 128 bit key encryption …

To begin, let’s create a [...]]]> There’s been a bit of buzz lately where MySQL databases have been compromised and user passwords being stored in plain text. Bad fu. It’s pretty simple to encrypt passwords in MySQL, here’s how you do it.

I’m using MySQL’s AES which is better than older standards. Basically 128 bit key encryption …

To begin, let’s create a demonstration table. e.g.
CREATE TABLE users ( user_name VARCHAR(20) NOT NULL, user_password BLOB );

So now we have a simple table with a user_name field and user_password field. e.g.

mysql> DESCRIBE users;
+---------------+-------------+------+-----+---------+-------+
| FIELD         | Type        | NULL | KEY | DEFAULT | Extra |
+---------------+-------------+------+-----+---------+-------+
| user_name     | varchar(20) | NO   |     | NULL    |       |
| user_password | blob        | YES  |     | NULL    |       |
+---------------+-------------+------+-----+---------+-------+

If we were creating a new user we would insert using the AES_ENCRYPT function. e.g.

INSERT INTO users VALUES ('tim.koopmans', AES_ENCRYPT('mypassword', 'Q4QjOyKANXj3qO'));

In this case I’ve used AES_ENCRYPT which encrypts a string and returns a binary string. The AES (Advanced Encryption Standard) algorithm is encoded with a 128-bit key. The second parameter is the key string, important to keep that secure!

On its own, this offers a good level of security. If the database was compromised the hacker will see information like this. e.g.

mysql> SELECT * FROM users;
+--------------+------------------+
| user_name    | user_password    |
+--------------+------------------+
| tim.koopmans | ??iu?hQ:???Ti?} |
+--------------+------------------+

The problem with this, is that the user_password field can still be ‘guessed’ via brute force methods. Every password that is the same i.e. ‘mypassword’ will also have the same binary string.

You can add a salt to this by prefixing random digits to the plain text password before you encrypt. For example, we can use the first 12 digits of a UUID. e.g.

INSERT INTO users VALUES ('joe.blow', AES_ENCRYPT(CONCAT(LEFT(UUID(),12), 'mypassword'), 'Q4QjOyKANXj3qO'));

The result set now looks like this if compromised.

mysql> SELECT * FROM users;
+--------------+----------------------------------+
| user_name    | user_password                    |
+--------------+----------------------------------+
| tim.koopmans | ??iu?hQ:???Ti?}                  |
| joe.blow     | ?M}69?H?0?H}?/{.?we|?A?P7ZD?R?   |
+--------------+----------------------------------+

Every password that is encrypted should look different which will further complicate dictionary attacks.

To determine a password from a stolen hash, an attacker cannot simply try common passwords (such as English language words or names). Rather, they must calculate the hashes of random characters (at least for the portion of the input they know is the salt), which is much slower.

For best results, the value/length of your salt should be kept secret (and separate from your database). You should also keep the key secret of course!

Now if we ever need to extract the password we use the AES_DECRYPT function. e.g.

mysql> SELECT SUBSTR(AES_DECRYPT(user_password, 'Q4QjOyKANXj3qO'),13) AS password FROM users WHERE user_name = 'joe.blow';
+------------+
| password   |
+------------+
| mypassword |
+------------+

In this case we know the first 12 digits were the salt so I used SUBSTR after decrypting the hash to get the original password.

A couple of caveats, encrypting passwords in this fashion is reversible i.e. If your key gets compromised, then the hacker has access to all your passwords. Another option is to reverse the plain text and key parameters of AES_ENCRYPT, where the user’s password now becomes the key, and the key becomes the plain text. Even if the database and key is compromised, it is hard to crack user passwords. You would not be able to recover lost passwords of course … Security through obscurity!

In this [...]]]> A common requirement when load testing is to populate subsequent requests with dynamic data from previous requests. If you’re thinking about using a database to store static test data, such as usernames, paswords, credit card numbers etc then read on to find out how to extract this data at runtime and populate JMeter variables.

In this example we’ll need to add a JDBC Connection Configuration to our test plan. This will define and control the way in which JMeter will access your database.

Notice I’ve left the defaults for connection pool properties, although you may need to tweak this depending on how you’ll access the data. I’ve used a JDBC connection URL that accesses a local mysql server listening on port 3306. The database in use is called watirgrid

I’ve also defined the JDBC Driver class and specified the username / password to connect with.

Now we can make a JDBC request using the JDBC Request sampler. It will look something like this:

The Variable Name matches the JDBC Connection Config Variable Name i.e. mysql

I’m using a select statement that will print out results in a JSON-like format. That query is as follows:

SELECT
CONCAT("[",
GROUP_CONCAT(
CONCAT("{email:'",email,"'"),
CONCAT(",login_count:'",login_count),"'}")
,"]")
AS json FROM users

In doing this, it will return a result set that looks like this:

+-------------------------------------------------+
| json                                            |
+-------------------------------------------------+
| [{email:'tim.koops@gmail.com',login_count:'1'}] |
+-------------------------------------------------+
1 row IN SET (0.00 sec)

Now we need to add a Post Processor Regex Extractor to the JDBC request as follows:

The reason we do this is so we can extract the results from the JSON string returned by the database query. For example, to extract and populate a variable called ${email} our PPRE would look like this:

Now we’re cooking! Add a Debug Sampler to confirm everything works as you’d expect. You should see results like this in the response data

email=tim.koops@gmail.com
email_g=1
email_g0=email:'tim.koops@gmail.com'
email_g1=tim.koops@gmail.com
mysql=org.apache.jmeter.protocol.jdbc.config.DataSourceElement$DataSourceComponentImpl@10d69502

Attached is a demo test plan: jdbc_example.jmx

We’re [...]]]> No, not the song but my adage for 2010 in my approach to performance testing in general. Readers of my old blog 90kts.com may have noticed the recent change to altentee.com. In 2009 I helped setup a boutique testing company called Altentee Pty Ltd. Read on for a quick advertisement about this new initiative …

We’re an Australian software performance and test automation specialist company based in Melbourne. We already have a global presence and employ some of the most experienced performance engineers available.

We pride ourselves on quality and cost effective solutions . Altentee offers alternative performance and test automation solutions using viable open source tool sets. This eliminates exorbitant license costs commonly associated with commercial tool sets. We’ve partnered with the team from BrowserMob.com for clients that want real-browser based performance testing. We’re also big fans of Hyperic for performance monitoring, Watir for browser automation and JMeter for protocol level performance tests.

Altentee’s mission is to deliver affordable and reliable performance and test automation alternatives to more traditional solutions.

Our goal is to deliver high quality, timely solutions whilst remaining affordable for any size organisation. Pay for the skills, not the tools and bring the focus back to your outcomes: delivering a fast, reliable system.

If you’re interested in what we do or know someone looking for performance and test automation solutions that won’t break the bank feel free to contact us via info@altentee.com

Essentially the EC2 snapshot is triggered like this:

This method is based on advice from here

Notice I [...]]]> So I’ve been working on a system where the MySQL instance on EC2 sporadically locks up (mysqld is zombied) during a snapshot process =)

Essentially the EC2 snapshot is triggered like this:

system("xfs_freeze -f /vol")                        and die;
system("ec2-create-snapshot $vol -K $key -C $crt ") and die;
system("xfs_freeze -u /vol")                        and die;

This method is based on advice from here

Notice I am doing an xfs_freeze of the entire volume on which the mysql data sits. It is intended to be used with volume managers and hardware RAID devices that support the creation of snapshots such as EBS.

The -f flag requests the specified XFS filesystem to be frozen from new modifications. When this is selected, all ongoing transactions in the filesystem are allowed to complete, new write system calls are halted, other calls which modify the filesystem are halted, and all dirty data, metadata, and log information are written to disk. Any process attempting to write to the frozen filesystem will block waiting for the filesystem to be unfrozen.

Note that even after freezing, the on-disk filesystem can contain information on files that are still in the process of unlinking. These files will not be unlinked until the filesystem is unfrozen or a clean mount of the snapshot is complete.

The -u flag is used to un-freeze the filesystem and allow operations to continue. Any filesystem modifications that were blocked by the freeze are unblocked and allowed to complete.

EC2 recommends this for ec2-create-snapshot:

When taking a snapshot of a file system, we recommend unmounting it first. This ensures the file system metadata is in a consistent state, that the ‘mounted indicator’ is cleared, and that all applications using that file system are stopped and in a consistent state. Some file systems, such as xfs, can freeze and unfreeze activity so a snapshot can be made without unmounting.

Clearly we don’t want to unmount the whole drive, so freezing the XFS volume is our best bet.

On a couple of occassions, we’ve seen the whole mysql server crash when a backup was taking place. The first symptoms displayed in logs is a long semaphore wait. This message is repeated for different threads:

mysqld[26852]: InnoDB: Warning: a long semaphore wait:

mysqld[26852]: --Thread 46912644213072 has waited at ./../include/trx0sys.ic line 101 for 241.00 seconds the semaphore:

mysqld[26852]: X-lock on RW-latch at 0x2aaaaf01cc08 created in file buf0buf.c line 497

mysqld[26852]: a writer (thread id 46912644213072) has reserved it in mode  wait exclusive

After about 10 minutes of this behaviour the server crashes. This is because 101 threads have made 101 connections where the max_connections=100. This only occurs whenever we are doing a snapshot in the background which makes me thing its a deadlock condition on the underlying filesystem.

mysqld[26852]: InnoDB: We intentionally crash the server, because it appears to be hung.

mysqld[26852]: 091207 20:16:41InnoDB: Assertion failure in thread 46912625965392 in file srv0srv.c line 2097

mysqld[26852]: InnoDB: We intentionally generate a memory trap.

mysqld[26852]: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

mysqld[26852]: InnoDB: If you get repeated assertion failures or crashes, even

mysqld[26852]: InnoDB: immediately after the mysqld startup, there may be

mysqld[26852]: InnoDB: corruption in the InnoDB tablespace. Please refer to

mysqld[26852]: InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html

mysqld[26852]: InnoDB: about forcing recovery.

mysqld[26852]: 091207 20:16:41 - mysqld got signal 11 ;

mysqld[26852]: This could be because you hit a bug. It is also possible that this binary

mysqld[26852]: or one of the libraries it was linked against is corrupt, improperly built,

mysqld[26852]: or misconfigured. This error can also be caused by malfunctioning hardware.

mysqld[26852]: We will try our best to scrape up some info that will hopefully help diagnose

mysqld[26852]: the problem, but since we have already crashed, something is definitely wrong

mysqld[26852]: and this may fail.

In panic mode, we restarted the whole server which was able to then recover from this error. But is there a more graceful recovery open to us?

Alestic have released a revised version of the ec2-create-snapshot called ec2-consistent-snapshot

Here are some of the ways in which the ec2-consistent-snapshot program has improved over the original:

Command line options for passing in AWS keys, MySQL access information, and more.
Can be run with or without a MySQL database on the file system. This lets you use the command to initiate snapshots for any EBS volume.
Can be used with or without XFS file systems, though if you don’t use XFS, you run the risk of not having a consistent file system on EBS volume restore.
Instead of using the painfully slow ec2-create-snapshot command written in Java, this Perl program accesses the EC2 API directly with orders of magnitude speed improvement.
A preliminary FLUSH is performed on the MySQL database before the FLUSH WITH READ LOCK. This preparation reduces the total time the tables are locked.
A preliminary sync is performed on the XFS file system before the xfs_freeze. This preparation reduces the total time the file system is locked.
The MySQL LOCK now has timeouts and retries around it. This prevents horrible blocking interactions between the database lock, long running queries, and normal transactions. The length of the timeout and the number of retries are configurable with command line options.
The MySQL FLUSH is done in such a way that the statement does not propagate through to slave databases, negatively impacting their performance and/or causing negative blocking interactions with long running queries.
Cleans up MySQL and XFS locks if it is interrupted, if a timeout happens, or if other errors occur. This prevents a number of serious locking issues when things go wrong with the environment or EC2 API.
Can snapshot EBS volumes in a region other than the default (e.g., eu-west-1).
Can initiate snapshots of multiple EBS volumes at the same time while everything is consistently locked. This has been used to create consistent snapshots of RAIDed EBS volumes.

Here’s hoping that this release fixes the problems with locking of the filesystem! Notice we don’t attempt to flush and lock since we are using the InnoDB engine.

Revised snapshot code looks like this:

# create snapshot
system("xfs_freeze -f /vol")                        and die;
system("ec2-consistent-snapshot --aws-access-key-id $key --aws-secret-access-key $secret --xfs-filesystem /vol $vol") and die;
system("xfs_freeze -u /vol")                        and die;

# auto complete word autocompleteword.automatic=1

# advanced find and replace find.replace.advanced=1

# force [...]]]> Well the subject goes without saying. I’ve been working on WinXP again lately and when I’m not around my favourite text editor on the mac (TextMate) I opt for SCiTE on windows. The following settings I find useful for this simple but powerful text editor. YMMV.


# auto complete word
autocompleteword.automatic=1

# advanced find and replace
find.replace.advanced=1

# force monospaced fonts
font.base=$(font.monospace)
font.small=$(font.monospace)
font.comment=$(font.monospace)
font.text=$(font.monospace)
font.text.comment=$(font.monospace)
font.embedded.base=$(font.monospace)
font.embedded.comment=$(font.monospace)
font.vbs=$(font.monospace)
font.monospace=font:DejaVu Sans Mono,size:8

# show the status bar
statusbar.visible=1