Collecting error reports from Visual Studio load tests

Generally Visual Studio load tests that call web performance tests are written to follow a happy path through the application. The goal is to have tests that are expected to pass so any errors are due to faults in the system under test. When the tests are data driven the data should be screened to exclude any values that always cause failures. Unfortunately that cannot always be achieved. On a recent project the data could not be screened before use. We had many discussions about attempting database resets between test executions but in the end we had to accept that some data records might be invalid.

Given that some tests might fail the project manager wanted the test results to include details of which tests failed, at what time, with what values from the data source, and with what error code. At first sight that data is readily available because a load test keeps logs for up to 200 (the default number) failing tests. These 200 logs could be studied to extract the details, but there is no easy way. Each log file must be opened by clicking on a link and then clicking through the various parts of the web test log to find the information. To analyze all 200 logs would require opening 200 log files, exploring 200 files and closing 200 files. That is much too much mouse clicking and mouse waving. It involves too much manual activity making it very error prone.

Trying to extract the log file details by program appears possible. The log files are stored in the SQL table 'LoadTestTestLog' within the load test results database. Using SQL the results can be exported to a CSV file where one field of each record contains a hexadecimal dump of the log file. Converting the hex to bytes is easy and viewing the results in a simple text editor shows encoded data with lots of embedded ASCII text. Some of that text appears to be parts of the log file. Using a "strings" command leaves the useful text hidden by a mass of other text.

The record appears to be a serialized class. My attempts to deserialize the data have been unsuccessful. The initial error is that the DLL required to decode the file cannot be found. So this approach does not work. Of course, even if it did work this approach would only show details of the first 200 failing tests. The number of saved logs could be increased but as each log file record is large that might mean filling up the database if several test runs were wanted.

Another approach is wanted and that will be the subject of another blog entry.

Small Steps in Software

Small Steps in Software gives me an opportunity to write a few words from time to time about the software I am working on. This first post is a bit of self congratulation about success in software related examination rather than about actual software. Today I sat and passed the Microsoft 70-498 Delivering Continuous Value with Visual Studio 2012 Application Lifecycle Management. One of my work colleagues took the same exam today and passed. So it was a good day for our employer who paid the fees and gave us some time to study.