CTRTEST Information

CTRTEST is a test program designed to allow the Extensible Performance Counter developer to test their performance DLL in an isolated scenario. Normally, the developer must run the DLL in some performance monitoring client (e.g. Performance Monitor) which calls the DLL in the context of lots of other functions making debugging and troubleshooting difficult.

CTRTEST allows the developer of performance DLL's to test, debug and stress their DLL's without the need or influence of any other program or function. CTRTEST also provides a way to stress test the DLL in a multi-threaded environment. Although function calls on a particular performance library are synchronized within the process, it is important that the library support features such as Open/Close ref-counting in order to operate correctly in a multiple threaded environment.  This is especially important in that this emulates the environment that the DLL will operate in when performance counters are monitored by a remote client.  This is also a very critical test since if the DLL fails during remote performance monitoring it can cause the entire system to fail (i.e. "Blue Screen").

CTRTEST also allows the developer to run their DLL's in combination with other specific Counter DLL's in order to test for possible interaction problems.

CTRTEST and Monitoring Clients

Testing with ctrtest.exe is a method of “black box” testing which attempts to establish a benchmark for determining whether a performance library can be considered to be “trusted”.  A trusted performance library will not be subjected to the level of run-time validation in Wbemperf.dll (via WMI) and Perflib (via RegQueryValuEx( HKey_PerformanceData ) ) that un-trusted performance libraries undergo.  The reduced amount of error checking means higher performance from applications retrieving data from trusted performance libraries.

Running CTRTEST

The command line syntax of CTRTEST is

CTRTEST <path to ini file>

where:

<path to ini file> is the location of the CTRTEST.INI file that contains the execution parameters.  If this is not provided, CTRTEST will default to the current working directory for the location of the CTRTEST.INI file.

CTRTEST.INI

The format of the INI file is as follows:

[main]
NumThreads=10
CycleCount=100
LoopCount=1
Random=1
StopOnError=11
NumObjects=2
Object0=perfproc
Object1=perfos
Counter1=238

NumThreads is the number of threads that will run during the test at the same time. Each thread will run the same "Open, Collect X n, Close" cycle that is specified by CycleCount.

CycleCount is the number of times CTRTEST will call Open, Collect and Close in that sequence. This entry must be a decimal number.

LoopCount is the number of times CTRTEST will call the collect function between the Open and Close functions during each cylce. This entry must be a decimal number.

Random controls randomization on each cycle.  If this value is non-zero, each time a cycle is run, CTRTEST will randomly choose a performance library to query.

StopOnError controls the behavior of ctrtest when an error is encountered.  A value of 0 indicates that the application should continue running after an error is encountered, a 1 will cause the threads to gracefully terminate and the application will stop.  A value of 2 will cause a debug breakpoint to be thrown, and processing will terminate immediately terminate at the offending error check.

NumObjects indicates how many performance DLL's CTRTEST should work with.

Objectn specifies the name of the service the performance counters are registered under. CTRTEST will only test counter DLL's that have been registered by LODCTR. This is also the name that your DLL will be listed with in the EXCTRLST list box. If your DLL supports more than one service entry, then you'll need to repeat the test for each service.

Countern specifies the query string that will be passed into the corresponding performance DLL's Collect function.  If no value is specified, CTRTEST will default to use "Global".

For example, in the sample INI file above, when CTRTEST executes, it will spawn 10 threads that will each perform 100 Open/Collect/Close Cycles.  During each cycle, Collect will be called once.  On each cycle on each thread, CTRTEST will randomly choose between the "perfproc" and "perfos" performance DLL's.   When the Collect function is called for "perfproc", CTRTEST will specify a query string of "Global".  When the Collect function is called for "perfos", CTRTEST will specify a query string of "238" (which in this case is the "Processor" object).

Testing with CTRTEST

While CTRTEST will spew statistics to the console (these can be redirected to a file for later processing). It's also important to monitor your test run using Performance Monitor. You should monitor the following counters from the Process object:

-- Private Bytes
-- Handle Count
-- Thread Count

While the test is running, these counters should not increase during the test. They should reach some operating level and stay there or below, but not slope up. If the plot of any of these counters goes up, then you probably have a resource leak somewhere in the DLL. You should run this test both from within a debugger as well as on it's own. Running in a debugger will allow you to catch any exceptions that may or may not be handled. Also running the test inside other tools such as NuMega's BoundsChecker can help spot memory and other resource leaks.

CTRTEST will also check that returned buffers do not have memory overwrite errors or that the returned buffer sizes make sense (e.g. less than or equal to the size of the specified buffer).  If it encounters these sorts of conditions, it will stop processing.

Before you attempt to run your counter DLL in a "retail" or "production" environment, it should run cleanly with the following settings (This is the bare minimum test):

[main]
NumThreads=10
CycleCount=100
LoopCount=100
NumObjects=1
Object0=<your service name here>

Ideally you'll run more thorough tests such as

[main]
NumThreads=10
CycleCount=1000
LoopCount=100
NumObjects=1
Object0=<your service name here>

You should also try settings such as the following to check for potential interaction problems:

[main]
NumThreads=10
CycleCount=1000
LoopCount=100
Randomt=1
NumObjects=5
Object0=<your service name here>
Object1=perfproc
Object2=perfos
Object3=perfnet
Object4=perfdisk

You can use EXCTRLST to get a list of all Performance DLL's on a system, if you are suspecting interaction problems with another DLL other than the base ones mentioned above.

Test Outline