Wednesday, 3 December 2008

Data Driven Testing in Visual Studio 2008 - Part 2

From Maarten Balliauw's blog: This is the second post in my series on Data Driven Testing in Visual Studio 2008. The first post focusses on Data Driven Testing in regular Unit Tests. This part will focus on the same in web testing.

Web Testing

I assume you have read my previous post and saw the cool user interface I created. Let's first add some code to that, focussing on the TextBox_TextChanged event handler that is linked to TextBox1 and TextBox2.

[code:c#]

public partial class _Default : System.Web.UI.Page
{
// ... other code ...

protected void TextBox_TextChanged(object sender, EventArgs e)
{
if (!string.IsNullOrEmpty(TextBox1.Text.Trim()) && !string.IsNullOrEmpty(TextBox2.Text.Trim()))
{
int a;
int b;
int.TryParse(TextBox1.Text.Trim(), out a);
int.TryParse(TextBox2.Text.Trim(), out b);

Calculator calc = new Calculator();
TextBox3.Text = calc.Add(a, b).ToString();
}
else
{
TextBox3.Text = "";
}
}
}

[/code]

It is now easy to run this in a browser and play with it. You'll notice 1 + 1 equals 2, otherwise you copy-pasted the wrong code. You can now create a web test for this. Right-click the test project, "Add", "Web Test...". If everything works well your browser is now started with a giant toolbar named "Web Test Recorder" on the left. This toolbar will record a macro of what you are doing, so let's simply navigate to the web application we created, enter some numbers and whatch the calculation engine do the rest:

Web Test Recorder

You'll notice an entry on the left for each request that is being fired. When the result is shown, click "Stop" and let Visual Studio determine what happened behind the curtains of your browser. An overview of this test recording session should now be available in Visual Studio.

Data Driven Web testing

There's our web test! But it's not data driven yet... First thing to do is linking the database we created in part 1 by clicking the "Add datasource Add Datasource" button. Finish the wizard by selecting the database and the correct table. Afterwards, you can pick one of the Form Post Parameters and assign the value from our newly added datasource. Do this for each step in our test: the first step should fill TextBox1, the second should fill TextBox1 and TextBox2.

Bind Form Post Parameters

In the last recorded step of our web test, add a validation rule. We want to check whether our sum is calculated correct and is shown in TextBox3. Pick the following options in the "Add Validation Rule" screen. For the "Expected Value" property, enter the variable name which comes from our data source: {{DataSource1.CalculatorTestAdd.expected}}

image

If you now run the test, you should see success all over the place! But there's one last step to do though... Visual Studio 2008 will only run this test for the first data row, not for all other rows! To overcome this poblem, select "Run Test (Pause Before Starting" instead of just "Run Test". You'll notice the following hyperlink in the IDE interface:

Edit Run Settings

Click "Edit run Settings" and pick "One run per data source row". There you go! Multiple test runs are now validated ans should result in an almost green-bulleted screen:

image

kick it on DotNetKicks.com

Thursday, 27 November 2008

Data Driven Testing in Visual Studio 2008 - Part 1

From Maarten Balliauw's blog: Last week, I blogged about code performance analysis in Visual Studio 2008. Since that topic provoked lots of comments (thank you Bart for associating "hotpaths" with "hotpants"), thought about doing another post on code quality in .NET.

This post will be the first of two on Data Driven Testing. This part will focus on Data Driven Testing in regular Unit Tests. The second part will focus on the same in web testing.

Data Driven Testing?

We all know unit testing. These small tests are always based on some values, which are passed throug a routine you want to test and then validated with a known result. But what if you want to run that same test for a couple of times, wih different data and different expected values each time?

Data Driven Testing comes in handy. Visual Studio 2008 offers the possibility to use a database with parameter values and expected values as the data source for a unit test. That way, you can run a unit test, for example, for all customers in a database and make sure each customer passes the unit test.

Sounds nice! Show me how!

You are here for the magic, I know. That's why I invented this nifty web application which looks like this:

Example application

This is a simple "Calculator" which provides a user interface that accepts 2 values, then passes these to a Calculator business object that calculates the sum of these two values. Here's the Calculator object:

[code:c#]

public class Calculator
{
public int Add(int a, int b)
{
return a + b;
}
}

[/code]

Create Unit Tests...Now right-click the Add method, and select "Create Unit Tests...". Visual Studio will pop up a wizard. You can simply click "OK" and have your unit test code generated:

[code:c#]

/// <summary>
///A test for Add
///</summary>
[TestMethod()]
public void AddTest()
{
Calculator target = new Calculator(); // TODO: Initialize to an appropriate value
int a = 0; // TODO: Initialize to an appropriate value
int b = 0; // TODO: Initialize to an appropriate value
int expected = 0; // TODO: Initialize to an appropriate value
int actual;
actual = target.Add(a, b);
Assert.AreEqual(expected, actual);
Assert.Inconclusive("Verify the correctness of this test method.");
}

[/code]

As you see, in a normal situation we would now fix these TODO items and have a unit test ready in no time. For this data driven test, let's first add a database to our project. Create column a, b and expected. These do not have to represent names in the unit test, but it's always more clear. Also, add some data.

Data to test

Test View Great, but how will our unit test use these values while running? Simply click the test to be bound to data, add the data source and table name properties. Next, read your data from the TestContext.DataRow property. The unit test will now look like this:

[code:c#]

/// <summary>
///A test for Add
///</summary>
[DataSource("System.Data.SqlServerCe.3.5", "data source=DataDirectory\\Database1.sdf", "CalculatorTestAdd", DataAccessMethod.Sequential), DeploymentItem("TestProject1\\Database1.sdf"), TestMethod()]
public void AddTest()
{
Calculator target = new Calculator();
int a = (int)TestContext.DataRow["a"];
int b = (int)TestContext.DataRow["b"];
int expected = (int)TestContext.DataRow["expected"];
int actual;
actual = target.Add(a, b);
Assert.AreEqual(expected, actual);
}

[/code]

Now run this newly created test. After the test run, you will see that the test is run a couple of times, one time for each data row in he database. You can also drill down further and check which values failed and which were succesful. If you do not want Visual Studio to use each data row sequential, you can also use the random accessor and really create a random data driven test.

Test results

Tomorrow, I'll try to do this with a web test and test our web interface. Stay tuned!

kick it on DotNetKicks.com

Monday, 10 November 2008

Integrating NUnit test results in Team Build 2008

When using Team Foundation Server 2008 and Team Build, chances are you are developing unit tests in Microsoft’s test framework which is integrated with Visual Studio 2008. This integration offers valuable data hen a build has been finished on the build server: test run results are published in the Team Foundation Server 2008 data warehouse and can be used to create detailed metrics on how your development team is performing and what the quality of the product being developed is.

Not all software development teams are using Microsoft’s test framework. Perhaps your team is using Team Foundation Server 2008 and creates (unit) tests using NUnit. By default, NUnit tests are not executed by the Team Build server nor are they published in the Team Foundation Server 2008 data warehouse. The following guide enables you to leverage the features Team Foundation Server 2008 has to offer regarding metrics, by customizing the build process with the necessary steps to publish test results.

(cross-posted on Maarten Balliauw's blog)

1. Prerequisites

Make sure the following prerequisites are present on your Team Build server (in addition to a default build server installation):

2. Registering NUnit framework in the global assembly cache (GAC)

For NUnit tests to be run in a Team Build script, make sure that the NUnit framework is registered in the global assembly cache (GAC). This can be achieved by copying the file C:\Program Files\NUnit 2.4.8\bin\nunit.framework.dll to C:\Windows\Assembly.

clip_image002

3. Customizing a build script

After installing all prerequisites, make sure you know all paths where these tools are installed before continuing.

The build script for a NUnit enabled build should be modified in several locations. First of all, the MSBuild Community Tasks target file should be referenced. Next, a new build step is added in the AfterCompile hook of the build script. This build step will run the NUnit tests in the compiled DLL’s, transform them to a Microsoft Test results file (*.trx) and publish this transformed file to the Team Foundation Server 2008.

Open the TFSBuild.proj file from source control and merge the following lines in:

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="DesktopBuild" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">

<!-- Do not edit this -->

<Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets" />

<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.targets" />

<ProjectExtensions>

<!-- ... -->

</ProjectExtensions>

<!-- At the end of file: -->

<ItemGroup>

<AdditionalReferencePath Include="$(ProgramFiles)\Nunit 2.4.7\bin\" />

</ItemGroup>

<Target Name="AfterCompile">

<!-- Create a Custom Build Step -->

<BuildStep TeamFoundationServerUrl="$(TeamFoundationServerUrl)" BuildUri="$(BuildUri)" Name="NUnitTestStep" Message="Running NUnit Tests">

<Output TaskParameter="Id" PropertyName="NUnitStepId" />

</BuildStep>

<!-- Get Assemblies to test -->

<ItemGroup>

<TestAssemblies Include="$(OutDir)\**\Calculator.dll"/>

</ItemGroup>

<!-- Run NUnit and check the result -->

<NUnit ContinueOnError="true" Assemblies="@(TestAssemblies)" OutputXmlFile="$(OutDir)nunit_results.xml" ToolPath="$(ProgramFiles)\Nunit 2.4.8\bin\">

<Output TaskParameter="ExitCode" PropertyName="NUnitResult" />

</NUnit>

<BuildStep Condition="'$(NUnitResult)'=='0'" TeamFoundationServerUrl="$(TeamFoundationServerUrl)" BuildUri="$(BuildUri)" Id="$(NUnitStepId)" Status="Succeeded" />

<BuildStep Condition="'$(NUnitResult)'!='0'" TeamFoundationServerUrl="$(TeamFoundationServerUrl)" BuildUri="$(BuildUri)" Id="$(NUnitStepId)" Status="Failed" />

<!-- Regardless of NUnit success/failure merge results into the build -->

<Exec Command="&quot;$(ProgramFiles)\nxslt-2.3-bin\nxslt2.exe&quot; &quot;$(OutDir)nunit_results.xml&quot; &quot;$(ProgramFiles)\MSBuild\NUnit\nunit transform.xslt&quot; -o &quot;$(OutDir)nunit_results.trx&quot;"/>

<Exec Command="&quot;$(ProgramFiles)\Microsoft Visual Studio 9.0\Common7\IDE\mstest.exe&quot; /publish:$(TeamFoundationServerUrl) /publishbuild:&quot;$(BuildNumber)&quot; /publishresultsfile:&quot;$(OutDir)nunit_results.trx&quot; /teamproject:&quot;$(TeamProject)&quot; /platform:&quot;%(ConfigurationToBuild.PlatformToBuild)&quot; /flavor:&quot;%(ConfigurationToBuild.FlavorToBuild)&quot;" IgnoreExitCode="true" />

<!-- If NUnit failed it's time to error out -->

<Error Condition="'$(NUnitResult)'!='0'" Text="Unit Tests Failed" />

</Target>

</Project>

4. Viewing test results

When a build containing NUnit tests has succeeded, results of this tests are present in the build log:

clip_image004

When clicking the test results hyperlink, Visual Studio retrieves the result file from Team Foundation Server 2008 and displays it in the test results panel:

clip_image006