At the beginning of the month I gave a 30 minute talk at Cukeup!, an imaginatively named conference for the Cucumber community.
The talk was titled Example.Examples,Examples:
Talk about BDD and people often think about Given, When and Then. Mention Specification by Example and the focus shifts to tables. This talk will focus on the latter, showing the benefits of giving examples in tabular form for both collaboration and automation.
Expect to see real-world scenarios from a trading system, and hear how they were produced through collaboration between Business Analysts and Developers. You will also see how the examples were automated using the TickSpec library which includes a number of novel extensions to the Gherkin language.
Specification by Example
Back in 2002, Ward Cunningham developed the Framework for Integrated Test or Fit for short. With Fit or Fitnesse customers provide examples of how their software should work using ordinary business tools like Excel or a Wiki. The examples can then be automated via test fixtures.
In 2006, Dan North introduced BDD (behaviour-driven development) with it’s given, when, then nomenclature used by JBehave, RSpec and later Cucumber. Here the focus was on plain text while maintaining support for examples in tables via Wiki style markup using pipe ‘I’ delimited rows and columns.
2011 saw Gojko Adzic’s Specification by Example book offer a collaborative approach, that illustrates requirements through examples and encompasses automation with both Fit and BDD based tools.
TickSpec
.Net has seen it’s fair share of BDD libraries like Cuke4Nuke, NBehave, Raconteur, SpecFlow, StorEvil and TickSpec. Of these, in the BDD tagged Nuget package download charts SpecFlow is the current leader. I tend to think of TickSpec as Specflow’s ugly cousin, at least on the outside, there’s no pretty website, no Visual Studio add-in and it’s not going to win a popularity contest any time soon. On the inside though, TickSpec does have some redeemable qualities: it’s a lightweight, dependency free, extensible library that supports and extends Cucumber’s Gherkin language. Where SpecFlow uses the official gherkin parser via IKVM, TickSpec features it’s own parser written in a few 100 lines of F#. TickSpec’s parser builds a simple Abstract Syntax Tree (AST) for extensibility, a feature according to Aslak Hellesoy, Cucumber aims to emulate in the future.
SpecFlow also provides syntax colouring of feature files in Visual Studio, a feature based on Cuke4VS which was originally used with Cuke4Nuke. Cuke4VS and Cuke4Nuke are no longer under active development, however you can install the entire SpecFlow bundle to get Cuke4VS like syntax colouring for .feature files that works in Visual Studio 2010 or 2012 and then if you prefer use an alternative library like TickSpec for automation. TickSpec uses just-in-time compilation via Reflection.Emit so that you can set breakpoints in feature files, it therefore does not require SpecFlow’s code generation step which you should be able to disable from the Properties menu.
That said I’d recommend when writing features to step away from tools like Visual Studio and focus on collaborating with business stakeholders in an environment that everyone is comfortable in like a whiteboard, good old paper, Excel or a wiki. Also remember the testing pyramid and consider that not all scenarios will need to be automated, so starting in an automation environment is somewhat premature.
Combinatorial Examples
Circa 2009, NUnit can generate test cases for all possible combinations of data items via the CombinatorialAttribute, in fact it is the default:
let [<Test; Combinatorial>] MyTest
([<Values(1,2,3)>] x,
[<Values("A","B")>] s) =
() // ...
Along similar lines TickSpec generates all combinations of examples for a scenario:
Scenario: Winning positions
Given a board layout:
| 1 | 2 | 3 |
| O | O | X |
| O | | |
| X | | X |
When a player marks X at <row> <col>
And viewed with a <rotation> degree rotation
Then X wins
Examples:
| row | col |
| middle | right |
| middle | middle |
| bottom | middle |
Examples:
| rotation |
| 0 |
| 90 |
| 180 |
| 270 |
Here for each example of row and column, each rotation is also specified. It’s a bit like using database normalization for your examples, where data values are not repeated, and then applying a cross join when using the data items.
TickSpec also introduces the idea of Shared Examples, which can be shared across all scenarios in a feature file, just as Gherkin Backgrounds are.
F# Step Definitions
Step definitions wire up feature files with code, typically with regular expressions. TickSpec lets you write step definitions in either C# or F#. I find F# gives the better experience as you can specify method names with white space and symbols for matching lines in scenarios which keeps syntactic noise to a minimum.
Taking Pacman scoring as an example:
Feature: Pacman score
Scenario: Pacman eats dots
When pacman eats a pac-dot
Then he scores 10
Scenario: Pacman eats power pellets
When pacman eats a power pellet
Then he scores 50
The step definitions can be defined in a class or as below in an F# module:
let [<BeforeScenario>] SetupScenario () =
properties <- []
let [<When>] ``pacman eats a pac-dot`` () =
properties <- PacDot::properties
let [<When>] ``pacman eats a power pellet`` () =
properties <- PowerPellet::properties
let [<Then>] ``he scores (.*)`` (points:int) =
let scored =
properties |> List.sumBy (function
| PacDot -> 10
| PowerPellet -> 50
)
Assert.AreEqual(points, scored)
Getting Started
If you’re interested in Specification by Example, then I’d recommend taking a look at:
If you decide to automate some examples, and like the sound of TickSpec I’d recommend starting by pulling the TickSpec.NUnit Nuget package into an F# library project. On the other hand if you find you’re not interacting with the business and spend all your time in Visual Studio I’d recommend doing plain old unit testing or using a fluent assertion library like Fluent Assertions, MSpec, FsUnit or NaturalSpec.