Phillip Trelford's Array

POKE 36879,255

TickSpec: One Year On

It’s coming up to the first anniversary for TickSpec, a library and framework designed to support Behaviour Driven Development (BDD).

The first check-in was on the 30th August 2010:

Barebones BDD framework that supports a subset of the Gherkin language (Given,When,Then).

Since then the original aims of the project have been met, i.e create a lightweight parser library and invocation framework with first class support for F#, C# and integration with the major unit testing frameworks.

At the start of the project the focus was on core features, e.g.:

Added dynamic code generation so that you can step through feature files with the debugger

Added table support

Added Tags support

After the core features came a set of project examples for major unit testing frameworks including NUnit, xUnit, MbUnit and the Silverlight Unit Testing Framework.

TickSpecSLUT

Then with a number of projects using TickSpec in production, it was time to get out and about to talk with people about BDD and demonstrate how Acceptance Tests can be automated with TickSpec. Most recently a Live Meeting in June for the F# Community:


 

Over the last year I’ve been fortunate to attend and speak at quite a few related conferences and community events:

And there’s more on the horizon:

    OSJCukeyFSharp

And there are plenty more related articles here on this blog:

Thanks to recent user feedback on TickSpec there is also now a New Release with:

  • 100x Performance Increase (thanks Jon Harrop)
  • Improved error messages (thanks Martin Trojer)
  • A better C# debug experience (thanks James Banton)

In summary, a year on TickSpec has become a mature open source library and framework for automating acceptance tests on the .Net framework.


Disclaimer

Please bare in mind that a BDD framework is a means to an end and not the end in itself.

In the words of Gojko Adzic from his excellent new book Specification By Example (chapter 4, pages 46-47):

Don’t make test automation the end goal

this approach often leads to tests that are too technical

Don’t focus on a tool

this approach often ends up with programmers using a non-technical tool, intended for executable specifications, to manage technical, developer-orientated tests. This is a waste of time.

And Liz Keogh’s wise words in the article Step Away from the Tools:

BDD isn’t about the tools.

You don’t need BDD tools to do BDD.

Start by having conversations, engaging both testers and business stakeholders

If you can’t have those conversations, stop. You can’t do BDD

BuyerUser Beware!

DDD Scotland

This weekend I made my first ever trip up to Glasgow to attend and present at the DDD Scotland conference, an event run by the community with the sessions voted for by the community.

I arrived on Friday afternoon in time for a quick walk around Glasgow’s rather impressive city centre before the speaker’s dinner. It appears I may not have been the only visitor from out-of-town on the day ;)

Police Box

The event

I started the morning with Ray Booysen’s excellent talk: Streams of Streams – Your Rx Prescription. Ray gave a great introduction to the library with plenty of live examples inside LINQPad. The Reactive Extensions (Rx) is basically LINQ for events.

Then shared the speaker’s room with Gary Short’s Ego before my talk on BDD with F#:

Code samples:

The event itself was not recorded but the following videos present some similar material:

My afternoon started with Paul Stack’s session SpecFlow - functional testing made easy. Paul gave a great intro to automating acceptance test for web apps using Specflow and Watir.

Then some important messages in Richard Dalton’s excellent session: How to build a framework, and why you almost never should. A quote from Richard’s slides:

Base the Framework on existing code, preferably multiple cases. Before Reuse… there must BE USE.

Thanks to all the organizers, speakers and attendees for making this a really fun weekend.

Resources:

Functional Style BDD Steps

    Could a BDD framework be implemented simply as a fold over a sequence of lines in a document?
    Or in the words of Apple:

This changes everything. Again

marsattacks

Or in the words of Gojko Azdic:

Like anything else in F#, the primary use of this feature is to send it to 10 competitors whose brains you want to melt.

Let inception commence.


Fold

As defined on Wikipedia the Fold (higher-order function):

In functional programming, fold, also known variously as reduce, accumulate, compress or inject, is a family of higher-order functions that iterate an arbitrary function over a data structure in some order and build up a return value.

Or more succinctly defined via the F# List.fold signature on MSDN

  • ('State -> 'T -> 'State) -> 'State -> 'T list –> 'State

Fold high-order function implementation examples in F# and C#:

F#
C#

let rec fold f acc = function

    | [] -> acc

    | x :: xs -> fold f (f acc x) xs

public static TAcc Fold<TItem,TAcc>

    (Func<TAcc, TItem, TAcc> f,

     TAcc acc,

     IEnumerable<TItem> items)

{

    foreach (var item in items)

        acc = f(acc, item);

    return acc;

}

 

 

Thusly a BDD framework can be thought of simply as a fold that iterates over a document passing the lines or steps to an arbitrary matching function.

Signature:  fold: (‘context –> step –> ‘context) –> step list –> ‘context

With the accumulator value threaded between calls to the matching function representing the context, which can be explicitly scoped to the scenario under test!


Pattern Match

Given the following Feature document “Addition.txt”:

feature

And an immutable calculator type:

type Calculator = { Values : int list } with
    member this.Push n = {Values=n::this.Values}
    member this.Add() = {Values=[List.sum this.Values]}
    member this.Top = List.head this.Values

 

When folding over the lines in each scenario of the feature:

let feature = parse "Addition.txt"
feature.Scenarios
|> Seq.iter (fun scenario ->
    scenario.Steps
    |> List.fold performStep (Calculator())            
    |> ignore
)

 

And the following matching function is defined:

let performStep (calc:Calculator) (step,line:LineSource) =
    match step with
    | Given "I have entered (.*) into the calculator" [Int n] ->
        calc.Push n                        
    | When "I press add" [] -> 
        calc.Add ()
    | Then "the result should be (.*) on the screen" [Int n] ->
        Assert.AreEqual(n,calc.Top)
        calc            
    | _ -> sprintf "Unmatched line %d" line.Number |> invalidOp

 

Then the feature test succeeds!

inception


Parser

Parsing of the feature file is achieved using TickSpec’s parser module. TickSpec is an Open Source BDD framework written in F#. Underneath TickSpec’s parser is also implemented as a fold (well technically actually a scan) over a Gherkin feature document using regular expressions and Active Patterns.

To support the pattern matching function above the following Active Patterns are required:

[<AutoOpen>]
module Patterns =
    open System.Text.RegularExpressions
    let Regex input pattern =
        let r = Regex.Match(input,pattern)
        if r.Success then 
           Some [for i = 1 to r.Groups.Count-1 do yield r.Groups.[i].Value]
        else None
    let (|Given|_|) (pattern:string) (step) =
        match step with
        | GivenStep input -> Regex input pattern        
        | WhenStep _ | ThenStep _ -> None
    let (|When|_|) (pattern:string) (step) =
        match step with
        | WhenStep input -> Regex input pattern        
        | GivenStep _ | ThenStep _ -> None    
    let (|Then|_|) (pattern:string) (step) =
        match step with
        | ThenStep input -> Regex input pattern        
        | GivenStep _ | WhenStep _ -> None
    let (|Int|) s = System.Int32.Parse(s)

 

All the code to TickSpec including the parser and this sample (in Examples/Functional) are available at: http://tickspec.codeplex.com

 


NUnit

Finally the tests scenarios can be easily run within an NUnit test runner using the TestCaseSource attribute to parameterize a test method:

[<TestFixture>]
type AdditionFixture () =
    [<Test>]
    [<TestCaseSource(typeof<ScenarioSource>,"Scenarios")>]
    member this.TestScenario (scenario:ScenarioSource) =
        scenario.Steps |> Seq.fold performStep (Calculator.Create())
        |> ignore  
    member this.Scenarios =       
        let feature = parse "Addition.txt"        
        feature.Scenarios
        |> Seq.filter (fun scenario ->
            scenario.Tags |> Seq.exists ((=) "ignore") |> not
        )

 

Launch the NUnit Test Runner and the test passes:

NUnit


References