TDD and Frameworks
Looking for some advise on improving our dev environment.
We are a .net house and this is what we currently have.
Visual Studio, SVN, JIRA for issues, backlog and TeamCity is our CI server.
Can anyone recommend an automated testing framework that is easy to use and write the acceptance tests. We are looking at using NUnit and perhaps Zest.
Our goal would be to build in VS, which kicks off TeamCity and the acceptance tests with the results then updated in JIRA. We are partially there, it is just the acceptance testing that is missing. We are wanting to do more test driven development.
Would it be worthwhile to use SpecFlow or similar in order to achieve BDD, and thereby subject the actual acceptance criteria to test automation?
That sounds interesting. I will look into it. My one concern is that I am hoping to write the tests myself, hence the need for "ease of use"
Acceptance Testing is generally done at the service or UI level. Are your services REST? What kind of UI do you have? Mobile phone? Web? PC app? other?
> My one concern is that I am hoping to write the tests myself, hence the need for "ease of use"
What is your automation/coding skill set? Any previous experience with automated testing tools?
Majority of our UI is web but also mobile. I don't have much coding ability but from what I have seen the acceptance tests can follow the Given then when syntax. I could manage that. The proper coder can then import those tests and develop against them.
That sounds like a reasonable strategy. I would highly recommend you try to follow the Test Automation Pyramid strategy:
Some tools to look into:
You should probably look into CodedUI as well. Also check out Greenpepper
Thanks a lot for the info. I hope that I am not missing something, but is the Service portion in the pyrimad essentially BDD? I say that as Mike explains the Service layer as something an application does in response to an input, would that not be considered its behavior, and so BDD?
Your reference to Telerik, are you talking about the Test Studio product?
Another question please there is a fair amount of effort doing BDD, it obviously makes sense when developing a "shelf" product, but when it comes to one off custom projects, I would assume that there is a size cut off, when a project is small enough that it no longer makes sense to do proper BDD. Is there a general rule of thumb that defines what the size/effort cut off level is?
> is the Service portion in the pyrimad essentially BDD?
The acceptance criteria of user stories can be specified in terms of demonstrable service behaviors, so a service layer *might* be tested using BDD. However it could also be subject to unit testing...the use of mock objects may be appropriate for stubbing out services.
In most cases, acceptance & end to end tests are similar. We generally define the end to end test cases as the acceptance criteria. I have done similar stuff in Java:
1. Unit tests utilizing mocking frameworks: JUnit + Mockito
2. Integration tests: JUnit + services executed against a target environment. Depending on the type of integration under test, you can leverage concepts like in-memory testing. ex: H2 driver for database integration, citrusframework for JMS, Web Service, etc
3. End to end / Acceptance testing with Selenium
All the above different tests can be implemented in TDD (JUnit) or BDD (JBehave, etc) style.
Depending on the product under test, it is crucial to mix and match the different testing frameworks available.
Most of them are better suited for service layer testing, few for integration testing & rare for end-to-end testing.
It is also important to leverage the TCK provided by any product.
Test execution is part of the CI cycle (Jenkins + maven). Test results (JBehave reports are nice) available via Jenkins.
End to end tests are executed as part of post deployment step.
Not exactly what you were asking but thought of sharing the experience.