CEWT #3 Why do we Test, and What is Testing Anyway?


Last Sunday 6th November 2016 I took part in CEWT at Jagex. Unlike the previous two CEWT’s I wasn’t presenting this time round.

CEWT stands for Cambridge Exploratory Workshop on Testing and is inspired by other similar workshops that happen globally. CEWT is a full day workshop with short presentations followed by more lengthy discussions.

There were twelve participants from various companies including; Jagex, Linguamatics, GMSL, Nokia, Redgate, Cambridge Consultants, DisplayLink, and Quest.

Jagex has a pretty cool interior and I was desperate to use the lift!

There were six presentations with the following abstracts around the theme of ‘Why do we Test, and What is Testing Anyway?’

Teach them to Fish by Michael Ambrose

Micheal started by stating what he, and other testers wanted. They wanted a more complex understanding of the system, to be able to do technical testing, and to leverage automation further. To get to this goal they required developers to take ownership of some areas of testing to free them up.

Micheal proposed that the training of developers would likely involve workshops to explain testing principles, types of testing, and crib sheets, etc.

Michael also warned of the implications of this. How will it be perceived? What does it mean for testers? How far will it go?

It’s worth noting that this experiment hasn’t happened yet so I would be keen to see how it has gone in a few months time.

Group discussion points:

  • How do we measure the success of this?
  • It was commented on that it could look like a more of a ‘blurring’ the lines of roles activity?

Who should do testing and what can they test? by James Coombes

This was as interesting approach. James mentioned how everyone at Nokia had a mantle of ‘I own quality’. He talked about how he was trying to see how all areas of the company could help testing and how testing could help them. This often resulted in pairing between disciples such as the ‘document writers’ pairing with ‘testers’ and so on.

Testing was seen very much as an activity.

James described how different people in the business, as well as customers, can find different issues. He also pointed out that this was often a supportive relationship with testers helping other functions.

James also highlighted how he visits different groups in the company to help them test in their day to day work.

My thoughts are that this approach could focus very much on bugs but what if quality for someone else is different? Does it work both ways?

Group discussion points:

  • It was interesting to see how each group can contribute towards quality and testing and what bugs they might find.
  • If we help others test then what is the test group for?

What is Testing? It depends … by Lee Hawkins

Lee highlighted James Bach and Michael Bolton’s definition on testing from the blog post ‘Testing and checking refined‘.

Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc.”

Lee pointed out that that this is helpful for testers but what about other stake holders?

Lee went on to describe how testing can be seen from various stake holders and also explored some common perceptions of what testing is, such as;

  • Testing is a cost center
  • Testing is dead
  • Testing as an information service
  • Testing is easy (anyone can do it)
  • Testing is a way to make money
  • Testing can be completely automated

Group discussion points:

  • Testing could well be easy if seen from the viewpoint from a customer if it’s what they would normally be doing as they used the product.
  • Sometimes testers help enforce certain views around testing such as it’s a cost center by writing endless test script that might only ever be run once.

Lee has written his own report of the day on his blog.

Testing is … by Aleksandar Simic

Aleksandar focused on showing all the ‘testing’ activities that he performed in a two day period while investigating & verification a bug. A lot of these activities taken in isolation might not be considered strictly testing but he argued that they could be testing when the output directly influenced and informed his testing.

The following are some examples over this two day period

  • Looking into the bug history
  • Grouping other bugs that may be able to be verified at the same time
  • Do I know the terminology?
  • Learning about the problem space
  • Exploring the problem space
  • Observation of 3rd party software
  • Testing while sleeping (automation)
  • Modelling
  • Dealing with interruptions
  • Asking questions of testers
  • Taking notes

He also mentioned how he had a ‘testing dairy’ to record his daily work and thoughts.

Group discussion points:

  • Learn enough of the domain knowledge to make your testing effective.
  • Gerald M. Weinberg’s book An Introduction to General Systems Thinking was referenced.
  • When having to context switch how can you cope with this?
    • Practice focus and defocusing
    • Learn how you and your brain functions and work with it
    • Except that interruptions happen and come at a cost

I test, therefore I am by Karo Stoltzenburg

One of my favorite talks of the day because Karo effectively explained why ‘she’ tests. Basically because she enjoys it and finds it rewarding.

Karo started with a standard definition and substituted ‘testing is’ with ‘when I grow up I want to’. She pointed out that this didn’t make a particularly appealing statement!

Karo explored different angles to the question in an attempt to understand why she tested, reflected back on her answers to expand a definition of testing that worked for her.

Group discussion points:

  • Could any other careers have the same characteristics that she likes in testing?
  • We tried to switch the definition around, and explored what that might suggest about the areas of testing Karo wouldn’t like.
  • How do we change the perception of the testing role?

Testing All the Way Down, and Other Directions by James Thomas

This was much more of a theoretical talk about James refined definition of testing and what it meant to him. He highlighting that while often testing can be seen as starting from one high level point and going deeper this isn’t always the case.

He referenced a huge blog post from Arborosa on a large collection of testing definitions.

James also mentioned a lovely definition from Rikard Edgren which was “Testing is simple; you understand what is important and you test it”

For a complete in depth review of James work I suggest you head over to his blog post from the day and the follow up one.

In summary James proposed a definition of testing as such;

“Testing is the pursuit of actual or potential incongruity”

Group discussion points:

  • What about removing testing from the definition and replacing it with another word. Would it still make sense? Like replacing testing with hacking? What might that tell us about the definition?

CEWT #3 GangFurther discussions:

After all the presentations there was a final round of discussions leading from topics raised during the whole day. These were around quality and what it means, domain knowledge, evaluating success, and how we ‘as testers’ can make a difference.

Just like the last three CEWT’s that I attended I got a great deal of enjoyment from the day and appreciated the time and thought people had put into the presentations and following discussions. CEWT #4 anyone?