Multipurpose testing

Everybody knows testing is important. Even though they are expensive in man-hours, it is way more expensive not to test.

As tests are the cornerstones of our development plan, our clients often ask why we are testing the way we do, what we are testing, and how they can prepare for a test.

Our answer is that we test in three ways, and make sure to get as much out of each test as possible:

Below is a comprehensive guide to why we test, and what you should keep in mind as you run tests in various stages of your project. But also to the way we try to get the most out of tests by using them to solve more than one problem at a time.

Early testing – Proof-of-concept

Primary purpose: To get ‘proof of concept’. If you are developing a new, relatively unique solution, you need to do prototype tests from very early on. Even the crudest mock-up will provide a tangible conversation piece that lets the team discuss how ideas match project objectives. The first test(s) help you make sure you are heading in the right direction.

Secondary purpose: Internal knowledge sharing. We often invite collegaues to our proof-of-concept tests. In this way more than the core development team get a taste of new ideas or mechanisms that might find their way into other projects. It also gives us more people with some familiarity with the project.

Participants: Participants can be limited to the core development team. Early tests need a safe space where everyone can give honest feedback without worrying about someone changing basic conditions or the scope of the project.. Do not use early tests to involve important stakeholders that are not already part of the development process.

Focus: At this point, you do not need to test the whole process. Focus on the absolute key parts and/or the parts that you feel the most uncertain about. If it is a process tool, this could be a central conversation piece (e.g. a set of cards, a visual model). If it is a game, you can test a single game round or parts of the central gameplay. If it is dilemma exercise, make one or two dilemmas to illustrate the format.

Degree of finish: For this, you do not need a lot of finished content. You should be able to test the concept with only 10-30% of the content ready. There is some debate as to whether you should have your prototype properly layouted. In many cases, the graphic design can wait, as long as you are able to illustrate the basic interaction with drawings or simplified models. If some visual representations are absolute key, it makes sense to bring an early version of the layout, but in many cases, this makes the prototype look more finished than it is, which may blur the focus of the test.

Process: Do not spend too much time on making the process super smooth and on keeping time. Instead, make sure you have plenty of time for discussions and for taking breaks and reiterating parts of the design. Often, early tests are just “walk-through” conversations. Other times you do a test for a few minutes and then this gradually turns into a more open conversation.

Challenges: When you test with the core development team, everyone will know the project and have opinions about possible solutions. As a result, the conversation will jump back and forth between actual testing (as if you were real participants) and meta-discussion (where you discuss solutions and next steps). If you need the participants to engage with the materials without launching into meta-discussions, you need to be very clear about that and you probably need to keep reminding the participants at your session.

Mid-project testing

Primary purpose: Adjusting the finer details of the concept + qualifying content and formats. If your project is based on an existing solution, you can skip early testing and start here. If your project involves new tailor-made content, you will need to do one or two tests mid-project.

Secondary purpose: Mid-project testing is also a great opportunity to invite key stakeholders. If you need buy-in or specific input from various stakeholders, inviting them to a test is a great way to make sure everyone is on the same side. This helps align your project with other important projects and agendas within the organisation. Some stakeholders can/should also be enrolled as reviewers later on.

Participants: As these tests are also about content, you should invite subject matter experts (SMEs) who will help you identify mistakes and clear up misunderstandings. Inviting SMEs to the test is also a great idea if you want to enrol them as reviewers later in in the project. They then start feeling ownership of the project, and since they have been introduced to the process and materials, it is easier for them to understand what to look for in review rounds.

Focus: These tests can cover the whole process or at least the central parts. You may still leave out the parts you know you can do without getting feedback, but these tests are also great occasions if you want to rehearse an introduction speech or process instructions. Make sure you test enough of the process to give your test persons a clear picture of the expected final participant experience. This goes for process as well as content and interaction design. If it is a game, you should run through at least half of the game to make sure that all participants understand the process and all representations.

Degree of finish: This is your chance to test large parts of the content, before you start finalising it. When you enter these test, your content should be 50-80% ready. You do not need to have done thorough proof reading, but it is important that all texts are written in the right tone of voice and with appropriate business lingo. At this point, the prototype should start looking like and working like the final product. This will make it easier for you to show the possible benefits to different stakeholders. You do not need the aesthetic finish or the production value of the final product, but the look and feel should be in place, and the interaction design should be finished. It is important that the materials do not cause unnecessary confusion.

Process: Mid-project tests should have a clear distinctions between actual testing and meta-discussions. At some points, you should focus on testing and have the participants play along without making comments. At other points, you should collect feedback. Make sure that the explicit feedback is not the only data you take away from the sessions. It may include valid points, but the pure nature of the discussion may lead participants to overemphasise some things and completely forget to mention others. In order to get the best information, you will need to take notes during the actual testing. Observe the participants’ discussions and actions. Are they having the right conversations? What are their thoughts on the choices they make? Also, observe how they navigate the interaction design. Are they getting confused about something? Are they using the materials as intended?

Challenges: If you invite stakeholders and subject matter experts to review unfinished content, you will almost inevitably show them something they do not like to see. Be ready to handle an expert who is frustrated that an important part is poorly explained or that you are not using the right lingo. And be ready to handle a stakeholder who feels that you are misrepresenting an important agenda. Make sure you listen to their feedback, note it down, promise that you will do something about, but also get the process back on track. Otherwise, these discussions may take forever and completely ruin the test session.

Late stage testing – Pilot test

Primary purpose: Testing at this stage is to confirm that you have right design and to identify minor adjustment you will need to make. In some cases, this may be the first actual use of your game or tool. This will label it as a “pilot”. As pilot is supposed to deliver real value, but a pilot version is also open for changes afterwards.

Secondary purpose: Late stage testing is a great way to introduce the game or tool to the people who have to work with it. We often use these tests to train facilitators or otherwise introduce the game to the people who have to handle the game.

Participants: Participants should be members of the real target group who have not seen any earlier versions of the design. If they know as little as possible beforehand, you will truly see how they understand the process and the inherent messages. It is a good idea to test more than one group. This gives you different results to compare, and if you run the different groups in parallel, it also stress-tests your facilitation a bit.

Focus: Late stage tests should include the whole process and almost all content. This should show you the full participant experience and the all the hassle it takes to run a smooth process as a facilitator.

Degree of finish: This should be as close to the finished product as possible. Maybe your prototype or pilot version does not live up to the final production quality, but everything should be fully layouted and you should have 80-100% content ready.

Process: The process should be as close to the final intended process as possible. Only stop the process if you absolutely have to. Instruct the participants to take notes as they make observations, and postpone all meta-discussions and comments until after you have been through the whole process. This will also give you a clearer picture of the timing of the whole process.

Challenges: At this point, you will probably not have time to make major changes. Make sure in your evaluation with the tests persons, not to promise to turn it all upside down afterwards. Within your own development team, do not panic if everything did not happen according to plan. Focus on where you can make quick fixes, where you can simplify, or how you can make up for various kinds of “noise” through better instructions and better facilitation.

Why test?

As designers, we cannot craft or control every aspect of the experience, because much power is left with the participants and their choices and actions. If they do not do or experience what they are supposed to, it is simply not a good enough design. If people get confused, frustrated or bored using the tool, there is something wrong with the tool, not the people. And if there is something wrong with a tool or a process, we would rather find out sooner than later.

Having been in this business for quite a few years, we have a lot of experience, standard solutions and general tricks, but we never rely solely on our ability to predict participant behaviour. Every organisation and every project has unique qualities that may surprise the designer. A new target group can have different references or a different culture than the previous one. Also, different projects may be influenced by different stakeholders and bearers of different organisational agendas.

If you are designing a game or an involvement process and your project plan does not include at least one test, you are running a very risky project.

When you are planning your tests, it is important that you think about what you need to test, whom to involve, and what to be aware of at different stages of the project.

A few general tips

• Make the test schedule early. This will provide a good backbone to your project plan and make it clear when you should plan reviews.
• Ensure you invite the right people in good time.
• For each test, be upfront about what you are testing and how you want the test persons to behave.
• Don’t forget to observe. The best feedback you can get is seeing and hearing the participants engage with your design. It is tempting to focus on just running the sessions and then taking feedback afterward. Don’t.
• Keep an open mind. A test is not an exam – it is a chance to improve. Make sure you take observations and participants seriously and that you are willing to make the necessary changes.
• Listen to participant feedback. When the participants provide feedback, they may have misunderstood something in the process, they may get too caught up in less important details, and their ideas for solutions may be impossible or just plain bad. If the test person who provides the feedback is not part of the development team, do not enter into long discussions, and don’t get tempted to make elaborate speeches justifying the current design. Acknowledge their input, and make sure you understand where it is coming from. This will help you as you move forward, both in terms of design and in terms of goodwill in the organisation