Conducting effective pilot testing as a product manager

Pushing out features into customers’ hands doesn’t guarantee value creation. We may think we know what customers want and how to collect business value from new features, yet we’re often wrong. It’s not about how fast you can ship new features; it’s about how fast you can learn what works and what doesn’t so you adapt course fast enough.

Product experiments, particularly pilot testing, can help accelerate this learning process.

In this blog, I explore how product teams can benefit from pilot testing. I will begin with what pilot testing looks like, its benefits, the steps to conducting one, and the tools and techniques you can bring to use. I’ll also share a case study.

What is pilot testing?

Pilot testing is a method for evaluating products or features with a controlled, selectively chosen group of users. The objective is to be as close as possible to those who will actually use the product so you uncover challenges and opportunities that could otherwise be missed in the development or prototype phases.

With pilot testing, product teams can quickly iterate the product until the control group receives the maximum benefit.

Pilot testing is often confused with beta testing, but there are key differences. Beta testing focuses on more users interacting, meaning the group is uncontrolled. Product teams can identify patterns and gather quantitative data. Pilot testing, on the other hand, helps product teams receive more qualitative feedback from a smaller, hand-picked group of users.

The two testing techniques — pilot testing and beta testing — complement each other, so it is normal to start with a pilot and then move to beta to increase confidence in market success.

Benefits of pilot testing

You may ask why you’d want to invest time in pilot testing if you have already interviewed users, conducted prototype sessions, and collected considerable feedback before creating new features.

Well, users surprise us. Interviews and prototypes give, at best, moderate but not enough evidence to justify an upfront investment without involving customers until the feature is incorporated into the product.

Pilot testing will help product teams with the following:

• Evidence strength — When real users interact with the product, you receive feedback based on reality, not opinions, ultimately making your evidence stronger
• Iteration — With a controlled group, product teams can quickly iterate the product and receive direct feedback on what works and what doesn’t
• Learn from reality — Pilot testing isn’t a lab exercise but a natural product use
• Focus — Once you run a pilot test, you can learn about valuable features and confusing elements or distractions, which empower you to create a more relevant product
• De-risk — The more you interact with users in a natural scenario, the more prepared you become to scale your product, so you can de-risk your product ideas

Steps to conduct a pilot test

Conducting a pilot test requires structure and alignment. Here’s what I like defining before starting the pilot:

1. Objective — Set the goal for your pilot testing. Make it as measurable as possible. For example, 75% of pilot users signed up for the product release. You want the pilot’s result to increase confidence, so make the goal related to it
2. Length — Define the duration of the pilot. Pilot testing takes time, as you need to collect feedback and iterate, so give reasonable time for it. Depending on your product, somewhere between 2 and 6 weeks tends to be enough
3. Milestones — It’s vital to measure the results continuously. If you measure only the goal, you may learn too late when you miss it. Set a few milestones to help product teams inspect and adapt
4. Team — Pilot testing requires focus to thrive. Have a dedicated team to manage such an experiment. Ideally, this team should be entirely focused on it
5. Pilot sample — Define the number of users you want to invite for your pilot testing. This will depend on your scenario. For B2C products, I’d recommend limiting to 20 in the early stages. For B2B, the story is different as it evolves more people. Getting five companies would be great, but you can be happy with three participants

Once you’ve delineated the basics of pilot testing and established your framework, it’s time to move on to execution and iteration. A lean startup approach can benefit you:

1. Build — Start by building the least amount necessary to run the pilot. The goal is to first build and learn, then scale. Good enough is better than perfect here
2. Measure — As you release the pilot, get closer to the participants. Observe how they use it (shadowing) and collect qualitative feedback (interviews, surveys). The goal is to uncover what’s helpful and what’s not
3. Learn — Reflect on what you measured and name the learnings. This should empower you to decide what to do for the next iteration
4. Rinse and repeat — Refine your product and repeat this cycle based on what you have learned. If you realize your pilot isn’t flying, you can drop it altogether. But if you identify tweaks that can make it valuable, go for it

Choosing the right tools for your pilot testing can be troublesome. But don’t bother too much about it — the goal is to learn from real users, not deploy the best tools. I recommend the following tools:

• Shadowing — If you can observe how they interact with your product, you will learn aspects you didn’t even know you didn’t know
• Moderated sessions — Once you recruit the pilot users and have sessions, moderate them to run specific tasks and help users think aloud. You will get a peek into their first reactions and see where they get confused
• Unmoderated sessions — These are similar to the previous ones but without moderation. Users receive the task and record how they perform it. This is scalable but lacks the interactive feedback loop of a moderated session
• Customer effort score (CES) — Add surveys to your product so users can share how easy or not it was to perform a task. This way, you will learn where they get stuck
• Data analytics: Add the minimum level of data collection to assess user engagement, conversion, and frequency of use. You want to understand how much your product resonates with your audience
• Heatmaps: Heatmaps help you understand where users put their attention and what they ignore. This will help you prioritize how to evolve your product and where to learn more in-depth from your users

Remember — your objective is to learn as fast as possible. The longer it takes to learn, the riskier your product idea becomes.

Challenges to pilot testing and their solutions

You will face different types of resistance and challenges when you’re pilot testing. I’ll share the three most common challenges and how you can avoid them:

1. First version trap

The pilot product version isn’t your first version. Treating it that way will mean taking too long to get it to users’ hands.

A common mistake is going into defining an MVP and then investing months into building it. With pilot testing, the goal is to build just enough for users to try and learn from the test. So, focus on the core aspects and leave the rest out.

If you choose to pilot test, remember that confronting your product against reality as fast as possible is mandatory. The advantage of a pilot is that it makes that possible with a small controlled group. You don’t need to fear repercussions. Embrace learning.

2. Lack of support

Some business stakeholders will inevitably be against pilot testing:

• I talked to 10 customers, and they want this!
• Our competitors have this feature. We need to build it.
• We don’t have time to waste. Let’s ship it!

The above will increase the risks of failure. In such cases, I suggest taking 10% of the overall budget to run a pilot and learn whether investing further makes sense. When you limit the investment, you will be more pragmatic and use the pilot to boost learning, enabling you to make better-informed decisions.

Strive to move from opinions to evidence.

3. Validation obsession

Many people confuse pilot testing with a validation of your product idea. That’s right up to a certain extent, but it’s a tricky approach.

We often fall prey to our biases when we try to prove ourselves right. We search for confirmation and usually find “positive signs” that our idea will fly. Yet, we fail to learn what we don’t know, missing valuable opportunities to make the product more relevant.

The goal isn’t to find reasons to build the way you want but to find reasons not to make it or to do it differently. Focus on falsifying your ideas instead of validating them. A learning attitude will help you uncover your blind spots.

Case study: Gmail use of pilot testing

Since early 2000, Google has developed a practice of letting employees invest a day a week into a side project, which gives them the flexibility to explore wild ideas.

In 2001, Paul Buchheit, a software engineer, had the idea of creating a different way of handling emails. He was frustrated with limited storage, organization, and search options, so he came up with Gmail. This quickly moved from a side project to the most used email worldwide, though.

Pilot testing played a pivotal role in the success of Gmail.

As the product evolved, Paul Buchheit ran pilot testing with internal users in early 2003. The idea was to learn how the product solved the challenges Paul himself encountered. The pilot testing revealed aspects like:

• Insufficient storage — As googlers used Gmail, the storage quickly filled up, which frustrated them
• Confusing design — Users struggled to understand how to interact with Gmail; it wasn’t intuitive enough
• Inaccurate search — The search results were sub-optimal, making it harder to find what users wanted

With the feedback, the team improved the project, adding 1 GB of storage, something which was rather unimaginable in 2003. After that, Googlers gained trust in the product, and it was time to finalize the pilot as it achieved its objectives.

After the successful pilot, though, Google didn’t release Gmail to everyone. It first went into a beta release, where users could get in only with invitations. The objective was to slowly scale and learn how the product solved users’ challenges at scale.

Gmail was, then, officially launched in 2004 after successfully completing beta testing. Today, it’s the most used email worldwide. The initial pilot was fundamental to its success because it reviewed core flaws before reaching a mass audience. When the product arrived in users’ hands, it was already a better, more polished, and consistent version.

Key takeaways

When you run pilot testing, most of your ideas will not go beyond it because you will discover that investing in them isn’t worth it. It’s normal to drop ideas after running a pilot.

And remember that beta testing isn’t the same as pilot testing. The first aims to address an uncontrolled group in a higher volume, collecting relevant quantitative data. The second targets a small controlled group, enabling you to gather qualitative data.

Pilot testing isn’t the first version of your product but a fast way of learning from real users in a controlled group. It aims to yield learning, not to prove yourself right. So, with pilot testing, you should strive to uncover your blind spots instead of letting false positives illude you.