Leader Spotlight: The importance of vigorous experimentation, with Maria Cuasay

Maria Cuasay is Director of Product, Growth at Ancestry. She began her career as an analyst on the venture capital team at the Business Development Bank of Canada before becoming employee number 9 at Opencare.com, a software company focused on connecting patients to exceptional dental care. Prior to joining Ancestry, Maria held growth and product leadership roles at Lyft and Elemy.

In our conversation, Maria talks about hitting goals and running experiments as fast as possible. She shares her approach to MVPs, which includes breaking down the product as far as it can go. She also discusses how she’s grown — both personally and professionally — to handle stressful situations as a leader.

From VC to product management

How did your early career experience in venture capital influence your approach to product management?

I sort of fell into venture capital. I’d just graduated from college and didn’t know what I wanted to do. It was also the Great Recession, which created a very difficult business environment. I wasn’t planning on being in tech at all, and this was still before the boom of companies like Airbnb and Uber. But that early start in venture capital turned out to be one of the best things for my career.

There were a couple of things that made my time in VC so beneficial. First, I became very comfortable with analytics and gained lots of experience reviewing companies and products. From the analytics perspective, I look at things very top-down. It sounds simple, but lots of people start in product thinking from the bottom up — they build the product and then they try to figure out how it would help the business as opposed to the other way around.

Second, I learned how to network. That’s not something a lot of people may put on their list because we tend to focus on hard skills. I learned how to find and get in touch with the right people and how to build a network. Eventually, I learned how to get in touch with heads of product. Now, whenever I’m dealing with a product problem, I reach out to my network.

Distilling the product down

You mentioned thinking about products from the top down. Could you describe your approach to building and testing MVPs? What’s your approach for balancing speed and quality — how do you know when it’s good enough?

I like to first think: what’s the most ideal thing we should build? I learned this approach from an engineer that I worked with, Dave Connolly. I was learning how to build products when we were creating an MVP of a referral product in one of my first companies. I was saying, “We have to build this, this, and this. And this is how they connect.” He stopped me and said, “What is the Cadillac of this MVP? What is your ideal solution? Once we’ve drawn that out, we can chop it up and turn it into a Toyota Corolla.”

There are multiple ways to think about the ideal solution. First, look at other products — don’t reinvent the wheel. Second, know the metrics that you’re trying to move and what they are. What are all the features that you need to build for this ideal experience? Third, what are customers looking for and what problems are they trying to solve? There’s going to be many, and that’s why you’ll need multiple features.

Once you’ve drawn it out, you can chop it up. Then, once you have something that you think you can’t distill down any further, you can figure out timelines. If it’s within the timeline you’re looking for, then move forward.

How do you know how far to ‘chop it up’ — how much is too much??

I think figuring out how far to chop it up is the biggest question. That’s where you need the three things. What does high-level data analysis tell you, what are the main things that customers are looking for and are those “must haves” in the product, and lastly what is your product intuition telling you?

For analytics, for example, if the most crucial thing to make your MVP work is driving traffic, then you need to prioritize entry points. Next, you need to verify what customers are looking for. I suggest interviewing a bunch of customers, listing out everything that they want or the biggest problems that they’re trying to solve, and then prioritizing which problems are the most important.

Let’s say you want to solve a problem with referrals and a user requests having the ability to select people to refer from their contact list. That’s not a must-have — it’s a nice-to-have, so I’d put it at the very bottom of the list. But a key component of referrals is having a referral code, so that would be the highest priority.

Experimenting in product development

Are there any tools or technologies that you lean on or that you’ve found effective and helpful in accelerating product development?

From an analytic standpoint, we use an experimentation platform called StatSig. One thing I want to call out is that with experimentation, interaction effects are a very real thing. When there are so many experiments going on, you have no idea what’s colliding. Having a tracker to see which of them are pulling down key metrics like revenue is important. Especially at larger companies, there could be a team that you don’t talk to every day that could be pulling down your numbers because of an experiment they’re running.

We also employ a lot of user research tooling. This has been very helpful to me personally and also to the team. There are a couple of ways to do user research. One is through moderated study where you interview the people and ask them what they’re thinking or feeling. There are also unmoderated studies where users go through your experiment and then share their thoughts.

You’ve emphasized the importance of vigorous experimentation. How do you balance that with the possibility that the experiments could disrupt customer experience?

One of the biggest issues that we have with so many experiments going on is that they can collide with each other. When that happens, the experiments get messed up. We’re still working on figuring out the best way to go about this. We ask PMs and engineers to talk to each other as much as possible. We now have a regularly scheduled meeting to connect about experiments that could potentially collide. There’s a lot of pieces to it but we’re working on cleaning the processes up.

What metrics do you rely on to test the impact and success of your product iterations?

This is our bread and butter. Our team has a revenue goal and we break that down to its core metrics. Let’s say we’re trying to drive $10 million in revenue. We back out how many customers we need to get to hit that revenue goal, and then how much traffic we need to get that many customers. If the conversion rate is really low from traffic to customers, we evaluate ways to increase the conversion rate.? We break it down further and further to determine the number of experiments that we need to run.

We usually have an assumption that we need to drive the conversion rate to, say, 10 percent. If an experiment is a win, it usually increases the conversion rate by only 2 percent. So we probably need at least five winning experiments to get it up to 10 percent. But how many experiments actually win? Only 20 percent. So, in that case, we’d need to run 25 experiments in order to get five winners. We break it down that far. This method allows the team to feel like they are in control of their destiny.

How long does the experiment take to run, on average?

The length of time really depends on the company and the type of users it has. We specifically do two weeks, but if we’re running out of time, we have the option to reduce that to one week so that we can get results faster. The reason why we do one to two weeks is because Ancestry’s user behavior is pretty high during the week as opposed to the weekend. If we only look at the data during the weekend, then it’s going to look pretty rough.

Hitting goals as fast as possible

When you’re hiring, what are some characteristics or traits that you look for?

The best growth PMs have three things: analytics skills, speed, and creativity. But it is very hard to find all three things in one person, so the big non-negotiable for me is being analytical. I feel like someone either has that or they don’t.

Speed and creativity, though, can depend on where the person is at. I believe creativity can be learned. I came from finance, and we didn’t need to come up with grand ideas. But I think being so exposed to very creative people helped me get there. It does take time to develop, though, creativity has a slow ramp-up. If you’re getting someone who’s a bit more early in their career, that’s going to be a two-year investment until they can really start to be creative on their own.

Speed is based on the environment that you’re in. For example, if you’re in a city like Omaha, Nebraska, people usually walk slowly. It’s a very slow-paced life. You’re probably also going to walk slowly too. But then if you travel to New York City, you have no choice but to walk very quickly to keep up with everyone else. That’s similar to our growth team — we move so quickly that members have no choice but to keep up. Someone said that joining our team is like jumping onto a bullet train.

If someone is not used to this type of momentum, it can definitely be a shock to the system to join our team. But I still think what’s really critical is how fast they think, and how efficient they are in analyzing things, coming up with good ideas, and executing them. And, at the end of the day, they may realize that our environment is too fast for them, and that’s OK.

Handling stress and creating a positive team environment

Setbacks are always part of life when you work in product. Could you share an example of a significant setback from some point in your career and the learnings from that?

My biggest setback wasn’t product-specific, it was around me as a leader. Earlier in my career, I struggled with building the right team environment, especially during highly stressful moments. This specific instance was during a very stressful time. It was at the end of the year and we were trying to hit our aggressive goals. It became so stressful that I wasn’t mindful of what I was saying to others and how I was reacting to things.

This is so important for any leader to learn to handle — especially in growth where stress levels can go so high that they’re not conducive to productivity or effectiveness. I had to take a step back and figure out the core problems of why that happened.

I have found meditation to be very helpful for regulating stress, and I’ve become very intentional about itn. No matter how stressful things become, I can always be very calm and make sure that the environment is both focused and fun at the same time. I’ve learned how to avoid projecting my stress onto other people, as well as how to be mindful of what I say and how I react, especially to other people’s opinions and ideas. This was the biggest setback I’ve had professionally, and I feel fortunate that these learnings came about fairly early in my career.

How do you lead by example in terms of managing stress and work-life balance? Do you have a process for checking in with team members?

I like to make use of my PTO and be a really good role model in that area. I want my team to know that it’s a good thing to take time off because if your boss is not taking time off, it’s very likely you won’t either. Modeling that out is a really good thing.

I’m also a very disciplined person. I have my morning routine, afternoon routine, and evening routine, and if I don’t shut that off, that motor is going to burn out. Additionally, I have a monthly career meeting with people on my team. It’s a mix of everything. I ask how they’re feeling and how things are going.

We also talk about where they would want to be in their career right now or how they’re progressing towards their yearly goals. It’s a good, safe space to chat with the PMs and see how they’re feeling. Sometimes people may not realize that they’re at risk for burning out, and so you’re going to have to identify that in the most mindful way possible.