Leader Spotlight: Building and training AI models, with Abner Rosales Castillo

Abner Rosales Castillo is Senior Director of Product Management Analytics at Experian. He began his career at American Express and later became a senior marketing manager at LexisNexis, a provider of legal, regulatory, and business information and analytics. From there, Abner worked in marketing and strategy at Talanx, an insurance group, before joining Dun & Bradstreet as VP of Product Management. He also co-founded two companies before joining Experian.

In our conversation, Abner talks about his team’s process for building and training machine learning and predictive analytics models. He also discusses how they adapt to changing regulations in an industry with such high oversight as consumer credit reporting.

Marketing vs. product management

You started in marketing and slowly transitioned into product management. How did that shift come about?

For me, it was more of an organic move to the product management world. I’d say product management is still a pretty new function, and I started more on the strategic side of the business. From there, I got into product-specific strategy. Eventually, that became product management, which I’ve applied to different industries. I’ve been going through all different verticals throughout my background, and that’s how I fell into product management solidly.

Do you see a lot of overlap in your marketing background compared to what you do as a product manager?

I was on the strategy side of marketing, which is not usually the most fun part of being a marketer. I would say it is very similar, but the main difference is that you’re driven by the consumer when you’re in product management. As a product person, you’re more focused on trying to solve an issue or pain point, or you’re trying to bring a solution to your clients at a very specific level, such as behavior or capability. In marketing, generally, you are more focused on the strategy of how to take something to market and the features that it would need to make sense within the market.

Segmenting data for machine learning

From your perspective, what does it really mean to be data-driven?

I think it’s about making decisions with facts. You’re putting a strategy in place along with a vision of your product. The roadmap is based on facts, not feelings, nor what the board or internal teams like. Being data-driven centers around driving your product based on facts. That is a critical path.

Data comes down to understanding behaviors. At Experian, we know for a fact how our clients and users are working. We know how they use our tools and what they want and don’t want, and from there, we can take action against those.

A big focus of your current role is spending time working with teams who are building LLM models. What is that like and how do you select the populations for those models?

Being in analytics for a while now, I’ve seen that machine learning has become a popular topic. It’s critical to find the right purpose, which population you’re going to go after, and what makes sense for the specific model that you’re building.

My day-to-day, the products that we build, and the overall experience have to do with what we are trying to satisfy. What exactly is the model going to be doing? And based on that, that’s how we find the right segmentation. We always stand from the broadest consumer and then drill down until we find the right one via segmentation, filtering, and optimization.

We test and train it, and that’s when we first understand whether the model will perform as we want it to. From there, we can make decisions as to whether to switch it or not within the segmentations. There are lots of different ways to create these sub-segments, and that’s where the team focuses a lot of their time. Most of the time, the results have more to do with finding the right segments and understanding data rather than the model itself.

Could you give an example to add some context to what a segment might be?

One example is by profiles of consumers. Which customers are more risky and which are less risky? We’ll have specific types of trades, for instance, because Experian has to do more with trades and building models. We look into the behaviors of those consumers and based on that break them down into different populations.

Segmenting by geography is a popular tactic, but we also want to be representative. We are in a very regulated industry, so it’s hard to build something specific to states or regions unless it’s a product specific to credit unions, for example. We can build models for those instances specifically, which we have in the past, but when we’re talking about broader models, we have to comply with multiple regulations. As a result, we can’t segment in the same way other industries can.

Building predictive analytics models

AI or ML models aren’t useful without good data, so how do you know if you have that?

Once we find a data set that we think will fit our purpose — or if we have a use case first and then go after that data — our team spends quite a lot of time cleaning the data. We want to really understand it all the way down to the meaning of every last column. Our goal is to be able to interpret that data in the best possible way. And out of that, we create our own aggregations or things that will make sense for this specific model. This is probably the lengthiest part of the entire process because this is where we need to truly understand every single aspect of the data set.

Our teams spend most of their time providing analytics within that data set. How many records do we have? How many are clean? How many are doubled? From the whole analysis that they implemented, they will come up with clean, spot-on data, and then they can start testing assumptions or hypotheses. Based on those hypotheses, we can make sense of our findings. At least in our world, we have to see a large volume of consumers. Our data set has to be big enough and representative enough for us to say, “OK, this is something we can work with.”

You also work with teams that build the predictive models for the credit products Experian offers. Could you talk a bit about prediction models as well?

For us, machine learning and prediction are part of the same notion. Machine learning is a methodology around how we get a specific outcome. When it comes to prediction, there is linear regression, machine learning, and a lot of other models within this concept that are part of that main goal — predicting or identifying propensity models. What is the likelihood of someone opening an email or accepting a discount code that we provide?

At the end of the day, prediction models fall under the same structure, data, cleaning exercises, and testing as machine learning. Machine learning just happens to be more structured and makes it easier for our teams to identify potential issues.

Keeping up with changing regulations

You mentioned regulation before, and that adds a tremendous amount of complexity to nearly all processes. Can you say more about that?

The three components we always look for when we build a model are observability, disputability, and explainability. Not every single data that we have or every type of variable that we build will automatically meet that criteria, but in any model that we build at Experian, we focus on the use case.

What are we trying to solve for? That’s a critical component for us. After we understand that, we think through the expected outcome. How would that play into the big picture for a lender? Score, for example, is a small piece within a full strategy. Once we understand the big picture, that’s where governance comes along, because everything that we build has different regulations that apply to it. It’s not one-size-fits-all.

The majority of our regulations — for use cases that have to do with giving the consumer a decision or offer , or that will undergo underwriting — fall under the Fair Credit Reporting Act (FCRA), Equal Credit Opportunity Act (ECOA), and Fair Lending,. For example, if we’re going to say no to your application, we need to be able to explain why, and we need to give you the opportunity to dispute that decision.

How do you stay up to date on the many changes in federal and state regulations?

Experian is well-equipped to provide our teams with all the changes. We see changes related to governance and relationships, but I am personally subscribed to every single governing institution that impacts us, such as FCRA and ECOA. That way, I’m always aware of what’s coming.

Stakeholder management and having the right mindset

You’ve had experience in several industries. How has your background in B2B SaaS informed your current work at Experian?

Many of the products that I manage have to do with data. Right now, Experian is going through a full transformation to try to be more agile in our delivery and create necessary cloud platforms for our clients. I come from the platform world, and I’ve worked with data for quite a while. The biggest differences that I see are how to read the market, how to read the user, and having the ability to go into a very detailed approach while also trying to see the big picture. Having strong metrics and regulation awareness is key because our decisions all impact that and vice versa.

My mindset around scalability and flexibility puts me in a good position to make decisions based on insights. Nowadays, we live in a more organized world when it comes to data, but I’m hoping to emphasize user-driven conversations more.

In your current role, do you spend a lot of time on stakeholder management?

Yes! I believe that a product manager is a mini-CEO of their product. They need to be able to see and feel what’s going on, and not only focus on the product itself but also understand the market, sales, engineering, etc. Otherwise, they might not build a product or roadmap that actually aligns with reality. Stakeholder management, as a whole, is at the core of this.

Expereian is in a highly regulated industry. Multiple key people play within specific products. We need to be able to manage expectations, as well as set the right tone for the team. We also need to get our partners to help us achieve what we’re trying to do, which is one of the most important functions of being a product manager. Stakeholder management is not a percentage of our activities, it’s integrated into the full end-to-end process of our work.

When you’re hiring for your team, what traits do you look for besides demonstrated experience?

For me, it’s about having the right mindset and a high level of awareness. For instance, if I were to hire tomorrow, I would be looking for people who have an awareness of the regulatory environment and skills related to problem-solving. Additionally, they would need to have strong analytical skills and be able to understand things beyond just headlines.

Stakeholder management is very hard to gauge in an interview, so I like to make sure the person has experience with that. Communication, across the board, is also important. Lastly, I look for someone who is great with documentation. Most people will not be in their current role forever. If something happens tomorrow, documentation is the key to maintaining momentum and moving forward. Those are things that are so important to the product management role and that I always look for them when hiring.