Machine Learning in the eCommerce Cybersecurity Space - Riskified, Leader in Frictionless Fraud Management

This week we sat down with Assaf Feldman, the Co-Founder and CTO of Riskified, to ask him about the development of Riskified’s data science models and machine learning best practices. Riskified provides eCommerce fraud-prevention SaaS solutions for enterprise merchants, using advanced machine learning algorithms to differentiate legitimate purchases from fraud.

Before co-founding Riskified, Assaf held several positions, including as an algorithm engineer at BillGuard, the Head of Research Technologies at Kinetic Global Markets, Lead Developer at monitor110.com, and VP of Engineering at Oddcast. He has extensive experience in building systems that use machine learning and innovative algorithms to solve complex problems.

As CTO of Riskified, Assaf leads over 140 data scientists, software engineers, product managers, and analysts. Under the leadership of Assaf and his co-founder, Eido Gal, Riskified has more than doubled its revenue in each of the last five years and began trading on the New York Stock Exchange in July 2021.

Our CEO, Yaron Singer, led the discussion.

Can you give us some background on Riskified and how you're using machine learning?

“Riskified is a platform of products for e-Commerce mainly, which deals with different risks along the customer journey. We're helping online merchants optimize their customer journey, from the login through checkout and the transaction.

When you're selling online, you have unique risks. Basically, every innovation you make as an e-commerce company to ensure a smoother and interesting user experience comes with some risk and an open door for abuse or fraud.

Riskified applies machine learning in all these junctions to help prevent fraud and be more accurate about what is legitimate and what is not legitimate. This facilitates a better customer experience and ensures that the legitimate ones make it all the way through and end up buying and remaining as real customers to our merchants.

That was our main offering for many years: detecting if an online transaction was being made by the owner of the payment method, for example, the credit card or the PayPal account. Later, we added products that help customers deal with being declined by their banks and how they recover that transaction. We also have a product called “Account Secure,” which is securing login by trying to assess if the user that is logging in is actually the owner of the account or if that's someone taking over their account.

We also have a new product that we're developing that is focused on abuse—all kinds of scenarios, for example where people are saying the package was not delivered even though it was delivered and are thus abusing policies. There is trust between the eCommerce merchant and the customer, and sometimes the customer takes advantage of them.

We've also developed our own AI platform to facilitate the whole lifecycle of training models and improve models and check different features and feature engineering approaches. At the end of the day, it's a platform that enables us to have dozens of data scientists working in parallel in different avenues.”

One of the things that you mentioned is model selection. How does that process work at Riskified?

“We’re not really in an endless loop of testing one against the other, but we are relying on different feedback loops coming from the field.

Lower performance is a trigger. We're also monitoring features and data drift, and things that are rooted in the data to trigger retraining. Once we have either retrained or improved a model, we have a system that checks the new version's performance versus the old version offline.

We then go into ‘shadow mode,’ where we let both models run in real-time in production and include some A/B Testing at this stage. When we're sure about it, we finalize the replacement. Later on, we also check retrospectively to see that our estimates of the improvements are actually correct.

We also have mechanisms, like control groups, that keep on checking the false decline rate—call it false negative or false positive, depending on how you look at your negative or positive. In our case—especially in the checkout—once we've declined an order, you never know if it's good or bad. So, it's tough to assess your false decline rate.

The transactions that we approve are guaranteed to our merchants, so every mistake that we make comes out of our pocket. It's a great way to keep the incentives aligned. We also guarantee an approval rate. This is one of the things I really like about the way we created the business model because it ended up making sure that we create a whole machine learning platform that is aligned with incentives of the merchants.”

What are a few best practices for building robust machine learning systems?

"It's not just data science—it's data science-meets-engineering, especially when you do it at scale, which I think not many companies are doing. That's kind of unique to us. We're running dozens of models in real-time in production that are changing all the time, with huge teams working on that.

It requires collaboration between research and data scientists and engineers and creating an organization and a platform that can support it. We also have to build the data storage, data architectures, and data strategies to enable this.

As the scale gets larger and larger, it's an enormous challenge that we’re continuing to improve on and adapt to. That's one of the biggest challenges and one of my main focuses."