Kristin Milchanowski shared some of the ways her team is working toward a return on AI and making models more trustworthy at American Banker's Digital Banking Conference.
Transcription:
Transcripts are generated using a combination of speech recognition software and human transcribers, and may contain errors. Please check the corresponding audio for the authoritative record.
Penny Crosman (00:00):
Hi, I'm Penny Crosman, tech editor at American Banker, and I'm here with Kristin Milchanowski, who is chief AI and data officer at BMO. And she's going to tell us about what she's been doing with AI at BMO. So thanks for coming.
Kristin Milchanowski (00:14):
Thanks for having me, Penny.
Penny Crosman (00:15):
Can you tell us about some of the top AI projects at your bank?
Kristin Milchanowski (00:19):
Absolutely. We are doing everything for business value. So we have hyper-personalization, automation and augmentation. And some of those big projects for us is one called M, and that's where we're transforming our investor line product, which is where individual clients come to make their trades and get advice. So that is our advice generating tool that we're underway building for our clients.
Penny Crosman (00:49):
And how are you getting a return on investment for AI?
Kristin Milchanowski (00:53):
Because we start every project out with the notion of business value first. So we collectively, business leaders get in a room and we agree that we believe a project's going to provide a certain amount of business value. We align on what those OKRs, those metrics are to measure that. And then we really go after the build of that project. And that's how we are ensuring that we get the return out of what we're doing. So we're not just building AI for the sake of AI. We're really pedantic about the business value.
Penny Crosman (01:28):
So Kristen, what is on your drawing board? What might we expect to see from BMO in the next five years?
Kristin Milchanowski (01:34):
Oh, five years, that's a big crystal ball. But I really believe that if we can optimize capital, leverage AI to optimize capital, if we can create so much quality hyper-personalization that we attract new quality clients in our portfolio. And then if we create a few new revenue streams like with our product M, that we will really be far ahead for our shareholders.
Penny Crosman (02:05):
So we've seen surveys showing that consumers distrust AI agents. How can a bank or fintech build trust in the AI agents that they launch?
Kristin Milchanowski (02:16):
One of the things that a bank has going for us is that we have been a highly regulated industry for a very long time. And so governance with excellence is something we do well. And one of the things that we're introducing with the new conversation around agents is agent governance and making sure that every agent has what we call a performance evaluation. And that performance evaluation is something that lives with that agent. It's making sure that that model is still producing the output that we expected it to. Because large language models, they degradate over time. And so you want to have that performance eval in place so that you can capture any moment that it starts to degradate a bit. But that performance eval is just like your kid's report card. And so that in an essence helps build trust that it is doing what you expected it to do and performing the way that you expected it to perform.
Penny Crosman (03:15):
And if an agent does degrade and gets a C or a D or an F, do you put it out of commission or do you retrain it or?
Kristin Milchanowski (03:26):
There's a lot of viable options that you can do, but the good thing is that we still have the human in the loops and the human's taking over at that point so that the process doesn't get interrupted basically. And so then we go back to the model and we retrain it or whichever step of the process that we feel is broken and we get it back online with a performance eval and you're off to the races again. The other big part to trust is just dialogue. So making sure that everyone has access to AI, making sure that they feel comfortable and trained and confident in how they're being upskilled and leaning into the conversation. So we offer a lot of training and a lot of just communication sessions with our teammates around the firm and also with our clients. So a lot of my time is spent with our clients just talking about our program and how we're going about it and ways that they could perhaps even adopt themselves.
Penny Crosman (04:23):
Well, Kristen, thank you so much for joining us today. This has been really interesting.
Kristin Milchanowski (04:27):
Thank you for having me, Penny, and thanks to American Banker for hosting this wonderful event.