Transcription:
Penny Crosman (00:03):
Welcome to the American Banker Podcast. I'm Penny Crosman. There are risks to using large language models like ChatGPT in regulated financial institutions, obviously, but there is also an opportunity cost to banning the use of ChatGPT altogether, according to Ryan Favro, a managing principal at Capco. Welcome Ryan.
Ryan Favro (00:24):
Good afternoon. Thanks for having me.
Penny Crosman (00:25):
Thanks for coming. So as you know, Deutsche Bank has blocked employees from using the ChatGPT website. JP Morgan Chase has reportedly restricted employees' use of ChatGPT . Why do you think this is happening? Do you think these banks are more concerned that employees will upload sensitive documents or information to ChatGPT or do you think they're more worried that they'll try to make ChatGPT do their work for them the way students get ChatGPT to do their homework? Or is there some other concern that's driving these bans or restrictions on the use of it in large banks?
Ryan Favro (01:09):
Good question to get started. So I think all of the above, but let's start at the high level and we'll double click number one, banks are extremely allergic to risk and I think there's a consensus there. I've yet to see a bank that is not allergic to risk, especially severely. So here you have this technology, you have ChatGPT there, there's other technology, but we'll just say ChatGPT to cover sort of broad spectrum and it's a black box, we don't know how it works. It can do some profound things. It feels like magic and especially when you don't understand how it actually functions. So your immediate reaction is, if I don't understand this, I need to make sure that I'm limiting my risk. So the banks say, okay, we're not going to allow this thing because if we're sending it information, where does that information go or how long does it go there?
(02:03)
Who can see it? What did they do with it so that nobody can answer those questions early on. Or even if there were answers to those questions, we hadn't seen precedent to see is this true And the banks, so the banks put the brakes on that. I'm not really certain that the banks were worried so much about ChatGPT doing their work for them and fundamentally at Capco and my personal position is we should stop wasting time on tasks that AI can do for us. So if I can get an AI, whether it's ChatGPT or any other tool to help me perform my job better so that I can do more in the number of hours I have in a day, so be it. Right? It helps the bottom line. So I think primarily they're allergic to risk. They didn't have or didn't enough information in which to proceed. So the prudent thing was to say let's stop and create an assessment.
Penny Crosman (03:01):
So we're starting to see banks start to experiment with large language models, the kind of AI engine that can process a lot of information. For instance we wrote yesterday about Westpac, which is letting employees and customers use a large language model [not ChatGPT, but software from Kasisto] to help them through the mortgage process. Customers will be able to ask questions about the forms they have to fill out and what's next and so forth. And it'll help employees by checking information about borrowers for them. So it taking a little bit of legwork out of the work that loan officers might have to do. And I think other banks are thinking about use cases like this. Are there any risks to that kind of thing where it's, what I'm seeing is kind of testing with employees first and then later pushing out to customers. Are there any unseen risks in that kind of approach?
Ryan Favro (04:15):
Yeah, I think there are, and again, I'm going to start broad and we'll double click on where this goes. So the first risk is the information that ChatGPT is trained on. So it's basically goes up till September, 2021. They sucked in as much of the internet as they possibly could and when they suck in the internet, that goes through into their system and then they have to train their system on what is factual or true and what is not. So the models that the banks are using, they will contain biases for variety of different reasons because a human being was involved in determining what the models believe is true or false. The employees that are using these tools may not understand that the models that they're drawing from the large language models are fixed in time. So if a policy's changed since the September, 2021 date, if policies change or there's new information, that may be a blind spot. Now there's ways to mitigate that from an engineering perspective. There are ways to get around, but out of the box, out of the box that's a problem.
I would be extremely concerned as well about the wrong information leaving the premises of my bank. So as an employee I start typing in, and this is theoretical, as an employee of a bank, I start to interact with the large language model and I start to tell it things about my customer or about my business that would be considered confidential or private information, personally identifiable inform information and I may inadvertently trip a regulatory rule and I've now sent data that I'm not allowed to send outside of my network. And the large language models are not on-prem, right? These are super computers that are shared resources amongst the customers of OpenAI. So that would be my major concern. I'm leaking data and I'm getting data that I'm not quite familiar with and I'm just trusting the system right out of the box and I'm making decisions with that without truly vetting it. Yeah,
Penny Crosman (06:32):
Those sound like significant risks. I do hear banks talk about enterprise large language models, so within the confines of the bank using only information that they provide. So now of course then you have less information to draw on, but it seems like that's at least something I hear people talking about to try to contain the risks of going outside. But that risk of a policy document being out of date I think is a very good one. And I guess you would have to have some sort of mechanism for making sure that documents are up to date, but then is there a way to delete the old, I'm not quite sure how that would work. So we talked in the beginning about these banks that are severely restricting or banning the use of ChatGPT in their organizations. Do you think there's an opportunity lost here or a unintended consequence of taking such a harsh stance?
Ryan Favro (07:43):
Oh absolutely. Right. So at Capco, our R and D team has been working with this technology for well over a year and we've seen efficiencies across the board. So at first we were sort of covertly using this because we wanted to see if people could tell if our output was augmented with AI or not. So for example, we were generating certain types of code that would take hours and we were doing it in 10 seconds. Now if you're not a software developer, the code it generates may or may not be good, you don't know. But as an engineer we could look at that code and go, 80% of it's pretty good, the other 20 percent's wrong, so let's make the adjustments, the fine fine detail adjustments. And then I've now saved myself seven hours and now I'm more productive because I can do more work.
(08:36)
That use case can apply to a lot of other areas that banks might be interested in. So for example, their HR departments may be interested in using AI to augment certain tasks like vetting resumes. So once you strip your PII off a resume, because don't want, we want to make sure we're not leaking information. We could do things like ask the AI to not decide if we should hire someone or not because that would be a bad use of ai, but we could ask AI to do something like, hey, for this resume and for this job, show me all the red flags. Show me all the red flags. Might as a recruiter I might be interested in knowing that it's not obvious. So forgetting insights and summarizations and taking large amounts of data and it breaking it into consumable chunks for humans to benefit from.
(09:28)
I mean that's fantastic. The other thing that worries me when they have outright bans and the bigger your organization, the more likely this is probably going to occur, employees will find a way to work around the restrictions you put in and as soon as they find those workarounds, we're back to where we started. All that risk is back on the table. And as an organization, you have no way of knowing that this is taking place until something goes wrong. So I could could be banned on my work computer, but I'm using my personal computer right next to me to manually do the side side of desk, side of desk access to ChatGPT, and I'm still leaking information or I'm still bringing in information that has no auditability. And we know the banks need to be audited. That's one of the big things that we can't do with ChatGPT right now is I have no audit trail. I don't know what information went out. I don't know what information came in or what decisions were made with that information.
Penny Crosman (10:27):
That reminds me of the WhatsApp problem where banks are supposed to record interactions with clients that have anything to do with stock trades or investment trades. And all the big banks almost have had instances of people using their personal phones and WhatsApp or similar apps to communicate with clients off the record. And the banks have all gotten big fines for them. So it's like a super common problem. So I can see that migrating to this case.
Ryan Favro (11:03):
And the employees aren't even malicious. It's human nature to just want to be efficient. In your head, you justify it. You say, if I use this tool or use WhatsApp or whatever the medium is, I'm saving my employer time and money because I'm more efficient. And everyone goes, oh, and nothing bad's going to happen, right? Oh, what's the worst that's going to happen? My client is happy I got the message across. Everyone goes home early on a Friday and we have a nice weekend, it's fine until it doesn't work. So when we ban it all, we're just asking for these sort of scenarios to inevitably come up. And the last thing I would want to do is have a really good employee end up in a position where their job's at risk because they violated some policy because we were too rigid.
Penny Crosman (11:50):
Yeah, that's a good point, too. So recently there was an AI hearing in Congress where Sam Altman at OpenAI, someone from IBM, there was a college professor there. They all talked about what regulation of AI could look like. I don't know if you caught that hearing, but did you have any thoughts on what was said there?
Ryan Favro (12:13):
So I didn't watch the whole hearing, so I don't want to provide too much of a precise statement on this, but I can tell you generally we see these types of hearings happen all the time whenever there's new disruptive technology. And usually you have to ask yourself why are they having the hearing? What's the problem? Is there a problem? Because so far there really hasn't been a problem that would require a government body to sit down and say, we need to construct some rules. Yet usually that's a reaction to something going wrong. So the question is why are they doing it?
(12:51)
Sam is a founder of OpenAI. Open AI is maybe in position number one today in this market. What better way to keep your position than to have regulations put in place that make it harder for competition to come in and be the next step in all the major industries? We see this so that's my personal gut feeling is that these are hearings to prop up or ensure that the incumbents stay incumbent and the folks that are on the regulatory side or the political side use it as a platform so that they can advance their agendas as well. That's usually how these things go. I mean, I look back in time and show me a hearing like this that didn't have some sort of ulterior agenda for both sides under the tent of whatever the hearing's about.
Penny Crosman (13:44):
Well, that's super cynical. I love it. And I think you're probably correct, and Sam Altman took most of those congressmen out to dinner the night before. So it was very friendly and at one point, one of the congressmen asked him if he wanted to be the person to set up a new entity that would regulate AI. So that was a little bit remarkable also. But there has been a lot of discussion about regulating AI. The White House has this blueprint for responsible AI. That's not the exact language, but they have a framework for how companies should think about the kinds of guardrails they put around AI. And certainly some of the bank regulators have talked about their concerns about AI, especially Rohit Chopra's talked a lot about the idea of AI in lending and that you can't use a black box, it's got to be explainable, it's got to be free of bias and so forth. Do you think that we're going to see any laws or new regulations anytime soon?
Ryan Favro (15:03):
Yeah. And let's start with a position, the starting point of all of this, what information went into the large language models in the first place? It didn't come out of thin air, it came out of other people's work. So the large language model model could be argued by some that it is a derivative of somebody else's work in intellectual property. So the question becomes, do the people that own that intellectual property that was sucked into this large language model, are they a stakeholder or a shareholder even in the money that's generated? How do they get paid? How are they compensated? How do our IP laws today protect people that have already created information that's ended up in these systems and how does it protect the people that have yet to create something that's ends up in one of these large language models? I'm not sure there has to be some balance between fair use and compensating IP owners.
(16:05)
I think that's going to probably come before everything else because that's where this all started. And we're already seeing lawsuits against generative images like some of the mid journeys and whatnot. We have artists doing class action lawsuits saying you are producing images using my visuals. In fact, there's many examples where you could see the watermarks, the Getty image watermarks in some of these generated images from these other generative platforms. So you can clearly see with your own eyes that intellectual property belonging to somebody's leaking into the new work that these machines are producing. So that's a problem. As far as regulation of use of ai, I think that's yet to be determined. What we're really seeing here is the think of AI to the knowledge or skilled worker as the industrial revolution was to the individual craftsperson, right? Because we're really doing is we're super charging the knowledge base or knowledge skilled type jobs.
(17:08)
The manual labor jobs are most likely going to be unaffected by AI, at least at this point. So if I'm a plumber, I'm not worried. If I'm an industry leading expert and I'm a high performing SME, the AI is going to function like a power booster and exponentially enhance my ability to perform and be productive. Conversely, if I'm on the lower end of the skilled workforce spectrum, AI is going to serve as an educational tool that's going to refine my ability being less experienced and therefore leveling the playing field. So I think those areas are going to come out on top. The middle is still to be determined. We're not quite sure what's going to happen there and as AI potentially impacts the middle, what do we do with those people? Or how does it play into regulatory terms from banking? Does your AI have to be on-prem? What happens if it's off-prem or if it's in a different country? So there's a whole variety of unknowns right now, but I think we should start with who owns the intellectual property that started the whole thing and how do you compensate them?
Penny Crosman (18:21):
Well, yeah, that question of which jobs will be eliminated is always interesting when you talk about any form of AI, and you were talking before about software developers using it to help them generate code, and I know both Goldman Sachs and Wells Fargo are using generative AI that way. Do you think that as banks use this for that kind of use, giving it to their software developers to make them more productive, does that mean you then need fewer software developers and could that apply to other areas as well? It just kind of makes people more effective, so you don't need as many of them.
Ryan Favro (19:03):
So let's start back at the intellectual property argument. So let's say I'm in a bank and I'm a software engineer, and I go to ChatGPT and I say, Hey, can you make me a function or an algorithm that does X, Y, Z, and it happens to give me code that belonged to somebody else, someone else. The large language models were trained on 150 million code projects. Some of those projects are probably going to have, they're not all open source, so they might have certain licensing terms that went with that code, but that code is now in the large language model, and that code now ends up in the bank's code and the bank goes, Hey, great. I saved time. I saved $800 of labor today because of ChatGPT on one employee, let's say. And what they don't know is there are lawyers in class action lawsuits or patent trolls as they're called that go around and find IP owners and say, Hey, your code is in open AI's large language model.
(20:09)
They're making money off your code, your intellectual property. Join our class action lawsuit so that we can get you compensated fairly right now. Then they go to open AI or they go to any large language model because again, this is wild, wild west. We don't know how these intellectual property laws are going to apply. And they say, Hey, we want to sue you for X number of dollars because we know that you are making money off our client's work and therefore you owe them whatever. And then the large language model provider says, hang on a second. We're just like Google, we're just indexing this stuff and passing it through. We're not claiming ownership of it. And then let's say that argument holds water. They go, great, give us the logs and tell us where my client's intellectual property went, who queried it, when and where can we find them?
(21:00)
And they get, maybe they're successful, they get the list, they get the logs, and they go through the logs and it's some kid in his basement, it's like some startup company. And then they see a bank and they go, bingo. And then all of a sudden this bank receives a demand letter. You owe my client money, pay now or stop using their code and shut off your bank, or whatever the scenario is. That keeps me up at night. And as an engineer, that really keeps me up at night because I love the benefits of ChatGPT, I absolutely like the ability for it to write code that I would write myself anyways, and then it just does in 10 seconds create a function that determines if somebody is the age of majority in the state they live in, right? That's not going to be an intellectual property conflict most likely.
(21:46)
But when I have untrackable access to things like ChatGPT and developer's hands, I have no way to audit where they got their code. We actually pivot and we say half of the development life cycle is writing automated unit tests. Those unit tests are going to be unique because they're based on your code. They test your codes like software to test software. So I can actually, we built tools that allow us to do the unit test generation in one click, and we basically save 50% of our dev time now that is huge. And we circumvent the risk of IP conflict, so we're okay still developing by hand, but we automate with automation.
Penny Crosman (22:28):
Interesting. So that idea of having ChatGPT write your code for you is pretty fraught with challenges, but this other use case of using it for testing is quite a bit safer or seems much less problematic, which yeah, makes sense. That's a nuance I hadn't really thought of.
Ryan Favro (22:49):
Beware of anything that's too good to be true. Right?
Penny Crosman (22:53):
Yeah, that's a good rule for life. Great. Well, this has been super interesting. Maybe we could have you back another time to talk about the later developments as this all carries through. But Ryan Favro, thanks so much for joining us today and for all of you and the audience, thank you for listening to the American Banker Podcast. I produced this episode with audio production by Wen-Wyst Jeanmary. Special thanks this week to Ryan Favro at Capco. Rate us, review us and subscribe to our content at www.americanbanker.com/subscribe. From American Banker, I'm Penny Crosman and thanks for listening.