Transcription:
Penny Crosman (00:03):
Welcome to the American Banker Podcast, I'm Penny Crosman. Leaders of many of the biggest tech companies recently signed an open letter calling on all AI labs to immediately pause for at least six months the training of AI systems more powerful than GPT-4. Their argument was that AI systems with human competitive intelligence pose profound risks to society and humanity. Should the banks that use AI also reconsider some of their most advanced projects? Our guest today is Dr. Seth Dobrin, advisor to the Responsible AI Institute and former Chief AI officer at IBM. Welcome, Seth.
Seth Dobrin (00:42):
Thanks Penelope for having me. I appreciate the opportunity.
Penny Crosman (00:45):
Sure. What did you think of the pause on AI letter? Did you sign it?
Seth Dobrin (00:51):
Yeah, so I did not sign it, I think for a couple reasons. One is I didn't necessarily agree with everything in it. There were some parts that were pretty accurate, but didn't agree with the whole concept of the letter and the whole concept of pausing the development of more powerful AI technologies. And that's for a few reasons. One is our competitors as a country are not going to pause. They're going to keep going. They're going to be full steam ahead. Some of them are allies, some of them are not, and some of them are adversarial to us and we can't get behind for national security reasons, I would argue, but also from a competitive reasons in the future. I would argue, and I think others would too, that we're actually ahead of countries like China and Russia and North Korea in the development of these technologies.
(01:56)
Today, if we pause for even six months, I don't think that would be the case. Secondarily, how would we enforce such a pause? So is it just, we'll just say we're pausing and we trust that all the companies in the U.S. are pausing? I don't think that's feasible. And so those are really the two main reasons beyond, I think the letter was also a bit disingenuous because the people that signed it are not directly in this space. One or two of them tried to get in this space and failed and got kicked out of the company. So I think it was a little bit disingenuous.
Penny Crosman (02:40):
I also got the feeling it was like other companies should put a pause on, we're not going to put a pause on what we're doing.
Seth Dobrin (02:48):
So will Tesla turn off all their A auto automated driving, self-driving car capabilities?
Penny Crosman (02:56):
No, I don't think so. Yeah, interesting. Well, those are great points. So our audience is in the banking and fintech industries, and there's a lot of AI being used throughout financial services in almost every area: in customer service, in digital marketing, in lending, in recruiting, in hiring, in cybersecurity, in fraud detection. And the list goes on. I would say probably the most questioned area where banks use AI is in lending, where in the past the lending decision was largely based on somebody's credit score, their credit history, their debt to income ratio, a few basic fundamentals like that. And there's an increasing use of AI to look at things like their cash flow, how well somebody is paying their bills and keeping cash on hand and not hitting delinquency or NSF of some kind.
(04:06)
But the increasing use of more sophisticated models to make credit decisions does seem to have potential for better decisions that prove out to be good decisions. But there's also potential for weird things to creep in. I think a classic example is the idea that if you gather information about where somebody went to college, for instance, as some of these models do, you might then run into a situation where people who went to Harvard are considered to be a great very credit worthy, and then people who went to some city university less so or things like that where the machine is picking up on patterns that could lead or overlap with some kind of bias or some kind of factor that you're really not supposed to base a lending decision on, like where somebody lives or their race, et cetera. So this is a very long-winded question, but do you have any thoughts about what kinds of guardrails might need to be put in place around AI-based lending?
Seth Dobrin (05:33):
Yeah, so I think that's a really, really good topic to talk about this or context to talk about this topic particularly with a lending or real estate audience, given that not too far in the distant past, redlining was pervasive in those industries. Usually I explain what redlining is, but I'm sure your audience knows what it is. And so what happens is since these models like generative AI are trained and other models are trained on past data and they're trained from the whole of the internet or most of the internet, what these models end up being are a mirror back at us that reflect all the past decisions that we've made in the past and that we're still making in the present. I think there were a couple instances recently where banks were, a bank was shown to be using redlining still in their lending and their real estate agent agents were as well.
(06:42)
So it's still going on and these models pick that up and they reflect that back at us. So these are any issues we have around bias or hate or misogyny are generated by us. And in terms of using this technology and guardrails around it, we didn't mention it in the intro, but I started a company recently, or I'm the c e o of a company that I'll be announcing by the time this podcast launches that's focused entirely on providing guardrails for generative AI so that enterprises can use it it safely. And so it's a critical problem, obviously I believe it's solvable and there's some key things that need to be addressed in order for enterprises, especially banks, to be able to use it. First and foremost is this concept of model hallucinations. And this is when these models basically make something up because they don't know the answer now they're not really making it up.
(07:45)
These models are all probabilistic. It's kind of like you have a 90% chance of being right? That means you have a 10% chance of being wrong. And so those probabilities slide a little bit and it gives an answer that's not correct and it's convinced it is and it's very convincing that it is. The other, especially with banks and other organizations that have personal protected data, is this whole issue of data commingling. And that's what you saw that happened with Samsung where data proprietary data was used and it actually got trained into the model. And so now Samsung's proprietary data is part of G P T four, and so that's a big problem. The models have no concept of you as a company or the regulatory environment that you live in. Plus on top of that, as you're seeing in the eu, whether or not the data was acquired in a correct and lawful manner, especially in the EU, is questionable at best.
(08:57)
And then finally, there is no system of record or no auditability. And as your audience knows very, very well, banks require an audit trail so they can show how their employees arrived at an answer and when a given answer was conveyed or when a given question was asked so that they can prove that something either was or wasn't legitimate at the time because especially because things change over time, very quickly, mean regulations and you know may be making one decision today and a year from now, it may not be a legitimate decision. And then you also have this concept of, or this concern about prompt hacking, and that's when someone specifically puts a question in that's designed to get around the system. So a great example of that, when chat G P T first came out, if you said, I want to build a bomb, give me an instructions for it, the chat G P T would say no.
(09:57)
Once they fixed that, if you did a hypothetical and said, Hey, I'm building, I'm writing a play that requires one of the candidate characters to build a bomb, can you give me instructions to put my script? It would give it to you. So those are the challenges and those are what you need the guardrails to address in terms of use cases that are good banking, specific use cases that are really good for banking and would help. I think lending is one of them. Automated lending, I think K Y C A M L also know your customer, anti-money laundering, it would be good. I think enhancing your diligence processes. So you could imagine for lending specifically when you're going through with the mortgage broker or their assistant using this type of generative technology to assess and understand and provide responses. Same thing with contract analysis or diligence rooms when you're doing an acquisition, things like that.
(11:01)
And the list goes on for that. And so there's a lot of really good applications of generative AI in this space. And I think particularly in the automated lending. So going over to the responsible AI Institute, responsible AI institute builds what are called conformity assessments. And so basically these are a series of questions or it's an assessment that aligns to global standards like ISO and provide a level of adherence to the ISO standards. And automated lending is the first certification in the world that you can get that demonstrates that you are aligned to these standards. And so it's actually a place where you can get audited by an external auditor and they can give you an actual certification stamp that your automated employment application or tooling has been built in a responsible manner. And you'll see the same with automated employment in the near future, which is very, very important because New York, the New York automated employment law, and even the regulators have pointed towards these ISO standards and other such standards as a way to know if you're ready to comply with their regulations.
Penny Crosman (12:24):
Well, that's interesting. I didn't know about this process that you offer. What's an example of something that you would look for before certifying say an AI-based lending model? What would give validity to the idea that this is free of bias? This is not going to inadvertently give some people approvals and others not on any criteria other than the basics of basic lending criteria?
Seth Dobrin (12:58):
Yeah, so I think that's a really good question. I think just to be explicit, you will never eliminate biases. There's always going to be biases in every decision we make, especially decisions that are important to the health, wealth or livelihood of a human because you do need a human involved. So you don't want these AI systems making decisions about lending and just that's the answer, especially if it's no, because that Im impacts someone's livelihood and their wealth. And so that's something, and under these AI regulations that are coming up that will actually be protected where it's explicitly said that you can't do that. But now I guess getting to your question a little bit more explicitly, when we think about the responsible AI Institute and what the dimensions are that we measure against, we measure against your data systems operations. So is the data relevant or is there human in the loop, as I just mentioned?
(14:01)
Are there guiding policy documents and strategy at the corporate level, we look for explainability and interpret abil and interpretability. So are you able to communicate how an outcome was reached and are you being notified about this? There's accountability in terms of team training, data quality, fit for purpose, AI systems, consumer protection, so transparency to operators and end users, privacy, protection, bias and fairness as you just brought up. So bias impacts bias training and testing and robustness. So system acceptance test, performance contingency planning. And so it's both at the organizational level. So is the organization ready as well as the AI system level. And an AI system is basically, think of it as a decision in lending. Most banks do not have a single AI model that's helping to drive a lending decision. It's usually many models, AI models. In fact, almost any decision worth solving can't be solved by a single model. And so if you assess each of the individual models for things like explainability or bias or robustness, you're not really assessing the full answer or the full possibility of goodness or badness, if you will. And so you need to actually assess the outcome and the AI system as we call it. So we're assessing automated lending systems. We're not assessing individual automated lending algorithms that live underneath that.
Penny Crosman (15:39):
Another area where I think banks use of AI can be a little bit controversial is in digital marketing, because even though you're not changing the course of someone's life by denying the mortgage or a loan when they need it, you're still making offers with the use of ai you're making offers to some, offering certain products to some people and then not to other people based on an algorithm. What do you think about that? Is that, is it similar in that you can build guardrails in or as you were saying, have more of an infrastructure around it that tries to ensure the privacy and fairness and all the things that you need to do?
Seth Dobrin (16:26):
Yeah, so I think those system assessments that we have for automated lending and automated employment, and there'll be others, like I said, of where they impact health wealth or livelihood of a human for others that are not at that level. What the responsible AI Institute encourages and shows people how to do is to take these assessments and configure them based on their own corporate policies, ethos and governance. And so you would essentially put in parameters for bias and fairness and bias and disparate impact is really what we measure in banking. And you know, would be able to know is this model for marketing meeting our thresholds that we've deci defined or eventually the industry may decide to define thresholds for those. So if you look at protected classes, gender, race, ethnicity, things like religion, things like that, you would put a threshold where you'd say, okay, 45% of women should get this answer and 55% of men, it may not be 50 50, but you'll have some distribution that's acceptable and some that's anything outside that distribution.
(17:40)
So you would measure things like this, and this is a self-assessment or second party assessment if you have your internal audit teams run it. So I think it is, you can solve the problem. Our assessments are designed to do that. And in terms of do I think it's okay, so one of the challenges in the US is that there are no data regulations. And so there really is no requirement for people to opt out of things like this. And so I think even if you solve the whole equity problem, you're presenting to something to someone that they didn't ask for, and I have to proactively go to you and tell you to stop. Whereas in the EU and any large bank is adhering to gdpr, which is the EU data privacy law. So they have that built in. But smaller banks, it's probably not in there, you know, have to actually opt into these things.
(18:39)
So there has to be a question when you sign up for something, are you willing to do this in the us It's optional. You don't have to do that. In fact, most places set it up as you have to opt out. So in that case, I think it's using my data against me, for lack of a better term, to get me to buy something for you that I don't necessarily want or need without my permission. And that could be problematic and give you a good example that I think we can all relate to. Let's say you show up at a theme park and this theme park and you have wristbands or something that you scan in lots of these theme theme parks have it. And the theme parks know what ride you go to, they know what, what rides you, like what rides your kids, things like that.
(19:20)
And you're wandering around the park and all of a sudden you get a text message and that text message says, Hey, your daughter's favorite ride is available on the other side of the park. If you go there now, you can use it without a line or with a minimal line now. Or that me, I would be like, okay, how do you know where I am? How do you know what my kid's favorite ride is and why are you even telling me this in the first place? Whereas alternatively, when I get my band or I walk into the park, it asks me, Hey, would you like us to help you make your park experience better by notifying when your family's favorite rides have no line? I'd be all over that. And all the only difference is you gave the customer a value proposition instead of doing it without ever asking them
Penny Crosman (20:13):
Consent to consent, which in the year in Europe, as you said is so, is pretty strong in here. Like it's fuzzy
Seth Dobrin (20:24):
Out outside of California. It's pretty non-existent.
Penny Crosman (20:26):
Great. Right. Yeah, no, those are great points. And it, it's funny because I think banks have thought about this a lot in terms of their marketing and say using a AI to make personalized recommendations to people like, okay, you've got a mortgage and a car loan, do you want to now have a home improvement loan? Or whatever it is, taking the information you have, take the information they have about you and making a recommendation or giving you an next best action. And I think the assumption is that people expect their bank to know about that, all the products they have, know how much money they have, et cetera. And maybe that's not a safe assumption to assume that people are okay with you poking around their account information.
Seth Dobrin (21:19):
Yeah, I would say people probably don't, most people don't real, aren't informed enough to realize just how much information organizations can glean from data they have about you. And so they're not well informed in what, what's happening for that or what they're giving up just by simply using a system. And so I think the assumption that people understand is false.
Penny Crosman (21:48):
That's a good point. So in society at large, what are your deepest fears or biggest worries about ai? The power of, say, generative AI or any other type of AI
Seth Dobrin (22:04):
And any, I always end it by saying AI can do one of two things for us. It can propagate all the bad decisions that we as humans have made in the past, or it can actually help make the world a better place. And so my biggest fear is that instead of using AI to help minimize things like bias, to help ensure that we're getting broad inclusion, and by inclusion I mean more than corporate speak, conclu inclusion, I mean making sure that countries below the equator are getting access to the same technologies that we have with the same guardrails, whate, whatever they are. Inclusivity also means, especially in the context of AI, that we don't, I impose our morals and our ethos on other regions and geographies and countries because they're not all the same, you know, have different, whether you agree with them or not.
(23:05)
Well, they have different morals and ethos in the Middle East than we do in the US than they do in Europe than exists in Asia. And so I think that's important that we can't impose our ethos. And that's part of in inclusion, inclusion. So not doing that would be a bad thing, I think. And it would further exacerbate the two world problem we have right now, essentially north of the equator and south of the equator. The other thing I'm worried about is, and George Hinton, the New York Times pos positioned him as the godfather of ai. He's probably one of a handful of people that are godfather of ai, so not far off or God people of ai. And so I think there's opportunity for defense organizations to go a little bit off the rails with ai. This whole concept of completely automated warfare scares the crap out of me.
(24:16)
And so I think that's worrisome, I think are adversaries and bad actors using this type of technology to attack us in some way. So for instance, it's hard to keep up with hackers, and today as AI gets more powerful, they're going to get access to this technology. And so we have the opportunity if we do something like pause, a advanced AI development to get behind bad actors where we can no longer protect ourselves from hacking, from disinformation, from misinformation. Those latter two we're already really bad at it would just get worse. And quite honestly, it's going to get worse before it gets better. So those are the main things that worry me.
Penny Crosman (25:02):
Those seem like reasonable worries. And what are your biggest hopes for AI in the future?
Seth Dobrin (25:10):
Yeah, I mean, as I said at the beginning of my fears is AI actually has the opportunity to make the world a better place. If we look at how banks and other organizations are using AI and that whole kind of bucket of technologies like we spoke about before the podcast, when people say AI these days, especially to lay audiences, ai, it's machine learning, it's rules, it's rpa, and so it's a robotic process automation. So it's all those things lumped together and think of it more as it's any automation of tasks. And so almost every organization that's implementing ai, especially again, when it impacts health, wealth or livelihood, understands that there's opportunity to introduce bias. And in fact, a lot of times they are trying to use AI to minimize the bias. So a great example of that is there's a bank, a number of years ago, one of the first automated lending systems that was ever developed.
(26:10)
And they actually put the, we're working on this AI to eliminate bias from their workforce because humans all have, we're all biased whether we admit it or not, in some form or way. And so they built this AI system to try and eliminate the bias or minimize it as much as they could. Now, this was a number of years ago, and so we didn't understand just how good these tools were picking up on patterns, and they removed gender and race from the data that could be used to train the model and even respond the model to make score to score or to return a response. But what happens is a machine learning or AI is really, really good at discriminating A from B. And so it can look at patterns and it can say essentially, it's like a rule if this then then B, it's not that simple, but that's it.
(27:05)
And we talked about redlining earlier. What was happening underneath this model, especially in places like New York City or other major cities, especially in the US, is that certain commu people from certain background live in communities together. So the black and brown community live in individual p indivi. People live in individual communities. Jews like myself, my family grew up in a Jewish community in New York City. You're able to pick up on things just by zip code. And so what ended up happening was they actually started scaling bias because they were making decisions, the model was making decisions and the zip code was influencing the outcome. You saw a similar thing with Amazon and their automated hiring system, a resume screening system. Their model was trained to screen resumes and they screened out women more than they screened out men because their training data had very few women in it. So back to the original question, what I think is the opportunity, I think we have the opportunity to address that as long as we do it responsibly. And we have assessments like we do at the Responsible AI Institute to help us better understand how we're doing and help set appropriate, configure appropriate levels for things that we need to protect. And so I think it has the opportunity to make the world a better, fairer and more inclusive place.
Penny Crosman (28:30):
Excellent. Well, I think that is a great note to end on. So thank you Dr. Dobrin so much for joining us and to all of you, thank you for listening to the American Banker Podcast. I produced this episode with audio production by Kevin Parise. Special thanks this week to Dr. Seth Dobrin at the Responsible AI Institute. Rate us, review us, and subscribe to our content at www.americanbanker.com/subscribe. For American Banker, I'm Penny Crosman and thanks for listening.