Lightning Talk: AI with Humans in the Loop: A Pragmatist's Guide to Results

Introduction by Bailey Reutzel, Strategy and Content, American Banker Live Media

Building an AI-native product for SMB underwriting taught us a simple truth: trust comes from collaboration between analysts and AI systems. This talk discusses practical steps to getting reliable gains. The secret: AI starts → analysts correct → AI learns—and repeats. This keeps analysts in control, writes back to your system of record, and compounds trust as correction rates fall. The result is more throughput, stronger auditability, and better decisions.

Transcription: 
Transcripts are generated using a combination of speech recognition software and human transcribers, and may contain errors. Please check the corresponding audio for the authoritative record. 

Matt Arderne (00:08):
Hi, that's overstating it. To Brian's point, you just spoke, we're very much back-office automation. We're not getting in front of customers, so just to diffuse that mental image. AI with humans in the loop—this is a quick lightning talk, something we learned: The one thing we got wrong about automation. My name's Matt Arderne. I'm the co-founder of CDev and we're building AI for SMB credit underwriting. I think Chris Bokman said yesterday there's speed and then there's a personal relationship, and we're really focusing on how we accelerate the underwriting process so that you can put more time into building relationships and a better banking experience. So this is the talk. It's relatively technical. There are aspects that are recent learnings that I've tried to simplify into something that is interesting and memorable, but please do keep in mind questions. We'll keep some time at the end for probably a fair bit of time for questions.

(01:15):
And yeah, so this is what we learned. The impossible task—this is where I started my career in financial services, working on automating financial spreads for complex multi-entity SMBs for a non-bank lender. It was a long-running challenge for me to try to automate this task working in a number of different approaches. What really made this task a challenge to automate was that there was a huge amount of nuance and discretion applied by the credit underwriters when they were underwriting these multi-entity SMBs. There was a huge variety in the formats. The data we were looking at was complex management financials. It wasn't standardized in any way. We were looking at it at a monthly level, so it was a huge backlog for the credit teams to get through all of these financial data sets to input into their credit spreading models. This felt like something that technology wasn't able to solve.

(02:20):
We spent a lot of time trying to figure out what we could do about it and it boiled down to flexibility. Very recently there was a big change in the flexibility of technology. We had a breakthrough: before we were losing 35 hours per analyst per month on this task, and after we were doing it in two hours. It was the same people at the same level of quality that they were comfortable with. It was human review of every single financial field, which really is essential. Getting to this point was kind of counterintuitive, not what we expected, and speaks to the breakthrough that we experienced and the title of the talk: human in the loop. That's the clue. This is a question we got asked a lot through building this product and through exploring AI for financial services and SMB credit underwriting: how accurate is your AI?

(03:19):
Is it 95%, 99, 99.19%? You kind of keep chasing down this accuracy question. What we discovered is that this is quite often the wrong question to ask. The reason is because it's point in time; it doesn't speak to a change. It's a static view of what's possible. This was a learning for us. The right question is: how fast can your AI learn? Human in the loop was the unlock. To speak to this a little bit more clearly, if the AI is learning fast—if you're able to get this capability to learn quickly and change rapidly and improve through each cycle—then whatever your current accuracy, you can be quite comfortable that it's going to continue improving while your AI is learning. Human in the loop is really the unlock there. Before I get into it, because "AI learning" speaks a little bit to AI sentience, that is really not the case.

(04:28):
It's more just a case of: we're doing a simple task, are you able to increase the accuracy by specifying that task in a better way? Human in the loop is the unlock where if it's badly specified, it's done badly. If it's well specified, then the task can really be effectively done. So this is the loop. Just very practically, this is what we spent a lot of time refining where an AI generates a draft financial spread. This is extracting the data out of the documents, out of these mixed format management financials and tax statements and their accounts receivable, and giving that to the AI to generate a first draft. The next steps are the most important: the analyst reviews and corrects. Reviewing is one thing—we've all put something into ChatGPT and asked to improve it, but what we've found to be extremely important is that the analyst reviews and corrects those changes.

(05:38):
So not only giving a review but making the corrections in the place that they need to be corrected, and those corrections are tracked. Then the AI can take that and improve the prompt or the task specification—the input guidance for how you want the financial spread. That improvement on the task specification based on corrections from an analyst is basically the mechanism by which the AI gets better on the next path. It's a relatively simple loop. What we've discovered is that looking for these types of feedback loops is everything. They give you a huge amount of value with quite a simple process, but without it, you don't benefit from the corrections. This changes things quite dramatically. The old model was wait until the AI is accurate enough to deploy and you are stuck with what is "accurate enough" and what you are willing to tolerate.

(06:39):
The new model is you deploy where it's currently working. You can do it in parallel; deploy it at 80% and let the analysts correct the 20%. That's essential. It's not that 80% is good enough—you need 100%, and that's where analysts correcting the balance is extremely important. The result of that is you're learning from day one. But to emphasize, the correction step is a requirement; it is essential and that requires quite a lot to get right. What you lose—what stops—is the manual copy and paste, which is a huge time sink. In our process improvements, that's where we see the biggest gains: reducing the copy and paste into an existing system that can't be manually scraped. You lose the need to read bank statements because they go straight in and get considered, and reconciling data formats is obviously a real pain.

(07:41):
What stays is what's important: understanding the business. Everyone has spoken about that. What we see as a byproduct is that the analysts build a deep understanding of the business. They manage the relationships better and they are better able to spot risk and understand the dynamics. What also stays are the trusted risk structures. This is really important: the way we've designed this, nothing has to change other than the very specific step of copying and pasting into a system. Everything else stays much the same and you have the same trusted team. Most importantly, it's that team making the final call. An interesting analogy is to think like a head chef. The head chef tastes everything before it leaves the kitchen.

(08:33):
The senior analyst is the head chef. They're ultimately their reputation; they're there to do that job, and the AI is just prep cooks. They're doing the chopping and dicing—all those things that are undifferentiated—but the senior analyst is really ultimately responsible. That's human in the loop with the human responsible and accountable. A quick point on the surprise: the breakthrough was with the hardest problem we were facing. The reason this was a surprise is because we had a number of problems and we didn't expect that this would be the place to have a breakthrough. It felt like there was too much of a human in the loop required to achieve this solution, but because of the human in the loop, we actually got a really satisfying and effective increase in speed for credit underwriting with a lot of safety wrapped into that with the human chef.

(09:31):
The thought is: is AI accurate enough? I think that's something that we will start thinking less about and start thinking more about: "Am I creating feedback loops?" because feedback loops change the AI accuracy very quickly. This is how trust compounds; how analysts start to understand the system they're working with is through many cycles. They build an intuition for where the AI needs a more careful review, and that trust compounds through the cycles. A suggestion, if you were interested in trying this, is to pick one tedious task that happens repeatedly and takes more than 30 minutes that your team hates doing. Run it in parallel, give the AI the first pass, have someone review it, and then track what gets corrected over a few passes.

(10:38):
That's the feedback loop. Important checks we have built our product around at CDev are: could my senior people spot errors in minutes? If it's taking them long to spot the errors, then it's not really being that effective. They need to be drawn to the errors and be able to move across a lot of data very quickly. Could I track correction rates? If I run 20 cycles, what percentage of those cycles have what percentage of error rates? And then, can the AI act on the corrections? That's really the secret sauce: if the AI can act on those corrections automatically and the next pass is better because of it, that's a powerful compounding improvement system.

(11:37):
The final thought was mentioned yesterday and it's becoming increasingly obvious: customer expectations have already changed. This is really interesting in the SMB space where they're rapid adopters of AI in a way that is exceptional. Small businesses are able to adopt things that work; AI can do a lot of work for them and they're able to adopt operational technology around AI easily. Their expectations have gone up dramatically and the question is: how do we meet those expectations while managing risk? From our perspective on the speed side, if we accelerate the process and give them a snappy answer while spending more time understanding their needs, then that's a powerful outcome. That's why we are leaning on this human in the loop as being an answer for us. Thank you very much, Matt Arderne, CDev co-founder.

(12:43):
I would love to hear from you if this resonates with your underwriting processes. Happy to chat about feedback loops, complex spreading, and what we're learning. It's a learning process and we're happy to share where we see the opportunities. We have time for questions. Thank you.

Audience Member One (13:08):
So give me an example of how it would work when you say that it provides the underwriting. What would we need to submit, enter, or scan in order for it to accomplish that?

Matt Arderne (13:22):
Good question. The types of things we're thinking about are a set of management accounts, financial statements, accounts receivable, and accounts payable—the typical things you would use to get a sense of the business's cash flows. Typically, that's a hugely manual process to copy and paste and consolidate into a credit decisioning system. We've built something that can do that copy and paste step really easily without losing the human essence of understanding the business as they do it. That process is really valuable for the credit underwriter; they're learning a lot about the business. We want that to happen fast without them losing touch with what they're looking at. We don't want it to be a black box or something they are unfamiliar with at the end of it. Great question. Thanks.

Audience Member Two (14:28):
Hey, Matt, great talk on AI. It's great to hear you covering this basis, but what I was trying to think of is what other areas can AI be used for when we're thinking about underwriting ops and different areas of SME lending?

Matt Arderne (14:43):
Nice. Thank you. Great question. One of the things that we see quite a lot of value in is just taking a different view on the data that you're collecting. Quite often there are opportunities to look at what you have already with a different lens. To the line chef example, you kind of have infinite capacity to allocate tasks. There are types of things that would be uneconomical to have historically done because there wouldn't necessarily be a return on them. A good example is digging into your base and seeing if there's cross-sell opportunities. That is a really good opportunity because it's quite time-consuming to identify the signals. You maybe wouldn't be doing that type of thing manually, but if you have this broad resource base that you can allocate those tasks to, it can be a really effective outcome.

Audience Member Three (15:57):
Question over there. My question is, could you add components like risk rating and a full write-up package to the spreading component? Could you add risk grading and then ultimately the entire underwriting package to this?

Matt Arderne (16:15):
Yes, exactly. We heard "crawl, walk, run" yesterday, and we're starting with the biggest time sync—the most time-consuming part of the process. Definitely, our intermediate next steps are: how do we build this into a full end-to-end? As the product gets more complicated, it gets harder to keep the human in the loop. That's why we're doing the steady process of making sure that each step considers the human in the loop rather than just being entirely automated. Those capabilities are there, and I think we just need to make sure that we keep the product evolving in a way that keeps the analyst in control and understanding what's going on, because that's what we think is the most important part. Sure. Thank you.

(17:17):
That's an excellent question. How can AI help us collect the information? That's actually where we started—a more front-office approach where we have the AI sending emails and following up when there's missing data. "DockChase" is what we call it. It's more sensitive because you need to be comfortable with AI doing a lot of the legwork for you. I think that's where we'll end. One of our work streams is how to make that reliable and safe. Why we've seen the breakthroughs on the more back-office and operational stuff is because it's much less sensitive and you can keep your experiments running in a more contained way. But I would say the two ends of our product are collecting, then analyzing, and then following up. There's a nice feedback loop there too. Thanks.

Audience Member Four (18:20):
I'm going to ask the last question then. This is something I wanted to ask on the panel yesterday and then I didn't have time. What's going to happen to the interns and low-level employees that need experience doing these more manual tasks to be able to level up in an organization if we have AI that is largely doing those processes?

Matt Arderne (18:43):
It's a good question. Interestingly, what we see is that the younger analysts are often the most intuitive around where the AI struggles. I think it's giving them the opportunity to work with these tools as native users. We've all come to accept mobile internet; it's a similar thing where they take to it much more easily. The challenge is giving them the right ecosystem and infrastructure to support them and build understanding. It's a good question and the one I have the least sense for how we do well, but I think it's probably the most important thing: nurturing those skills alongside this new support agent. The great thing is that they never have to do the extremely tedious tasks. A lot of us had to do things that are now redundant, like memorizing massive lists. You wouldn't need to do that now, but there's still a lot of work for them to learn around problem-solving. Good question. I don't know the full answer.