Banks and nonbanks are heavily competing to build better products for less affluent and underserved consumers. But the hurdles to success are mounting.

Five years after the worst of the financial crisis, many Americans have fled, or been forced out of, the traditional banking system; the Federal Deposit Insurance Corp. in 2011 estimated that one in five households, or about 50 million Americans, are underbanked. This has created many opportunities to sell financial services to poor and underserved consumers, but banks that do so are facing several regulatory challenges and stiffening competition from within and outside the industry.

"The underbanked have gone mainstream," says Jennifer Tescher, head of the nonprofit Center for Financial Services Innovation. "There's just been so much development that involves very mainstream players, and you're continuing to see a boatload of new players from Silicon Valley or otherwise."

Banks, startups, investors and nonprofits are all looking for the next big innovation in financial services, for reasons both altruistic and revenue-driven. For example, at Accion International's Center for Financial Inclusion, the Bill & Melinda Gates Foundation is working with companies including Citigroup (NYSE:C), Visa (NYSE:V), MasterCard (MA) and Western Union (WU) to broaden financial services for — and the potential profits from — underserved people worldwide.

But all of this innovation is happening under the watchful eyes of regulators, who are increasingly scrutinizing the prepaid cards, payday loans, remittances and other products largely marketed to underbanked customers. That scrutiny is dampening some innovation, former regulator Raj Date argues.

"So far you've seen institutions almost in a reflexive defensiveness — it's as though the entire industry decides to get into the fetal position because things are changing. That's not smart," says Date, the former deputy director of the Consumer Financial Protection Bureau and founder of the consumer banking advisory firm Fenway Summer LLC. "Anything that changes the status quo is an opportunity, and someone's going to see that opportunity and drive a truck through it."

Date will be one of the speakers this week at the Underbanked Financial Services Forum, an annual conference co-sponsored by CFSI and American Banker. Industry members there will be debating how best to cater to the underserved, stay ahead of the mounting competition and yet not run afoul of regulators. Here are four questions they should ask.

1) What's ahead in regulation, and how much will it change the ground rules?

Regulators are clamping down on several products that banks and nonbanks have long sold to low-income customers. Payday loans have drawn the most fire recently; in April, the Office of the Comptroller of the Currency and the Federal Deposit Insurance Corp. issued tighter guidelines for the versions that banks call "deposit-advance" loans. Some industry members expect the guidelines to essentially wipe out the products, which are sold by Wells Fargo (WFC), U.S. Bancorp (USB), Regions Financial (RF) and others.

Remittances, prepaid cards and overdraft services on checking accounts have also drawn regulators' attention in recent months, and the consumer banking industry in general is still figuring out how to develop new products without violating Consumer Financial Protection Bureau rules. Most bankers now accept the reality of working under the new regulator, but "there's a long way to go" to really get used to it, Tescher says. "In particular pockets and particular arenas, there's still a lot of lack of clarity that's dampening innovation."

2) What can technology do, and what can't it do, to help the underserved?

Many of the so-called underbanked have ditched the traditional bank branch for the computer screen or, especially, the mobile phone. Smartphones now let Americans do anything from making deposits to wiring money, and banks and nonbanks alike have embraced mobile and online technology as a way to cut costs and improve customer satisfaction.

Technology has also made it much easier for banks to gather data on their customers, and to tailor products to specific groups of people. JPMorgan Chase (JPM), for example, has split its retail customers into segments based on profitability and other measures, and developed the Liquid prepaid card largely to serve its 8.7 million customer households with less than $75,000 in annual income.

"Your ability to gather and harness and use data to make decisions is just dramatically better now. That should have a real impact on the ability to better decision mass-market customers," Date says. "There's a lot to like about this moment in time in terms of gathering and using data."

Both banks and startups are trying to develop mobile technology specifically for underbanked customers, offering alternatives to payday loans, hybrid prepaid bank accounts or apps that encourage people to save more money. But questions remain as to whether many of these variations on established financial services can address customers' needs cheaply, effectively and legally. Technology can solve some problems, but it cannot remedy the underlying poverty and unemployment that has forced many people out of the banking system. For companies trying to make a profit from serving these customers without preying on them or appearing to do so, innovation has its risks as well as its rewards.

3) How much should banks worry about outside competition and disintermediation?

Banks may once have had a near-monopoly over how people store, spend and save their money, but they are facing the threat of disintermediation in several markets. When trying to sell products to financially underserved customers, the banking industry needs to worry about competition beyond storefront payday lenders and remittance providers.

"Disintermediation has been a long time in the making, first with lending with credit, then with payments and basic transactions, and now with savings. I think they're feeling the heat on all fronts," Tescher says.

Some of the most successful recent innovations have come from big traditional financial services companies: JPMorgan and American Express (AXP) have introduced prepaid cards that essentially replace full-service bank accounts. But Amex is working with industry bête noire Wal-Mart (WMT), the retailer that has long provided financial services to many low-income and underserved Americans without having a banking license. And many other startups and tech companies are eyeing banks' turf, with companies from Google and PayPal to Simple and Moven now competing to process payments and take deposits online.

4) What's the next big product?

Prepaid cards have had the biggest success in helping the underbanked break through to the mainstream, thanks largely to JPMorgan's Liquid and the Bluebird card from American Express and Wal-Mart.

Short-term credit is another area where many companies are trying to innovate, with fewer advances so far. The pressure over deposit advances and payday loans, as well as regulations of overdraft fees, have opened what Tescher and Date call an opportunity for a better, more customer-friendly version of a short-term loan. But that may be a nearly impossible task.

KeyCorp (KEY), for example, has offered a variation that appears to address some consumer advocates' complaints about payday loans. But while KeyCorp described the product as profitable, it has not discussed specifics and thus scale or sustainability.

Tescher says she has seen little progress over the past year in building a better payday loan: "I don't think anyone has yet to find what good is," she says.