To be great at customer experience, incumbent financial institutions have to navigate the "paradox of consistency." On the one hand, driving out the inconsistency that creates negative experiences is one of today's biggest priorities. On the other hand, great experiences in the future will be all about inconsistency, tailored to individual needs and preferences — requiring very different skills, technology and processes.

When financial institutions look at the average experience they provide and compare it with the average experience of their competitors, there do not tend to be big differences. But when providers look at the distribution of experiences within their own institutions, there is usually a wide range.

For example, one provider wanted to look at the impact of improving experience on selling future products. They used a 1-10 scale to gather feedback on the experience. Going from "1" to "5" nearly doubled their future conversion rates. Going from "5" to "8" did not make much of a difference. And they saw that when they really hit it out of the park with a "9" or "10," there was another big lift.

The payback on improving customer experience in financial services is often this sort of s-curve. In particular, much of the opportunity sits in finding the bad experiences that hit some customers and not others, and weeding them out. Making things consistently good is itself a kind of greatness.

But that isn't enough. Here's why in personal terms: My mom and I are different people. Today, we go to the same bank and use very similar products. We use the same mobile app and website and get the same highly generic advice. But my mom is a grandmother near retirement and I'm in the middle of my career, with a young daughter. Treating us the same does not make much sense. We're in such different places in our lives and have very different preferences for how we do, well, pretty much everything.

Newer players have figured this out. For example, Ellevest provides investment services specifically targeting younger professional women. And SoFi focuses on recent graduates of high-end colleges and graduate schools, allowing them to offer student loans, mortgages and wealth management products wrapped in an experience and with advice tailored to their customers' specific needs.

If a client of SoFi is looking for a new job that will value that expensive degree, SoFi has a service to help with that. SoFi has even thrown parties to help their customers find dates with people of similar educational backgrounds. (That takes "know your customer" to a whole new level.)

Some of the best institutions are navigating the paradox of consistency by treating today's inconsistency as a gift. For example, there are key elements of the mortgage origination journey where two customers may view the same experience completely differently. Some customers want the processor to explain every step in detail. Others prefer a much briefer and to-the-point approach.

For providers, this kind of inconsistency typically occurs accidentally. But once it is clear it works for some customers and not others, the opportunity lies in making accidental inconsistency intentional. Indeed, more and more companies, both in financial services and beyond, are using the preferences identified through customer feedback to route service calls to agents with a particular interaction style.

The next step is not just better matching of agents to customers, but rather to design distinct journeys for different customer segments. Done right, incumbents have two advantages over newer technology-enabled players.

First, incumbents can offer differentiated experiences while taking advantage of their scale, such as in their common product platform, back-office and compliance capabilities.

Second, unlike the tech company that has to guess at the start what works for different customer segments, incumbents can mine all of the natural experiments that lie in today's inconsistency to figure out what works for whom.

And so this is the paradox of consistency. How do providers balance what's required to create consistently high-quality experiences with the creativity and complexity associated with meeting diverse needs and preferences? The answer affects organizational structure, technology, compliance and culture, among other issues. The best way through starts with seeing today's inconsistency as an opportunity as much as a challenge.

Robert Schiff is a vice president and general manager at Medallia, the global SaaS-based customer experience technology company. The views expressed are his own.