A handful of startup core systems providers are promising cutting-edge technology and an innovative approach to banking. But can they make a dent in a market dominated by a few entrenched firms?

The latest player to enter the fray is Thought Machine. The London company came out of stealth mode last week to announce Vault OS, a cloud-based core banking software that uses blockchain-like technology and machine learning capabilities to offer banks greater efficiency and digital capabilities.

A newcomer to the core banking market like Thought Machine in some ways faces less of a barrier to entry: New technology like cloud computing means there's a lot less infrastructure to invest in, such as mainframes. Also, observers say there is a growing restlessness with existing solutions. Still, dislodging banks from their longstanding relationship with the incumbent core vendors seems like a gargantuan task.

 "The market is longing for new entrants to shake things up, so there's plenty of reasons to hope these new firms make it," said Brad Smith, a managing director with Cornerstone Advisors. "But they're competing against vendors with 40 years of product-development lead time. On the one hand, that means they can make the case [that existing vendors] are archaic, but on the other hand it means [existing vendors] still have got 40 years of listening to client feedback and conducting user groups and really understanding their customers."

The problem with the status quo is that it is holding banks back from investing in new technology, said Paul Taylor, founder and chief executive of Thought Machine. Banks spend so much time maintaining and patchwork upgrading their current systems that it makes serving customers in the digital world untenable.

Banks' core systems "are typically built on decades-old technology," Taylor said. "We're coming from an agile development world."

In fact, Taylor and his team, made up of several former Google engineers, did not set out to get into the core banking space. They originally focused on developing products to improve digital user experience, but kept finding antiquated core systems that wouldn't support new technology.

"We'd show banks what we were working on, but we'd continually hit the same problem," said Taylor, who joined Google in 2010 after it bought Phonetic Arts, a speech synthesis technology company he co-founded. "They'd say, 'Your idea is cool, but there's no way it will work with our current core banking engine.' "

Thought Machine is initially targeting U.K. and European banks but eventually wants to sell technology to banks all over the world, including the U.S.

Taylor said the company understands it will be difficult to break into the market. As such, it is starting small, focusing on retail banks, especially small-to-midsize ones.

"We're aware that there are incumbents happy with the way the market is, but that's a challenge we need to face," Taylor said.

An American company also staring down that challenge is Nymbus in Miami Beach, Fla. Though many bank executives are hesitant to undertake core conversions — especially with newer providers — enough are realizing the benefit of a modern core that breaking in is a bit easier, said Nymbus' founder, Alexander Lopatine.

"Of course, there are many bank CEOs that are more conservative when it comes to replacing a core, especially those that may be a couple of years before retirement and don't want the hassle," Lopatine said. "At the same time, there are enough that understand a modern bank must be digital, and a bank can't be digital with a core built in the last century."

Nymbus, which launched 18 months ago, offers a cloud-based platform that uses application programming interfaces to connect with partners to provide things like bill payment and personal financial management, Lopatine said.

"What we're offering is bank-in-a-box," he said.

The company is set to unveil its first three bank customers later this year, Lopatine said. It has specifically been targeting banks with assets of less than $10 billion since they tend to be more nimble, Lopatine added.

Other new entrants in the core banking space include Corezoid, in Redwood City, Calif., which was spun out of a Ukrainian Bank. Corezoid, which formally launched in May, is courting U.S. banks with a cloud-based platform that relies on APIs to cut out the hardcoding typically needed to create banking services.

Targeting small-to-midsize retail banks, as Nymbus and Thought Machine are doing, is a smart move, Smith said.

"They tend to be easier conversions, and less complex," he said.

Although industry observers believe the core vendor market could use more competition, conversations often return to how hard it will be for them to gain traction. Besides the incumbency issues, core conversions are massive undertakings that can be expensive, disruptive and risky.

"There's a lot of bank CEOs dissatisfied with their core vendor, but, at the end of the day, they are just unwilling to pull the trigger to go through what will be a time-consuming and costly process," Smith said.

Further, he said, there are only a small number of banks each year whose contracts are up. It also takes about 18 months from when a bank makes a decision to actually implementing a new core system, he said.

"This is just not an easy business to scale," Smith said. "It's not Uber."

To be sure, the well-established core vendors are not resting on their laurels. Several core providers have innovation labs around the world and are looking for ways to partner with fintech companies to improve the industry's ability to provide digital services. The U.K.-based Misys, a longtime financial technology vendor, announced in April that it would move its full product suite — including core systems — to the cloud. Also, as Smith noted, the core providers often buy core banking providers, such as Fiserv's 2013 acquisition of Open Solutions. Fiserv, FIS and Jack Henry did not immediately respond to inquiries for this story.

Whether it comes from new providers or legacy ones, banks will start to realize it costs less to replace a core system than to try to compete in a digital world with legacy technology, said Mark Schwanhausser, director of omnichannel financial services at Javelin Strategy & Research.

"If you want to win long term, and be a truly digital bank creating the highly personalized service that leads to longstanding customer relationships, you need to have better, real-time systems," he said.