One of the ostensible selling points of blockchain technology is its potential to bring greater transparency to financial markets. But this feature has turned out to be a bug for the institutions that would use the technology.

In the original bitcoin blockchain, transactions are recorded on a public ledger for the whole world to see but users are pseudonymous, identified only by alphanumeric addresses that look like a cat walked on the keyboard. In the private, or "permissioned," blockchains being developed for the industry, only known, trusted entities can participate. This may alleviate concerns about bad actors using the system, but it doesn't address worries about tipping one's hand to the competition if everyone involved can see who did what.

"Confidentiality is the key issue holding things back — that's part of the sticking point for the banks," said Keith Horowitz, an analyst at Citigroup.

So the major startups in this budding field are trying to restore privacy in distributed ledgers, by allowing users to encrypt sensitive data stored on them or to leave such information off the chains, or some combination. The leave-it-out strategy addresses another issue with the blockchain model: storing large volumes of transaction data can be costly for institutions executing hundreds of thousands of trades each day.

[Get up to speed on distributed ledgers, cryptocurrencies and the bleeding edge of fintech at American Banker's third annual Blockchains + Digital Currencies conference July 28 in New York. Click here to read the agenda and register.]

A potential drawback of both approaches is they could undermine one benefit of blockchains, even if they preserve other advantages (such as eliminating redundant data entry and removing single points of failure).

"If regulated financial institutions implement the technology in a compartmentalized way that shields regulators' mission-critical information, then regulators will have missed the once-in-a-lifetime opportunity to gain necessary tools for keeping the system safe and sound," wrote Caitlin Long, a former Morgan Stanley executive and self-described blockchain evangelist, in a recent blog post.

How blockchain vendors address the interrelated questions of privacy and storage has become a point of differentiation in this young market. Some say sensitive information should never be published; some say all data should be published, even if some of it must be concealed.

Digital Asset Holdings, the outfit run by former JPMorgan Chase executive Blythe Masters, falls squarely in the former camp.

"No private contractual data should be stored on a distributed ledger, encrypted or otherwise," Masters said by email. "Shared ledgers should contain the bare minimum information, interpretable only by those with a need and right to know, to permit notification, synchronization and confirmation."

Symbiont, a developer of software for self-executing smart contracts, takes pretty much the opposite approach.

"We've been utilizing very sophisticated cryptographic techniques to execute on our thesis that everything — and I mean everything — must be on ledger," said Mark Smith, Symbiont's chief executive.

His competitors have "punted on this concept that says 'we're going to take things off the ledger in order to provide for some type of mythical confidentiality,'" Smith said. "When you do that you remove a large portion of the value of the distributed ledger and you remove the concept that all entities are sharing information."

When someone chooses to encrypt information on a blockchain, it usually looks like a time stamp of the transaction, and not much more. For example, one record might show that a certain participant on the network added something to the ledger on a given day and time, with a link. The wider network would not be able to view that information, because it's encrypted. But the party who uploaded it could give a private key to someone else (say, a regulator), who would use it to decrypt and view the information.

To encrypt or not depends on whether data is considered sensitive, and that's difficult to determine when most blockchain startups and the institutions studying their technology have yet to decide the best applications and use cases for it.

"This is an emerging technology with a huge number of potential applications, and the type and volume of data stored on any given distributed or shared ledger would be dependent on its purpose and the required level of regulatory oversight," said Charley Cooper, a managing director at R3 CEV, the bank consortium. "Ultimately this technology will dramatically improve transparency in financial markets." But R3 is also designing for confidentiality.

In September the firm hired IBM's former executive architect of banking innovation, Richard Gendal Brown, who has since been building a distributed ledger platform called Corda to record and manage important, critical information between regulated financial institutions. In R3's view, that information is financial agreements.

Corda has "no unnecessary global sharing of data: only those parties with a legitimate need to know can see the data within an agreement," chief technology officer Richard Gendal Brown said in a blog post introducing the platform. "We reject the notion that all data should be copied to all participants, even if it is encrypted."

The Hyperledger Project, an initiative to advance blockchain technology for recording and verifying transactions across multiple industries, is building technology for more general purposes, where all details of a trade are shared with all members of the network. Brian Behlendorf, the project's executive director, said Hyperledger doesn't really take a perspective on the right way to use a ledger, as it will differ from application to application, but pointed to data storage volume as one motivation for keeping information off the ledger.

"It costs a lot to store a lot of data on a blockchain," Behlendorf said.

In the worst case scenario, off-chain data could disappear entirely. URLs fade out, people lose their domain names. It's hard to preserve data for more than a decade, Behlendorf said, and that's another reason he focuses on solutions that write unencrypted information into the chain. Sharing keys to decrypt text is not only hard to manage; it means that when someone shares the key, the information is no longer really private.

"When it comes to using ledgers as a system of record for tracking assets, encrypted messages don't make that possible," Behlendorf said. Storing encrypted information on the blockchain is useful only to the extent of showing a timestamp and proving information was transferred at that time.

Further, "writing encrypted text is not a good idea for a blockchain in which sharing that data with regulators is part of what you want to do — which is a reason regulators should spend the time and money getting to understand the technology and working with the industry to co-create a system," Behlendorf said.

That need for dialogue and transparency should be a two-way street. As much as the industry needs to design technology accessible by regulators, regulators need to work with financial institutions to access information in a way that doesn't compromise the integrity and security of the blockchain system.

"It certainly has been a very tense relationship [between regulators and the financial industry] for the last 10 years," Behlendorf said. "But that's perhaps why blockchains are a chance to reboot that relationship to do something more constructive."

And although the industry intuitively understands regulators will ultimately have access to the information they need, regulators still need to allow blockchains to develop and work effectively, said Kevin Petrasic, a partner at White & Case and head of the law firm's global financial institutions advisory practice.

"The tremendous challenge with blockchain is the ease with which you can potentially protect and anonymize information so you can understand that the transaction occurred but you can't always see the details," Petrasic said. "I think the challenge for the industry is the regulators still have the right to access that information and they will ask for — and under the law, be provided — the ability to access that information to make sure an institution isn't doing anything they're not supposed to be doing."

Commissioner Chris Giancarlo of the Commodity Futures Trading Commission said he doesn't personally have a view on the open or closed evolution of blockchain. As a regulating body, the CFTC would seek to be a participant on the blockchain with a node that would allow it to see all transactions whether they're closed or open. (Incidentally, most blockchain companies do allow regulators to sit in on observer nodes or are communicating with them directly through the development process.)

"We should absolutely be participating as an observer in a lot of the use case testing going on right now," Giancarlo said. "We need to up our game in terms of our technological understanding of the evolution of the technology; but we also want to understand how the technology can deliver on the promise of counterparty credit exposure transparency — the lack of which underlined the financial crisis and Title VII sought to deliver."

Title VII of the Dodd-Frank Act sought to provide regulators visibility into that counterparty credit exposure through swap data repositories that would collect information so in the event of a future crisis the regulators could see those exposures. Today, Giancarlo said, there still isn't a complete picture of the global swap transaction marketplace. He has been highly vocal this year about blockchains' promise for real-time, accurate reporting of information.

"It could only serve our purposes if we had the full transparency that Congress gave us the authority to require from Title VII," he said. "Whether it's open or closed to other market participants is for the marketplace to decide. Whatever evolves, we will need access to both."

Giancarlo said U.S. regulators should do more to encourage fintech innovation, citing those in the U.K., Australia and Singapore as being far ahead on the blockchain learning curve and therefore being better equipped to provide a welcome environment for technology innovators. The government should not dictate the technological evolution, he added, but should not hold the industry back as its global peers move forward.

It's likely to be a delicate balancing act.

"That's what the tension really is," Petrasic said. "How do regulators integrate with financial institutions to get the information they need in a way that doesn't negate the value of the blockchain?"