State Street's Big (Data) Dig

In 2009, when he was State Street Corp.'s president and chief operating officer, Jay Hooley was trying to help a major institutional client with a significant blind spot. The customer wanted to calculate its exposure to a particular market. Getting the numbers turned into a painful exercise as State Street's middle- and front-office staffers reconciled disparate data sets housed in different client systems and in nine of State Street's 29 global locations.

It took "several hours to get that answer," says Hooley.

"I knew we were on common accounting, cash and security systems," he says, but having such dispersed information meant that Boston-based State Street couldn't fulfill one of the most pressing needs to come out of the financial crisis-namely, "that the people in decision support in risk management needed that answer instantaneously," Hooley says.

"As I think about that moment, that's what we're solving for."

Like the famed road construction project that reshaped Boston's central freeway artery, State Street is in the midst of a Big Dig of its own-a project that remaps the routes to a custom-built data warehouse. It is along these new roadways that the firm will be able to transport real-time, properietary market analysis to clients, who in turn will be invited to shift their accounting, security and database systems onto State Street's private cloud.

It's a massively expensive initiative-estimated at $100 million a year for three years-but Hooley, now the firm's chairman and CEO, projects the bank eventually will reap $575 million to $625 million in cost savings by automating tasks such as software development. (State Street, which has more than $23 trillion in custodial assets under management, writes nearly all of its own software for asset management, trading and transaction services.)

The private cloud also promises a new revenue stream from clients that choose to park their data and back-office applications on the network, culling them as needed on an on-demand basis.

The project is still a year away from completion, but it's already being closely watched in the industry, not only because of State Street's influence in financial services as a major custodian bank, but because of the general direction it may provide as to the way big data can be turned into a revenue generator.

Even as more institutions buy into the promise that big data holds, there are burgeoning questions about how to attack its complexity, and about what can be mined, packaged and sold from all the information that will be streaming into back offices.

"The pure volume of computing power that we use and volume of data we were storing, it was growing faster than business was growing. Faster than revenue, faster than productivity gains," says Chris Perretta, State Street's chief information officer, who was one of the early proponents of the project alongside the firm's chief architect, Kevin Sullivan, and its recently retired chief innovation officer, Madge Meyer. "That puts pressure on our discretionary spending [for technology], which for us isn't so discretionary."

Getting an idea of what big data looks like requires the imagination of the astronomer or geologist who can better fathom what to most people would be unimaginable segments of distance or time. Big data is about information stores of an inconceivably large size. Included in the data is all of the "metadata" that tags, catalogues and references the information itself.

According to IBM, each day another 2.5 quintillion bytes of data are created, meaning that more than 90 percent of all the data that's ever been compiled by humanity (probably counting stone tablets and cave drawings) has been created just within the past two years.