With success in many markets riding on innovative technology, banks are turning to the tricky game of venture investing.