Bigger banks are farming out broker-dealer services, as regulatory compliance costs rise.