Alabama's BancTrust Financial has abandoned a bid to raise private equity funds. As bank-acquirers circle, a deal could say a lot about the state of M&A.