A flurry of new mortgage rules are overwhelming home lenders — some they feel are too strict and others too ill-defined to decipher.