Concerns are growing that the government's tougher stance on late fees could hit Capital One Financial Corp.'s bottom line harder than the other credit card companies.

The credit card issuer turned bank reaps more income from late fees than top competitors, as its relatively large customer base of less-creditworthy borrowers tend to fall behind on payments. And concerns are mounting on Wall Street that the Federal Reserve's proposal aimed at keeping late fees from exceeding a customer's minimum payment could dent this income stream.

Goldman Sachs downgraded its Capital One investment rating to "neutral" from "buy" on Friday, one day after SunTrust Robinson Humphrey did the same, with both firms mentioning possible restrictions on late fees.

A Capital One spokeswoman declined to comment on the downgrades.

Subscribe Now

Access to authoritative analysis and perspective and our data-driven report series.

14-Day Free Trial

No credit card required. Complete access to articles, breaking news and industry data.