These are challenging times to be a bank director. As banks grapple with changing customer demands and contend with shrinking margins, reduced fee income and higher regulatory costs, directors are being asked to do more than ever to help their banks navigate this new landscape.

But in our view, directors can get so preoccupied with dealing with these challenges that they can become complacent in one key area: CEO and leadership succession. And the brutal reality for financials is that at a time when the cost of complacency is rising, the size of the fix is widening.

Why is complacency costly? First, when unprepared for CEO succession, boards are more likely to hire a CEO from the outside, but studies show that the return on investment of this option isn't favorable to shareholders over the long term. According to a 2013 study by Booz & Co., the median to shareholders by insider CEOs over the past dozen years was 3.9%, compared with 0.65% by outsiders.

Second, transition costs are substantial — Greg Steinhafel, the recently departed CEO of Target, will reportedly receive $55 million on his way out.

Third, it creates unnecessary turnover. When CEO transitions occur, top executives who wanted the job frequently leave the company. While some level of turnover is expected and natural, failed successions leave unnecessary turnover in their wake.

Executing a successful CEO succession is neither glamorous nor rocket science. It just takes time and hard work by the incumbent CEO, the board and the internal candidates who are developing the skills required to take the helm. Preparation involves proactive movement of executives into key roles, real-time knowledge of the external market and high-intensity development of potential successors. Here are five keys to success:

1. Know what you need tomorrow, not just what's available now. Don't just ask, "Who's ready to succeed the incumbent?" Also ask, "What does the company need in future CEOs, and do our first-, second- and third-generation internal fit that profile?" Make moves that position leaders to learn the skills necessary in the future. In 2013, BlackRock promoted Charles Hallack, then 49, to co-president and Rob Goldstein, then 40, to chief operating officer. The moves put younger executives into key positions after half of the co-founders left or retired from active roles.

CEO James Gorman added several executives to the operating committee to maintain a balance of managers from different units who are capable of rotating across businesses in the future.

2. Experience matters. Don't just ask, "How long until they are ready?" Also ask, "What specific experiences are internal candidates being given to prepare them for a job they've never done before?" They also need exposure to tough constituents and will benefit from real-world experience presenting to analysts, investors, the board, unions and employees.

John Stumpf succeeded Richard Kovacevich, first as CEO and then as chairman of Wells Fargo (WFC). Stumpf was a highly regarded internal candidate who had previously held a variety of senior positions inside Wells. Similarly at U.S. Bancorp (USB), Jerry Grundhofer passed the torch to Richard Davis through an established succession plan. Davis was a key member of the team that created U.S. Bancorp through a series of acquisitions.

3. Know the external market. Don't just ask, "Which executive search firm would we use if we went outside?" Also ask, "Which executives in other companies/industries fit best with our future CEO profile, and how likely would they be to come on board?" Convergence in many industries means that the board should be tracking executives in other industries, not just their own. Many banks are under pressure to innovate, and they may need dip into other talent pools in order to find the right person.

4. Firsthand knowledge beats talent reports every time. Boards shouldn't just ask for names of internal candidates; they should get to know them formally and informally, beyond presentations. When considering internal candidates, Directors should have personal, direct knowledge of the heirs apparent. Ajay Banga, the CEO of MasterCard, sets up formal and informal interactions between the company's top talent and the Board on a regular basis.

5. Don't fear the horse race. Former General Electric CEO Jack Welch is perhaps the best known proponent of setting up the "horse race," but he did so with an extraordinary amount of interaction between the candidates and the board. It was a process that took six and a half years and is still widely regarded as one of the most comprehensive and transparent succession plans in recent memory.

The opposite was true when Chuck Prince succeeded Sandy Weill as CEO of Citigroup (NYSE: C) in 2003 and as chairman in 2006. Prince resigned in November 2007 at the onset of the financial crisis then his hastily appointed successor, Vikram Pandit, abruptly resigned in 2012, paving the way for current chief Michael Corbat. Weill has said that, in hindsight, he wishes he had fostered competition among Citi's top managers for the CEO post rather than handing the job to Prince.

While many factors go into share price appreciation, it is interesting to note that three of the largest banks faced succession issues before and during the crisis. Wells Fargo and U.S. Bancorp handled this challenge well. As such, they preserved their strategic focus and have significantly outperformed Citi, which is on its third CEO since late 2007.

Peter Thies is president and senior partner at the River Group, a management consulting firm specializing in CEO succession and M&A integration. Ollie Sommer is a partner at the firm.