A. Financial Sector Assessment Program (FSAP) and Financial Sector Stability Assessments (FSSA)

6. The FSAP/FSSA is a major initiative undertaken by the IMF, in collaboration with the World Bank, in response to the crises of the late 1990s. It is designed to strengthen bilateral surveillance of member countries and contribute to crisis prevention by helping to identify financial sector vulnerabilities and generate policy recommendations to overcome them. The experience under this initiative was last reviewed by the Executive Board in March 2003, based on an internal joint assessment by IMF and World Bank staff. Approximately 70 assessments are expected to be completed by the end of FY 2004. An independent evaluation of this important, and resource-intensive, activity is therefore timely.

7. The proposed evaluation could focus on areas of broad program design and integration with surveillance functions, as well as implementation aspects, including:

Program design

• How has the Fund's involvement in financial sector stability issues aligned with its core macro-stability mandate? What has been the overarching framework guiding the design of the FSSA/FSAP and its integration into the Fund's surveillance function?
  
  
• What has been the role of the Fund in financial sector surveillance vis-à-vis other IFIs? What has been the division of labor, including on leadership for LDCs, emerging countries, and advanced economies; and on development, and soundness and stability issues?

Implementation

• Risk Assessment: How well has financial sector assessment under the FSAP and associated FSSAs identified the major risk factors in a country's financial system? Are best approaches to diagnose vulnerabilities being used?
  
  
• Addressing Vulnerabilities: Have FSSAs proposed appropriate and feasible measures to address vulnerabilities? Are recommendations articulated as a coherent "blueprint" for reform with proper regard to suitability, sequencing, and implementation capacity? Have actions/reforms been effectively carried out? How has the Fund operated to induce countries to address financial sector vulnerabilities in its surveillance and UFR operations? Has the exercise led to a better understanding of TA needs in the context of strengthening the financial sector, and has this led to appropriate IMF response in the provision of TA?
  
  
• Assessment of Value Added: How do country authorities assess the FSAP/FSSA exercise? Has it deepened the quality and coverage of discussions with authorities on financial sector issues? Have authorities perceived an effective reduction of vulnerabilities? How has the private sector reacted to the new information published in FSSAs and how has it impacted their activities and risk perceptions?
  
  
• How effective has been the coordination efforts with the World Bank? Is there an appropriate balance between industrialized and developing countries covered, given resource requirements and potential benefits? Does the FSAP/FSSA achieve its goals in a cost effective manner? Is the current degree of transparency (which involves voluntary publication of FSSAs) appropriate? What has been the role of the FSAP and FSSAs in recent crisis intervention cases?

8. The evaluation would likely have to be conducted at two levels. Some issues would be examined on the basis of a cross country assessment of experience in all the FSAP cases. This cross-country assessment would have to be supplemented by in-depth study of a sample of country cases.

9. Since the FSAPs are undertaken jointly with the World Bank, it would be desirable to undertake the study in parallel with an evaluation by the World Bank's Operations Evaluation Department of the Bank's experience in this regard. Preliminary discussions with OED have been held to this end.

B. A Case Study of Turkey

10. Turkey provides another example of a capital account crisis involving large access, similar in some ways to the cases in Korea, Indonesia, and Brazil already studied by IEO and the study of Argentina which is currently under way. The country experienced severe financial instability in 1999 leading to the request for a stand-by arrangement for SDR 9.3 billion (US$12.7 billion) at the end of 1999, to protect the fixed exchange rate which was viewed as an essential nominal anchor. The amount under the arrangement was subsequently augmented, but the effort to maintain the exchange rate peg proved unsustainable and was abandoned in February 2001 when the lira was floated. The 1999 arrangement, with SDR 3.3 billion undrawn, was cancelled in February 2002, and a new SBA for SDR 12.8 billion was put in place. This program is expected to end in December 2004.

11. An evaluation of Turkey's experience, including the process by which key decisions on IMF-supported programs were taken, should provide additional lessons on the effectiveness of Fund surveillance in crisis prevention and the Fund's approach to crisis resolution. It would be most useful if the study could cover a longer period after the initial crisis than was possible in the earlier IEO studies of crisis cases, as this would enable the IEO to focus on the post-crisis period in greater depth. The exact period to be covered under the evaluation will be chosen so as to avoid interfering with ongoing operations. The experience with the original 1999 arrangement which was cancelled in February 2002 can be examined without interfering with ongoing operations. The evaluation could be extended to cover issues relating to the second 2002, arrangement after December 2004, when the present arrangement is set to expire.

12. The evaluation would seek to address several questions related to the adequacy of IMF diagnosis and policy advice, including:

(i) Surveillance in the pre-crisis period will be critically examined to assess whether it identified the problems which surfaced later, especially regarding inflation stabilization, the use of the exchange rate as a nominal anchor, and consideration of vulnerabilities, risks, and exit strategies.

(ii) Were the programs well designed from the point of view of assessing fiscal sustainability, both before the crisis and during the period of crisis management?

(iii) Did program design focus adequately on the structural factors that have impaired fiscal adjustment and have been a source of financial sector vulnerability?

(iv) Were programs designed on the basis of a realistic assessment of the extent to which the proposed reforms were owned by the authorities, as well as the authorities' capacity to implement key measures?

(v) Did crisis management, including the defense of the exchange rate regime, take adequate account of banking sector problems?

(vi) How well did IMF policy advice in the post-crisis period cope with the challenges of stabilization and continuing vulnerabilities, including debt sustainability and banking soundness? Parallels with Argentina in this respect would be interesting.

C. The IMF's Approach to Capital Account Liberalization

13. The issue of liberalization of the capital account has been the focus of a great deal of attention and generated considerable criticism of the IMF. Liberalization of capital account transactions is not an explicit part of the mandate of the Fund but policies relating to liberalization of the capital account have figured in Fund surveillance. Critics have argued that the IMF has encouraged emerging market countries to liberalize capital flows in a manner that has subjected them to severe stress because of the unexpected volatility of these flows. Some have argued that rapid liberalization, with insufficient attention to sequencing and establishing the preconditions for effective integration with global financial markets, has been responsible for a great deal of the financial instability and associated economic distress experienced by many emerging market countries.

14. The IMF has argued that its approach to capital account liberalization has become much more pragmatic after the East Asian crisis and there is now a clear recognition of the need to build strong financial systems as a precondition for capital account liberalization. The proposed evaluation would assess whether indeed the Fund's policy advice on the scope, pace and composition of capital account liberalization has changed after the late 1990s. The evaluation would be largely based on a selective study of surveillance and program documents in a representative subset of countries before 1997 and after 1999, supplemented by an examination of other internal documents, if any.

15. Specific issues which could be addressed by the evaluation include the following:

D. Structural Conditionality in IMF-Supported Programs

16. The IMF was criticized in many quarters for the widening of both the scope and the volume of structural conditionality over the last decade. The Managing Director responded to this criticism by issuing, in the Fall of 2000, a set of interim guidelines for staff on streamlining conditionality and, shortly thereafter, launching a thorough review of its conditionality. This led to the adoption by the Executive Board of revised conditionality guidelines in September 2002. These guidelines call for streamlining conditionality to achieve parsimony while ensuring that conditionality covers all policies critical to the achievement of macro-economic objectives. They recognize that critical policies would include policies for which other institutions, notably the World Bank, have primary responsibility. They also emphasize the importance of ownership for conditionality to be effective.

17. The evaluation could examine the internal consistency of the new guidelines, including (a) the possible tension between emphasizing parsimony and also covering all policies critical to the achievement of macroeconomic objectives, and (b) the tension between an approach that emphasizes country-owned adjustment programs but also retains the traditional dependence on conditionality. It could assess the extent to which the new guidelines have brought about a significant change in actual practice in focusing on the four core areas of structural conditionality—fiscal, trade, financial and privatization. This would require comparison of the situation before September 2002 (as already documented in a staff review) and the current position reflected in programs approved after the September 2002 guidelines were issued. Specific issues to be investigated could include:

• What is the evidence about the effectiveness of structural conditionality? How have programs dealt with the issue of ownership and what are the implications for the design of structural conditionality? In addressing these issues, and others listed below, the evaluation will need to distinguish between PRGF and non-PRGF cases and for the former would draw upon any lessons emerging from the ongoing evaluation of the PRSP/PRGF Initiative.
  
  
• How has the potential conflict between the objective of achieving parsimony and the objective of ensuring that conditionality cover all policies critical to achieving macroeconomic objectives of the program been resolved in practice? Have programs identified core essential elements and non-core desirable elements while limiting conditionality explicitly to the former?
  
  
• How have staff approached the task of assessing macroeconomic relevance in deciding whether or not to include particular structural measures in IMF conditionality?
  
  
• Has the concept of division of labor led to a redesign of programs to give greater emphasis to the areas of conditionality that are within the competence of the IMF, with less emphasis being given to the others. Where other areas of conditionality remain important, how is the objective of parsimony achieved in collaboration with other institutions?
  
  
• How has enhanced collaboration with the World Bank in the design of conditionality been implemented in practice?

E. Private Sector Involvement (PSI) in Crisis Resolution

18. The need to involve the private sector effectively in the resolution of financial crises in order to contain the size of access and also to create market-based systems of resolution with the right mix of incentives has been much discussed. In principle, an evaluation of recent country programs could be undertaken to address issues such as:

• How are private market creditors being involved in the resolution of financial crises and are principles in this respect being applied consistently? Is debt sustainability/solvency analyzed in a consistent manner? Is there a tendency to be over-optimistic about the expected return to normalcy, e.g., expected reductions in risk premia?
  
  
• What has been the catalytic role of Fund-supported programs and official financing packages in mobilizing private capital (in terms of both volume and terms of financing)? Why in many cases were private flows less than assumed in the program?
  
  
• How effective have been the various concerted efforts, short of a standstill, to secure private sector involvement in major crisis uses? What has been the outcome in terms of burden-sharing among creditors? Are further institutional changes required to facilitate PSI?
  
  
• What lessons do the early cases suggest about the impact of concerted PSI on the real economy on the resumption of voluntary flows, and on the availability of financing to other countries?

While these are important issues, an argument against undertaking an evaluation at this stage is that policy in this area is currently being discussed and there are significant differences within the official community on how the issues should be handled. The management initiative on introducing a Sovereign Debt Restructuring Mechanism was potentially an important element of private sector involvement but there is no agreement on this proposal as yet. There is general agreement on the usefulness of Collective Action Clauses (CAL), but too little experience.

F. The Role of Multilateral Surveillance

19. Multilateral surveillance is a core activity of the IMF that aims to provide a basis for analyzing the forces driving the world economy, identifying global vulnerabilities, and advising on appropriate policies, especially in the most systemically important countries. Potential issues to be addressed would include the following:

G. An Additional Country Case Study

20. The IEO could undertake an in-depth examination of a recent country program with the focus on the effectiveness of program design and conditionality. The primary goal would not be to assess whether the IMF's actions were right or wrong in a particular case (although that would be an additional outcome), but to investigate in some detail what the case suggests about the IMF's recent approach to program design as well as the structure and focus of its conditionality. Programs chosen could be from either 2000 or 2001, allowing evaluation of outcomes in the post-program period. Two possible options could be considered:

(i) An investigation of program design in a middle-income country, but not one of the major "capital account crisis" cases, which are being examined in other evaluations. Possibilities would include Bulgaria, Peru, Romania, Sri Lanka, or the Ukraine.

Appendix