Menu of possible evaluation topics

IEO's work focuses on issues important to the IMF's members and relevant to the Fund's mandate.

IEO evaluations could thus cover the full range of IMF activities—and even extend, in principle, to issues relevant to the IMF's mandate but not now part of its activities.

The Director of IEO will prepare the work program in the light of consultations with the IMF's Executive Board and management and with informed and interested parties outside.

Numerous topics have already surfaced in our initial consultations, in the five major categories of IMF activity. We welcome your comments on the composition of our proposed work program by November 5, 2001.

Surveillance

IMF supported programs

Technical assistance and training

Internal processes and governance

Research

The menu of possible topics is likely to expand as we continue with our internal and external consultations. We will then narrow this broad menu to a concrete work program for review by the Executive Board.

The work program for the first three full years will consist of around twelve studies. Before we undertake individual studies, we will discuss detailed terms of reference setting out questions to be addressed with interested parties.

Surveillance

Surveillance, a core activity of the Fund, taking up about 40% of staff resources, is now even more important given the emphasis on crisis prevention.

Possible studies:

1. The IMF's role and effectiveness in crisis prevention based on an examination of recent Article IV reports including all the recent capital account crisis cases. Are "early warning" procedures effective, and do they influence the Fund's advice and countries' policies?

2. Review of Financial System Stability Assessments (FSSA). What lessons can be learned from the experience of the first two years? Are they identifying the key vulnerabilities and proposing remedies in a cost-effective manner?

3. Follow-up of the 1999 external review of surveillance. Specifically, the review could focus on how the recommendations agreed to by the Executive Board have been implemented.

4. Effectiveness of IMF surveillance of industrial countries. Is the scope of surveillance appropriate and what is its value added?

5. The IMF's role in multilateral surveillance, including the World Economic Outlook and the International Capital Markets Report exercises.

6. Role and effectiveness of regional surveillance (European Community, other regional groupings).

7. The IMF's approach to liberalization of the capital account. Possible topics include whether the Fund's policy advice on the pace of capital account liberalization, and its sequencing with other reforms especially vis-à-vis the financial sector has been appropriate and consistent across countries. How has policy changed in the light of experience with capital account crises?

8. The IMF's work on standards and codes of good practice (in collaboration with other agencies). Is the approach effective in building institutional capacity and reducing vulnerability in member countries?

IMF-supported programs

Three types of studies are possible in this category of IMF activity: reviewing individual country programs, broad policy issues cutting across programs, and lending facilities and related issues.

Possible studies:

Individual country programs

A sample of programs from the past few years could be chosen for review. The past three to four years yield a potential universe of about 70 programs. Questions to be addressed would include: was surveillance effective in identifying vulnerabilities prior to the onset of the crisis; was the program design appropriate; was the negotiating process conducive to country ownership; were the key objectives of the program achieved and, if not, why not; and what are the lessons to be learned; and how effectively is post-program surveillance being carried out? The countries chosen should reflect the diversity of country and crisis types:

1. Capital account crises (Argentina, Brazil, Ecuador, Indonesia, Korea, Thailand, Turkey). Some stakeholders have suggested that IEO should evaluate all programs where exceptionally large access to Fund resources is involved.

2. Low-income and highly indebted cases (Bolivia, Burkina Faso, Cameroon, Honduras, Mozambique, Nicaragua, Niger, Senegal, Tanzania, Uganda, Vietnam, Zambia).

3. Transition countries (Bulgaria, Hungary, Poland, Russia, Ukraine, other FSU countries).

4. Selected countries that have had repeat programs (Bolivia, Madagascar, Philippines, Zambia). Are there problems with program design that contribute to such repeat usage?

Broad policy issues cutting across programs

This category covers particular policy issues and what can be learned about these issues from cross-country experience. The following are some possibilities:

1. Exchange rate policies in IMF-supported programs and the Fund's policy advice on exchange rates as part of surveillance. Has the Fund's policy advice on exchange rate regimes and associated policies been consistent across countries? Has the design of exit strategies from exchange rates pegs been appropriate? Have the contractionary consequences of devaluations, stemming from their balance sheet consequences, been underestimated in program design? Has financial program design adapted effectively to inflation targeting regimes?

2. Fiscal adjustment in IMF-supported programs. Does fiscal adjustment take sufficient account of longer-term goals (for growth, poverty reduction) and is it implemented in a sustainable manner? Has the potential, immediate adverse impact of programs on particular vulnerable groups been adequately assessed and taken into account in program design?

3. Policies toward financial sector stability and financial sector restructuring. In addition to the effectiveness of IMF policy advice in helping member countries avoid financial crises (also discussed above under surveillance), possible issues could include whether the design of financial sector restructuring packages has been appropriate, how the fiscal consequences have been handled, and how the macroeconomic consequences of corporate restructuring were taken into account in program design.

4. Debt reduction and debt sustainability issues. Possible issues (either in the context of the HIPC cases or more generally) could include whether the approach to assessing the sustainability or unsustainability of debt positions (external or public debt) has been appropriate and consistent across countries; whether debt sustainability analyses in surveillance reports have been adequate; what use has been made of the sustainability analysis; and what can be learned from `best practice' approaches.

5. Has IMF support of a country's program had a positive "catalytic effect," in terms of generating additional external financing flows within a specific timeframe? Are there objective measures of this catalytic effect? What factors influence the impact on market credibility?

6. The nature and effectiveness of conditionality and issues involving the "ownership" of national/IMF-supported programs. Also, IMF policy toward structural conditionality has been modified recently; a review of the impact of the new policy could be undertaken at the end of the first two-year experience (that is, in FY2004).

7. Why do many IMF-supported programs remain uncompleted and what difference does it make? Are there particular aspects of program design (e.g. optimism of projections, extent of conditionality) that have a strong influence on the probability of completion? Do outcomes depend on the extent to which programs are completed and what lessons can be learned from uncompleted programs?

8. Conditionality with respect to trade policies. Has the IMF adopted a consistent approach to trade policies in the design of conditionality? Is the approach consistent between the surveillance stage and the program stage?

9. Private sector involvement in crisis resolution. This is a growing area of concern in which policy is still evolving but there are lessons to be learnt from experience thus far. Does the existing experience suggest that some forms of PSI are likely to be more successful than others? Many of the questions involved are also closely related to the size of the IMF financial support, including through the Supplemental Reserve Facility (SRF) for countries undergoing crises that are centered primarily in the capital account.

10. Experience with privatization in IMF-supported programs. Possible questions to be addressed could include: was the sequencing with regard to implementation of regulatory frameworks appropriate? How realistic was the timeframe for privatization? What was the impact on prices for services and investment in the privatized sectors? And what was the social impact?

Lending facilities and related issues

1. The role of the Poverty Reduction and Growth Facility (PRGF) and the need to evolve special procedures to overcome problems in achieving stated objectives in countries with PRGF-supported programs has been much discussed. Although an internal review is currently under-way, there is scope for an independent review beginning perhaps in late FY2003. The review could address such issues as: Have all the lessons from the ESAF reviews and "best practices" on particular policy issues been incorporated into PRGF adjustment strategies? Has the increased emphasis on country ownership resulted in real changes in the approach to negotiations? Has program design effectively incorporated the analysis of the social impacts of major reforms? Has program design and monitoring improved the targeting of spending in key sectors relevant to growth and poverty reduction? Have resources been effectively channeled to social sectors?

2. An external review of the Enhanced Heavily Indebted Poor Country (HIPC) Initiative would be important once a "critical mass" of countries reach their completion points.

3. The Contingent Credit Line would also be a candidate for evaluation, but only after there is sufficient concrete operational experience with country cases.

4. The strategy vis-à-vis member country arrears to the IMF.

5. The IMF's role in countries emerging from conflicts.

Technical assistance and training

Important for capacity building, especially in low-income countries, technical assistance is also crucial for crisis prevention in the more advanced emerging market economies.

Possible studies:

1. A followup of the 1999 internal review of technical assistance. This could include an assessment of whether there are effective and consistent internal systems for evaluating technical assistance advice and for setting priorities.

2. Does technical assistance help improve national ownership of Fund supported programs? Has it been effective in improving program implementation or in enhancing crisis prevention?

3. The effectiveness of technical assistance in such areas as:

• Tax policy and revenue administration.

• Expenditure policy and expenditure management.

• Banking supervision and financial stability.

• Debt and external reserve management.

Internal processes and governance

The IMF's effectiveness in pursuing its objectives depends clearly on the procedures it adopts both for operational work and Executive Board supervision.

Possible studies:

1. Are the IMF's internal review systems adequate? For example, are there adequate mechanisms for early internal re-assessments of the adequacy of program design?

2. Is World Bank-IMF collaboration effective? How can it be improved, given their distinct operational approaches and objectives? Is the division of labor/degree of overlap between the two institutions appropriate?

3. Do staff papers on country programs contain the necessary information and analysis for the Board to make an informed judgment on the probability of success?

Research

Research is a very important part of the IMF's work, but since the Research Department is currently being restructured, this need not be a high priority area for review. A follow-up on the recommendations of the 1999 External Evaluation of Research Activities could be considered for the longer term.

Joint evaluations

Where an activity undertaken by the IMF is part of a broader multilateral initiative, it may be appropriate for IEO to undertake joint evaluations—either with the World Bank's Operations Evaluations Department (OED) or other agencies.

Activities suitable for joint evaluation with OED include the Financial Stability Assessment Program (FSAP) process, Poverty Reduction Strategy Papers (PRSPs) and PRGF-supported programs, and experiences with privatization. Other agencies could be involved in the evaluation of the Fund's work on standards and codes of good practices.

Preliminary discussions relating to cooperation have been held with the World Bank's Operations Evaluation Department (OED), and there is agreement in principle.

Criteria for setting priorities

The work program will include some evaluations of specific country programs, some of broader thematic issues, and recent major new initiatives by the IMF. The emphasis will be on quality and depth—not on sheer numbers of studies—to maximize the potential for learning.

In order to narrow down and select topics for the work program, the following criteria may be used:

For country programs

• Geographic and economic diversity of country cases is desirable for learning and for ensuring relevance for all categories of member countries. Special attention could go to countries making contrasting policy choices, exemplifying alternative approaches (Thailand, compared with the non-program case of Malaysia).

• Programs with exceptionally large access could be given priority

• Country cases could include a mix of cases viewed as "relative failures" (to learn from mistakes) and as "more successful" (the basis for defining best practice).

• Country cases likely to generate useful inputs for the "thematic" evaluations that are also on the work program should have priority.

For thematic issues

• Immediate policy relevance to a wide range of the Fund's membership.

• Where there are widely different views about the appropriate approach.

• IEO's comparative advantage in conducting the study (through access to internal information).

For new initiatives

Major new IMF initiatives should be evaluated as soon as there is enough experience to ensure useful feedback into Fund policies. This makes the Poverty Reduction and Growth Facility (PRGF), the Financial System Stability Assessment (FSSA) process, and the IMF's work on standards and codes good prospects. There will inevitably be a trade-off between waiting for experience to be accumulated and undertaking reviews early enough for the results to provide a timely input into policy formulation.

Timetable for formulating the work program

October: Comments of external stakeholders on this status note, including the menu of proposed topics, are invited by November 5, 2001. This period will also be used for continuing consultations with Fund staff and the Executive Board.

Mid-November: A draft work program for the next three years will be submitted to management for comment.

Early December: Review of the work program by the Executive Board.

Late December: The final work program will be posted on the IEO's website.

Consultation and external communication

In carrying out its mandate, the IEO is free to consult with any individuals or groups it deems necessary, inside and outside the IMF.

Reaching four main audiences

Consultations to date

Input on the terms of reference for evaluation studies

Consultation during the evaluation

Dissemination of findings

Reaching four main audiences. The IEO's terms of reference indicate that it should address four main audiences:

• Fund staff and management, to enhance learning culture.

• Those interested in the Fund's activities, to enhance its external credibility.

• The public in all member countries, to promote greater understanding of the work of the Fund.

• The Executive Board, to strengthen its institutional governance and oversight responsibilities.

The multiplicity of audiences has implications for the extent and manner of consultation with different groups of stakeholders.

Consultations to date. Informal discussions are under way with Executive Directors and with staff of the IMF's area and functional departments. Meetings have also been held with academics, policy analysts, government authorities, and representatives of NGOs in Boston, Washington, London, Paris and Berlin. Consultations—with a broad range of representatives from African countries took place at the time of the African Forum on Poverty Reduction Strategies in Dakar, Senegal in mid-September. We hope to arrange similar consultations in Asia and Latin America.

Input on the terms of reference for evaluation studies. Many internal and external observers emphasize that appropriate definition of the terms of reference for evaluation studies is critical to ensuring that all key issues are addressed. External commentators have specifically requested an opportunity to provide inputs into defining the terms of reference of individual studies included in the work program. Consistent with the independence of the IEO, all final decisions on the terms of reference for individual studies must rest with the IEO. But there is a good case for establishing explicit procedures for obtaining inputs on the terms of reference of individual studies in the final program, as well as inputs on substantive issues addressed by the study.

Consultation during the evaluation. The best approach to ensuring adequate consultation with staff, management, member country governments, and external groups is likely to vary depending on the evaluation project, and will evolve with experience. The following general approach is being considered:

• Draft terms of reference for each study—identifying the questions to be addressed and also, to the extent practicable, the methodology—would be prepared and comments sought from the Executive Board, staff, management, and member country governments (especially for evaluations involving individual countries). In addition, draft terms of reference would be posted on IEO's Web site to elicit comments from external observers. The final terms of reference, as determined by IEO after these consultations, would also be posted on the Web site. Other means of communicating the terms of reference may also be used, especially in cases involving countries with limited access to the Internet.

• The IEO evaluation team would conduct its own research and interact with concerned parties inside and outside the IMF. For evaluations involving individual countries, consultations are expected to be held with the authorities and with others. The evaluation team would be open to input on substantive issues covered by the terms of reference from all interested parties. IEO's Web site will be used to elicit written comments.

• Draft reports will be submitted to management, and (where appropriate) the country authorities, for their comments. The final reports will then be circulated to the Executive Board, along with comments of management and country authorities (where relevant).

• To enhance the IMF's ability to learn from IEO evaluations, each report will identify lessons and make recommendations. The recommendations should be specific enough to allow management, and subsequently the Executive Board, to indicate whether they agree (in which case they may wish, at their discretion, to indicate specific actions they plan to take) or disagree (with an explanation of the reason for disagreement).

Dissemination of findings. There is a strong presumption that IEO reports will be published promptly (within the constraints imposed by the need to respect the confidentiality of information provided to the Fund by members) unless the Executive Board were in exceptional circumstances to decide otherwise. The work program to be presented to the Board in December will discuss a strategy for disseminating evaluation results and lessons.

The IEO annual report, will include a summary of suggestions for evaluation topics received from within and outside the Fund, along with a discussion of priorities set by IEO and a summary of Executive Board discussions of the whole program. The Report will also set out the main conclusions and recommendations of evaluation reports completed during the year. And it will provide an opportunity to assess the progress in implementing specific recommendations and in tackling issues identified in earlier evaluations.