![]()
|
![]() |
![]() |
![]() |
October 19, 2001
Defining the 2002-2005 Work ProgramThe Independent Evaluation Office is engaging in a broad consultation as it prepares its work program for the next three years. Menu of possible evaluation topics Criteria for setting priorities Timetable for formulating the work program Consultation and external communication October 2, 2001 status note on the IEO work program in PDF format. |
Menu of possible evaluation topicsIEO's work focuses on issues important to the IMF's members and relevant to the Fund's mandate. IEO evaluations could thus cover the full range of IMF activities—and even extend, in principle, to issues relevant to the IMF's mandate but not now part of its activities. The Director of IEO will prepare the work program in the light of consultations with the IMF's Executive Board and management and with informed and interested parties outside. Numerous topics have already surfaced in our initial consultations, in the five major categories of IMF activity. We welcome your comments on the composition of our proposed work program by November 5, 2001. Technical assistance and training Internal processes and governance The menu of possible topics is likely to expand as we continue with our internal and external consultations. We will then narrow this broad menu to a concrete work program for review by the Executive Board. The work program for the first three full years will consist of around twelve studies. Before we undertake individual studies, we will discuss detailed terms of reference setting out questions to be addressed with interested parties. Surveillance Surveillance, a core activity of the Fund, taking up about 40% of staff resources, is now even more important given the emphasis on crisis prevention. Possible studies:
2. Review of Financial System Stability Assessments (FSSA). What lessons can be learned from the experience of the first two years? Are they identifying the key vulnerabilities and proposing remedies in a cost-effective manner? 3. Follow-up of the 1999 external review of surveillance. Specifically, the review could focus on how the recommendations agreed to by the Executive Board have been implemented. 4. Effectiveness of IMF surveillance of industrial countries. Is the scope of surveillance appropriate and what is its value added? 5. The IMF's role in multilateral surveillance, including the World Economic Outlook and the International Capital Markets Report exercises. 6. Role and effectiveness of regional surveillance (European Community, other regional groupings). 7. The IMF's approach to liberalization of the capital account. Possible topics include whether the Fund's policy advice on the pace of capital account liberalization, and its sequencing with other reforms especially vis-à-vis the financial sector has been appropriate and consistent across countries. How has policy changed in the light of experience with capital account crises? 8. The IMF's work on standards and codes of good practice (in collaboration with other agencies). Is the approach effective in building institutional capacity and reducing vulnerability in member countries? IMF-supported programs Three types of studies are possible in this category of IMF activity: reviewing individual country programs, broad policy issues cutting across programs, and lending facilities and related issues. Possible studies: Individual country programs A sample of programs from the past few years could be chosen for review. The past three to four years yield a potential universe of about 70 programs. Questions to be addressed would include: was surveillance effective in identifying vulnerabilities prior to the onset of the crisis; was the program design appropriate; was the negotiating process conducive to country ownership; were the key objectives of the program achieved and, if not, why not; and what are the lessons to be learned; and how effectively is post-program surveillance being carried out? The countries chosen should reflect the diversity of country and crisis types:
2. Low-income and highly indebted cases (Bolivia, Burkina Faso, Cameroon, Honduras, Mozambique, Nicaragua, Niger, Senegal, Tanzania, Uganda, Vietnam, Zambia). 3. Transition countries (Bulgaria, Hungary, Poland, Russia, Ukraine, other FSU countries). 4. Selected countries that have had repeat programs (Bolivia, Madagascar, Philippines, Zambia). Are there problems with program design that contribute to such repeat usage? Broad policy issues cutting across programs This category covers particular policy issues and what can be learned about these issues from cross-country experience. The following are some possibilities:
2. Fiscal adjustment in IMF-supported programs. Does fiscal adjustment take sufficient account of longer-term goals (for growth, poverty reduction) and is it implemented in a sustainable manner? Has the potential, immediate adverse impact of programs on particular vulnerable groups been adequately assessed and taken into account in program design? 3. Policies toward financial sector stability and financial sector restructuring. In addition to the effectiveness of IMF policy advice in helping member countries avoid financial crises (also discussed above under surveillance), possible issues could include whether the design of financial sector restructuring packages has been appropriate, how the fiscal consequences have been handled, and how the macroeconomic consequences of corporate restructuring were taken into account in program design. 4. Debt reduction and debt sustainability issues. Possible issues (either in the context of the HIPC cases or more generally) could include whether the approach to assessing the sustainability or unsustainability of debt positions (external or public debt) has been appropriate and consistent across countries; whether debt sustainability analyses in surveillance reports have been adequate; what use has been made of the sustainability analysis; and what can be learned from `best practice' approaches. 5. Has IMF support of a country's program had a positive "catalytic effect," in terms of generating additional external financing flows within a specific timeframe? Are there objective measures of this catalytic effect? What factors influence the impact on market credibility? 6. The nature and effectiveness of conditionality and issues involving the "ownership" of national/IMF-supported programs. Also, IMF policy toward structural conditionality has been modified recently; a review of the impact of the new policy could be undertaken at the end of the first two-year experience (that is, in FY2004). 7. Why do many IMF-supported programs remain uncompleted and what difference does it make? Are there particular aspects of program design (e.g. optimism of projections, extent of conditionality) that have a strong influence on the probability of completion? Do outcomes depend on the extent to which programs are completed and what lessons can be learned from uncompleted programs? 8. Conditionality with respect to trade policies. Has the IMF adopted a consistent approach to trade policies in the design of conditionality? Is the approach consistent between the surveillance stage and the program stage? 9. Private sector involvement in crisis resolution. This is a growing area of concern in which policy is still evolving but there are lessons to be learnt from experience thus far. Does the existing experience suggest that some forms of PSI are likely to be more successful than others? Many of the questions involved are also closely related to the size of the IMF financial support, including through the Supplemental Reserve Facility (SRF) for countries undergoing crises that are centered primarily in the capital account. 10. Experience with privatization in IMF-supported programs. Possible questions to be addressed could include: was the sequencing with regard to implementation of regulatory frameworks appropriate? How realistic was the timeframe for privatization? What was the impact on prices for services and investment in the privatized sectors? And what was the social impact? Lending facilities and related issues
2. An external review of the Enhanced Heavily Indebted Poor Country (HIPC) Initiative would be important once a "critical mass" of countries reach their completion points. 3. The Contingent Credit Line would also be a candidate for evaluation, but only after there is sufficient concrete operational experience with country cases. 4. The strategy vis-à-vis member country arrears to the IMF. 5. The IMF's role in countries emerging from conflicts. Technical assistance and training Important for capacity building, especially in low-income countries, technical assistance is also crucial for crisis prevention in the more advanced emerging market economies. Possible studies:
2. Does technical assistance help improve national ownership of Fund supported programs? Has it been effective in improving program implementation or in enhancing crisis prevention? 3. The effectiveness of technical assistance in such areas as:
Internal processes and governance The IMF's effectiveness in pursuing its objectives depends clearly on the procedures it adopts both for operational work and Executive Board supervision. Possible studies:
2. Is World Bank-IMF collaboration effective? How can it be improved, given their distinct operational approaches and objectives? Is the division of labor/degree of overlap between the two institutions appropriate? 3. Do staff papers on country programs contain the necessary information and analysis for the Board to make an informed judgment on the probability of success? Research Research is a very important part of the IMF's work, but since the Research Department is currently being restructured, this need not be a high priority area for review. A follow-up on the recommendations of the 1999 External Evaluation of Research Activities could be considered for the longer term. Joint evaluationsWhere an activity undertaken by the IMF is part of a broader multilateral initiative, it may be appropriate for IEO to undertake joint evaluations—either with the World Bank's Operations Evaluations Department (OED) or other agencies. Activities suitable for joint evaluation with OED include the Financial Stability Assessment Program (FSAP) process, Poverty Reduction Strategy Papers (PRSPs) and PRGF-supported programs, and experiences with privatization. Other agencies could be involved in the evaluation of the Fund's work on standards and codes of good practices. Preliminary discussions relating to cooperation have been held with the World Bank's Operations Evaluation Department (OED), and there is agreement in principle. Criteria for setting prioritiesThe work program will include some evaluations of specific country programs, some of broader thematic issues, and recent major new initiatives by the IMF. The emphasis will be on quality and depth—not on sheer numbers of studies—to maximize the potential for learning. In order to narrow down and select topics for the work program, the following criteria may be used: For country programs
For thematic issues
For new initiatives Major new IMF initiatives should be evaluated as soon as there is enough experience to ensure useful feedback into Fund policies. This makes the Poverty Reduction and Growth Facility (PRGF), the Financial System Stability Assessment (FSSA) process, and the IMF's work on standards and codes good prospects. There will inevitably be a trade-off between waiting for experience to be accumulated and undertaking reviews early enough for the results to provide a timely input into policy formulation. Timetable for formulating the work programOctober: Comments of external stakeholders on this status note, including the menu of proposed topics, are invited by November 5, 2001. This period will also be used for continuing consultations with Fund staff and the Executive Board. Mid-November: A draft work program for the next three years will be submitted to management for comment. Early December: Review of the work program by the Executive Board. Late December: The final work program will be posted on the IEO's website. Consultation and external communicationIn carrying out its mandate, the IEO is free to consult with any individuals or groups it deems necessary, inside and outside the IMF. Input on the terms of reference for evaluation studies Consultation during the evaluation Reaching four main audiences. The IEO's terms of reference indicate that it should address four main audiences:
The multiplicity of audiences has implications for the extent and manner of consultation with different groups of stakeholders. Consultations to date. Informal discussions are under way with Executive Directors and with staff of the IMF's area and functional departments. Meetings have also been held with academics, policy analysts, government authorities, and representatives of NGOs in Boston, Washington, London, Paris and Berlin. Consultations—with a broad range of representatives from African countries took place at the time of the African Forum on Poverty Reduction Strategies in Dakar, Senegal in mid-September. We hope to arrange similar consultations in Asia and Latin America. Input on the terms of reference for evaluation studies. Many internal and external observers emphasize that appropriate definition of the terms of reference for evaluation studies is critical to ensuring that all key issues are addressed. External commentators have specifically requested an opportunity to provide inputs into defining the terms of reference of individual studies included in the work program. Consistent with the independence of the IEO, all final decisions on the terms of reference for individual studies must rest with the IEO. But there is a good case for establishing explicit procedures for obtaining inputs on the terms of reference of individual studies in the final program, as well as inputs on substantive issues addressed by the study. Consultation during the evaluation. The best approach to ensuring adequate consultation with staff, management, member country governments, and external groups is likely to vary depending on the evaluation project, and will evolve with experience. The following general approach is being considered:
Dissemination of findings. There is a strong presumption that IEO reports will be published promptly (within the constraints imposed by the need to respect the confidentiality of information provided to the Fund by members) unless the Executive Board were in exceptional circumstances to decide otherwise. The work program to be presented to the Board in December will discuss a strategy for disseminating evaluation results and lessons. The IEO annual report, will include a summary of suggestions for evaluation topics received from within and outside the Fund, along with a discussion of priorities set by IEO and a summary of Executive Board discussions of the whole program. The Report will also set out the main conclusions and recommendations of evaluation reports completed during the year. And it will provide an opportunity to assess the progress in implementing specific recommendations and in tackling issues identified in earlier evaluations. |