Trick or Treat? Equity Concerns in the Preliminary Feasibility Study of the Republic of Korea†
Abstract
As a project appraisal tool, the preliminary feasibility study (PFS) has contributed to enhancing the efficiency of public investment decision-making in the Republic of Korea over the last two decades. To overcome the limitations of the efficiency-oriented cost-benefit analysis, the PFS accommodates equity concerns among regions, namely balanced regional development (BRD) analysis. This study attempts to gauge the contributions of BRD analysis to PFS results. Specifically, it addresses how effectively policy efforts to promote decision-making have been implemented in the PFS stage while also considering the balance between equity and efficiency in terms of the trade-off between them, the degree to which they influence the results, and whether the consideration of equity is in fact actually reflected in seriously underdeveloped regions as intended. The study finds that the PFS results over the last two decades have been largely in line with the background and policy objectives. Based on the findings of the study, needs for institutional improvement are suggested, including enhancements in the analysis of regional economic ripple effects and taking into account the psychological factors pertaining to the evaluators in the overall judgment.
Keywords
Equity-efficiency Trade-off, Balanced Regional Development, Project Appraisal Preliminary Feasibility Study
JEL Code
H43, H54, R53, D61
I. Introduction
The preliminary feasibility study (PFS) was devised and first carried out in 1999 for efficient and objective public investment management in the course of overcoming the 1997 Asian financial crisis in the Republic of Korea. The ex ante appraisal of large-scale public investment projects is considered to have contributed greatly to securing fiscal soundness and increasing the efficiency of public investments by preventing unnecessary budget waste over the past 20 years.
More precisely, the PFS has helped the budget authority to make informed decisions by providing high-quality information and has raised the quality of discussions during the negotiation process by alleviating the information asymmetry between the budget authority and line ministries. In addition, the PFS of the Republic of Korea has been recognized for its objectivity and transparency, which were secured by continuously improving the evaluation methodology and providing a buffer from political influence due to its management by an independent center of excellence, namely the Korea Development Institute (KDI)’s Public and Private Infrastructure Investment Management Center (PIMAC).
However, there have also been a number of negative assessments, as the line ministries and local governments that want to implement the project in relatively underdeveloped areas would be at a disadvantage due to the lack of sufficient prerequisites. In the course of the operation of the PFS over the last two decades, the response to this problem has been to include not only an efficiency-oriented cost-benefit analysis (CBA) in the name of “economic analysis” but also an equity-centered “balanced regional development (BRD) analysis” when making comprehensive recommendations to the budget authority.
The point of the BRD analysis is to supplement the limitations of the CBA by adding extra points to projects to be implemented in relatively underdeveloped regions. Moreover, by steadily increasing the range of prior weights devoted to the BRD analysis, the consideration of equity among regions has been heightened as part of the effort to utilize the trade-off between equity and efficiency as pursued by BRD and CBA analyses, respectively.
The purpose of this study is to present a quantitative assessment of the contribution of the BRD analysis to PFS results by analyzing the PFS cases conducted over the past 20 years. In particular, we seek to provide insight into how effectively policy efforts to promote decision-making have been implemented in the PFS stage, considering the balance between equity and efficiency in terms of the trade-off between them, how much they have influenced the results, and whether the consideration of equity was actually reflected in the seriously underdeveloped regions as intended. This will enable us to evaluate the past performance, draw implications, and make policy suggestions for future institutional improvement.
This study finds that the results of PFSs over the last two decades have been largely in line with the background and policy objectives of considering BRD systematically and gradually expanding its weight. However, because the analysis was at times limited to the discovery of facts, an in-depth investigation to identify reasons for specific results remains outside of the scope of this study and will be left as a future research project. Moreover, due to analytical burden, samples were analyzed rather than surveyed in some cases, which means that a cautious interpretation is required.
Based on the results of this study, we attempted to make suggestions that we feel will improve the PFS system in the future, especially considering the significant changes since 2019. Policy suggestions can be summarized as improvements in the analysis of regional economic ripple effects and the consideration of the psychological factors of the evaluators in the overall judgment.
The rest of this paper is structured as follows. Section II provides an overview of the BRD analysis in the PFS, including why it is needed, its evaluation elements, and chronological changes. Section III attempts to evaluate the effectiveness of the BRD analysis according to three aspects: (a) whether the policy intention was effectively materialized in the PFS, (b) whether the results were influenced by the equity-efficiency trade-off, and (c) whether equity was of greater concern in more severely underdeveloped regions in relation to the policy goals. Finally, Section IV is devoted to summarizing the policy implications and making policy suggestions.
II. Overview of the BRD Analysis in the PFS
A. Need for the BRD Analysis
Using the analytic hierarchy process (AHP) as a tool to help with decision making by facilitating pairwise comparisons of multiple evaluation components, the overall judgement of a PFS follows a method of synthesizing the results of three evaluation components: economic feasibility, policy adequacy, and contribution to BRD. During this process, while none of the components takes precedence over others, the component with the highest prior weight is economic feasibility. By using the term “economic,” the analysis of this component is sometimes confused with a “financial” analysis that compares revenue and expenditures according to the project implementation. However, the main methodology of the economic analysis is a CBA.
A CBA compares the “social cost” and “social benefit” that occur over the entire lifecycle of the project and is not related to the revenue obtained as a result of operating the project. For example, in the case of a national museum, where admission is free, revenue from its operation is obtained mainly by selling souvenirs or food and beverages. Accordingly, the revenue will be very small compared to the construction and operation costs. Therefore, according to a financial analysis, the construction of the national museum will be difficult to secure. However, if we use a CBA to estimate social benefits, such as willingness to pay (WTP) for museum visits or its mere existence, we can find projects that secure economic feasibility. This logic applies equally to economic infrastructure, such as national roads that do not collect user fees and even toll roads where users are in fact charged.
In this way, an economic analysis by means of a CBA can be a stand-alone criterion for judging the feasibility of a project. In countries such as Denmark, Greece, Ireland, Portugal, and Spain, transportation projects have been evaluated solely on the basis of a CBA (Leleur, 2000, cited in Park et al., 2001, Table II-1, p.38).
In many cases, the method of calculating benefits in a CBA is to take the product of the unit value and the quantity of the identified benefit item, where the quantity is calculated based on the estimated demand. In general, demand estimates may be favorable in areas that are already developed. In other words, already developed regions tend to have relatively large populations with high incomes; accordingly, ceteris paribus, it is likely that the demand for the project to be estimated as relatively high. Such a phenomenon may be more noticeable when the benefits are concentrated in a specific area, such as a metro or a science museum, than when the benefits are distributed in various areas, such as inter-regional roads and railroads.
Therefore, relying only on the results of a CBA to determine the feasibility of a project and allocate budget for it can ultimately result in exacerbating regional disparities, or the so-called “the rich get richer and the poor get poorer” phenomenon, by allocating projects in areas that have already been developed. Of course, the logical basis of a CBA is the Kaldor-Hicks efficiency, in which the members of society who are worse off due to the implementation of the project can be compensated “virtually” to be at least indifferent to the status quo by the excess of the net social benefits when the social benefits outweigh the social cost. Thus, even if projects are concentrated in already developed areas, it can be said that no logical problem exists as long as proper compensation is provided in any form from the benefited areas to the disadvantaged ones. However, in practice, it is difficult to provide such compensation accurately. Moreover, if indirect ripple effects not reflected in the CBA or effects that are difficult to quantify are considered, the concentration of the project may become a major problem.
In this regard, in the PFS, the “regional backwardness analysis” was included as a mechanism to improve the equity of budget allocation by adding points to underdeveloped areas. In other words, by examining the relative backwardness of cities and counties across the country and ranking them, additional points can be given to regions with low rankings (relatively backward). Accordingly, an equity-efficiency trade-off is inevitable. The CBA is a scientific tool that has continually improved its methodology with regard to curbing inefficient projects, and the efficiency evaluation function can be weakened by the consideration of equity, and vice versa.
In addition, considering the relative size of the added-value creation effect in certain regions when carrying out projects, the “regional economic ripple effect analysis,” which gives additional points when the effect is large, is also considered. This is based on the “cost-effectiveness” of selecting an alternative that has a greater effect for the same cost input or a less expensive alternative for the same effect.
B. Data and the Econometric Methodology
In the BRD analysis of the PFS, the RBI is calculated according to the weighted sum of selected indicators that can reveal the extent of development in the region in question. As shown in Table 1, there were initially five indicators that make up the RBI, but this number was expanded to eight. The indicator values are updated periodically (for a detailed definition and for the source of the indicators, see KDI, 2008, Table 4-2, p.100).
TABLE 1
CHRONOLOGY OF THE REGIONAL BACKWARDNESS INDEX
Source: Adapted from Kim and Cho (2018), Table 4, p.294.
The RBI for region r at time t is calculated as
where Wi is the (time-invariant) weight assigned to indicator i, and 
in which 
is the indicator i’s value in region r at time ti(t), and 
and Siti(t) are the average and standard deviation of indicator i, respectively (Korea Development Institute, 2008, p.99 and p.101). The subscript ti(t) represents the most recent time point before time t for which the data of indicator i are available. Meanwhile, the weight Wi was set as shown in Table 2 as a result of an expert survey.
C. Estimating the Ripple Effect in the Regional Economy
In the regional economic ripple effect analysis, the region-specific value-added effect of the project implementation is estimated using the interregional input-output (IRIO) model, which is constructed by combining estimates such as the value-added by region and sector, regional final demand, and regional trade factors. The “IRIO index” is calculated by comparing it with the gross regional domestic product (GRDP) in the region, after which a qualitative evaluation is conducted when a comprehensive judgment of the feasibility of the project is made.
Table 3 compares the hypothetical results of the calculation of the IRIO index. It is assumed that two projects with different total project cost (TPC) levels (high and low cost alternatives) will be implemented and that two regions with different GRDP levels (Provinces A and B) will be considered. When projects with the same TPC (KRW 206.94 billion) are implemented in different regions, the difference in the size of the induced value-added in the region is smaller than that between the GRDPs in the two regions, resulting in a difference in the IRIO index. The IRIO index is a value obtained by dividing the induced value-added in the region by the GRDP of the region. Accordingly, even if the same project is pursued, the index value of the region where the GRDP is low is generally large. If two projects with different TPCs are implemented in the same area (Province B), the value of the IRIO index increases when a relatively large project is implemented, as the GRDP is fixed. In summary, we can conclude that, ceteris paribus, a lower GRDP and a higher TPC would be advantageous in the regional economic ripple effect analysis.
D. Status Enhancement of the BRD Analysis
The BRD analysis, composed of the regional backwardness analysis and the regional economic ripple effect analysis, was separately considered from the beginning of the introduction of the PFS to overcome the limitations of a CBA, as discussed above. However, the status of the BRD analysis gradually increased over time. There are two main status enhancements. First, in the AHP analysis used for making a comprehensive judgment, the hierarchy of the regional backwardness and regional economic ripple effect analyses corresponding to the BRD analysis was upgraded from the second to the first tier. Second, the weight to the BRD analysis has increased since it was upgraded to the first tier.
The new stratification of the BRD analysis began in 2006, as shown in Figure 1. The regional backwardness and the regional economic ripple effect analyses, which were included as a policy analysis item before that point in time, were separated out to the new first-tier BRD analysis after 2006, which changed the AHP analysis structure. This measure was taken to make the results of the BRD analysis more explicit and complementary in the same phase as the results of an economic analysis.
However, the relative pre-weight of the BRD analysis was set to be lower than that of the economic analysis (40-50 percent) by assigning pre-weights in the range of 15-25 percent in 2006 (hereafter, the term “pre-weight” refers to the specified range of weight of each analysis component so that evaluators can provide a weight value of their choosing within it). As discussed above, this can be understood as not overriding the efficiency in view of the trade-off between equity and efficiency. Since then, the consideration of equity has been strengthened, which can be confirmed by the trend of the increasing pre-weight of the BRD analysis since 2006, as shown in the lower right area of Figure 1. The pre-weight of the BRD analysis has gradually increased over several years and, after 2019, the BRD analysis was changed to give additional points only for non-Seoul Metropolitan Areas (non-SMAs). Arguably, this was intended to strengthen equity consideration across the country, as there are many relatively developed areas that receive deductions rather than extra points in the SMA, while non-SMAs consist of relatively less developed areas that are the beneficiaries of additional points in the BRD analysis.
III. Evaluating the Effectiveness of the BRD Analysis in the PFS
In this section, we attempt to measure the effectiveness of the BRD analysis based on the experience of considering the component corresponding to concern for equity in the overall judgment of the feasibility of the project. In so doing, we take into account three aspects of effectiveness: (a) whether the policy intention was effectively materialized in the study, (b) whether the results were influenced by the trade-off between equity and efficiency, and (c) whether equity was of greater concern in more severely underdeveloped regions according to the policy goals.
First, in the context of the increasing magnitude of the range of weights of the BRD analysis within the PFS since 2006, we examine whether such a change in the pre-weight has influenced the weight actually assigned by evaluators. Through this exercise, it is possible to understand whether the policy intention that was pursued considering equity more in the PFS was effectively implemented in reality.
Second, we analyze the pattern by which the evaluation results according to efficiency standards are reversed by considering equity due to the trade-off between equity and efficiency. There may be a case in which economic feasibility is not secured as a result of an economic analysis according to the efficiency standard but the overall feasibility is concluded by summing up the results of the BRD analysis according to the equity consideration, and vice versa. The occurrence of such discrepancies between the economic and overall feasibilities is referred to as a “feasibility reversal.” By observing the frequency and trend of such reversals, it is possible to obtain a policy implication that seeks to achieve a balance between equity and efficiency.
Third, we evaluate the differential contribution of the BRD analysis according to the development of the region in question. When the policy goal of introducing and strengthening the BRD analysis is to give greater consideration to equity in the PFS, the results of the BRD analysis should actually be more vigorously reflected in areas with severe backwardness.
The data obtained for analysis consist of a total of 704 PFS results that were completed between 1999 and 2018. The results include the completion year of the PFS; the type and scale (TPC) of the project; the benefit-to-cost ratio (BCR); evaluation points of economic, policy, and BRD analyses; and the AHP score. Some specific analyses may use part of the data depending on the context, and the range of data used at each time will be specified.
A. Effect of Raising the Pre-weight of the BRD Analysis
To examine whether the policy intention to strengthen the consideration of equity gradually was actually implemented in the PFS, we analyzed the trend of the actual assigned weights in the results of the PFS since 2006, when the BRD analysis was upgraded to the first tier. Figure 2 shows the annual averages of assigned weights to economic, policy, and BRD analyses in the AHP evaluations during the overall evaluation of the 449 PFSs conducted in the period of 2006-18. It can be seen that the annual average weight actually assigned to the BRD analysis by AHP evaluators gradually increases during the period.
FIGURE 2.
TREND OF THE PRE-WEIGHTS ON FIRST-TIER AHP EVALUATION COMPONENTS: 2006-2018
Note: The shaded area represents the 95 percent confidence interval.
For a clearer understanding of the increasing trend, we carried out quantitative analyses using ordinary least square (OLS) regression models. In addition to the time when the PFS is conducted, additional factors possibly affecting the actual weight on the BRD analysis include the type, location, scale, and economic feasibility of the project, among others. Table 4 shows the dependent and explanatory variables used in the regression models depending on data availability.
The types of projects were treated as dummy variables to determine whether road, railroad, construction, water resources, and port projects caused differences in weight assignments to the BRD analysis compared to other types. The benchmark group here included airports, industrial complexes, and health facilities, to name a few. A project may be carried out across several regions, especially network projects such as roads and railroads, making it difficult therefore to identify regional differences clearly. Therefore, by using a dummy variable, we attempted to compare cases in which the project site includes a region in the SMA (i.e., Seoul, Incheon, and Gyeonggi-do) to those where it does not. There may be several variables indicating the size of the project, but the total project cost (TPC)—the only information that can be obtained from the data—is included in trillion Korean won (KRW) units. One of the 449 PFSs in the data was missing the TPC value. Lastly, we included the benefit-to-cost ratio (BCR) obtained as a result of the CBA to examine the impact of the results, in that the evaluator knows the economic feasibility analysis results when performing the AHP evaluation and weighting the BRD analysis.
One may argue that the year variable potentially suffers from the endogeneity problem if many new AHP evaluators in the later comprehensive evaluations assign greater weights to the BRD analysis as compared to earlier evaluations systematically for certain unidentified and unobserved reasons. However, a trend in this direction is highly unlikely to exist considering the composition of the evaluators in the comprehensive evaluation. For all PFS cases, the Executive Director and the Director of PIMAC participate in the comprehensive evaluations throughout the data period. Other AHP evaluators in the comprehensive evaluations consist of the project manager who is in charge of the study and outside experts such as university professors and engineering field specialists who have repeatedly conducted PFSs in many cases. Moreover, in practice, consistency between PFSs has been emphasized within PIMAC, with clear instructions at the beginning of each wave of PFSs. In this case, individual effects other than policy changes on the pre-weights of the BRD analysis may be limited. Unfortunately, it is impossible to investigate this issue further due to data limitations in that information pertaining to individual evaluators is not disclosed.
Table 5 shows the results of regression analyses using different combinations of these variables as explanatory variables. The increase in the BRD weights as the evaluation year passed, which is our main concern, was statistically significant in all model specifications. In model V, which is the full model using all explanatory variables, the weight given to the BRD analysis increases by 0.371 percent points on average when the evaluation time was one year later and is statistically significant at the 1 percent significance level.
TABLE 5
REGRESSION RESULTS OF THE ACTUAL WEIGHT ON THE BRD ANALYSIS I: YEAR EFFECT
Note: Numbers in parentheses refer to the heteroscedasticity-robust standard errors. Asterisks indicate statistical significance: *, p < 0.1; **, p < 0.05; ***, p < 0.01.
If the project site includes a region in the SMA, the actual weight to the BRD analysis decreases by 1.332 percent points compared to non-SMA projects. The causes of this outcome need to be analyzed further in different ways, but it shows that economic feasibility is considered to be more important for projects in the SMA.
Looking at the changes according to the type of project, roads and railroads, which are network-type projects, did not show statistically significant differences from the benchmark group. The same results were found for water resource and port projects. Only for cultural and tourism projects did the estimation result reveal that the evaluators assigned smaller weights to the BRD analysis by 1.297 percent points as compared to the benchmark group, which is statistically significant at the 1 percent significance level.
Both the scale and economic feasibility of a project affect the weight to the BRD analysis, which is statistically significantly in a negative direction. The larger the scale (TPC) is, the lower the weight becomes to the BRD analysis. This result suggests that evaluators tend to place more emphasis on economic feasibility for relatively large projects. The negative effect of BCR on the weight in the BRD analysis implies that there is no evidence that evaluators, ceteris paribus, attempted to compensate for the lack of economic feasibility with the BRD analysis. We will revisit this issue later in this paper.
Similarly, to examine the changes in the actual weighting directly according to changes in the pre-weights for the BRD analysis, we conducted OLS using dummy variables for periods of institutional changes, as shown in Figure 1. In so doing, the data for the years in which the pre-weights changed (2009, 2011, and 2016) were excluded from the analysis because which pre-weight was used was unclear, before or after the change. Then, given that the second period contains only one year (2010), the period was set as a benchmark group.
Table 6 displays the results of regression analyses using different combinations of explanatory variables with dummy variables for periods. Upon an examination of the change in the actual weight of each period, which is our main interest, there was no statistically significant difference between 2010 and the period immediately after it (Period III: 2012-2015). However, the periods before 2010 and after Period III were statistically significantly different from 2010. The actual weight to the BRD analysis in Period I (2006-2008) and Period IV (2017-2018) is statistically significantly smaller and greater than that in 2010, respectively. More precisely, the actual weight to the BRD analysis in Period I was 1.778 percent points lower than those in Periods II and III (2010-2015 excluding 2011) on average in Model V. The same weight in Period IV was 3.026 percent points higher than those in Periods II and III. These results imply that a greater increase of the lower-bound of the pre-weight range is needed compared to the increase of its upper bound for the policy intention to be effective. That is, failing to find a statistically significant difference between Periods II and III may be a sign that a five percentage point increase of the lower bound of the pre-weight range is not sufficient to observe any significant change. In contrast, the increase of the upper bound given the identical magnitude from Period I to Period II leads to a statistically significant difference.
TABLE 6
REGRESSION RESULTS OF THE ACTUAL WEIGHT TO THE BRD ANALYSIS II: EFFECT BY PERIOD
Note: Numbers in parentheses refer to the heteroscedasticity-robust standard errors. Asterisks indicate statistical significance: *, p < 0.1; **, p < 0.05; ***, p < 0.01.
The effects of other explanatory variables were similar to those in the results of the previous analysis shown in Table 5. The only difference was that ports had a lower actual weight to the BRD analysis compared to the benchmark group at the 10 percent significance level.
From these results, we can infer that the policy intention to strengthen the consideration of the BRD analysis has been realized. The results of the first set of analyses shown in Table 5 confirmed that the actual weight to the BRD analysis increased in a statistically significantly manner by year. Furthermore, as a result of the second set of analyses shown in Table 6, a statistically significant increase was found between Periods I and III, despite the assignment of the same weight within the range of 20 to 25 percent in Periods I through III.
B. Feasibility Reversal
Next, we examined how the feasibility reversal effect was expressed when the result of the BRD analysis corresponding to the equity criteria reversed the result of the economic analysis of the efficiency criterion in the overall evaluation of the PFS. To this end, among the total of 704 studies completed between 1999 and 2018, 696 cases that included a comprehensive evaluation using AHP were analyzed. At this time, if a PFS assumed several scenarios and an AHP was conducted for each scenario, each was regarded as a stand-alone comprehensive evaluation. After dividing the projects with and without economic feasibility into BCR greater than or equal to and less than one, and dividing the overall feasibility determination result based on the AHP score of 0.5, the results were obtained, as shown in Table 7.
TABLE 7
COMPARISON BETWEEN ECONOMIC AND OVERALL FEASIBILITIES
Note: Cases include 696 PFSs (including scenarios) in which the AHP evaluations were carried out during 1999-2018.
When economic feasibility is secured (BCR ≥ 1) and overall feasibility is secured (AHP ≥ 0.5), there is no change between the two feasibility outcomes, or the feasibility is “retained.” Likewise, if neither economic feasibility nor comprehensive feasibility is secured (BCR < 1 and AHP < 0.5), the feasibility is also “retained.” These cases of retention accounted for 44.7 percent and 36.5 percent of the total, respectively. When combined, they constituted 81.2 percent of the total. Feasibility reversals occurred in the remaining 18.8 percent, and this fact alone suggests that the role of the BRD analysis based on equity standards was significant.
Feasibility reversal cases can be divided into two categories. If economic feasibility is secured but overall feasibility is not secured (BCR ≥ 1 and AHP < 0.5), it can be said that the secured economic feasibility based on efficiency is “offset.” The opposite case (BCR < 1 and AHP ≥ 0.5) is termed the feasibility “supplement” case. What is interesting here is that when feasibility reversal occurs, there are approximately 21.3 times more feasibility supplement cases than feasibility offset cases. That is, there is very large asymmetry with regard to feasibility reversals.
To look at this more closely, we used the conditional probability to obtain the results shown in Table 8. In other words, when economic feasibility is considered as an independent event and overall feasibility is deemed as a dependent event, when economic feasibility is secured, the probability that overall feasibility will also be secured is 98.15 percent, and, conversely, the probability that feasibility will be offset is only 1.85 percent. In contrast, if economic feasibility is not secured, the probability of the occurrence of feasibility supplementation with overall feasibility is relatively high at 32.99 percent.
TABLE 8
CONDITIONAL PROBABILITY OF CHANGE IN FEASIBILITY
Note: Cases include 696 PFSs (including scenarios) in which the AHP evaluations were carried out during 1999-2018.
Although the cause of this asymmetry is not clear and a more in-depth investigation is necessary, we can conjecture two possible reasons. First, this may be caused by the systematic method used, in which the results of BRD analysis are reflected in the comprehensive evaluation. In the comprehensive evaluation using AHP, a transformation formula is used to match the RBI to the unit of the AHP scores. This standard score conversion formula is composed of polynomials of regional backwardness rankings by city and county (LIR) and rankings by metropolitan city and province (MIR), as
where
and
(KDI, 2008, p.174). From this formula, we find that the lower the ranking (the greater the value of LIR) is, the more the standard score is reflected, relative to linearity, if the ranking of the regional backwardness is lower than an area ranked as average. Therefore, the formula is structured to assign additional points to severely underdeveloped areas in a marginally increasing manner, which may be advantageous with regard to feasibility supplement. However, the structure of AHP scoring is not linear with respect to the scale in pairwise comparisons and cannot therefore be determined until a closer examination is conducted. Such a review is outside of the scope of this study and will be left as a future research project.
Second, the psychological factor of the AHP evaluator may have played a role during the comprehensive evaluation. In other words, if economic feasibility is not secured when the location of the project is underdeveloped, there is a possibility that the AHP evaluator assigns a relatively low weight to the economic analysis taking into account CBA limitations, such as the limited benefit estimated from the low demand forecasted in an underdeveloped area. In this case, underdeveloped areas are given extra points by the BRD analysis, and the influence of the additional points can be greater when the BRD analysis is assigned a greater weight. In the opposite case, even if the project is planned to be implemented in a developed area, it can be burdensome for the AHP evaluator to consider the equity as high so as to oppose the implementation of the project in the context of economic feasibility as a result of the CBA based on scientific methodology. This conjecture also requires a closer analysis and review for verification after collecting proper data and is thus also left as future work.
Meanwhile, among the feasibility reversal cases, we examined the differences according to CBA results in 128 feasibility supplement cases that showed higher frequencies than the offsets. Figure 3 shows the distribution of the CBA results of these cases. At this time, the proportions of cases for which economic feasibility were not secured but those with BCR ≥ 0.9 and BCR ≥ 0.8 were found to be 55.5 percent and 78.1 percent, respectively. All such cases for which BCR < 0.6 were completed before 2006, the year in which the BRD analysis was upgraded to the first tier. As can be intuitively expected, feasibility supplement was effective as the project approaches economic feasibility (i.e., as the BCR moves closer to 1 from below this value). In addition, it can be seen that the frequency of the occurrence of feasibility reversal decreases rapidly as the project moves away from economic feasibility (i.e., as the BCR decreases).
FIGURE 3.
DISTRIBUTION OF CBA RESULTS IN FEASIBILITY SUPPLEMENT CASES
Note: Cases include 696 PFSs (including scenarios) in which AHP evaluations were carried out during 1999-2018.
Lastly, in order to determine how changes in the pre-weight of the BRD analysis affected the occurrence of feasibility reversal, the ratio of feasibility reversal among all projects by year was divided into feasibility supplements and offsets. These results are reported in Figure 4, in which the solid line represents the mid-point value of the allowed range of pre-weights of the BRD analysis in the overall evaluation. After 2006, when the BRD analysis was upgraded to the first tier in the AHP structure, feasibility offset did not occur, except in 2006 and 2009, and the feasibility offset ratios for these years were less than 4 percent. In addition, it should be noted that the feasibility supplement ratio revealed a declining trend since 2006 despite the increase in the mid-point value of the pre-weight range of the BRD analysis. More precisely, the feasibility supplement ratio and the mid-point of the pre-weight range of the BRD analysis had an inverse correlation, as the correlation coefficient obtained was -0.244. Considering the increasing trend of the actual weight for the BRD analysis as confirmed in Table 5 and Table 6, we can expect that the likelihood of feasibility supplement will increase as the pre-weight of the BRD analysis becomes adjusted upward, as shown in Figure 1. Counterintuitively, such a uniform effect has not been largely expressed in reality. Due to the different characteristics of each project, available data cannot accurately identify the cause of this outcome. Nevertheless, from the above results, we can infer that the method of increasing the pre-weight of the BRD analysis over the past, for instance, ten years did not significantly improve the issue of equity impairment. This is indicated above as a limitation of the CBA.
FIGURE 4.
RATIO OF FEASIBILITY REVERSAL BY YEAR
Note: 1) Cases include 696 PFSs (including scenarios) in which the AHP evaluations were carried out during 1999-2018, 2) The bars representing the feasibility supplements are categorized by their colors to distinguish the periods divided in Figure 1, 3) The solid line represents the mid-point value of the allowed range of pre-weight of the BRD analysis in the overall evaluation.
However, care must be taken not to misinterpret the result of this weak improvement. This result should not be misunderstood as the economic feasibility of the project to be implemented in the underdeveloped area being very poor, and even if the weight to the BRD analysis was increased, it would remain insufficient to supplement the feasibility. In contrast, in such a case, all other things being equal during the period of 2006-2018, the proportion of the feasibility supplement should increase as the weight to the BRD analysis increases. Therefore, such a result suggests that there is no persistent relationship between the outcomes of economic and BRD analyses.
Due to data limitations, it was not possible to carry out a detailed analysis of these contentions, but we can conjecture that a “learning effect” has arisen in the planning process of the line ministry since 2006. As a result, it is possible that the economic feasibility has gradually improved. As the PFS system was institutionalized, the level of effort and preparation by line ministries may have generally increased in the conception and planning stages of the project, thereby increasing the economic feasibility of the project (i.e., expanding the benefits compared to the costs). In such a case, even if an attempt is made to lower the weight of the judgment based on efficiency standards by increasing the pre-weight of the BRD analysis based on equity, the probability of feasibility supplementation may decrease as the economic feasibility results gradually improve.
For a better understanding of this, Figure 5 summarizes the CBA results (BCRs) by year, where the line and shaded area represent the annual average and range of the BCRs, respectively. The data include 448 out of the total 449 PFSs carried out during 2006-2018, excluding an outlier in 2006 where the BCR was exceptionally high. From this figure, two major observations can be made. First, it shows that the average of the CBA results is maintained without much change at around 1 throughout the data period of 2006-2018. To examine this finding more closely, we tested whether there are differences in the CBA results by year and found no statistically significant differences. In view of this, the tendency for the BCR to increase over time during the period of 2006-2018 is not confirmed.
Second, although the scope of the CBA results differs from year to year, it can be seen that the maximum BCR value for each year increased compared to those of the previous years. This is particularly true after 2017. In addition, as a result of the CBA, the proportion of the total in which economic feasibility is secured (ratio of BCR ≥ 1) is plotted by year in Figure 6. This figure shows that the proportion tends to gradually increase over time in a manner similar to that seen in Figure 5. As a result of a simple regression analysis of the proportion to the year, it was found that the proportion of cases securing economic feasibility increased by 0.85 percent each year during 2006-2018, with this result statistically significant at the 1 percent significance level. Note that the result does not mean that the economic feasibility of individual projects becomes higher over time but rather that the proportion of projects evaluated as economically feasible for each year increases. Therefore, these results should not be misinterpreted as the probability of securing economic feasibility of individual projects, and it should be noted that the probability is independent of the passage of time, as discussed above.
Moreover, the results obtained, as shown in Figure 5 and Figure 6, confirm that the learning effect of line ministries described above is being expressed to some extent. In other words, the probability that the economic feasibility of the project is secured does not change from year to year, but as time passes, the proportion of projects with high BCR planned increased and thus the economic feasibility by year also tended to increase. From the perspective of feasibility reversal, as shown in Figure 3, feasibility supplement was effective when the BCR was close to 1; thus, what is left is to determine the difference by year for these cases. For convenience of the discussion, a project in which economic feasibility is not secured but whose BCR is close to 1 will be referred to as a “marginally economically feasible project” (MEFP). The lower bound, i.e., the least amount of BCR of the project that should be the cut-off standard in order to be an MEFP project, is unclear. Therefore, referring to the results in Figure 3, we will take into account projects with BCR values greater than or equal to 0.7, 0.8, or 0.9 but less than 1. Within these standards, the proportion of feasibility supplement by year and three ratios of MEFPs according to the different definitions above are plotted in Figure 7.
FIGURE 5.
BENEFIT-TO-COST RATIO BY YEAR
Note: 1) Cases include 448 PFSs (including scenarios) carried out during 2006-2018, excluding an outlier in 2006, 2) The solid line and shaded area represent the annual average and range of BCRs, respectively.
FIGURE 6.
PROPORTION OF PFSS SECURING ECONOMIC FEASIBILITY (BCR ≥ 1)
Note: Cases include 449 PFSs (including scenarios) carried out during 2006-2018.
FIGURE 7.
PROPORTION OF FEASIBILITY SUPPLEMENT AND MARGINALLY ECONOMICALLY FEASIBLE PROJECTS BY YEAR
Note: Cases include 449 PFSs (including scenarios) carried out during 2006-2018.
In all cases with the three MEFP criteria, the proportion of MEFPs is similar to that of the feasibility supplement by year, especially when 0.7 ≤ BCR < 1. In addition, the proportion of feasibility supplement by year and the ratio of MEFPs showed very high positive interdependence with correlation coefficients of 0.855, 0.768, and 0.582 for standards of BCR greater than or equal to 0.7, 0.8, and 0.9, respectively.
Summarizing the above findings, we can conclude that the weight to the BRD analysis has gradually increased over time since 2006, whereas the proportion of feasibility supplementation has decreased. Moreover, this result occurred because the proportion of MEFPs decreased.
C. Contribution of the Regional Backwardness Analysis
Subsequently, we determine the degree to which the regional backwardness analysis, as one of two elements of the BRD analysis, has contributed to the comprehensive evaluation. To this end, we examined the effect of the regional backwardness rankings in the area where the project in question is to be implemented considering the overall evaluation process and results. Moreover, we targeted a sample because it was not possible to gather evaluator-level information of all projects studied in the past 20 years due to the vastness of the data and time constraints. We therefore selected 237 projects with identified regions among 247 projects commissioned in 2005-2010 for the PFS.
Figure 8 reports the proportion of the BRD analyses classified with the regional backwardness ranking. Municipalities (i.e., cities and counties) are ordered by their regional backwardness ranking, and each third of them grouped and divided correspondingly into upper (developed), middle, and lower (underdeveloped) regions. The proportion of the BRD analysis is higher in areas in which the project site is located in the lower regions. Quantitatively, the weight of the BRD analysis was found to be 1.2 percent points higher in the middle regions than in the upper regions and the 1.4 percent points higher in the lower regions than in the middle regions. More precisely, the weight of the BRD analysis in lower regions shows a statistically significant difference from those in the other regions while the difference between those in upper and middle regions are not statistically significant from the results of two sample t-tests of the equality of the weights.
FIGURE 8.
REGIONAL BACKWARDNESS RANKING AND SHARE OF FIRST-TIER EVALUATION COMPONENTS IN THE ANALYTIC HIERARCHY PROCESS
Note: Cases include region-specified 237 out of 247 PFSs (including scenarios) carried out during 2005-2010.
The result suggests that evaluators using AHP for comprehensive evaluations tend to consider BRD actively based on equity in less developed regions, which is in line with the asymmetry of the feasibility reversal. This occurs because if the project site is in a relatively developed region, a penalty is given in the BRD analysis during the comprehensive evaluation. Hence, if the proportion of the BRD analysis in the developed area is high, the probability of feasibility offset will increase. It is encouraging that the practice has met the purpose of introducing the BRD analysis, as it accounts for a larger portion of the overall evaluation when the region is less developed.
Moreover, Figure 8 shows that the increase in the weight of the BRD analysis for underdeveloped regions in the comprehensive evaluation reduces both the weights of the economic and policy analyses. On average, the BRD analysis proportion increased by 1.2 percent points in the middle regions compared to the upper regions. Simultaneously, the economic analysis and policy analysis showed decreased values by 0.9 percent points and 0.3 percent points, respectively. In addition, the BRD analysis proportion increased by 1.4 percent points in the lower regions compared to the middle regions, while the corresponding ratio of economic analysis and policy analysis decreased by 0.3 percent points and 1.1 percent points. Compared to the upper and lower regions, the BRD analysis proportion increased by 2.6 percent points and the economic analysis and policy analysis proportion decreased similarly to each other (-1.2 percent points and -1.4 percent point, respectively). As a result, it was confirmed that the trade-off between equity and efficiency was mainly considered between the upper and middle regions, but not between the upper or middle regions and the lower regions. This suggests that improvement is needed to focus on the trade-off between equity and efficiency in line with the purpose of introducing the BRD analysis in the future.
Lastly, the ratio of securing overall feasibility according to the ranking of regional backwardness is shown in Figure 9. This figure divides projects according to economic feasibility: a group for which economic feasibility is secured (BCR ≥ 1), the MEFP group (0.8 ≤ BCR < 1), and a group lacking economic feasibility (BCR < 0.8). Two sample t-tests for equal proportions of overall feasibility reveal that the differences between all proportions within regions and groups categorized by economic feasibility are statistically significant at the 1 percent significance level, except for the difference between the MEFP group and the group lacking economic feasibility in the lower group.
FIGURE 9.
REGIONAL BACKWARDNESS RANKING AND PROPORTION OF OVERALL FEASIBILITY
Note: Cases include region-specified 237 out of 247 PFSs (including scenarios) carried out during 2005-2010.
As economic feasibility falls in the upper and middle regions, the ratio of securing overall feasibility also decreases. In the lower regions, however, the ratio of securing economic feasibility was highest in the MEFP group, followed by the group that secured economic feasibility and the group which lacked economic feasibility. For each project group, in groups with and without economic feasibility, the ratio of securing overall feasibility in the middle regions was highest, whereas in the MEFP group, lower regions’ proportion of securing overall feasibility was remarkably higher than those of the other two regions. It should also be noted that the ratio of securing overall feasibility of the upper regions is low, with a large gap, in the group that lacks economic feasibility. It is difficult to make a direct and clear comparison due to the different characteristics of each project. However, this outcome is intuitive because when economic feasibility of the project is largely lacking and the project is promoted to the upper region, its evaluation result is deducted further from the BRD analysis. For cases with BCR ≥ 0.8, it can be said that the reflection of regional backwardness is effective for the MEFP group (0.8 ≤ BCR < 1). This is again in line with the purpose of introducing the BRD analysis to overcome the limitations of the CBA.
IV. Policy Implications and Suggestions
A. Policy Implications
This study aimed to investigate the effects of the BRD analysis in PFSs. To this end, it focused on three aspects; (a) whether the policy intention was effectively fulfilled at the actual stage of the PFS, (b) whether the consideration of BRD influenced the actual results according to the trade-off between equity and efficiency, and (c) whether the consideration of equity in less developed regions was actually reflected to a large extent.
First, with reference to the effective fulfillment of the policy intention, it was confirmed that the actual weight given by AHP evaluators was raised according to the change in the situation where the pre-weight of the BRD analysis in the PFS was steadily increased. Therefore, through the introduction of the BRD analysis, the policy intention to give more consideration to equity was effectively realized.
Second, by examining the different forms of feasibility reversal, we analyzed the changes in the evaluation results according to the efficiency criteria by considering equity in relation to the equity-efficiency trade-off. Interestingly, the case of feasibility supplementation, in which overall feasibility is secured although economic feasibility is not, accounted for a very high proportion compared to the opposite case, i.e., the feasibility offset case. Possible reasons are that the structure of AHP used for the comprehensive evaluation may be a structure that is advantageous for feasibility supplementation or that the psychological factors of the AHP evaluators may have played a role. However, more in-depth studies will be needed to confirm these conjectures.
In addition, feasibility supplementation occurred more frequently when the project was closer to the threshold of economic feasibility (BCR = 1), as expected. This is encouraging in that the introduction of the BRD analysis resulted in the intended effective equity-efficiency trade-off.
However, we found that the overall feasibility supplement ratio by year decreased over time despite the increase in the magnitude of the pre-weight of the BRD analysis. This is not intuitive at first glance when taking into account the purpose of raising the pre-weight of the BRD analysis, but this result can be explained by the fact that the proportion of MEFPs was decreasing. As described above, feasibility supplementation frequently occurs for MEFPs. As the proportion of MEFPs becomes smaller, the feasibility supplement ratio decreases.
Meanwhile, AHP evaluators on average lowered the weights of both economic and policy analyses while they increased the weight to the BRD analysis. This suggests that it is necessary to improve the system in a way that focuses more specifically on the equity-efficiency trade-off. In this way, weight adjustments between an economic analysis and a BRD analysis can occur effectively with less interference of the weight to the policy analysis.
Third, as a result of examining how the contribution of the BRD analysis varies depending on the development level of the target region of the project, the BRD analysis results were more actively reflected in less developed areas, in accordance with the policy objective. Moreover, for MEFPs in particular, the BRD analysis appeared to help increase the possibility of securing overall feasibility, as the proportions of securing overall feasibility among MEFPs were in the reverse order of development.
In conclusion, the BRD analysis has played a significant role in the course of conducting PFSs over the past 20 years, and our results were mostly in line with the policy objective and background to take into account equity among regions and to increase its weight in public investment decisions. However, because the system has been implemented in a different way since 2019, the results of this analysis have implications for future applications of the system.
The change in the PFS process that occurred in 2019 led to a differentiation between SMA and non-SMAs when considering BRD. The BRD analysis is not included in the PFS of projects to be implemented in the SMA, and the pre-weight of the BRD analysis in the PFS for non-SMAs has been increased to 30-40 percent. In addition, the comprehensive evaluation using the AHP is carried out by the PFS Committee, which is composed of private experts and installed under the Ministry of the Economy and Finance. From the perspective of the analysis, this change has resulted in the exclusion of the analysts who conducted the PFS from the AHP evaluation. The analysts participated in the AHP evaluation because they have a high level of understanding and expertise in relation to the project in question. Under the new system, a disadvantage is that the understanding of the project in question is relatively low and the accuracy of the comprehensive judgment may be limited due to the lack of expertise. Nevertheless, a positive aspect of the change is that it conforms to the general principle that the budget process is a political process and that analysts do not make (political) decisions (Boardman et al., 2011, p.15).
Rather than discussing whether the changes in the system were positive or negative, we present a standard by which to measure how the contribution of the BRD analysis will change in future comprehensive evaluations as a result of the changes. Before the institutional change in 2019, the evaluators participated in the AHP evaluations after already knowing the results of all of the evaluation components. Under the new scheme, they assess the feasibility with respect to policy aspects in AHP evaluations without knowing the results of the economic analysis, and this is reflected exogenously afterward. Previously, Table 8 revealed the conditional probabilities according to the current comprehensive evaluation scheme. Thus, in line with the changed procedures, we can consider the conditional probabilities in which the comprehensive evaluation result is an independent event and the economic analysis result is the dependent event. Table 9 shows the results of the analysis of such conditional probabilities according to Bayes’ theorem.
TABLE 9
CONDITIONAL PROBABILITY OF CHANGE IN FEASIBILITY TO BE COMPARED UNDER THE NEW COMPREHENSIVE EVALUATION SCHEME
Note: Cases include 696 PFSs (including scenarios) in which the AHP evaluations were carried out during 1999-2018.
According to the previous scheme, the probability that economic feasibility is also present is 71.3 percent when overall feasibility is secured. In contrast, in the absence of overall feasibility, the probability that economic feasibility is also absent is 97.7 percent. In the case of feasibility reversal, the probability of the occurrence of feasibility supplementation is 28.7 percent, whereas that of feasibility offset is only 2.3 percent. As discussed above, it is appropriate to use the conditional probabilities in Table 9 to compare the outcomes under the new scheme with those under the previous scheme when a sufficient amount of data is accumulated from the implementation of the changed system in the future. Through this, it will be possible to determine if the implementation of the changed system can more effectively achieve the desired policy goal.
B. Policy Suggestions
Summarizing the results of the analysis and discussions in this study, several suggestions are given below to improve the PFS system. First, there are a few aspects that require attention given how the application of the BRD analysis has changed in the overall judgment. It is necessary to accumulate the information obtained through the implementation of the new scheme for assigning points to non-SMAs and then to derive future improvements accordingly. This is important because there is a possibility that a different pattern will be realized with respect to the results of the application of the previous scheme in this study.
It is also necessary to remove the unnecessary “strategic bias” of evaluators by considering the psychological factors of the AHP evaluators in the overall judgment. If the project is promoted in a non-SMA, an evaluator who does not know the quantitative results of the economic analysis may behave strategically in two opposite directions. On the one hand, because the evaluator knows that the project is being promoted in a non-SMA, it is possible to increase the weight of the BRD analysis. The evaluator can accept the differentiated treatment between SMA and non-SMA cases as a signal to take a favorable position regarding the promotion of the project in a non-SMA from the perspective of BRD, even if it is economically infeasible. Accordingly, if the evaluator makes a qualitative judgement that the economic feasibility of the project is insufficient, she can respond with the feasibility supplement in mind to make up for it.
On the other hand, because the evaluators do not know quantitative results of the economic analysis, they can lower the weight of the BRD analysis to avoid a burden that causes a feasibility reversal outcome. It is critical to eliminate such bias, as the overall judgment requires the assignment of reasonable weights from an expert point of view by synthesizing the characteristics of the project and various results of the analysis.
Second, efforts are needed to tackle issues with the regional economic ripple effect analysis. As discussed in Section 2, the current regional economic ripple effect analysis contains both the limitations of the input-output (IO) model, in which the IRIO model is nested, and the GRDP. The problem with the IO model is that only positive effects are reflected linearly as a ripple effect. In other words, negative effects of the project are not reflected, and when the size of the project is doubled, the ripple effect is estimated to be exactly doubled. In addition, it has been pointed out that the GRDP has problems of representation and adequacy as an indicator of the economic power in the region. For example, for a company with production bases in various regions, the final production result may all be assigned to the region in which the head office is located, and there may be differences due to inconsistencies between the production location and the workers’ residences.
As long as the current methodology is maintained, fundamental improvements may be difficult due to the above-mentioned limitations. Therefore, efforts will need to be made further to identify both positive and negative effects centered on the relevant sector of the project being promoted. As an outcome-oriented approach, it will be possible to examine the gap between the national average and the level of the development of the region in which the project is being targeted, and how the implementation of the project can help to close this gap.
In addition, the current regional economic ripple effect is concentrated on the effect during the construction period. It is necessary to expand this period to include the effect of the entire operation period as much as possible, although understanding this effect can be very challenging due to various uncertainties.
In particular, if the CBA method is not directly related to demand estimates, it is necessary to review the regional impact in the BRD analysis carefully. For example, when estimating the benefits by applying the WTP elicitation method via a survey, such as via the contingent valuation method in the CBA, the calculation of the benefits is not directly linked to the estimate of demand.
Third, it is possible to consider how to integrate the policy analysis and BRD analysis into what is termed a “social value analysis” in the mid- to long-term period. In this way, there can be an improvement in the system in which decisions are made that considers the balance between economic value based on the CBA and social value that encompasses other aspects. In the United Kingdom, through the enactment of the Social Value Act, considering additional benefits by reflecting social values (economic, social, and environmental) during public procurement efforts is obligatory. If the BRD is an important social value, as stated in Article 123 of the Constitution of the Republic of Korea, it will be worth reviewing the change in the system in such a direction.
Notes
This paper is a revised version of Lee, J., 2020, “Roles and Performances of the Balanced Regional Development Analysis in the Preliminary Feasibility Studies” in A Study on Decision Makings of Public Investment Projects, Korea Development Institute (in Korean). I am grateful to two anonymous referees and seminar participants at the PIM Conference Seoul 2019 for their helpful comments. All errors are my own.
References
, & (2018). A Study of the Validity of Preliminary Feasibility Studies: Focused on General Guidelines. Korean Journal of Public Administration, 56(2), 279-311, in Korean, https://doi.org/10.24145/KJPA.56.2.11.