An Unsuccessful Reform on the Local Public Contracts Law in Korea^†

I. Introduction

Local public contracts procured by local governments, local-government-owned enterprises or other local agencies are very important in the national economy, especially considering their size. As of 2019, local public contracts amounted to at least 55 trillion Korean won, representing one third of all public contracts procured by either local governmental bodies or central governmental bodies.

Therefore, relieving the fiscal burden of local governmental bodies by applying a well-designed auction format to the local public procurement process should be an important concern in the Korean economy. Traditionally, the first price auction was used in the local public procurement process in order to minimize winning bid prices. However, this approach has been heavily criticized in that too much bid price competition can result in poor ex-post performance. For instance, an inefficient construction company may win a procurement auction by submitting an unreasonably low bid and, once an initial contract is signed, the company then fails to provide the promised services or requires from the procurer an additional payment or subsidy as a condition for continuing its obligations. That is, a winning bid in the first price auction could be ‘too low to be true.’

There are two approaches to overcome the aforementioned ‘too-low-to-be-true’ problem of first price auctions. The first approach is to augment the simple first price auction with ex-post screening. Once the lowest bidder is determined, the bidder is subject to ex-post screening by the procurer. If the lowest bidder passes this screening process, it can finally be awarded the procurement contract. However, if it fails to pass the screening, the second lowest bidder is then subject to ex-post screening. The second approach to resolve the too-low-to-be-true problem is to replace the first price auction with an average price auction under which a bidder wins the auction if her bid is closer to the average of all submitted bids compared to competitors’ bids. Under this average price auction, bidders do not have an incentive to under-cut competitors’ bids. Instead, it is optimal to match what others will submit. As bid price competition is restrained, this nonstandard auction format results in a higher winning bid. As the winner is paid more and the winner is selected randomly out of bidders who bid the same price, the likelihood that an inefficient company wins the auction and demonstrates poor ex-post performance is expected to be reduced. Although this average price auction is not well known in the economic literature, it has been used or is still in use in a number of local governmental bodies, including those in Italy, the United States, China, Japan, Colombia, Peru, Chile, and Switzerland (e.g., Sweet ,1994; Albano et al., 2006; Bajari et al., 2014; Decarolis, 2018).

A hybrid of the two aforementioned approaches is the average price auction with screening. In Korea, Local Public Contracts Law required local governmental bodies to apply a version of the first price auction with ex-post screening until April 30 of 2016. However, this existing auction method was recently replaced with the Comprehensive Evaluation Method, which is going into effect on May 1, 2016. This new auction method is essentially an example of the average price auction with screening. The government of Korea announced that this new method is intended to mitigate the typical ex-post problems, including frequent renegotiations and cost overruns which arise after the initial contract is signed, associated with the existing method (see MOEF, 2015; PPS, 2019).

Interestingly, the existing theoretical literature predicts that the average price auction with screening (APAS) is economically dominated by the first price auction with screening (FPAS) due to two reasons. Firstly, winning bids under the APAS method will be higher than that under the FPAS method due to a restricted bid competition. Secondly, the ex-post performance under the APAS method will not be better than that under the FPAS as ex-post screening process exists in both methods (see Decarolis, 2014).

Based on this theoretical prediction, I empirically examine the effect of the newly introduced comprehensive evaluation method on winning bids and ex-post performance measures, specifically the number of changes in contracts and the cost overruns. The empirical analysis finds two main results. First, the newly introduced average price auction with screening increases winning bids. Second, the new method has no material impact on the number of changes in contracts nor the cost overruns, which suggesting that neither the frequent renegotiation problem nor the cost overrun problem is resolved. These findings are consistent with the theoretical literature.

This paper is related to a number of existing studies. One of the most closely related papers is that by Decarolis (2014), which examines an Italian public procurement dataset and finds that the replacement of the existing average price auction (without screening) by a first price auction with (weak) screening in the end reduces winning bids while exacerbating the cost overrun problem. The current paper differs from Decarolis (2014) in the sense that I consider an average price auction with screening, while Decarolis (2014) considers an average price auction without screening. The aforementioned theoretic prediction that the average price auction with screening is economically dominated by the first price auction with screening holds only when the average price auction is combined with a screening process.

Kang and Kim (2017) empirically study Korean public procurement data. They consider central public contracts procured by central governmental bodies rather than local governmental bodies and examine the comprehensive examination method, which came into effect on January 1, 2016, for central public contracts. The comprehensive examination method is similar to the comprehensive evaluation method considered in this paper in that both are examples of the average price auction with screening. However, the detailed scoring rules of bid price and nonprice characteristics are different. Moreover, the former is applied to central public contracts while the latter is applied to local public contracts. Without conducting a regression analysis but by comparing the average winning bid under the new method with that under the existing method, they find that winning bids increase due to the new auction method.

Kim (2012) examines the primary factors that determine a winner in the existing first price auction with screening using a Korean central public contracts dataset. He finds that almost every qualified bidder was assigned the highest possible score on non-price aspects in the screening process. This result implies that the first price auction with screening is in effect equal to a simple first price auction without screening given that the screening process cannot distinguish good from poor companies. Although Kim (2012) does not consider the average price auction with screening at all, his findings imply that even the existing first price auction with screening is not perfect. By considering this important caveat, I shall carefully discuss policy implications of the main findings of the current paper.

The current paper is also related to Spulber (1990), McAfee and McMillan (1986), and Bajari et al. (2014) in that these papers emphasize the ex-post moral hazard problem of the first price auction. Spulber (1990) and McAfee and McMillan (1986) depict the moral hazard problem theoretically, while Bajari et al. (2014) examines the problem empirically. In addition, the current paper is related to Lewis and Bajari (2011a; 2011b) in the sense that all of these empirical papers agree on the conclusion that a scoring auction that evaluates both the bid and quality of a bidder is better than a simple first price auction in the realm of public procurement.

The organization of the current paper is as follows. Section 2 reviews the institutional background of local public contracts in Korea. Section 3 provides an overview of related theoretical studies and formulates testable hypotheses based on the theoretical predictions. Section 4 examines empirically the effect of the comprehensive evaluation method on winning bids and on ex-post performance outcomes. Finally, Section 5 draws some policy implications and presents concluding remarks.

II. Institutional Background

Local governments and local-government-owned enterprises frequently procure contracts for construction. In Korea, these local governmental bodies must comply with the rules and regulations stipulated by the Local Public Contracts Law when they procure these contracts.

Until April 30, 2016, local governmental bodies were obliged to apply a type of first price auction augmented by an after-bid screening process when procuring construction contracts whose estimated prices are no less than 10 billion KRW.1 The procurement process consists of two phases. In the first phase, construction companies submit bids, and the lowest bidder is selected as the preferred bidder. In the second phase, the procurer screens this preferred bidder in order to assess whether this bidder is in fact capable of conducting the required construction tasks in a timely manner with a reasonable degree of quality. If the preferred bidder passes the screening, it becomes the winner and signs the contract with the procurer. However, if the preferred bidder is screened out, the procurer selects the second lowest bidder as the preferred bidder and proceeds with the screening process.

However, an amendment of the Local Public Contracts Law was passed and, therefore, a new method of procurement auction went into effect, starting on May 1, 2016. The new method is called the comprehensive evaluation method, which is basically a type of average price auction augmented by an after-bid screening process. This new method applies to almost every construction contract whose ‘estimated price’ is equal to or higher than 30 billion KRW. Under this method, construction companies submit bids. Then, the procurer evaluates the score on price (=bid) and the score on the non-price quality of each bidder and awards the contract to the bidder whose total score is the highest. Interestingly, the price score moves higher, as the difference between a bid and the average bid becomes smaller, where the average bid is the average of all submitted bids. That is, a bidder can receive the highest score on price if its bid is closer to the average bid relative to any competitor’s bid. Consequently, an optimal bidding strategy is to match what others may bid, and the worst strategy is to submit a bid lower than competitors’ bids. The scoring process for non-price quality in the new auction method is nearly identical to the screening process of the existing auction method in the sense that both processes rely on quantitative evaluations and the evaluation criteria and performance measurements are virtually identical. Accordingly, the newly introduced comprehensive evaluation method can be seen as an average price auction with screening, while the existing method is a type of first price auction with screening. It should also be noted that this existing method still applies to construction contracts whose estimated prices are between 10 and 30 billion KRW.

According to the government of Korea, the motivation behind the replacement of the existing auction method with the comprehensive evaluation method is to mitigate ‘ex-post problems’ caused by the first price auction rule within the existing method. The Ministry of Economy and Finance (MOEF) explicitly admitted that the first price auction rule should be abandoned in order to prevent ex-post problems such as frequent renegotiations and cost overruns after the initial contract is signed (see MOEF, 2015). It has been argued that a construction company has an incentive to submit an unreasonably low bid to win the auction and then soon after strongly request a renegotiation of the terms of the contract and often threaten the procurer that continuation of construction is possible only if the procurer pays the contractor more than the initially agreed-upon amount. Another goal is to subsidize the construction industry. The Korea Public Procurement Service (PPS) announced that a primary objective of the comprehensive evaluation method is to boost the profitability of the construction industry (see PPS, 2019).

III. Theory Overview and Hypothesis Formulation

Although the first price auction is standard in economics and practice, it has been criticized for the related trade-off between winning bids and ex-post performance (i.e., too-low-bid-to-be-true problem). According to Spulber (1990), auctions for contracts should be distinguished from auctions for goods because in the former case, a transaction does not take place immediately after the determination of a winner, and there is cost uncertainty. Given that it takes several months or years to complete the work required by a contract, a shock could attack the construction company in the meantime, possibly increasing the cost of construction. However, contracting parties cannot predict with certainty whether such a cost shock will occur when initially drawing up the contract. Due to this cost uncertainty, adverse selection, moral hazard, and the ‘winner’s curse’ could arise with auctions involving contracts. Similarly, McAfee and McMillan (1986) show that the first price auction may be feasible for awarding a contract at a low price but may also cause moral hazard of the winner ex-post. Decarolis (2018) also provides a stylized model in which a bidder gambles on the final cost of a project. In his model, the final cost is the sum of a bidder’s private cost and an added unforeseeable cost. By submitting a very low bid, an inefficient bidder can be awarded a project even if its private cost is higher than competitors. Once after the project begins, the added cost realizes as high or low. On the upside, the added cost becomes low and hence the inefficient bidder gets some return. However, on the down side, the added cost is sufficiently high that the bidder is not affordable to complete the project. In this case, the bidder will make a default on the project and hence loses nothing. In the end, inefficient bidders are awarded and could commit ex-post moral hazard under the first price auction.

In order to mitigate the aforementioned problem of an overly low bid being ‘too good to be true’ in a first price auction, one of two approaches can be utilized. The first approach is to augment a simple first price auction with ex-post screening. After selecting the lowest bidder as a preferred bidder, the procurer can determine by screening whether the bidder can sufficiently fulfill its obligation. The second approach is to replace the first price auction with an average price auction. Decarolis (2018) and Conley and Decarolis (2016) find that winning bids under average price auctions are generally higher than those under a first price auction, as bidders could avoid price competition in order to win under the average price auction. If they lower their bids, the chance of winning also decreases. Alternatively, if they match what they believe others will bid, the odds that they will win improve. These findings suggest that the usual ex-post problem of the first price auction is less likely to occur in the average price auction given that the winning bidder is repaid more and is therefore in a better position to overcome an ex-post cost shock.

One may also consider a hybrid of the previous two approaches – the average price auction augmented by screening. The comprehensive evaluation method introduced on May 1, 2016 in the Local Public Contracts Law in Korea is such an example. However, according to Decarolis (2014), average price auctions with screening are economically dominated by first price auctions with screening. As a screening process exists under both auction formats, there is not much of a difference with regard to the ex-post problem. Therefore, the average price auction with screening worsens the financial burden of the procurer by curbing bid price competition without leading to any improvement in the ex-post performance, as opposed to the first price auction with screening.

Although the ex-post performance can be evaluated by a number of performance measures, theoretical and empirical studies in the literature emphasize the importance of cost overruns as one of the leading performance measures. The more the cost of construction increases after the initial contract period, the poorer the ex-post performance becomes. In addition, frequent renegotiations after the initial contract could also indicate poor ex-post performance. If a construction contract changes frequently, it implies that the winning bidder is not very adaptable to unforeseen changes in the construction environment, which means a lack of capability to fulfill its obligations. The government of Korea also considered frequent changes in contracts after the initial contract is signed as an example of an ex-post problem.

Based on this theoretical prediction in the comparison of the average price auction with screening and the first price auction with screening and based on the discussion of the ex-post performance measures, I can formulate the following two testable hypotheses.

IV. Empirical Analysis: Price Performance

I utilize a contract-level cross-sectional dataset containing information on winning bids for 985 construction contracts procured by local governments or local-government-owned enterprises through standard competitive bidding processes during the period between May 2013 and December 2018. This period is roughly equal to the three-year window of May 1, 2016, when the new comprehensive evaluation method was introduced. The raw data is obtained from the Procurement Information Open Portal, which is operated by the Korea Public Procurement Service.2

I restrict the dataset so that it only contains construction contracts for which the ‘estimated price’ is no less than 10 billion Korean won. This is done because 10B KRW has long been considered in practice to be a threshold that distinguishes between normal-scale and small-scale construction projects. The Local Public Contracts Law also followed this convention and therefore applied a standard auction format to every construction contract whose estimated price exceeds 10B KRW but applied only a largely simplified version of the standard format to every other construction contract whose estimated price did not reach 10B KRW. However, since May 1, 2016, the comprehensive evaluation method has replaced the aforementioned standard method for any contract whose estimated price exceeds 30 billion KRW, while the existing standard method continues to be applied to every contract whose estimated price lies between 10B and 30B KRW. In this sense, I consider contracts higher than 30B KRW as a ‘treatment’ group while other contracts in the range of 10B to 30B KRW are the ‘control’ group in the following analysis.

However, it is also important to note that contracts between 10B and 30B KRW in fact form a reasonably good but not an ideal control group given that until May 1, 2016, one version of the first price auction with screening was applied to this group while another version of the first price auction with screening was applied to contracts exceeding 30B KRW. The two versions are similar in the sense that the lowest bidder is the first to be screened, but if it fails to pass the screening process, the second lowest bidder is then screened. However, in the first version, the procurement authority assesses the ability to conduct the assigned construction contract (i.e., ability screening) in the screening process, whereas in the second version, the authority assesses the appropriateness of the winning bid (i.e., bid screening) during the screening process. In the following, I provide a detailed explanation of this second version; some bidders may offer bids that are too high or too low such that they may be unable to properly conduct the assigned construction work at such low bids. For instance, a construction company may submit an unreasonably bid so low that it cannot even cover their costs. If such a company wins the auction, obviously it cannot properly conduct the required work. To prevent such an event from taking place, a procurement authority initially checks whether the winning bid lies in a predetermined safe interval. If the bid is outside of the safe interval, it is a signal that the bid is too high or too low. In such a case, the winning bidder must explain in front of a screening committee that it is still able to fulfill the required work even with this seemingly unreliable bid. The winning bidder is finally awarded the contract only if the committee determines that the explanation is reasonable enough. The gist is that ability screening in the first version and bid screening in the second share an identical goal: preventing an unreliable bidder from winning an auction by offering an unreasonably low bid. In addition, the two versions are slightly different in that the first version places a lower bound on bids, meaning that bidders are excluded if they bid lower than this bound. Although the second version does not utilize the same lower bound, the safe intervals used in the bid screening process effectively install similar lower bounds. I do not argue that the first and second versions are perfectly identical auction formats, but I assume that the two versions are at least reasonably similar. Likewise, the literature does not make an explicit distinction between first price auctions with ability screening and first price auctions with bid screening (see Bajari et al., 2014; Decarolis, 2014). The following empirical results based on the price and non-price effects of the comprehensive evaluation method obtained from the difference-in-differences estimation should be understood in this context.

Table 1 summarizes descriptive statistics of the variables of interest. The winning bid is the ratio of a winner’s bid price to the reserve price. The average winning bid of 985 construction contracts is 80.28%. A dummy variable D2016.5 equals one if the given contract is procured after May 1, 2016, and zero otherwise. Another dummy variable D30B equals one if the estimated price of the given contract is greater than or equal to 30B KRW. Of 985 contracts, 51.9% were procured after May 1, 2016, and 15.2% were contracts for which the estimated price exceeds 30B KRW. The estimated price and the reserve price are similar but not identical. The former is publicly disclosed before bidding while the latter is kept secret until all sealed bids are opened. The number of bidders is on average 182, which implies that bidding is usually very competitive.

TABLE 1

DESCRIPTIVE STATISTICS

Figure 1 compares the (within-quarter average) winning bids of the treatment group and control group. The control group consists of contracts whose estimated price is 10B to 30B KRW for which the comprehensive evaluation method is not applied. The treatment group contains contracts whose estimated price is greater than or equal to 30B KRW for which the new method has been applied since May 1, 2016. Before its experimental application, the winning bid of the treatment group is approximately 75% (as a percentage of the reserve price), which is approximately 6% below the winning bid of the control group. However, with the experimental application as described above, the winning bid of the treatment group suddenly rises above the winning bid of the control group.

FIGURE 1.

WINNING BID BEFORE AND AFTER THE COMPREHENSIVE EVALUATION METHOD

Note: The red real line represents the average winning bid (as a percentage of the reserve price) of construction contracts whose estimated price exceeds or equal to 30 billion KRW. The average is obtained by taking the average of all winning bids for contracts procured in the same quarter. The blue dotted line represents the average winning bid of contracts whose estimated price lies between 10B and 30B KRW.

I shall estimate the following empirical model:

where winning_bid_it is the winner’s bid price for contract i procured at time t as a percentage of reserve price. is a dummy variable that equals 1 if and only if the bidding process for contract i begins on or after May 1, 2016. is a dummy variable that equals 1 if and only if the estimated price for contract i is equal to or higher than 30 billion KRW.

X_it is a vector of control variables consisting of the logarithm of the estimated price, the number of bidders, the type of bidding competition, the type of auctioneer, the type of winner, a dummy variable for an urgent offer, and a dummy variable for a compulsory bid consortium.

There are three types of bidding competition: ‘general,’ ‘restricted,’ and ‘nominated.’ ‘General’ means there is virtually no entry regulation. ‘Restricted’ means only qualified bidders – whose headquarters are located in the same area where the construction will be taken place or who have enough experience in similar construction projects – can submit bids. ‘Nominated’ means only a few bidders designated by the auctioneer can participate in the bidding process, although the sample size of contracts subject to nominated bidding competition is very small.

There are two types of auctioneers: local governments and local-government-owned enterprises. Moreover, there are three types of winning bidders: major companies, middle market enterprises, and SMEs.

The dummy variable for an urgent offer equals 1 if and only if the auctioneer fast-tracks the bidding process when the related construction is urgent. The dummy variable for a compulsory bid consortium equals 1 if and only if the Local Public Contracts Law requires bidders to form a consortium and invite at least one local SME to be a member of the consortium.

In addition, I consider a number of fixed effects, including ‘region,’ ‘construction type,’ and ‘year.’ ‘Region’ is a municipal-level variable, such as Seoul, Busan, or Gyeongi Province. ‘Construction type’ examples include civil engineering, architecture, and plants, among others.

The coefficient of interest is β₁ , which measures the effect of the comprehensive evaluation method on the winning bid. Here, I examine this effect via a difference-in-differences estimation. Table 2 illustrates the difference-in-differences estimation. Recall that the new scoring method has applied contracts whose estimated prices are equal to or exceed 30B KRW since May 1, 2016. Therefore, the estimated effect of the difference in time on the winning bid for the treatment group is β₁ + β₂ , while the estimated effect of the same difference in time on the winning bid for the control group is β₂. Thus, difference-in-differences β₁ measures the effect of the new method on the winning bid.

TABLE 2

IDENTIFICATION BY DIFFERENCE-IN-DIFFERENCE

Table 3 shows the estimation results of empirical model (1) for a number of model specifications. The first three specifications are baseline specifications. The first specification ‘Simple’ is the simplest possible difference-in-differences model specification. The second specification ‘FE’ adds to ‘Simple’ the region fixed effect and construction fixed effect as additional control variables. The third specification ‘All’ adds to ‘FE’ some additional control variables including the number of bidders, the type of bidder, and the type of auctioneer.

TABLE 3

REGRESSION RESULTS ON WINNING BID 1

Note: 1) The dependent variable is the winning bid, 2) The type of bidding competition is either ‘general’ or ‘restricted.’ The estimated coefficient of ‘restricted’ is calculated relative to the estimated coefficient of ‘general.’ The type of auctioneer is either ‘local government’ or ‘local-government-owned enterprise (i.e., SOE).’ The estimated coefficient of auctioneer-SOE is calculated relative to the estimated coefficient of local government. The type of bidder is a major company, a middle market enterprise (MME), or a SME. The estimated coefficients of MMEs and SMEs are calculated relative to the estimated coefficient of a major company, 3) The standard error is the Huber-White-Sandwich robust standard error. *, **, and *** represent the 10%, 5%, and 1% level of significance, respectively.

In any of the first three baseline specifications, the estimated coefficient for the interaction term of the two dummy variables and is positive and significant. That is, the introduction of the comprehensive evaluation method, which is a version of the average price auction with screening, is associated with an increase in the winning bid. This positive association is statistically significant at the 1% level and is also economically significant as the magnitude is approximately 7.5%, which is meaningfully large especially in the current low-interest-rate environment. Recall from Table 1 that the average reserve price of construction contracts contained in the dataset is 24.8 billion KRW, meaning that a local government or a local-government-owned enterprise should pay on average 1.86 billion KRW more for each construction contract due to the new method. Therefore, the introduction of the new auction scoring rule significantly increases the fiscal burden borne by local governmental bodies.

The signs of the coefficient estimates of other control variables are also consistent with intuition. The winner is paid the less as the number of competing bidders grows simply because the bidding is then more competitive. In contrast, the winner is paid the more if it is not a major company but a middle-market enterprise or a SME. This presumably stems from the fact that the Local Public Contracts Law requires auctioneers to provide some privileges to non-major companies.

In the next three specifications, I conduct a number of robustness checks. In the first three baseline specifications, I do not control for year fixed effect (i.e., time) or the estimated price (i.e., scale). This is done because the two dummy variables and already control for a critical time and a critical level of scale, respectively. However, one can argue that non-critical times and scales should also be controlled. In response, I add the year fixed effect and/or the logarithm of the estimated price as additional control variables. The main result is robust to these changes in control variables. Interestingly, the winning bid turns out to be negatively associated with the scale variable. This presumably occurs because bidding is generally more competitive if the estimated price is the larger and hence the gain from winning is greater.

The difference-in-differences estimation will be the more accurate, the more similar is the control group to the treatment group. In this regard, as a robustness check, I shall confine my attention to a certain type of construction contract. Of 843 construction contracts containing information about the type of construction, 67% are civil engineering construction projects, 25% are architecture construction projects, and only less than 10% are other types of construction projects.

Table 4 shows the results when the estimation is conducted separately for civil engineering and architecture constructions. For civil engineering, which is the most important type of construction in terms of its share out of all construction projects, the comprehensive evaluation method is found to be positively associated with the winning bid, and this outcome is statistically significant at the 1% level for all five specifications. The magnitude is approximately 8%, which is roughly identical to the magnitude of 7.5% calculated for all construction projects. For architecture, which is the second most important type of construction, the new method is still positively associated with the winning bid at a strongly statistically significant level. The economic significance is even stronger as the estimated effect is 11.2% of the reserve price. However, it is important to note that this magnitude is obtained without controlling for a number of relevant variables due to the lack of sample of architecture construction contracts.

TABLE 4

REGRESSION RESULTS ON WINNING BID 2

Note: 1) The dependent variable is the winning bid, 2) The first five model specifications consider only civil engineering construction contracts and the last model specification considers only architecture construction contracts, 3) The row for ‘Control variable’ is marked ‘Yes’ if the number of bidders, the type of competition, the dummy variable for urgent offer, the dummy variable for compulsory consortium, the type of auctioneer, and the type of winner are all controlled. The row is marked as ‘No’ if all of these variables are not controlled, 4) The standard error is the Huber-White-Sandwich robust standard error. *, **, and *** represent the 10%, 5%, and 1% level of significance, respectively.

In this paper, I interpret an increase in the winning bid as a rise in the fiscal pressure on local governmental bodies. However, this interpretation is invalid if an auctioneer lowers the reserve price in response to the new auction scoring rule, simply because the auctioneer’s payment to the winner equals the product of the winning bid and the reserve price. In fact, some industry practitioners argue that local governments will cut reserve prices in order to minimize foreseeable fiscal shocks. In this regard, I examine how the reserve price changes in response to the introduction of the comprehensive evaluation method. Table 5 shows the estimation results, where the dependent variable is not the winning bid but the logarithm of the reserve price. These results show that the reserve price is not associated with the new auction scoring rule, which means a rise in the winning bid can be interpreted as an increase in the financial burden on local governments.

TABLE 5

REGRESSION RESULTS ON RESERVE PRICES

Note: 1) The dependent variable is the logarithm of the reserve price, 2) The row for ‘Control variable’ is marked ‘Yes’ if the number of bidders, the type of competition, the dummy variable for an urgent offer, the dummy variable for a compulsory consortium, the type of auctioneer, and the type of winner are all controlled. The row is marked as ‘No’ if all of these variables are not controlled, 3) The standard error is the Huber-White-Sandwich robust standard error. *, **, and *** represent the 10%, 5%, and 1% level of significance, respectively.

V. Empirical Analysis: Ex-Post Performance

The first dataset used in the previous section contains information only about what happens on the bid opening and contract signing dates, but it does not have information about changes in contracts. Fortunately, the Procurement Information Open Portal also provides a separate dataset based on which one can trace out how an initial contract evolves over time. However, this second dataset is not ready to be used in an empirical analysis given that renegotiated contracts are not properly linked to the corresponding initial contract. I have matched each set of renegotiated contracts to their corresponding initial contract manually and generated a new contract-level cross-sectional dataset that contains the number of contract renegotiations and the amounts of cost overruns for roughly 1,337 construction contracts that were initially procured during the period between August 1, 2012, and June 30, 2020, which is a three-year and nine-month event window at around May 1, 2016. I merge this second dataset with the first dataset. However, the information pertaining to the number of bidders, urgent offers, and the type of bidder (i.e., whether it is a large company, middle-market enterprise, or a SME) cannot be utilized because there is no related identifier in the second dataset.

Table 6 provides summary statistics of the variables reflecting changes in contracts. Even if the two parties agree and sign a contract, they could afterward renegotiate over the initial contract for some reason. The number of changes (1Y) is the number of official revisions made in one year since the initial contract date. These changes are recorded officially and hence the related new contract overrides the initial one. A typical contract changes on average 0.337 times, 1.271 times, and 2.967 times after six months, one year, and two year from the initial contract date, respectively. The maximum number of changes made for one year is nine. Cost overrun (1Y) refers to the cumulative changes in the cost of construction which occurred one year after the initial contract date. Cost overruns are measured as a percentage of the reserve price. (The reserve price is set only once at the bid opening time.) Thus, the cost of a construction contract increases on average by 0.386% of the reserve price one year since the initial contract date and by 2.823% for two years. Although the average of the cost overrun (1Y) is relatively small, this does not mean that cost overruns are not a significant issue in public procurement. The standard deviation of the cost overrun (1Y) is 8.055, which is approximately 20 times larger than its mean. This implies that once a cost overrun arises, its magnitude is meaningfully large. Consistent with this notion, the minimum and maximum of the cost overrun (1Y) in absolute terms turn out to be very large.

TABLE 6

DESCRIPTIVE STATISTICS

Note: The dataset contains information about changes in contractual terms which occurred up to February 28, 2020. Therefore, it is possible to trace out changes in contractual terms six months from an initial contract date only if the initial contract date is no later than August 30, 2019. For this reason, the numbers of changes (6M) and cost overruns (6M) are calculated for each contract whose initial contract date is between January 1, 2013, and August 30, 2019, which is a three-year and four-month window relative to May 1 2016. Similarly, the numbers of changes (1Y) and cost overruns (1Y) are calculated for each contract whose initial contract date is between July 1, 2013, and February 28, 2019, which is a two-year and ten-month window relative to May 1 2016. Additionally, the numbers of changes (2Y) and cost overruns (2Y) are calculated for each contract whose initial contract date is between July 1, 2014, and February 28, 2018, representing a one-year and ten-month window relative to May 1, 2016. However, all of the regression results presented in this section are robust to this subsampling.

In the following paragraphs, I examine whether the newly introduced comprehensive evaluation method improves the reliability of a construction contract. To this end, I use two measurements of reliability: the number of changes and the cost overruns. If these variables decrease according to the new method, it suggests an improvement in the reliability.

A. Number of Changes

Figure 2 illustrates the movement of the number of changes (1Y) before and after May 1, 2016, when the comprehensive evaluation method was first introduced for construction contracts whose estimated prices are equal to or exceed 30B KRW. The number of changes in this treatment group (i.e., the red solid line) does not appear to be meaningfully different from the number of changes in the control group (i.e., the blue dotted line) before and after the critical time. This suggests that the new auction format does not have an effect on the number of changes in contracts.

FIGURE 2.

NUMBER OF CHANGES IN CONTRACTS BEFORE AND AFTER THE COMPREHENSIVE EVALUATION METHOD

Note: The two lines represent the average number of changes in contracts one year since the initial contract date. The average is obtained by taking the average of all numbers of changes for contracts procured in one specific quarter. The solid real line represents the average number of changes in contracts whose estimated price is equal to or greater than 30 billion KRW. The blue dotted line represents the average number of changes in contracts whose estimated price is in the range of 10B to 30B KRW.

In order to examine the effect of the new auction format on the reliability of procurement contracting, I estimate the following ordered Probit model, as the dependent variable is the number of changes (1Y), which is an ordered variable for which the minimum and maximum orders are 0 and 11, respectively. The coefficient of interest is β₁ , which measures the difference-in-differences associated with the new auction format. The vector of the control variables X_it consists of the PPI (producer price index) growth, the logarithm of the estimated price, the type of bidding competition, the type of auctioneer, and a dummy variable for a compulsory bid consortium.

Table 7 shows the estimation result. It turns out that the interaction of the two dummy variables and has no statistically significant association with the number of changes (1Y) in any of the six model specifications. This result suggests that the comprehensive evaluation method is ineffective with regard to mitigating frequent changes in construction contracts, which is in stark contrast to what the government of Korea expected when it initially adopted the new method. The number of changes, however, is positively associated with the PPI growth. This presumably occurs because the Local Public Contracts Law allows renegotiation if the PPI growth for the first three months after the initial contract date is high enough. Moreover, the dependent variable is negatively associated with the variable ‘Restricted.’ A plausible explanation here is that the initial contract is more favorable to the winning bidder if the competition is restricted as opposed to unrestricted and, hence, the winner is less likely to require renegotiation.

TABLE 7

ORDERED PROBIT REGRESSION RESULTS OF NUMBER OF CHANGES 1

Note: 1) The dependent variable is the number of changes in contracts one year since the initial contract date, 2) PPI growth is the growth rate of the producer price index during the three months after the initial contract date, 3) The type of bidding competition is either ‘general’ or ‘restricted.’ The estimated coefficient of ‘restricted’ is calculated relative to the estimated coefficient of ‘general.’ The type of auctioneer is either ‘local government’ or ‘localgovernment- owned enterprise (i.e., SOE).’ The estimated coefficient of auctioneer-SOE is calculated relative to the estimated coefficient of local government, 4) The standard error is the Huber-White-Sandwich robust standard error. *, **, and *** represent the 10%, 5%, and 1% level of significance, respectively.

Below, I conduct a number of robustness checks. In the first set of robustness checks, I control for the contract size. In the baseline estimation, I place relatively small contracts for which the estimated price is 30B KRW and relatively large contracts for which the estimated price is 200B KRW into the same treatment group. Although the same set of procurement rules is applied to these two contracts, one may expect that the characteristics of these two types are quite different. As a response, I focus on a subsample which contains contracts for which the estimated price is in a small range around at 30B KRW. The first three columns in Table 8 suggest that the comprehensive evaluation method is still ineffective in reducing the number of changes in contracts.

In the second set of robustness checks, I trace out changes in contracts for different time intervals. One can argue that one year is not long enough to trace out changes in contracts fully. Others may be interested in examining changes in a shorter time interval than one year, as incomplete contracts could be renegotiated early after the initial contract date. In response, I consider the numbers of changes in contracts six months, one year, and two years from the initial contract date. The last three columns in Table 8 show that the coefficient of interest is still insignificant.

TABLE 8

ORDERED PROBIT REGRESSION RESULTS OF NUMBER OF CHANGES 2

Note: 1) For the first three specifications, the dependent variable is the number of changes in contracts one year since the initial contract date. The first specification considers contracts whose estimated prices are within 10B KRW relative to 30B KRW. That is, contracts with estimated prices from 20B to 40B KRW are considered. The second specification considers contracts with estimated prices within 15B KRW relative to 30B KRW. The third specification considers contracts with estimated prices within 20B KRW relative to 30B KRW, 2) For the next three specifications, the dependent variable is the number of changes in contracts six months, one year, and two years since the initial contract date, respectively, 3) The row for ‘Control variables’ is marked ‘Yes’ if PPI growth, the type of competition, the dummy variable for compulsory consortium, and the type of auctioneer are all controlled. The row is marked ‘No’ if all of these variables are not controlled, 4) The standard error is the Huber-White-Sandwich robust standard error. *, **, and *** represent the 10%, 5%, the 1% level of significance, respectively.

In the third set of robustness checks, I consider two most important types of construction, civil engineering and architecture. Table 9 suggests that the comprehensive evaluation method could not effectively cause any difference in the number of changes in either type of construction contract.

TABLE 9

ORDERED PROBIT REGRESSION RESULTS OF NUMBER OF CHANGES 3

Note: 1) For the first three specifications, I consider only civil engineering construction contracts. The dependent variable is the number of changes in contracts six months, one year, and two years since the initial contract date, respectively, 2) For the next two specifications, I consider only architecture construction contracts. The dependent variable is the number of changes in contracts one year and two years from the initial contract date, respectively, 3) The row for ‘Control variables’ is marked ‘Yes’ if PPI growth, the type of competition, the dummy variable for compulsory consortium, and the type of auctioneer are all controlled. The row is marked ‘No’ if all of these variables are not controlled, 4) The standard error is the Huber-White-Sandwich robust standard error. *, **, and *** represent the 10%, 5%, and 1% level of significance, respectively.

B. Cost Overrun

Cost overrun is a phenomenon during which the cost of construction increases unexpectedly after an initial contract is signed. A cost overrun is more likely to arise if the winning bidder is not capable of conducting the required work or if the winner does not put much effort into completing its mission. The government of Korea predicted that the comprehensive evaluation method could reduce the cost overrun problem. To ascertain whether this expectation is realized, I estimate the following empirical model (3). The dependent variable is the cost overrun as a percentage of the reserve price. The vector of control variables X_it is identical to that used in empirical model (2).

Table 10 shows the estimation results for a number of model specifications. In every such specification, it turns out that the new auction format is not statistically significantly associated with the cost overrun for the period of one year since the initial contract date. That is, this suggests that using the new method is futile with regard to reducing the cost overrun problem.

TABLE 10

REGRESSION RESULTS OF COST OVERRUN 1

Note: 1) The dependent variable is the cost overrun as a percentage of the reserve price one year since the initial contract date, 2) PPI growth is the growth rate of the producer price index during the three months since the initial contract date. The type of bidding competition is either ‘general’ or ‘restricted.’ The estimated coefficient of ‘restricted’ is calculated relative to the estimated coefficient of ‘general.’ The type of auctioneer is either ‘local government’ or ‘local-government-owned enterprise (i.e., SOE).’ The estimated coefficient of auctioneer-SOE is calculated relative to the estimated coefficient of local government, 3) The standard error is the Huber-White-Sandwich robust standard error. *, **, and *** represent the 10%, 5%, and 1% level of significance, respectively.

I also check the robustness of the baseline estimation result. In the first three columns in Table 11, I focus on a subsample of contracts whose estimated prices lie in small ranges around at 30B KRW. In the next three columns, I consider not the just one-year change in construction cost but also the six-month and two-year changes. In addition, I confine my attention to the two most important types of construction, i.e., civil engineering and architecture projects, as presented in Table 12. The estimation results for all of these specifications suggest that the new auction method is ineffective with regard to its ability to resolve the cost overrun problem.

TABLE 11

REGRESSION RESULTS OF COST OVERRUN 2

Note: 1) For the first three specifications, the dependent variable is the cost overrun as a percentage of the reserve price one year since the initial contract date. The first specification considers contracts whose estimated prices are within 10B KRW relative to 30B KRW. That is, contracts with estimated prices from 20B to 40B KRW are considered. The second specification considers contracts with estimated prices within 15B KRW relative to 30B KRW. The third specification considers contracts with estimated prices within 20B KRW relative to 30B KRW, 2) For the next three specifications, the dependent variable is cost overrun as a percentage of the reserve price six months, one year, and two years since the initial contract date, respectively, 3) The row for ‘Control variables’ is marked ‘Yes’ if the PPI growth, the type of competition, the dummy variable for compulsory consortium, and the type of auctioneer are all controlled. The row is marked ‘No’ if all of these variables are not controlled, 4) The standard error is the Huber-White-Sandwich robust standard error. *, **, and *** represent the 10%, 5%, and 1% level of significance, respectively.

TABLE 12

REGRESSION RESULTS OF COST OVERRUN 3

Note: 1) For the first three specifications, I consider only civil engineering construction contracts. The dependent variable is the cost overrun as a percentage of the reserve price six months, one year, and two year since the initial contract date, respectively. For the next two specifications, I consider only architecture construction contracts. The dependent variable is cost overrun as a percentage of the reserve price one year and two years from the initial contract date, respectively, 2) The row for ‘Control variables’ is marked ‘Yes’ if PPI growth, the type of competition, the dummy variable for compulsory consortium, and the type of auctioneer are all controlled. The row is marked ‘No’ if all of these variables are not controlled, 3) The standard error is the Huber-White-Sandwich robust standard error. *, **, and *** represent the 10%, 5%, and 1% level of significance, respectively.

Table 6 shows that the average of the cost overruns for one year after the initial contract date is as low as 0.386% of the reserve price, whereas its standard deviation is as large as 8.055%. In fact, the cost overrun problem does not occur frequently. It arises only for 42% of construction contracts contained in the dataset. However, once a cost overrun takes place, its magnitude is meaningfully large. Moreover, some construction contracts contain very large cost overruns. In this regard, it may be useful to determine how the new auction format affects the upper and lower quantiles of the cost overrun rather than its average. Therefore, I consider a quantile regression model in which the dependent variable is the 95th, 80th, 20th, and 5th quantiles of the cost overrun while the independent variables are identical to those in model (3). Table 13 shows the result of this quantile regression. It was found that the comprehensive evaluation method is not statistically significantly associated with any of the four upper and lower quantiles.

TABLE 13

QUANTILE REGRESSION RESULTS OF COST OVERRUNS

Note: 1) The dependent variable is the 95th, 80th, 20th, and 5th quantiles of the cost overrun as a percentage of the reserve price for one year, 2) The row for ‘Control variables’ is marked ‘Yes’ if PPI growth, the type of competition, the dummy variable for compulsory consortium, and the type of auctioneer are all controlled. The row is marked ‘No’ if all of these variables are not controlled, 3) The Standard error is the Huber-White-Sandwich robust standard error. *, **, and *** represent the 10%, 5%, and 1% level of significance, respectively.

VI. Empirical Analysis: Simple Regression

Hitherto I examined the effect of the newly adopted comprehensive evaluation method on price and ex-post performance outcomes based on the difference-in-differences framework in which contracts of 10B to 30B KRW are used as a control group. Until the adoption of the new method, the first price auction with ability screening was applied to this control group, while the first price auction with bid screening was applied to the treatment group. In this paper, I consider that the first price auctions with bid screening and ability screening are slightly different but very similar in the sense that both are examples of the first price auction with screening. However, one can argue that they are different methods and hence the difference-in-differences framework is less meaningful.

In this section, as a robustness check, I estimate a simple regression model (4) below in which the independent variable of interest is CEM_it , which equals 1 if the comprehensive evaluation method is used to procure contract i but 0 if another auction method is used. That is, if the CEM_it dummy is 1, it means that the average price auction with screening is applied in procurement. However, if the CEM _it dummy is 0, one of many versions of the first price auction with screening is applied in procurement. The dependent variable y_it is the winning bid, number of changes, or cost overruns.

See Table 14. The estimation results suggest that the comprehensive evaluation method increases the winning bid and has no effect on the number of changes or cost overruns. These results are in line with what I find in the previous sections based on the DID framework. Although the magnitude of the estimated price effect based on the simple regression framework is slightly smaller than that based on the DID framework, the directions are equivalent.

TABLE 14

SIMPLE REGRESSION RESULTS

Note: 1) The dependent variables in the models 1, 2, and 3 are the winning bid, number of changes, and cost overruns, respectively, 2) The independent variable of interest is CEM, which is 1 if the comprehensive evaluation method is used and 0 otherwise, 3) Models 1 and 3 are simple OLS models, while model 2 is an ordered Probit model, 4) The control variables used in Model 1 are Number of bidders, Restricted, Urgent offers, Compulsory consortium, Auctioneer-SOE, Winner-MME, Winner-SME, and Log estimated price (see Table 3), 5) The control variables used in the Model 1 and 2 are PPI growth, Restricted, Compulsory consortium, Auctioneer-SOE and Log estimated price (see Tables 7 and 10), 6) Standard error is the Huber-White-Sandwich robust standard error. *, **, and *** represent the 10%, 5%, and 1% level of significance, respectively.

VII. Conclusion and Policy Implications

The main findings of this paper are as follows. First, the replacement of the existing first price auction with screening by an average price auction with screening results in an increase in winning bids by about 7.5% of the reserve price. Secondly, despite this reform on the auction format, frequent changes in contracts and cost overruns are not reduced. These findings imply that the reform was unsuccessful as it increases the fiscal burden on local governmental bodies without making any improvement in the ex-post performance of winning bidders. These results are consistent with the theoretical literature, which shows that average price auctions with screening are dominated by first price auctions with screening.

There is an important caveat to consider when interpreting the empirical results here. The results are obtained by considering contracts exceeding 30B KRW as a treatment group and contracts from 10B to 30B KRW as a control group, as first price auctions with screening were applied to both groups until May 1, 2016. However, contracts in the range of 10B to 30B KRW represent a reasonably good but not ideal control group as a first price auction with bid screening was applied to the treatment group while a first price auction with ability screening was applied to the control group until May 1, 2016. Bid screening and ability screening are not identical procedures and the control group for this reason is not ideal. However, the two screening methods are similar in that during the bid screening process, the procurement authority examines the relationship between the winning bid and the ability of the winner. That is, both bid screening and ability screening are conducted with the same objective of preventing incompetent companies from winning an auction by offering too low bid, and in this regard I believe that the control group is reasonably good. In addition, the two first price auction versions are slightly different in that the one with ability screening utilizes a single lower bound for admissible bids while the other with bid screening effectively places a number of similar lower bounds. The empirical results on the price and non-price effects of the comprehensive evaluation method obtained in this paper using the difference-in-differences estimation should be understood in this context.

The theoretical literature and this paper’s empirical findings imply that the comprehensive evaluation method should be repealed and replaced by a version of the first price auction with screening. However, this paper does not argue that we must return to the existing version of the first price auction with screening, as this method has its own serious problems. Related to this, Kim (2012) observes that under this existing method, almost every qualified bidder receives the maximum score on non-price qualities given that the existing screening process is not enough effective with regard to its ability to distinguish between efficient bidders and inefficient bidders. Therefore, this paper proposes the adoption of a new version of the first price auction with screening in which the screening process is intensified. Determining how to intensify the screening process could be an important future research agenda.