Interesting legislative proposal to make procurement of AI conditional on external checks

Procurement is progressively put in the position of regulating what types of artificial intelligence (AI) are deployed by the public sector (ie taking a gatekeeping function; see here and here). This implies that the procurement function should be able to verify that the intended AI (and its use/foreseeable misuse) will not cause harms—or, where harms are unavoidable, come up with a system to weigh, and if appropriate/possible manage, that risk. I am currently trying to understand the governance implications of this emerging gatekeeping role to assess whether procurement is best placed to carry it out.

In the context of this reflection, I found a very useful recent paper: M E Kaminski, ‘Regulating the Risks of AI’ (2023) 103 Boston University Law Review forthcoming. In addition to providing a useful critique of the treatment of AI harms as risk and of the implications in terms of the regulatory baggage that (different types of) risk regulation implies, Kaminski provides an overview of a very interesting legislative proposal: Washington State’s Bill SB 5116.

Bill SB 5116 is a proposal for new legislation ‘establishing guidelines for government procurement and use of automated decision systems in order to protect consumers, improve transparency, and create more market predictability'. The governance approach underpinning the Bill is interesting in two respects.

First, the Bill includes a ban on certain uses of AI in the public sector. As Kaminski summarises: ‘Sec. 4 of SB 5116 bans public agencies from engaging in (1) the use of an automated decision system that discriminates, (2) the use of an “automated final decision system” to “make a decision impacting the constitutional or legal rights… of any Washington resident” (3) the use of an “automated final decision system…to deploy or trigger any weapon;” (4) the installation in certain public places of equipment that enables AI-enabled profiling, (5) the use of AI-enabled profiling “to make decisions that produce legal effects or similarly significant effects concerning individuals’ (at 66, fn 398).

Second, the Bill subjects the procurement of the AI to approval by the director of the office of the chief information officer. As Kaminski clarifies: ‘The bill’s assessment process is thus more like a licensing scheme than many proposed impact assessments in that it envisions a central regulator serving a gatekeeping function (albeit probably not an intensive one, and not over private companies, which aren’t covered by the bill at all). In fact, the bill is more protective than the GDPR in that the state CIO must make the algorithmic accountability report public and invite public comment before approving it’ (at 66, references omitted).

What the Bill does, then, is to displace the gatekeeping role from the procurement function itself to the data protection regulator. It also sets the specific substantive criteria the regulator has to apply in deciding whether to authorise the procurement of the AI.

Without getting into the detail of the Washington Bill, this governance approach seems to have two main strengths over the current emerging model of procurement self-regulation of the gatekeeping role (in the EU).

First, it facilitates a standardisation of the substantive criteria to be applied in assessing the potential harms resulting from AI adoption in the public sector, with a concentration on the specific characteristics of decision-making in this context. Importantly, it creates a clear area of illegality. Some of it is in line with eg the prohibition of certain AI uses in the Draft EU AI Act (profiling), or in the GDPR (prohibition of solely automated individual-decision making, including profiling — although it may go beyond it). Moreover, such an approach would allow for an expansion of prohibited uses in the specific context of the public sector, which the EU AI Act mostly fails to tackle (see here). It would also allow for the specification of constraints applicable to the use of AI by the public sector, such as a heightened obligation to provide reasons (see M Fink & M Finck, ‘Reasoned A(I)dministration: Explanation Requirements in EU Law and the Automation of Public Administration‘ (2022) 47(3) European Law Review 376-392).

Second, it introduces an element of external (independent) verification of the assessment of potential AI harms. I think this is a crucial governance point because most proposals relying on the internal (self) assessment by the procurement team fail to consider the extent to which such approach ensures (a) adequate resourcing (eg specialism and experience in the type of assessment) and (b) sufficient objectivity in the assessment. On the second point, with procurement teams often being told to ‘just go and procure what is needed’, moving to a position of gatekeeper or controller could be too big an ask (depending on institutional aspects that require closer consideration). Moreover, this would be different from other aspects of gatekeeping that procurement has progressively been asked to carry out (also excessively, in my view: see here).

When the procurement function is asked to screen for eg potential contractors’ social or environmental compliance track record, it is usually at arms’ length from those being reviewed (and the rules on conflict of interest are there to strengthen that position). Conversely, when the procurement function is asked to screen for the likely impact on citizens and/or users of public services of an initiative promoted by the operational part of the organisation to which it belongs, things are much more complicated.

That is why some systems (like the US FAR) create elements of separation between the procurement team and those in charge of reviewing eg competition issues (by means of the competition advocate). This is a model reflected in the Washington Bill’s approach to requiring external (even if within the public administration) verification and approval of the AI impact assessment. If procurement is to become a properly functioning gatekeeper of the adoption of AI by the public sector, this regulatory approach (ie having an ‘AI Harms Controller’) seems promising. Definitely a model worth thinking about for a little longer.

Cartels in public procurement: A short comment on Heimler's (2012) J Comp L & Econ 8(3): 1-14

Prof. Alberto Heimler has recently published the interesting piece 'Cartels in Public Procurement' (2012) J Comp L & Econ 8(3): 1-14 [available, but maybe for subscribers only, here]. In his paper, Prof. Heimler discusses the specific features of bid-rigging as a particularly stable instance of collusion and presents some proposals to reduce the administrative burden and increase the incentives for procurement officials to track potential instances of bid rigging and to report them to the competition authorities, even on the basis of a mere suspicion (ie without need to provide full proof of the infringement). 

The abstract of his piece shows these general ideas:

Public procurement markets differ from all others because quantities do not adjust with prices but are fixed by the bidding authority. As a result, there is a high incentive for organizing cartels (where the price elasticity of demand is zero below the base price) that are quite stable because there are no lasting benefits for cheaters. In such circumstances, leniency programs are unlikely to help discovering cartels. Since all public procurement cartels operate through some form of bid rotation, public procurement officials have all the information necessary to discover them (although they have to collect evidence on a number of bids), contrary to what happens in normal markets where customers are not aware of the existence of a cartel. However, in order to promote reporting, the structure of incentives has to change. For example, the money saved from a cartel should at least, in part, remain with the administration that helped discover it and the reporting official should reap a career benefit. In any case, competition authorities should create a channel of communication with public purchasers so that the public purchasers would know that informing the competition authority on any suspicion at bid rigging is easy and does not require them to provide full proof.

This 'mainstream' description of his paper is perfectly in line with most economic and legal scholarship in this field and his work is an interesting reminder of the need to increase the liaison between public procurement and competition authorities, as well as to create a set of incentives (or a dedicated position) for public buyers to act as competition watchdogs of sorts or, more generally, as competition advocates [along the same lines, see A Sanchez Graells, Public Procurement and the EU Competition Rules (Oxford, Hart Publishing, 2011) 385-389]. Moreover, Prof. Heimler offers a couple of interesting insights that should be taken into consideration in the design of effective public procurement systems against bid rigging.

On the one hand, Prof. Heimler clearly indicates the diverging financial interests in bidding rings as opposed to 'general' cartels, which make leniency programs (potentially) less effective in this type of market settings:

Contrary to what happens in normal markets, bid-rigging cartels are much more stable. While in normal markets, quantities and prices are found simultaneously, in bidding markets, quantities are set by the organizer of the bid and the bidding is just used to find the lowest price associated with those quantities. Bid riggers know that by reducing prices (with respect of the agreed ones), they do not achieve any increase in the quantities sold. Rather, they just increase their profit at the expense of competitors and, most importantly, only for one bid. Once there is defection for one bid, the cheater knows (because of the transparency rules in public procurement) that he will be discovered and competition will prevail for all future bids. As a result of these characteristics, partly structural and partly rule-based, the incentive to cheat in bid rigging is much less pronounced than in normal markets (where cheating can be kept secret, at least for some time) (p. 7).

On the other hand, the level of transparency that is structurally implicit in public procurement settings makes it much easier for (properly trained) procurement officials to detect instances of bid rigging and to react:

Contrary to normal cartels, where the participating firms agree on prices or on territories so that customers face an information gap with respect to competitive prices, bid rigging in public procurement requires that the participating firms agree on the bid participation strategy (who wins and at what price; who will participate today; and who wins and who participates in future bids). As a result, bid riggers leave a lot of evidence on the strategies pursued that a well-trained public administration official could indeed identify. As a result, while a public procurement cartel is stable on the supply side, it could be discovered by due diligence on the demand side. This is the opposite of what happens with private market cartels (p. 12).

However, Prof. Heimler also includes a couple of final recommendations to make bid rigging more difficult that, in my opinion, would raise more issues than they would solve. Indeed, he proposes that:

There are also some very important procedural and legal steps that should be taken to make bid rigging much more difficult.

The first is to centralize purchases (or make sure that bids are not made artificially too small so that the construction of a large infrastructure project cannot be easily divided up among all the firms in the industry). This way, the information on the different bids can be found within the same organization so that any irregularity across different bids can be more easily identified. Furthermore, a centralized purchasing agency can organize bids of higher value (purchasing for a number of administrations) so that bids would be more infrequent and bid-rigging agreements would be more difficult to maintain.

Also, the rules that favor small firms in their participation in tenders, in which individually they would not be able to participate because of their small size, should be made much more rigorous. In particular, temporary consortia should only be allowed if comprised by firms producing complementary goods or services, while simple horizontal consortia should be prohibited. In fact, temporary consortia between rivals are very often a tool for enforcing a cartel more so than a way to increase competition (p. 13, emphasis added).

In my opinion, the first objective (centralization of information to make detection easier) can be attained simply by improving reporting and analysis mechanisms (along the lines of Articles 83 to 87 in the 2011 proposal for new EU public procurement Directives, now significantly reduced), rather than by conducting centralized procurement (which can lead to market foreclosure and other knock-on effects that are detrimental in economic terms).

Regarding the second proposal, I do not see how restricting SME's participation through consortia (ie limiting participation to larger companies) would reduce rather than increase the likelihood of collusion--since it would be equivalent to creating an oligopolistic (sub)market for larger companies, to which those large(r) contracts would be reserved. 

Hence, I would strongly recommend not taking any of those two actions, at least until some further (empirical) research is conducted in this field.

US GAO report on streamlined use of strategic sourcing: Again, on exercising public buyer power

The US Government Accountability Office has issued the "Strategic Procurement: Improved and Expanded Use Could Save Billions in Annual Procurement Costs" report (Sept 2012, http://www.gao.gov/products/GAO-12-919), where it analyses the procurement activities of the Departments of Defense (DoD), Homeland Security (DHS), Energy, and Veterans Affairs (VA) during 2011 and finds that US Federal Agencies are not reaping the benefits of a more strategic exercise of their buyer power.

According to GAO, the federal agencies included in the report leveraged only a fraction of their buying power through strategic sourcing (a process that moves an agency away from numerous individual procurements to a broader aggregate approach) and achieved limited savings. "In fiscal year 2011, the four largest federal departments accounted for 80 percent of the $537 billion in federal procurement spending, but reported managing about 5% or $25.8 billion through strategic sourcing efforts. These agencies reported savings of $1.8 billion—less than one-half of one percent of procurement spending."

GAO considers this situation unsatisfactory because "While strategic sourcing may not be suitable for all procurement spending, leading companies strategically manage about 90 percent of their procurements and report annual savings of 10 percent or more. Further, most agencies’ efforts do not address their highest spending areas such as services, which may provide opportunities for additional savings."

Therefore, GAO issues a series of recommendations for a more strategic use of the leverage that the high volume of expenditure provides to the largest federal agencies. In particular, GAO refers to the DoD Office of the Undersecretary of Defense's 2010 "Better Buying Power" Guidelines (http://tinyurl.com/DoDBetterBuyingPower) which are designed in pro-competitive terms and indicate to procurement officials that they have to promote real competition if they truly want to achieve savings and obtain long-term superior procurement results.

I find these guidelines interesting and worth reading, particularly as regards this:

Real competition is the single most powerful tool available to the Department to drive productivity. Real competition is to be distinguished from a series of directed buys or other contrived two-source situations which do not harness the full energy of competition. Competition is not always available, but evidence suggests that the government is not availing itself of all possible competitive situations.[...]

 

Remove obstacles to competition. In recent years, the Department has achieved the highest rates of competition in its history. Having said that, the fact is that a significant fraction of those competitive procurements have involved what is termed “ineffective competition,” since only one offer to a solicitation was received even when publicized under full and open competition. This occurs in about $55 billion of Department contracts annually. One step the Department can take is to mitigate this loss of savings from the absence of competition. A common practice has been to conclude that either a bid or proposal submitted by a single offeror in response to a full and open competition met the standard for adequate price competition because the bid or proposal was submitted with the expectation of competition. As a result, no certified cost or pricing data was requested, no cost or price analysis was undertaken, and often, no negotiations were conducted with that single offeror. Henceforth I expect contracting officers to conduct negotiations with all single bid offerors and that the basis of that negotiation shall be cost or price analysis, as the case may be, using non-certified data. 

 

A more important approach is to remove obstacles to competitive bidding. For example, the Air Force’s PEO for Services reviewed the Air Force's Design and Engineering Support Program (DESP) for effective competition. She found 39 percent of the task order competitions under the Indefinite Delivery/Indefinite Quantity (IDIQ) contract resulted in one bid. The Air Force team undertook an analysis to determine why they were getting the one bid and made two changes. First, they amended their source selection methodology so that technical, cost, and past performance factors were more equally weighted. No one factor can be less than 25 percent or more than 50 percent. This served to lessen the advantage of the incumbent contractor since the technical factor could not overshadow past performance and cost. Second, the team provided a monthly report to all DESP IDIQ holders listing all known requirements in the pipeline. The report includes sufficient information to allow contractors to evaluate whether or not to bid and to start to prepare a bid package. The team has effectively added an additional 45 days to the time a requirement is made known to the potential offerors and the bid due date. These two changes have reduced the percentage of task orders receiving one bid by 50 percent. The team continues to evaluate its processes to further reduce the percentage. 

 

Each service component and agency has a competition advocate. I am directing each competition advocate to develop a plan to improve both the overall rate of competition and the rate of effective competition. Those plans should establish an improvement rate of at least 2 percent per year for overall competition and an improvement rate of at least 10 percent per year for effective competition. Those plans are to be approved by the CAEs. The Department’s competition advocate shall brief me on the overall progress being made to achieve those goals.

Even if some of the recommendations are hinting towards potential exploitation of suppliers (such as the mandate to negotiate when a single offer is received), the particularities of the defense industry (where supplier concentration is high and increasing over time) may justify them as an exercise of countervailing seller and buyer power. In any case, in my view, the importance of the message particularly lies in the need to find creative ways of lifting barriers to effective participation (particularly by revising tender requirements) and the existence and key role of the competition advocates within each of the federal agencies conducting major procurement activities. 

In my opinion, the creation of a similar position within main domestic procurement agencies would be desirable [see Sanchez Graells, Public Procurement and the EU Competition Rules (Oxford, Hart Publishing, 2011) 387-388], but this is an issue that, unfortunately, has not found space in the current revision of the EU public procurement Directives (where the more general proposal to create oversight bodies under Art 87 of the 2011 Proposal has been scrapped from the July 2012 Compromise text, most likely due to lack of funding and/or to concerns about the Member States organisational autonomy). 

However, in view of the evidence reported at the other side of the Atlantic, maybe we will at some point realize the relevance of having dedicated officials overseeing the competitiveness of public procurement processes (whether we call them competition advocates, auditors or something else is a discussion for another day). As GAO points out, the potential economic benefits should act as a strong incentive to move in that direction. Particularly in times of economic crisis, it seems clear that you need to invest (in human capital) if you want to save and, ultimately, to grow.