Digital procurement, PPDS and multi-speed datafication -- some thoughts on the March 2023 PPDS Communication

The 2020 European strategy for data ear-marked public procurement as a high priority area for the development of common European data spaces for public administrations. The 2020 data strategy stressed that

Public procurement data are essential to improve transparency and accountability of public spending, fighting corruption and improving spending quality. Public procurement data is spread over several systems in the Member States, made available in different formats and is not easily possible to use for policy purposes in real-time. In many cases, the data quality needs to be improved.

To address those issues, the European Commission was planning to ‘Elaborate a data initiative for public procurement data covering both the EU dimension (EU datasets, such as TED) and the national ones’ by the end of 2020, which would be ‘complemented by a procurement data governance framework’ by mid 2021.

With a 2+ year delay, details for the creation of the public procurement data space (PPDS) were disclosed by the European Commission on 16 March 2023 in the PPDS Communication. The procurement data governance framework is now planned to be developed in the second half of 2023.

In this blog post, I offer some thoughts on the PPDS, its functional goals, likely effects, and the quickly closing window of opportunity for Member States to support its feasibility through an ambitious implementation of the new procurement eForms at domestic level (on which see earlier thoughts here).

1. The PPDS Communication and its goals

The PPDS Communication sets some lofty ambitions aligned with those of the closely-related process of procurement digitalisation, which the European Commission in its 2017 Making Procurement Work In and For Europe Communication already saw as not only an opportunity ‘to streamline and simplify the procurement process’, but also ‘to rethink fundamentally the way public procurement, and relevant parts of public administrations, are organised … [to seize] a unique chance to reshape the relevant systems and achieve a digital transformation’ (at 11-12).

Following the same rhetoric of transformation, the PPDS Communication now stresses that ‘Integrated data combined with the use of state-of the-art and emerging analytics technologies will not only transform public procurement, but also give new and valuable insights to public buyers, policy-makers, businesses and interested citizens alike‘ (at 2). It goes further to suggest that ‘given the high number of ecosystems concerned by public procurement and the amount of data to be analysed, the impact of AI in this field has a potential that we can only see a glimpse of so far‘ (at 2).

The PPDS Communication claims that this data space ‘will revolutionise the access to and use of public procurement data:

• It will create a platform at EU level to access for the first time public procurement data scattered so far at EU, national and regional level.

• It will considerably improve data quality, availability and completeness, through close cooperation between the Commission and Member States and the introduction of the new eForms, which will allow public buyers to provide information in a more structured way.

• This wealth of data will be combined with an analytics toolset including advanced technologies such as Artificial Intelligence (AI), for example in the form of Machine Learning (ML) and Natural Language Processing (NLP).’

A first comment or observation is that this rhetoric of transformation and revolution not only tends to create excessive expectations on what can realistically be delivered by the PPDS, but can also further fuel the ‘policy irresistibility’ of procurement digitalisation and thus eg generate excessive experimentation or investment into the deployment of digital technologies on the basis of such expectations around data access through PPDS (for discussion, see here). Policy-makers would do well to hold off on any investments and pilot projects seeking to exploit the data presumptively pooled in the PPDS until after its implementation. A closer look at the PPDS and the significant roadblocks towards its full implementation will shed further light on this issue.

2. What is the PPDS?

Put simply, the PPDS is a project to create a single data platform to bring into one place ‘all procurement data’ from across the EU—ie both data on above threshold contracts subjected to mandatory EU-wide publication through TED (via eForms from October 2023), and data on below threshold contracts, which publication may be required by the domestic laws of the Member States, or entirely voluntary for contracting authorities.

Given that above threshold procurement data is already (in the process of being) captured at EU level, the PPDS is very much about data on procurement not covered by the EU rules—which represents 80% of all public procurement contracts. As the PPDS Communication stresses

To unlock the full potential of public procurement, access to data and the ability to analyse it are essential. However, data from only 20% of all call for tenders as submitted by public buyers is available and searchable for analysis in one place [ie TED]. The remaining 80% are spread, in different formats, at national or regional level and difficult or impossible to re-use for policy, transparency and better spending purposes. In order (sic) words, public procurement is rich in data, but poor in making it work for taxpayers, policy makers and public buyers.

The PPDS thus intends to develop a ‘technical fix’ to gain a view on the below-threshold reality of procurement across the EU, by ‘pulling and pooling’ data from existing (and to be developed) domestic public contract registers and transparency portals. The PPDS is thus a mechanism for the aggregation of procurement data currently not available in (harmonised) machine-readable and structured formats (or at all).

As the PPDS Communication makes clear, it consists of four layers:
(1) A user interface layer (ie a website and/or app) underpinned by
(2) an analytics layer, which in turn is underpinned by (3) an integration layer that brings together and minimally quality-assures the (4) data layer sourced from TED, Member State public contract registers (including those at sub-national level), and data from other sources (eg data on beneficial ownership).

The two top layers condense all potential advantages of the PPDS, with the analytics layer seeking to develop a ‘toolset including emerging technologies (AI, ML and NLP)‘ to extract data insights for a multiplicity of purposes (see below 3), and the top user interface seeking to facilitate differential data access for different types of users and stakeholders (see below 4). The two bottom layers, and in particular the data layer, are the ones doing all the heavy lifting. Unavoidably, without data, the PPDS risks being little more than an empty shell. As always, ‘no data, no fun’ (see below 5).

Importantly, the top three layers are centralised and the European Commission has responsibility (and funding) for developing them, while the bottom data layer is decentralised, with each Member State retaining responsibility for digitalising its public procurement systems and connecting its data sources to the PPDS. Member States are also expected to bear their own costs, although there is EU funding available through different mechanisms. This allocation of responsibilities follows the limited competence of the EU in this area of inter-administrative cooperation, which unfortunately heightens the risks of the PPDS becoming little more than an empty shell, unless Member States really take the implementation of eForms and the collaborative approach to the construction of the PPDS seriously (see below 6).

The PPDS Communication foresees a progressive implementation of the PPDS, with the goal of having ‘the basic architecture and analytics toolkit in place and procurement data published at EU level available in the system by mid-2023. By the end of 2024, all participating national publication portals would be connected, historic data published at EU level integrated and the analytics toolkit expanded. As of 2025, the system could establish links with additional external data sources’ (at 2). It will most likely be delayed, but that is not very important in the long run—especially as the already accrued delays are the ones that pose a significant limitation on the adequate rollout of the PPDS (see below 6).

3. PPDS’ expected functionality

The PPDS Communication sets expectations around the functionality that could be extracted from the PPDS by different agents and stakeholders.

For public buyers, in addition to reducing the burden of complying with different types of (EU-mandated) reporting, the PPDS Communication expects that ‘insights gained from the PPDS will make it much easier for public buyers to

• team up and buy in bulk to obtain better prices and higher quality;

• generate more bids per call for tenders by making calls more attractive for bidders, especially for SMEs and start-ups;

• fight collusion and corruption, as well as other criminal acts, by detecting suspicious patterns;

• benchmark themselves more accurately against their peers and exchange knowledge, for instance with the aim of procuring more green, social and innovative products and services;

• through the further digitalisation and emerging technologies that it brings about, automate tasks, bringing about considerable operational savings’ (at 2).

This largely maps onto my analysis of likely applications of digital technologies for procurement management, assuming the data is there (see here).

The PPDS Communication also expects that policy-makers will ‘gain a wealth of insights that will enable them to predict future trends‘; that economic operators, and SMEs in particular, ‘will have an easy-to-use portal that gives them access to a much greater number of open call for tenders with better data quality‘, and that ‘Citizens, civil society, taxpayers and other interested stakeholders will have access to much more public procurement data than before, thereby improving transparency and accountability of public spending‘ (at 2).

Of all the expected benefits or functionalities, the most important ones are those attributed to public buyers and, in particular, the possibility of developing ‘category management’ insights (eg potential savings or benchmarking), systems of red flags in relation to corruption and collusion risks, and the automation of some tasks. However, unlocking most of these functionalities is not dependent on the PPDS, but rather on the existence of procurement data at the ‘right’ level.

For example, category management or benchmarking may be more relevant or adequate (as well as more feasible) at national than at supra-national level, and the development of systems of red flags can also take place at below-EU level, as can automation. Importantly, the development of such functionalities using pan-EU data, or data concerning more than one Member State, could bias the tools in a way that makes them less suited, or unsuitable, for deployment at national level (eg if the AI is trained on data concerning solely jurisdictions other than the one where it would be deployed).

In that regard, the expected functionalities arising from PPDS require some further thought and it can well be that, depending on implementation (in particular in relation to multi-speed datafication, as below 5), Member States are better off solely using domestic data than that coming from the PPDS. This is to say that PPDS is not a solid reality and that its enabling character will fluctuate with its implementation.

4. Differential procurement data access through PPDS

As mentioned above, the PPDS Communication stresses that ‘Citizens, civil society, taxpayers and other interested stakeholders will have access to much more public procurement data than before, thereby improving transparency and accountability of public spending’ (at 2). However, this does not mean that the PPDS will be (entirely) open data.

The Communication itself makes clear that ‘Different user categories (e.g. Member States, public buyers, businesses, citizens, NGOs, journalists and researchers) will have different access rights, distinguishing between public and non-public data and between participating Member States that share their data with the PPDS (PPDS members, …) and those that need more time to prepare’ (at 8). Relatedly, ‘PPDS members will have access to data which is available within the PPDS. However, even those Member States that are not yet ready to participate in the PPDS stand to benefit from implementing the principles below, due to their value for operational efficiency and preparing for a more evidence-based policy’ (at 9). This raises two issues.

First, and rightly, the Communication makes clear that the PPDS moves away from a model of ‘fully open’ or ‘open by default’ procurement data, and that access to the PPDS will require differential permissioning. This is the correct approach. Regardless of the future procurement data governance framework, it is clear that the emerging thicket of EU data governance rules ‘requires the careful management of a system of multi-tiered access to different types of information at different times, by different stakeholders and under different conditions’ (see here). This will however raise significant issues for the implementation of the PPDS, as it will generate some constraints or disincentives for an ambitions implementation of eForms at national level (see below 6).

Second, and less clearly, the PPDS Communication evidences that not all Member States will automatically have equal access to PPDS data. The design seems to be such that Member States that do not feed data into PPDS will not have access to it. While this could be conceived as an incentive for all Member States to join PPDS, this outcome is by no means guaranteed. As above (3), it is not clear that Member States will be better off—in terms of their ability to extract data insights or to deploy digital technologies—by having access to pan-EU data. The main benefit resulting from pan-EU data only accrues collectively and, primarily, by means of facilitating oversight and enforcement by the European Commission. From that perspective, the incentives for PPDS participation for any given Member State may be quite warped or internally contradictory.

Moreover, given that plugging into PPDS is not cost-free, a Member State that developed a data architecture not immediately compatible with PPDS may well wonder whether it made sense to shoulder the additional costs and risks. From that perspective, it can only be hoped that the existence of EU funding and technical support will be maximised by the European Commission to offload that burden from the (reluctant) Member States. However, even then, full PPDS participation by all Member States will still not dispel the risk of multi-speed datafication.

5. No data, no fun — and multi-speed datafication

Related to the risk that some EU Member States will become PPDS members and others not, there is a risk (or rather, a reality) that not all PPDS members will equally contribute data—thus creating multi-speed datafication, even within the Member States that opt in to the PPDS.

First, the PPDS Communication makes it clear that ‘Member States will remain in control over which data they wish to share with the PPDS (beyond the data that must be published on TED under the Public Procurement Directives)‘ (at 7), It further specifies that ‘With the eForms, it will be possible for the first time to provide data in notices that should not be published, or not immediately. This is important to give assurance to public buyers that certain data is not made publicly available or not before a certain point in time (e.g. prices)’ (at 7, fn 17).

This means that each Member State will only have to plug whichever data it captures and decides to share into PPDS. It seems plain to see that this will result in different approaches to data capture, multiple levels of granularity, and varying approaches to restricting access to the date in the different Member States, especially bearing in mind that ‘eForms are not an “off the shelf” product that can be implemented only by IT developers. Instead, before developers start working, procurement policy decision-makers have to make a wide range of policy decisions on how eForms should be implemented’ in the different Member States (see eForms Implementation Handbook, at 9).

Second, the PPDS Communication is clear (in a footnote) that ‘One of the conditions for a successful establishment of the PPDS is that Member States put in place automatic data capture mechanisms, in a first step transmitting data from their national portals and contract registers’ (at 4, fn 10). This implies that Member States may need to move away from manually inputted information and that those seeking to create new mechanisms for automatic procurement data capture can take an incremental approach, which is very much baked into the PPDS design. This relates, for example, to the distinction between pre- and post-award procurement data, with pre-award data subjected to higher demands under EU law. It also relates to above and below threshold data, as only above threshold data is subjected to mandatory eForms compliance.

In the end, the extent to which a (willing) Member State will contribute data to the PPDS depends on its decisions on eForms implementation, which should be well underway given the October 2023 deadline for mandatory use (for above threshold contracts). Crucially, Member States contributing more data may feel let down when no comparable data is contributed to PPDS by other Member States, which can well operate as a disincentive to contribute any further data, rather than as an incentive for the others to match up that data.

6. Ambitious eForms implementation as the PPDS’ Achilles heel

As the analysis above has shown, the viability of the PPDS and its fitness for purpose (especially for EU-level oversight and enforcement purposes) crucially depends on the Member States deciding to take an ambitious approach to the implementation of eForms, not solely by maximising their flexibility for voluntary uses (as discussed here) but, crucially, by extending their mandatory use (under national law) to all below threshold procurement. It is now also clear that there is a need for as much homogeneity as possible in the implementation of eForms in order to guarantee that the information plugged into PPDS is comparable—which is an aspect of data quality that the PPDS Communication does not seem to have at all considered).

It seems that, due to competing timings, this poses a bit of a problem for the rollout of the PPDS. While eForms need to be fully implemented domestically by October 2023, the PPDS Communication suggests that the connection of national portals will be a matter for 2024, as the first part of the project will concern the top two layers and data connection will follow (or, at best, be developed in parallel). Somehow, it feels like the PPDS is being built without a strong enough foundation. It would be a shame (to put it mildly) if Member States having completed a transition to eForms by October 2023 were dissuaded from a second transition into a more ambitious eForms implementation in 2024 for the purposes of the PPDS.

Given that the most likely approach to eForms implementation is rather minimalistic, it can well be that the PPDS results in not much more than an empty shell with fancy digital analytics limited to very superficial uses. In that regard, the two-year delay in progressing the PPDS has created a very narrow (and quickly dwindling) window of opportunity for Member States to engage with an ambitions process of eForms implementation

7. Final thoughts

It seems to me that limited and slow progress will be attained under the PPDS in coming years. Given the undoubted value of harnessing procurement data, I sense that Member States will progress domestically, but primarily in specific settings such as that of their central purchasing bodies (see here). However, whether they will be onboarded into PPDS as enthusiastic members seems less likely.

The scenario seems to resemble limited voluntary cooperation in other areas (eg interoperability; for discussion see here). It may well be that the logic of EU competence allocation required this tentative step as a first move towards a more robust and proactive approach by the Commission in a few years, on grounds that the goal of creating the European data space could not be achieved through this less interventionist approach.

However, given the speed at which digital transformation could take place (and is taking place in some parts of the EU), and the rhetoric of transformation and revolution that keeps being used in this policy area, I can’t but feel let down by the approach in the PPDS Communication, which started with the decision to build the eForms on the existing regulatory framework, rather than more boldly seeking a reform of the EU procurement rules to facilitate their digital fitness.

Micro-purchases as political football? -- some thoughts on the UK's GPC files and needed regulatory reform

The issue of public micro-purchases has just gained political salience in the UK. The opposition Labour party has launched a dedicated website and an aggressive media campaign calling citizens to scrutinise the use of government procurement cards (GPCs). The analysis revealed so far and the political spin being put on it question the current government’s wastefulness and whether ‘lavish’ GPC expenses are adequate and commensurate with the cost of living crisis and other social pressures. Whether this will yield the political results Labour hopes for is anybody’s guess (I am sceptical), but this is an opportunity to revisit GPC regulation and to action long-standing National Audit Office recommendations on transparency and controls, as well as to reconsider the interaction between GPCs and procurement vehicles based on data analysis. The political football around the frugality expected of a government in times of economic crisis should not obscure the clear need to strengthen GPC regulation in the UK.

Background

GPCs are debit or credit cards that allow government officials to pay vendors directly. In the UK, their issue is facilitated by a framework agreement run by the Crown Commercial Service. These cards are presented as a means to accelerate payment to public vendors (see eg current UK policy). However, their regulatory importance goes beyond their providing an (agile) means of payment, as they generate the risk of public purchases bypassing procurement procedures. If a public official can simply interact with a vendor of their choice and ‘put it on the card’, this can be a way to funnel public funds and engage with direct awards outside procurement procedures. There is thus a clear difference between the use of GPCs within procurement transactions (eg to pay for call-offs within a pre-existing framework agreement) and their use instead of procurement transactions (eg a public official buying something off your preferred online retailer and paying with a card).

Uses within procurement seem rather uncontroversial and the specific mechanism used to pay invoices should be driven by administrative efficiency considerations. There are also good reasons for (some) government officials to hold a GPC to cover the types of expenses that are difficult to procure (eg those linked to foreign travel, or unavoidably ‘spontaneous’ expenses, such as those relating to hospitality). In those cases, GPCs substitute for either the need to provide officials with cash advances (and thus create much sounder mechanisms to control the expenditure, as well as avoiding the circulation of cash with its own corruption and other risks), or to force them to pay in advance from their private pockets and then claim reimbursement (which can put many a public sector worker in financial difficulties, as eg academics know all too well).

The crucial issue then becomes how to control the expenditure under the GPCs and how to impose limits that prevent the bypassing of procurement rules and existing mechanisms. From this perspective, procurement cards are not a new phenomenon at all, and the challenges they pose from a procurement and government contracting perspective have long been understood and discussed—see eg Steven L Schooner and Neil S Whiteman, ‘Purchase Cards and Micro-Purchases: Sacrificing Traditional United States Procurement Policies at the Alter of Efficiency’ (2000) 9 Public Procurement Law Review 148. The UK’s National Audit Office (NAO) also carried out an in-depth investigation and published a report on the issue in 2012.

The regulatory and academic recommendations seeking to ensure probity and value for money in the use of GPCs as a (procurement) mechanism generally address three issues: (1) limits on expenditure, (2) (internal) expenditure control, and (3) expenditure transparency. I would add a fourth issue, which relates to (4) bypassing existing (or easy to set up) procurement frameworks. It is worth noting that the GPC files report provides useful information on each of these issues, all of which requires rethinking in the context of the UK’s current process of reforming procurement law.

Expenditure limits

The GPC files show how there are three relevant value thresholds: the threshold triggering expenditure transparency (currently £500), the maximum single transaction limit (currently £20,000, which raised the pre-pandemic £10,000), and the maximum monthly expenditure (currently £100,000, which raised the pre-pandemic limits if they were lower). It is worth assessing these limits from the perspective of their interaction with procurement rules, as well as broader considerations.

The first consideration is that the £500 threshold triggering expenditure transparency has remained fixed since 2011. Given a cumulative inflation of close to 30% in the period 2011-2022, this means that the threshold has constantly been lower in comparative purchase parity. This should make us reconsider the relevance of some of the findings in the GPC files. Eg the fact that, within its scope, there were ‘65,824 transactions above £500 in 2021, compared to 35,335 in 2010-11’ is not very helpful. This raises questions on the adequacy of having a (fixed) threshold below which expenditure is not published. While the NAO was reluctant to recommend full transparency in 2011, it seems that the administrative burden of providing such transparency has massively lowered in the intervening period, so this may be the time to scrape the transparency threshold. As below, however, this does not mean that the information should be immediately published in open data (as below).

The single transaction limit is the one with the most relevance from a procurement perspective. If a public official can use a GPC for a value exceeding the threshold of regulated procurement, then the rules are not well aligned and there is a clear regulatory risk. Under current UK law, central government contracts with a value above £12,000 must be advertised. This would be kept as the general rule in the Procurement Bill (clause 86(4)), unless there are further amendments prior to its entry into force. This evidences a clear regulatory risk of bypassing procurement (advertising) obligations through GPC use. The single transaction limit should be brought back to pre-pandemic levels (£10,000) or, at least, to the value threshold triggering procurement obligations (£12,000).

The maximum monthly expenditure should be reassessed from an (internal) control perspective (as below), but the need to ensure that GPCs cannot be used to fraction (above threshold) direct awards over short periods of time should also be taken into consideration. From that perspective, ensuring that a card holder cannot spend more than eg £138,760 in a given category of goods or services per month (which is the relevant threshold under both current rules and the foreseen Procurement Bill). Current data analytics in basic banking applications should facilitate such classification and limitation.

(internal) expenditure controls

The GPC files raise questions not only on the robustness of internal controls, but also on the accounting underpinning them (see pp 11-12). Most importantly, there seems to be no meaningful internal post-expenditure control to check for accounting problems or suspected fraudulent use, or no willingness to disclose how any such mechanisms operate. This creates expenditure control opacity that can point to a big governance gap. Expenditure controls should not only apply at the point of deciding who to authorise to hold and use a GPC and up to which expenditure limit, but also (and perhaps more importantly), to how expenditure is being carried out. From a regulatory theory perspective, it is very clear that the use of GPCs is framed under an agency relationship and it is very important to continuously signal to the agent that the principal is monitoring the use of the card and that there are serious (criminal) consequences to misuse. As things stand, it seems that ex post internal controls may operate in some departments (eg those that report recovery for inappropriately used funds) but not (effectively) in others. This requires urgent review of the mechanisms of pre- and post-expenditure control. An update of the 2012 NAO report seems necessary.

Expenditure transparency

The GPC files (pp 10-11) show clear problems in the implementation of the policy of disclosing all expenditure in transactions exceeding £500, which should be published published monthly, 2 months in arrears, despite (relatively clear) guidance to that effect. In addition to facilitating the suppression of the transparency threshold, developments in the collection and publication of open data should also facilitate the rollout of a clear plan to ensure effective publication without the gaps identified in the GPC files (and other problems in practice). However, this is also a good time to carefully consider the purpose of these publications and the need to harmonise them with the publication of other procurement information.

There are conflicting issues at hand. First, the current policy of publishing 2 months in arrears does not seem justified in relation to some qualified users of that information, such as those with an oversight role (or fraud investigation powers). Second, in relation to the general public, publication in full of all details may not be adequate within that time period in some cases, and the publication of some information may not be appropriate at all. There are, of course, intermediate situations, such as data access for journalists of research academics. In relation to this data, as well as all procurement data, this is an opportunity to create a sophisticated data-management architecture that can handle of multi-tiered access to different types of information at different times, by different stakeholders and under different conditions (see here and here).

bypassing procurement frameworks

A final consideration is that the GPC files evidence a risk that GPCs may be used in ways that bypass existing procurement frameworks, or in ways that would require setting up new frameworks (or other types of procurement vehicle, such as dynamic purchasing systems). The use of GPCs to buy goods off Amazon is the clearest example (see pp 24-25), as there is nothing in the functioning of Amazon that could not be replicated through pre-procured frameworks supported by electronic catalogues. In that regard, GPC data should be used to establish the (administrative) efficiency of creating (new) frameworks and to inform product (and service) selection for inclusion therein. There should also be a clear prohibition of using GPCs outside existing frameworks unless better value for money for identical products can be documented, in which case this should also be reported to the entity running the relevant framework (presumably, the Crown Commercial Service) for review.

Conclusion

In addition to discussions about the type and level of expenditure that (high-raking) public officials should be authorised to incur as a political and policy matter, there is clearly a need and opportunity to engage in serious discussions on the tightening of the regulation of GPCs in the UK, and these should be coordinated with the passage of the Procurement Bill through the House of Commons. I have identified the following areas for action:

• Suppression of the value threshold triggering transparency of specific transactions, so that all transactions are subjected to reporting.

• Coordination of the single transaction threshold with that triggering procurement obligations for central government (which is to also apply to local and other contracting authorities).

• Coordination of the maximum monthly spend limit with the threshold for international advertising of contract opportunities, so that no public official can spend more than the relevant amount in a given category of goods or services per month.

• Launch of a new investigation and report by NAO on the existing mechanisms of pre- and post-expenditure control.

• Creation of a sophisticated data-management architecture that can handle of multi-tiered access to different types of information at different times, by different stakeholders and under different conditions. This needs to be in parallel or jointly with proposals under the Procurement Bill.

• There should also be a clear prohibition of using GPCs outside existing frameworks unless better value for money for identical products can be documented. GPC data should be used to inform the creation and management of procurement frameworks and other commercial vehicles.

More Nuanced Procurement Transparency to Protect Competition: Has the Court of Justice Hit the Brakes on Open Procurement Data in Antea Polska (C-54/21)?

** This comment was first published as an Op-Ed for EU Law Live on 8 December 2022 (see formatted version). I am reposting it here in case of broader interest. **

In Antea Polska (C-54/21), the Court of Justice provided further clarification of the duties incumbent on contracting authorities to protect the confidentiality of different types of information disclosed by economic operators during tender procedures for the award of public contracts. Managing access to such information is challenging. On the one hand, some of the information will have commercial value and be sensitive from a market competition perspective, or for other reasons. On the other hand, disappointed tenderers can only scrutinise and challenge procurement decisions reliant on that information if they can access it as part of the duty to give reasons incumbent on the contracting authority. There is thus a clash of private interests that the public buyer needs to mediate as the holder of the information.

However, in recent times, procurement transparency has also gained a governance dimension that far exceeds the narrow confines of the tender procedures and related disputes. Open contracting approaches have focused on procurement transparency as a public governance tool, emphasising the public interest in the availability of such information. This creates two overlapping tracks for discussions on procurement transparency and its limitations: a track concerning private interests, and a track concerning the public interest. In this Op-Ed, I examine the judgment of Court of Justice in Antea Polska from both perspectives. I first consider the implications of the judgment for the public interest track, ie the open data context. I then focus on the specifics of the judgment in the private interest track, ie the narrower regulation of access to remedies in procurement. I conclude with some broader reflections on the need to develop the institutional mechanisms and guidance required by the nuanced approach to procurement transparency demanded by the Court of Justice, which is where both tracks converge.

Procurement Transparency and Public Interest

In the aftermath of the covid-19 pandemic, procurement transparency became a mainstream topic. Irregularities and corruption in the extremely urgent direct award of contracts could only be identified where information was made public, sometimes after extensive litigation to force disclosure. And the evidence that slowly emerged was concerning. The improper allocation of public funds through awards not subjected to most (or any) of the usual checks and balances renewed concerns about corruption and maladministration in procurement. This brought the spotlight back on proactive procurement transparency as a governance tool and sparked new interest in open data approaches. These would generate access to (until then) confidential procurement information without the need for an explicit request by the interested party.

A path towards ‘open by default’ procurement data has been plotted in the Open Data Directive, the Data Governance Act, and the new rules on Procurement eForms. Combined, these measures impose minimum open data requirements and allow for further ‘permissioned’ openness, including the granting of access to information subject to the rights of others—eg on grounds of commercial confidentiality, the protection of intellectual property (IP) or personal data (see here for discussion). In line with broader data strategies (notably, the 2020 Data Strategy), EU digital law seems to gear procurement towards encouraging ‘maximum transparency’—which would thus be expected to become the new norm soon (although I have my doubts, see here).

However, such ‘maximum transparency’ approach does not fit well the informational economics of procurement. Procurement is at its core an information or data-intensive exercise, as public buyers use tenders and negotiations to extract private information from willing economic operators to identify the contractor that can best satisfy the relevant needs. Subjecting the private information revealed in procurement procedures to maximum (or full) transparency would thus be problematic, as the risk of disclosure could have chilling and anticompetitive effects. This has long been established in principle in EU procurement law—and more generally in freedom of information law—although the limits to (on-demand and proactive) procurement transparency remain disputed and have generated wide variation across EU jurisdictions (for extensive discussion, see the contributions to Halonen, Caranta & Sanchez-Graells, Transparency in EU Procurements (2019)).

The Court’s Take

The Court of Justice’s case law has progressively made a dent on ‘maximum transparency’ approaches to confidential procurement information. Following its earlier Judgment in Klaipėdos regiono atliekų tvarkymo centras (C-927/19), the Court of Justice has now provided additional clarification on the limits to disclosure of information submitted by tenderers in public procurement procedures in its Judgment in Antea Polska. From the open data perspective, the Court’s approach to the protection of public interests in the opacity of confidential information are relevant.

Firstly, the Court of Justice has clearly endorsed limitations to procurement transparency justified by the informational economics of procurement. The Court has been clear that ‘the principal objective of the EU rules on public procurement is to ensure undistorted competition, and that, in order to achieve that objective, it is important that the contracting authorities do not release information relating to public procurement procedures which could be used to distort competition, whether in an ongoing procurement procedure or in subsequent procedures. Since public procurement procedures are founded on a relationship of trust between the contracting authorities and participating economic operators, those operators must be able to communicate any relevant information to the contracting authorities in such a procedure, without fear that the authorities will communicate to third parties items of information whose disclosure could be damaging to those operators’; Antea Polska (C-54/21, para 49). Without perhaps explicitly saying it, the Court has established the protection of competition and the fostering of trust in procurement procedures as elements inherently placed within the broader public interest in the proper functioning of public procurement mechanisms.

Second, the Court has recognised that ‘it is permissible for each Member State to strike a balance between the confidentiality [of procurement information] and the rules of national law pursuing other legitimate interests, including that … of ensuring “access to information”, in order to ensure the greatest possible transparency in public procurement procedures’; Antea Polska (C-54/21, para 57). However, in that regard, the exercise of such discretion cannot impinge on the effectiveness of the EU procurement rules seeking to align practice with the informational economics of procurement (ie to protect competition and the trust required to facilitate the revelation of private information, as above) to the extent that they also protect public interests (or private interests with a clear impact on the broader public interest, as above). Consequently, the Court stressed that ‘[n]ational legislation which requires publicising of any information which has been communicated to the contracting authority by all tenderers, including the successful tenderer, with the sole exception of information covered by the [narrowly defined] concept of trade secrets [in the Trade Secrets Directive], is liable to prevent the contracting authority … from deciding not to disclose certain information pursuant to interests or objectives [such as the protection of competition or commercial interests, but also the preservation of law enforcement procedures or the public interest], where that information does not fall within that concept of a trade secret’; Antea Polska (C-54/21, para 62).

In my view, the Court is clear that a ‘maximum transparency’ approach is not permissible and has stressed the duties incumbent on contracting authorities to protect public and private interests opposed to transparency. This is very much in line with the nuanced approach it has taken in another notable recent Judgment concerning open beneficial ownership data: Luxembourg Business Registers (C‑37/20 and C‑601/20) (see here for discussion). In Antea Polska, the Court has emphasised the need for case-by-case analysis of the competing interests in the confidentiality or disclosure of certain information.

This could have a significant impact on open data initiatives. First, it comes to severely limit ‘open by default’ approaches. Second, if contracting authorities find themselves unable to engage with nuanced analysis of the implications of information disclosure, they may easily ‘clam up’ and perpetuate (or resort back to) generally opaque approaches to procurement disclosure. Developing adequate institutional mechanisms and guidance will thus be paramount (as below).

Procurement Transparency and Private Interest

In its more detailed analysis of the specific information that contracting authorities need to preserve in order to align their practice with the informational economics of procurement (ie to promote trust and to protect market competition), the Court’s views in Antea Polska are also interesting but more problematic. The starting point is that the contracting authority cannot simply take an economic operator’s claim that a specific piece of information has commercial value or is protected by IP rights and must thus be kept confidential (Antea Polska, C-54/21, para 65), as that could generate excessive opacity and impinge of the procedural rights of competing tenderers. Moving beyond this blanket approach requires case-by-case analysis.

Concerning information over which confidentiality is claimed on the basis of its commercial value, the Court has stressed that ‘[t]he disclosure of information sent to the contracting authority in the context of a public procurement procedure cannot be refused if that information, although relevant to the procurement procedure in question, has no commercial value in the wider context of the activities of those economic operators’; Antea Polska (C-54/21, para 78). This requires the contracting authority to be able to assess the commercial value of the information. In the case, the dispute concerned whether the names of employees and subcontractors of the winning tenderer should be disclosed or not. The Court found that ‘in so far as it is plausible that the tenderer and the experts or subcontractors proposed by it have created a synergy with commercial value, it cannot be ruled out that access to the name-specific data relating to those commitments must be refused on the basis of the prohibition on disclosure’; Antea Polska (C-54/21, para 79). This points to the emergence of a sort of rebuttable presumption of commercial value that will be in practice very difficult to overcome by a contracting authority seeking to disclose information—either motu proprio, or on the request of a disappointed tenderer.

Concerning information over which confidentiality is claimed on the basis that it is protected by an IP right, in particular by copyright, the Court stressed that it is unlikely that copyright protection will apply to ‘technical or methodological solutions’ of procurement relevance (Antea Polska, C-54/21, para 82). Furthermore, ‘irrespective of whether they constitute or contain elements protected by an intellectual property right, the design of the projects planned to be carried out under the public contract and the description of the manner of performance of the relevant works or services may … have a commercial value which would be unduly undermined if that design and that description were disclosed as they stand. Their publication may, in such a case, be liable to distort competition, in particular by reducing the ability of the economic operator concerned to distinguish itself using the same design and description in future public procurement procedures’; Antea Polska (C-54/21, para 83). Again, this points to the emergence of a rebuttable presumption of commercial value and anticompetitive potential that will also be very difficult to rebut in practice.

The Court has also stressed that keeping this type of information confidential does not entirely bar disclosure. To discharge their duty to give reasons and facilitate access to remedies by disappointed tenderers, contracting authorities are under an obligation to disclose, to the extent possible, the ‘essential content’ of the protected information; Antea Polska (C-54/21, paras 80 and 84). Determining such essential content and ensuring that the relevant underlying (competing) rights are adequately protected will also pose a challenge to contracting authorities.

In sum, the Court has stressed that preserving competing interests related to the disclosure of confidential information in procurement requires the contracting authority to ‘assess whether that information has a commercial value outside the scope of the public contract in question, where its disclosure might undermine legitimate commercial concerns or fair competition. The contracting authority may, moreover, refuse to grant access to that information where, even though it does not have such commercial value, its disclosure would impede law enforcement or would be contrary to the public interest. A contracting authority must, where full access to information is refused, grant that tenderer access to the essential content of that information, so that observance of the right to an effective remedy is ensured’; Antea Polska (C-54/21, para 85). Once again, developing adequate institutional mechanisms and guidance will thus be paramount (as below).

Investing in the Way Forward

As I have argued elsewhere, and the Antea Polska Judgment has made abundantly clear, under EU procurement (and digital) law, it is simply not possible to create a system that makes all procurement data open. Conversely, the Judgment also makes clear that it is not possible to operate a system that keeps all procurement data confidential (Antea Polska, C-54/21, para 68).

Procurement data governance therefore requires the careful management of a system of multi-tiered access to different types of information at different times, by different stakeholders and under different conditions. This will require investing in data and analysis capabilities by public buyers, which can no longer treat the regulation of confidentiality in procurement as an afterthought or secondary consideration. In the data economy, public buyers need to create the required institutional mechanisms to discharge their growing data governance obligations.

Moreover, and crucially, creating adequate data governance approaches requires the development of useful guidance by the European Commission and national competition authorities, as well as procurement oversight bodies. The Court of Justice’s growing case law points to the potential emergence of (difficult to challenge) rebuttable presumptions of justified confidentiality that could easily result in high levels of procurement opacity. To promote a better balance of the competing public and private interests, a more nuanced approach needs to be supported by actionable guidance. This will be very important across all EU jurisdictions, as it is not only jurisdictions that had embraced ‘maximum transparency’ that now need to correct course—but also those that continue to lag in the disclosure of procurement information. Ensuring a level playing field in procurement data governance depends on the harmonisation of currently widely diverging practices. Procurement digitalisation thus offers an opportunity that needs to be pursued.

New CJEU case law against excessive disclosure: quid de open data? (C‑54/21, and joined C‑37/20 and C‑601/20)

In the last few days, the Court of Justice of the European Union (CJEU) has delivered two judgments imposing significant limitations on the systematic, unlimited disclosure of procurement information with commercial value, such as the identity of experts and subcontractors engaged by tenderers for public contracts; and beneficial ownership information. In imposing a nuanced approach to the disclosure of such information, the CJEU may have torpedoed ‘full transparency’ approaches to procurement and beneficial ownership open data.

Indeed, these are two classes of information at the core of current open data efforts, and they are relevant for (digital) procurement governance—in particular in relation to the prevention of corruption and collusion, which automated screening requires establishing relationships and assessing patterns of interaction reliant on such data [for discussion, see A Sanchez-Graells, ‘Procurement Corruption and Artificial Intelligence: Between the Potential of Enabling Data Architectures and the Constraints of Due Process Requirements’ in S Williams & J Tillipman (eds), Routledge Handbook of Public Procurement Corruption (forthcoming)]. The judgments can thus have important implications.

In Antea Polska, the CJEU held that EU procurement rules prevent national legislation mandating all information sent by the tenderers to the contracting authorities to be published in its entirety or communicated to the other tenderers, with the sole exception of trade secrets. The CJEU reiterated that the scope of non-disclosable information is much broader and requires a case-by-case analysis by the contracting authority, in particular with a view to avoiding the release of information that could be used to distort competition. Disclosure of information needs to strike an adequate balance between meeting good administration duties to enable the right to the effective review of procurement decisions, on the one hand, and the protection of information with commercial value or with potential competition implications, on the other.

In a related fashion, in Luxembourg Business Registers, the CJEU declared invalid the provision of the Anti-Money Laundering Directive whereby Member States had to ensure that the information on the beneficial ownership of corporate and other legal entities incorporated within their territory was accessible in all cases to any member of the general public—without the need to demonstrate having a legitimate interest in accessing it. The CJEU considered that the disclosure of the information to undefined members of the public created an excessive interference with the fundamental rights to respect for private life and to the protection of personal data.

In this blog post, I analyse these two cases and reflect on their implications for the management of (big) open data for procurement governance purposes, in particular from an anti-corruption perspective and in relation to the EU law data governance obligations incumbent on public buyers.

There is more than trade secrets to procurement confidentiality

In Antea Polska and Others (C-54/21, EU:C:2022:888), among other questions, the CJEU was asked whether Directive 2014/24/EU precludes national legislation on public procurement which required that, with the sole exception of trade secrets, information sent by the tenderers to the contracting authorities be published in its entirety or communicated to the other tenderers, and a practice on the part of contracting authorities whereby requests for confidential treatment in respect of trade secrets were accepted as a matter of course.

I will concentrate on the first part of the question on full transparency solely constrained by trade secrets—and leave the ‘countervailing’ practice aside for now (though it deserves some comment because it creates a requirement for the contracting authority to assess the commercial value of procurement information in the wider context of the activities of the participating economic operators, at paras 69-85). I will also not deal with the discrepancy between the concept of ‘trade secret’ under the Trade Secrets Directive and the concept of ‘confidential information’ in Directive 2014/24 (which the CJEU clarifies, again, at paras 51-55).

The issue of full transparency of procurement information subject only to trade secret protection raises an interesting question because it concerns the compatibility with EU law of a maximalistic approach to procurement transparency that is not peculiar to Poland (where the case originated) but shared by other Member States with a permissive tradition of access to public documents [for in-depth country-specific analyses and comparative considerations, see the contributions to K-M Halonen, R Caranta & A Sanchez-Graells, Transparency in EU Procurements. Disclosure Within Public Procurement and During Contract Execution (Edward Elgar 2019)].

The question concerns the interpretation of multiple provisions of Directive 2014/24/EU and, in particular, Art 21(1) on confidentiality and Arts 50(4) and 55(3) on the withholding of information [see my comments to Art 21 and 55 in R Caranta & A Sanchez-Graells, European Public Procurement. Commentary on Directive 2014/24/EU (Edward Elgar 2021)]. All of them are of course to be interpreted in line with the general principle of competition in Art 18(1) [see A Sanchez-Graells, Public Procurement and the EU Competition Rules (2nd edn, hart 2015) 444-445].

In addressing the question, the CJEU built on its recent judgment in Klaipėdos regiono atliekų tvarkymo centras (C‑927/19, EU:C:2021:700), and reiterated its general approach to the protection of confidential information in procurement procedures:

‘… the principal objective of the EU rules on public procurement is to ensure undistorted competition … to achieve that objective, it is important that the contracting authorities do not release information relating to public procurement procedures which could be used to distort competition, whether in an ongoing procurement procedure or in subsequent procedures. Since public procurement procedures are founded on a relationship of trust between the contracting authorities and participating economic operators, those operators must be able to communicate any relevant information to the contracting authorities in such a procedure, without fear that the authorities will communicate to third parties items of information whose disclosure could be damaging to those operators’ (C-54/21, para 49, reference omitted, emphasis added).

The CJEU linked this interpretation to the prohibition for contracting authorities to disclose information forwarded to it by economic operators which they have designated as confidential [Art 21(1) Dir 2014/24] and stressed that this had to be reconciled with the requirements of effective judicial protection and, in particular, the general principle of good administration, from which the obligation to state reasons stems because ‘in the absence of sufficient information enabling it to ascertain whether the decision of the contracting authority to award the contract is vitiated by errors or unlawfulness, an unsuccessful tenderer will not, in practice, be able to rely on its right .. to an effective review’ (C-54/21, para 50).

The Court also stressed that the Directive allows Member States to modulate the scope of the protection of confidential information in accordance with their national legislation, in particular legislation concerning access to information [Art 21(1) Dir 2014/24, C-54/21, para 56]. In that regard, however, the CJEU went on to stress that

‘… if the effectiveness of EU law is not to be undermined, the Member States, when exercising the discretion conferred on them by Article 21(1) of that directive, must refrain from introducing regimes … which undermine the balancing exercise [with the right to an effective review] or which alter the regime relating to the publicising of awarded contracts and the rules relating to information to candidates and tenderers set out in Article 50 and 55 of that directive … any regime relating to confidentiality must, as Article 21(1) of Directive 2014/24 expressly states, be without prejudice to the abovementioned regime and to those rules laid down in Articles 50 and 55 of that directive’ (C-54/21, para 58-59).

Focusing on Art 50(4) and Art 55(3) of Directive 2014/24/EU, the CJEU stressed that these provisions empower contracting authorities to withhold from general publication and from disclosure to other candidates and tenderers ‘certain information, where its release would impede law enforcement, would otherwise be contrary to the public interest or would prejudice the legitimate commercial interests of an economic operator or might prejudice fair competition’ (para 61 and, almost identically, para 60). This led the Court to the conclusion that

‘National legislation which requires publicising of any information which has been communicated to the contracting authority by all tenderers, including the successful tenderer, with the sole exception of information covered by the concept of trade secrets, is liable to prevent the contracting authority, contrary to what Articles 50(4) and 55(3) of Directive 2014/24 permit, from deciding not to disclose certain information pursuant to interests or objectives mentioned in those provisions, where that information does not fall within that concept of a trade secret.

Consequently, Article 21(1) of Directive 2014/24, read in conjunction with Articles 50 and 55 of that directive … precludes such a regime where it does not contain an adequate set of rules allowing contracting authorities, in circumstances where Articles 50 and 55 apply, exceptionally to refuse to disclose information which, while not covered by the concept of trade secrets, must remain inaccessible pursuant to an interest or objective referred to in Articles 50 and 55’ (paras 62-63).

In my view, this is the correct interpretation and an important application of the rules seeking to minimise the risk of distortions of competition due to excessive procurement transparency, on which I have been writing for a long time [see also K-M Halonen, ‘Disclosure rules in EU public procurement: balancing between competition and transparency’ (2016) 16(4) Journal of Public Procurement 528].

The Antea Polska judgment stresses the importance of developing a nuanced approach to the management, restricted disclosure and broader publication of information submitted to the contracting authority in a procurement procedure. Notably, this will create particular complications in the context of the design and rollout of procurement open data, especially in the context of the new eForms (see here, and below).

Transparency for what? Who really cares about beneficial ownership?

In Luxembourg Business Registers (joined cases C‑37/20 and C‑601/20, EU:C:2022:912, FR only—see EN press release on which I rely to avoid extensive own translations from French) the CJEU was asked to rule on the compatibility with the Charter of Fundamental Rights—and in particular Articles 7 (respect for private and family life) and 8 (protection of personal data)—of Article 30(5)(c) of the consolidated version of the Anti-Money Laundering Directive (AML Directive), which required Member States to ensure that information on the beneficial ownership of corporate and other legal entities incorporated within their territory is accessible in all cases to any member of the general public. In particular, members of the general public had to ‘be permitted to access at least the name, the month and year of birth and the country of residence and nationality of the beneficial owner as well as the nature and extent of the beneficial interest held.’

The CJEU has found that the general public’s access to information on beneficial ownership constitutes a serious interference with the fundamental rights to respect for private life and to the protection of personal data, which is exacerbated by the fact that, once those data have been made available to the general public, they can not only be freely consulted, but also retained and disseminated.

While the CJEU recognised that the AML Directive pursues an objective of general interest and that the general public’s access to information on beneficial ownership is appropriate for contributing to the attainment of that objective, the interference with individual fundamental rights is neither limited to what is strictly necessary nor proportionate to the objective pursued.

The Court paid special attention to the fact that the rules requiring unrestricted public access to the information result from a modification of the previous regime in the original AML Directive, which required, in addition to access by the competent authorities and certain entities, for access by any person or organisation capable of demonstrating a legitimate interest. The Court considered that the suppression of the requirement to demonstrate a legitimate interest in accessing the information did not generate sufficient benefits from the perspective of combating money laundering and terrorist financing to offset the significantly more serious interference with fundamental rights that open publication of the beneficial ownership data entails.

Here, the Court referred to its judgment in Vyriausioji tarnybinės etikos komisija (C‑184/20, EU:C:2022:601), where it carried out a functional comparison of the anti-corruption effects of a permissioned system of institutional access and control of relevant disclosures, versus public access to that information. The Court was clear that

‘… the publication online of the majority of the personal data contained in the declaration of private interests of any head of an establishment receiving public funds … does not meet the requirements of a proper balance. In comparison with an obligation to declare coupled with a check of the declaration’s content by the Chief Ethics Commission … such publication amounts to a considerably more serious interference with the fundamental rights guaranteed in Articles 7 and 8 of the Charter, without that increased interference being capable of being offset by any benefits which might result from publication of all those data for the purpose of preventing conflicts of interest and combating corruption’ (C-184/20, para 112).

In Luxembourg Business Registers, the CJEU also held that the optional provisions in Art 30 AML Directive that allowed Member States to make information on beneficial ownership available on condition of online registration and to provide, in exceptional circumstances, for an exemption from access to that information by the general public, were not, in themselves, capable of demonstrating either a proper balance between competing interests, or the existence of sufficient safeguards.

The implication of the Luxembourg Business Registers is that a different approach to facilitating access to beneficial ownership data is required, and that an element of case-by-case assessment (or at least of an assessment based on categories of organisations and individuals seeking access) will need to be brought back into the system. In other words, permissioned access to beneficial ownership data seems unavoidable.

Implications for open data and data governance

These recent CJEU judgments seem to me to clearly establish the general principle that unlimited transparency does not equate public interest, as there is also an interest in preserving the (relative) confidentiality of some information and data and an adequate, difficult balance needs to be struck. The interests in competition with transparency can be either individual (fundamental rights, or commercial value) or collective (avoidance of distortions of competition). Detailed and comprehensive assessment on a case-by-case basis is required.

As I advocated long ago, and recently reiterated in relation to the growing set of data governance obligations incumbent on public buyers, under EU law,

‘It is thus simply not possible to create a system that makes all procurement data open. Data governance requires the careful management of a system of multi-tiered access to different types of information at different times, by different stakeholders and under different conditions. While the need to balance procurement transparency and the protection of data subject to the rights of others and competition-sensitive data is not a new governance challenge, the digital management of this information creates heightened risks to the extent that the implementation of data management solutions is tendentially ‘open access’ (and could eg reverse presumptions of confidentiality), as well as in relation to system integrity risks (ie cybersecurity)’ (at 10, references omitted).

The CJEU judgments have (re)confirmed that unlimited ‘open access’ is not a viable strategy under EU law. It is perhaps clearer than ever that the capture, structuring, retention, and disclosure of governance-relevant procurement and related data (eg beneficial ownership) needs to be decoupled from its proactive publication. This requires a reconsideration of the open data model and, in particular, a careful assessment of the implementation of the new eForms that only just entered into force.

Registration open: TECH FIXES FOR PROCUREMENT PROBLEMS?

As previously announced, on 15 December, I will have the chance to discuss my ongoing research on procurement digitalisation with a stellar panel: Eliza Niewiadomska (EBRD), Jessica Tillipman (GW Law), and Sope Williams (Stellenbosch).

The webinar will provide an opportunity to take a hard look at the promise of tech fixes for procurement problems, focusing on key issues such as:

• The ‘true’ potential of digital technologies in procurement.

• The challenges arising from putting key enablers in place, such as an adequate big data architecture and access to digital skills in short supply.

• The challenges arising from current regulatory frameworks and constraints not applicable to the private sector.

• New challenges posed by data governance and cybersecurity risks.

The webinar will be held on December 15, 2022 at 9:00 am EST / 2:00 pm GMT / 3:00 pm CET-SAST. Full details and registration at: https://blogs.gwu.edu/law-govpro/tech-fixes-for-procurement-problems/.

Save the date: 15 Dec, Tech fixes for procurement problems?

If you are interested in procurement digitalisation, please save the date for an online workshop on ‘Tech fixes for procurement problems?’ on 15 December 2022, 2pm GMT. I will have the chance to discuss my ongoing research (scroll down for a few samples) with a stellar panel: Eliza Niewiadomska (EBRD), Jessica Tillipman (GW Law), and Sope Williams (Stellenbosch). We will also have plenty time for a conversation with participants. Do not let other commitments get on the way of joining the discussion!

More details and registration coming soon. For any questions, please email me: a.sanchez-graells@bristol.ac.uk.

Urgent: 'no eForms, no fun' -- getting serious about building a procurement data architecture in the EU

‘EU Member States only have about one year to make crucial decisions that will affect the procurement data architecture of the EU and the likelihood of successful adoption of digital technologies for procurement governance for years or decades to come’. Put like that, the relevance of the approaching deadline for the national implementation of new procurement eForms may grab more attention than the alternative statement that ‘in just about a year, new eForms will be mandatory for publication of procurement notices in TED’.

This latter more technical (obscure, and uninspiring?) understanding of the new eForms seems to have been dominating the approach to eForms implementation, which does not seem to have generally gained a high profile in domestic policy-making at EU Member State level despite the Publications Office’s efforts.

In this post, I reflect about the strategic importance of the eForms implementation for the digitalisation of procurement, the limited incentives for an ambitious implementation that stem from the voluntary approach of the most innovative aspects of the new eForms, and the opportunity that would be lost with a minimalistic approach to compliance with the new rules. I argue that it is urgent for EU Member States to get serious about building a procurement data architecture that facilitates the uptake of digital technologies for procurement governance across the EU, which requires an ambitious implementation of eForms beyond their minimum mandatory requirements.

eForms: some background

The EU is in the process of reforming the exchange of information about procurement procedures. This information exchange is mandated by the EU procurement rules, which regulate a variety of procurement notices with the two-fold objective of (i) fostering cross-border competition for public contracts and (ii) facilitating the oversight of procurement practices by the Member States, both in relation to the specific procedure (eg to enable access to remedies) and from a broad policy perspective (eg through the Single Market Scoreboard). In other words, this information exchange underpins the EU’s approach to procurement transparency, which mainly translates into publication of notices in the Tenders Electronic Daily (TED).

A 2019 Implementing Regulation established new standard forms for the publication of notices in the field of public procurement (eForms). The Implementing Regulation is accompanied by a detailed Implementation Handbook. The transition to eForms is about to hit a crucial milestone with the authorisation for their voluntary use from 14 November 2022, in parallel with the continued use of current forms. Following that, eForms will be mandatory and the only accepted format for publication of TED notices from 25 October 2023. There will thus have been a very long implementation period (of over four years), including an also lengthy (11-month) experimentation period about to start. This contrasts with previous revisions of the TED templates, which had given under six months’ notice (eg in 2015) or even just a 20-day implementation period (eg in 2011). This extended implementation period is reflective of the fact that the transition of eForms is not merely a matter of replacing a set of forms with another.

Indeed, eForms are not solely the new templates for the collection of information to be published in TED. eForms represent the EU’s open standard for publishing public procurement data — or, in other words, the ‘EU OCDS’ (which goes much beyond the OCDS mapping of the current TED forms). The importance of the implementation of a new data standard has been highlighted at strategic level, as this is the cornerstone of the EU’s efforts to improve the availability and quality of procurement data, which remain suboptimal (to say the least) despite continued efforts to improve the quality and (re)usability of TED data.

In that regard, the 2020 European strategy for data, emphasised that ‘Public procurement data are essential to improve transparency and accountability of public spending, fighting corruption and improving spending quality. Public procurement data is spread over several systems in the Member States, made available in different formats and is not easily possible to use for policy purposes in real-time. In many cases, the data quality needs to be improved.’ The European Commission now stresses how ‘eForms are at the core of the digital transformation of public procurement in the EU. Through the use of a common standard and terminology, they can significantly improve the quality and analysis of data’ (emphasis added).

It should thus be clear that the eForms implementation is not only about low level form-filling, but also (or primarily) about building a procurement data architecture that facilitates the uptake of digital technologies for procurement governance across the EU. Therefore, the implementation of eForms and the related data standard seeks to achieve two goals: first, to ensure the data quality (eg standardisation, machine-readability) required to facilitate its automated treatment for the purposes of publication of procurement notices mandated by EU law (ie their primary use); and, second, to build a data architecture that can facilitate the accumulation of big data so that advanced data analytics can be deployed by re-users of procurement data. This second(ary) goal is particularly relevant to our discussion. This requires some unpacking.

The importance of data for the deployment of digital technologies

It is generally accepted that quality (big) data is the primary requirement for the deployment of digital technologies to extract data-driven insights, as well as to automate menial back-office tasks. In a detailed analysis of these technologies, I stress the relevance of procurement data across technological solutions that could be deployed to improve procurement governance. In short, the outcome of robotic process automation (RPA) can only be as good as its sources of information, and adequate machine learning (ML) solutions can only be trained on high-quality big data—which thus conditions the possibility of developing recommender systems, chatbots, or algorithmic screens for procurement monitoring and oversight. Distributed Ledger Technology (DLT) systems (aka blockchain) can manage data, but cannot verify its content, accuracy, or reliability. Internet of Things (IoT) applications and software oracles can automatically capture data, which can alleviate some of the difficulties in generating an adequate data infrastructure. But this is only in relation with the observation of the ‘real world’ or in relation to digitally available information, which quality raises the same issues as other sources of data. In short, all digital technologies are data-centric or, more clearly, data-dependent.

Given the crucial relevance of data across digital technologies, it is hard to emphasise how any shortcomings in the enabling data architecture curtail the likelihood of successful adoption of digital technologies for procurement governance. With inadequate data, it may simply be impossible to develop digital solutions at all. And the development and adoption of digital solutions developed on poor or inadequate data can generate further problems—eg skewing decision-making on the basis of inadequately derived ‘data insights’. Ultimately, then, ensuring that adequate data is available to develop digital governance solutions is a challenging but unavoidable requirement in the process of procurement digitalisation. Success, or lack of it, in the creation of an enabling data architecture will determine the viability of the deployment of digital technologies more generally. From this perspective, the implementation of eForms gains clear strategic importance.

eForms Implementation: a flexible model

Implementing eForms is not an easy task. The migration towards eForms requires a complete redesign of information exchange mechanisms. eForms are designed around universal business language and involve the use of a much more structured information schema, compatible with the EU’s eProcurement Ontology, than the current TED forms. eForms are also meant to collect a larger amount of information than current TED forms, especially in relation to sub-units within a tender, such as lots, or in relation to framework agreements. eForms are meant to be flexible and regularly revised, in particular to add new fields to facilitate data capture in relation to specific EU-mandated requirements in procurement, such as in relation with the clean vehicles rules (with some changes already coming up, likely in November 2022).

From an informational point of view, the main constraint that remains despite the adoption of eForms is that their mandatory content is determined by existing obligations to report and publish tender-specific information under the current EU procurement rules, as well as to meet broader reporting requirements under international and EU law (eg the WTO GPA). This mandatory content is thus rather limited. Ultimately, eForms’ main concentration is on disseminating details of contract opportunities and capturing different aspects of decision-making by the contracting authorities. Given the process-orientedness and transactional focus of the procurement rules, most of the information to be mandatorily captured by the eForms concerns the scope and design of the tender procedure, some aspects concerning the award and formal implementation of the contract, as well as some minimal data points concerning its material outcome—primarily limited to the winning tender. As the Director-General of the Publications Office put it an eForms workshop yesterday, the new eForms will provide information on ‘who buys what, from whom and for what price’. While some of that information (especially in relation to the winning tender) will be reflective of broader market conditions, and while the accumulation of information across procurement procedures can progressively generate a broader view of (some of) the relevant markets, it is worth stressing that eForms are not designed as a tool of market intelligence.

Indeed, eForms do not capture the entirety of information generated by a procurement process and, as mentioned, their mandatory content is rather limited. eForms do include several voluntary or optional fields, and they could be adapted for some voluntary uses, such as in relation to detection of collusion in procurement, or in relation to the beneficial ownership of tenderers and subcontractors. Extensive use of voluntary fields and the development of additional fields and uses could contribute to generating data that enabled the deployment of digital technologies for the purposes of eg market intelligence, integrity checks, or other sorts of (policy-related) analysis. For example, there are voluntary fields in relation to green, social or innovation procurement, which could serve as the basis for data-driven insights into how to maximise the effects of such policy interventions. There are also voluntary fields concerning procurement challenges and disputes, which could facilitate a monitoring of eg areas requiring guidance or training. However, while the eForms are flexible, include voluntary fields, and the schema facilitates the development of additional fields, is it unclear that adequate incentives exist for adoption beyond their mandatory minimum content.

Implementation in two tiers

The fact that eForms are in part mandatory and in part voluntary will most likely result in two separate tiers of eForms implementation across the EU. Tier 1 will solely concern the collection and exchange of information mandated by EU law, that is the minimum mandatory eForm content. Tier 2 will concern the optional collection and exchange of a much larger volume of information concerning eg the entirety of tenders received, as well as qualitative information on eg specific policy goals embedded in a tender process. Of course, in the absence of coordination, a (large) degree of variation within Tier 2 can be expected. Tier 2 is potentially very important for (digital) procurement governance, but there is no guarantee that Member States will decide to implement eForms covering it.

One of the major obstacles to the broad adoption of a procurement data model so far, at least in the European Union, relates to the slow uptake of e-procurement (as discussed eg here). Without an underlying highly automated e-procurement system, the generation and capture of procurement data is a main challenge, as it is a labour-intensive process prone to input error. The entry into force of the eForms rules could serve as a further push for the completion of the transition to e-procurement—at least in relation to procurement covered by EU law (as below thresholds procurement is a voluntary potential use of eForms). However, it is also possible that low e-procurement uptake and generalised unsophisticated approaches to e-procurement (eg reduced automation) will limit the future functionality of eForms, with Member States that have so far lagged behind restricting the use of eForms to tier 1. Non life-cycle (automated) e-procurement systems may require manual inputs into the new eForms (or the databases from which they can draw information) and this implies that there is a direct cost to the implementation of each additional (voluntary) data field. Contracting authorities may not perceive the (potential) advantages of incurring those costs, or may more simply be constrained by their available budget. A collective action problem arises here, as the cost of adding more data to the eForms is to be shouldered by each public buyer, while the ensuing big data would potentially benefit everyone (especially as it will be published—although there are also possibilities to capture but not publish information that should be explored, at least to prevent excessive market transparency; but let’s park that issue for now) and perhaps in particular data re-users offering for pay added-value services.

In direct relation to this, and compounding the (dis)incentives problem, the possibility (or likelihood) of minimal implementation is compounded by the fact that, in many Member States, the operational adaptation to eForms does not directly concern public sector entities, but rather their service providers. e-procurement services providers compete for the provision of large volume, entirely standardised platform services, which are markets characterised by small operational margins. This creates incentives for a minimal adaptation of current e-sending systems and disincentives for the inclusion of added-value (data) services potentially unlikely to be used by public buyers. Some (or most) optional aspects of the eForm implementation will thus remain unused due to these market structure and dynamics, which does not clearly incentivise a race to the top (unless there is clear demand pull for it).

With some more nuance, it should be stressed that it is also possible that the adoption of eForms is uneven within a given jurisdiction where the voluntary character of parts of the eForm is kept (rather than made mandatory across the board through domestic legislation), with advanced procurement entities (eg central purchasing bodies, or large buyers) adopting tier 2 eForms, and (most) other public buyers limiting themselves to tier 1.

Ensuing data fragmentation

While this variety of approaches across the EU and within a Member State would not pose legal challenges, it would have a major effect on the utility of the eForms-generated data for the purposes of eg developing ML solutions, as the data would be fragmented, hardly representative of important aspects of procurement (markets), and could hardly be generalisable. The only consistent data would be that covered by tier 1 (ie mandatory and standardised implementation) and this would limit the potential use cases for the deployment of digital technologies—with some possibly limited to the procurement remit of the specific institutions with tier 2 implementations.

Relatedly, it should be stressed that, despite the effort to harmonise the underlying data architecture and link it to the Procurement Ontology, the Implementation Handbook makes clear that ‘eForms are not an “off the shelf” product that can be implemented only by IT developers. Instead, before developers start working, procurement policy decision-makers have to make a wide range of policy decisions on how eForms should be implemented’ in the different Member States.

This poses an additional challenge from the perspective of data quality (and consistency), as there are many fields to be tailored in the eForms implementation process that can result in significant discrepancies in the underlying understanding or methodology to determine them, in addition to the risk of potential further divergence stemming from the domestic interpretation of very similar requirements. This simply extends to the digital data world the current situation, eg in relation to diverging understandings of what is ‘recyclable’ or what is ‘social value’ and how to measure them. Whenever open-ended concepts are used, the data may be a poor source for comparative and aggregate analysis. Where there are other sources of standardisation or methodology, this issue may be minimised—eg in relation to the green public procurement criteria developed in the EU, if they are properly used. However, where there are no outside or additional sources of harmonisation, it seems that there is scope for quite a few difficult issues in trying to develop digital solutions on top of eForms data, except in relation to quantitative issues or in relation to information structured in clearly defined categories—which will mainly link back to the design of the procurement.

An opportunity about to be lost?

Overall, while the implementation of eForms could in theory build a big data architecture and facilitate the development of ML solutions, there are many challenges ahead and the generalised adoption of tier 2 eForms implementations seems unlikely, unless Member States make a positive decision in the process of national adoption. The importance of an ambitious tier 2 implementation of eForms should be assessed in light of its downstream importance for the potential deployment of digital technologies to extract data-driven insights and to automate parts of the procurement process. A minimalistic implementation of eForms would significantly constrain future possibilities of procurement digitalisation. Primarily in the specific jurisdiction, but also with spillover effects across the EU.

Therefore, a minimalistic eForms implementation approach would perpetuate (most of the) data deficit that prevents effective procurement digitalisation. It would be a short-sighted saving. Moreover, the effects of a ‘middle of the road’ approach should also be considered. A minimalistic implementation with a view to a more ambitious extension down the line could have short-term gains, but would delay the possibility of deploying digital technologies because the gains resulting from the data architecture are not immediate. In most cases, it will be necessary to wait for the accumulation of sufficiently big data. In some cases of infrequent procurement, missing data points will generate further time lags in the extraction of valuable insights. It is no exaggeration that every data point not captured carries an opportunity cost.

If Member States are serious about the digitalisation of public procurement, they will make the most of the coming year to develop tier 2 eForms implementations in their jurisdiction. They should also keep an eye on cross-border coordination. And the European Commission, both DG GROW and the Publications Office, would do well to put as much pressure on Member States as possible.

More detail on the UK's procurement transparency ambitions -- some comments and criticisms

© GraceOda / Flickr.

On 30 June 2022, the UK Government’s Cabinet Office published the policy paper ‘Transforming Public Procurement - our transparency ambition’ (the ‘ambitions paper’, or the ‘paper’). The paper builds on the Green Paper and the Government’s response to its public consultation, and outlines ‘proposals to dramatically improve transparency of UK public contracts and spending’. The ambitions paper provides a vision well beyond the scant (almost null) detail in the Procurement Bill (clause 88), which is attracting a number of proposed amendments to try to enshrine in law the basic elements now spelled out in the paper.

In this post, I reflect on the need to amend the Procurement Bill to bind (successive) UK Governments to the current transparency aspirations. I also comment on other aspects of the paper, including persistent issues with the lack of granularity in planned access to procurement data, which I already raised in relation to the Green Paper (see here, Q27 and Q29, and here).

A necessary amendment of the Procurement Bill

The additional level of detail in the paper is welcome and helpful in understanding how the UK plans to operationalise its procurement transparency ambitions. However, a first point to make is that the publication of the ambitions paper should in no way deactivate concerns on the insufficiency of the Procurement Bill to ensure that a significant change in the way procurement information is captured and disseminated in the UK is achieved. In particular, the wording of clause 88(1) has to change.

It is nowhere close to good enough to simply have a weak enabling clause in legislation, stating that ‘An appropriate authority may by regulations make provision requiring certain information to be shared in a particular way, including through a specified online system’. The obvious first shortcoming is that the authority may do so, which also means it may not do so. The second is that the indication of a specified online system as a possible particular way of sharing information seems to take us back quite a few years. If not online (and if not as open data), how would a transparency aspiration be commensurate to the UK’s commitment to e.g. the open contracting data standard?.

Given the high level of aspiration in the paper, a more solid legal grounding is required. My proposal, which builds on discussions with the open contracting community, as well as the amendment already tabled by Baroness Hayman of Ullock, would be to amend clause 88(1) of the Procurement Bill, so it reads:

'An appropriate authority shall by regulations make provision requiring certain information to be shared through a specified online system. Such online system shall, at a minimum, establish and operate a freely accessible, machine-readable and licence-free digital register for all public procurement notices under this Act, wherein all information will be regularly updated in accordance with the time limits for the publication notices set out in the Act.'

Comments on the aspirations paper

Once the general commitment to having single digital register is strengthened, we can move on to consider the detail of what (and how) should be published in the register, what should be kept for restricted use, and what further transparency-related interventions can build upon it—e.g. the creation of a dashboard with useful data analytics, or the interconnection of the register with other sources of e.g. relevant anti-corruption information (for discussion, see here). There are some indications of what the UK aspires to do, but also some lack of clarity in the paper, and some clear risks of undesirable knock-on effects from the maximalist approach to procurement transparency it embraces.

Vision

The aspirations paper indeed starts from a maximalist position, indicating that the vision is ‘to create a fully transparent public procurement system’. However, there are two clear limitations to that approach.

First, the proposal itself includes a proportionate approach to transparency requirements: ‘we want to ensure that we are only asking for the most detailed information - contract documents, performance markings etc - from the largest contracts, in order to maintain transparency without bogging procurement teams down in unnecessary bureaucracy for low-value contracts’. This immediately means that a potentially large volume of (local) procurement will not be subjected to (some aspects) of the new transparency regime. Moreover, as the Procurement Bill stands, there would also be significant exclusions from important transparency obligations e.g. in relation to light touch contracts (see here, section 7, issues #21 on performance-related KPIS and non-performance notices, and #23 on modification notices). That already falls short of generating a ‘fully transparent’ procurement system, precisely in relation to the award of contracts where the risk of capture can be high.

Second, the publication of procurement information remains subjected to the general exclusions and carve-outs resulting from i.a. the Freedom of Information Act 2000 (FOIA). Interestingly, the ambitions paper does not refer to it at all, despite the Green Paper having made clear that, in the absence of FOIA reform (which is not sought), ‘only data which would be required to be made available under FOIA … would be publishable’ (at 167). Regardless of the paper’s silence on the issue, FOIA will continue to play a significant role in establishing which level of detail is disclosed, in particular in relation to disclosure of information not captured as a matter of mandatory disclosure in the relevant (award) notices, and perhaps even in relation to that.

The importance of preserving commercial confidentiality in the procurement setting is clear, and was also a clear focus of concern in the Green Paper consultation, leading e.g. to the Cabinet Office dropping its initial ambition of publishing tenders received in procurement procedures. As the Government’s response stressed: ‘We have considered the potential impact of public disclosure of information, such as (but not limited to) tenders. The feedback we received from stakeholders was that publishing tenders at this stage could prejudice future competitions that may run if the initial one is aborted and re-run for any reason, as bids will have been disclosed to the competition. As a result, we will not require disclosure of tenders submitted in a procurement’ (at 221).

Therefore, the system will not (and should not be) fully transparent. What is more useful is to see what the vision wants to enable in relation to procurement data and related analytics and insights. The vision indicates that the UK Government would like for everyone ‘to be able to view, search and understand what the UK public sector wants to buy, how much it is spending, and with whom’. This is a more realistic aspiration that does not necessarily entail total transparency and, given some safeguards and a more granular approach to the disclosure of differing levels of detail in the information (see here and discussion below), it should be welcome. Ultimately, the Government wants the future platform to help people understand:

1. current and future procurement opportunities created in the UK public sector; including pipelines of future work. [This should open up opportunities within the public sector to small businesses, driving down prices, increasing innovation and improving the business landscape across the country];

2. how much money the public sector spends on purchasing essential goods and services. [This should] allow taxpayers to see how much is being spent through procurement on and in their local area, who it is spent with and how it is delivering on local priorities. [Moreover, this should show] which routes to market are available to contracting authorities, and how much has been spent through each of those. [This should] give contracting authorities the data they need to collaborate better, drive value for money and identify cost savings in their procurements, so they can monitor for signs of waste and inefficiency;

3. which contracts finished on time and on budget–and which did not. [This means providing more detail across] the true lifecycle of government contracts, including how much the final amount spent on a contract differs from its original intended value, or how often contracts have been extended;

4. which companies have been excluded from winning future work due to fraud, corruption or persistent poor performance; [and]

5. who is really benefiting from public money - not just the companies winning contracts but the ownership of those companies

This list (which regroups the longer and slightly repetitive list in the paper, as well as aggregate the purpose for the disclosure of specific information) points to three categories. First, a category where the information is purely notice-based (categories 1, 4). Second, a category where the related insights should be easily derived from the information included mandatory notices (categories 2 and 3). Third, a category (mainly 5) that concerns non-procurement information and will require either (a) embedding disclosure obligations in the procurement life-cycle (thus raising the red tape and participation costs), or (b) interconnection with non-procurement databases.

The first category is relatively unproblematic, although there is an inherent tension between the disclosure of planned procurement opportunities and the facilitation of collusive practices (more details below).

The second category probably points at the need of considering the extent to which data dashboards should differentiate between different users, including the level of detail (and timeliness) of the information published in each of them (also discussed below).

The third category points at the need to consider issues of design and interoperability of the platform, as it would be preferable for it to be susceptible of plugging into other databases. Moreover, there are other (anti-corruption) functionalities that could be enabled, such as cross-checks against databases of political donations to identify potentially problematic relationships between procurement awardees and political donors. In relation to this category, and to anti-corruption efforts more generally, the ambitions paper is not particularly ambitious. However, the creation of a solid procurement data architecture on the basis of OCDS could facilitate those extensions in the future.

The future platform

The ambitions paper indicates that the Government seeks to operationalise the new transparency regime through two main elements (as the ‘tell us once’ supplier register is a parallel and distinct intervention):

• The introduction of a number of new procurement ‘notices’, covering the entire procurement lifecycle from planning through to contract expiry

• A digital platform which will display all of this information publicly, with API access to data published to the Open Contracting Data Standard (OCDS). Once we have completed the core notice development, over time we also plan to build a number of useful registers, and explore integrating commercial data analysis tools

What this means is that the future platform will initially simply bring into one place what is currently published across a scattered landscape of transparency tools (see section 3.1 in the paper). That is an improvement, but the more significant change will only come when register and dashboard insights get developed. Importantly, however, the design of these registers and dashboards need to be very carefully considered and linked back to the intended (and likely) use by different audiences. However, the ambitions paper does not seem to consider this need and rather seeks to establish a system accessible to any type of data user on an undifferentiated form (see section 4.4).

Research has shown that most of the gains from procurement transparency concern ex ante disclosure of information [M Bauhr et al, ‘Lights on the shadows of public procurement: Transparency as an antidote to corruption’ (2020) 33(3) Governance 495-523]. Conversely, the publication of ex post information is particularly risky in relation to e.g. anticompetitive practices, as well as corruption, and can generate limited benefits as it is unlikely that there will be a sustained level of engagement with that information by most stakeholders with a theoretical motivation to engage in procurement oversight [N Köbis, C Starke and I Rahwan, ‘The promise and perils of using artificial intelligence to fight corruption’ (2022) 4 Nature Machine Intelligence 418-424].

In that regard, it is particularly problematic that the aspirations paper seems to indicate that the UK Government would be publishing (in real time, for everyone to see) information such as: ‘Analysis of bid and win rates, analysis of supplier & bidder beneficial ownership patterns, general market trends analysis’. This should concern regulators such as the Competition and Markets Authority, as well as the Serious Fraud Office. While the latter should absolutely have access to that information and market intelligence, its public disclosure (in detail, with no time lag) could be counterproductive and help, rather than hinder, corrupt and collusive practices. In that regard, it is of paramount importance that those authorities (and others, such as the National Audit Office) are involved in the design of the system—which is not entirely clear from the ‘user-centric’ approach embraced in the aspirations paper (see section 4.1).

A multi-layered level of transparency

In relation to these risks and issues, it is necessary to reiterate a call for a more nuanced and discriminating approach than the one that transpires from the aspirations paper. As stressed in the response to the Green Paper consultation (here Q29), while it can but be endorsed that the platform needs to be created, and the data automatically fed into it in accordance with OCDS and other technical interoperability requirements, a key feature of the new system should be its multi-layered level of access/transparency.

Analysis carried elsewhere (see here) supports a nuanced approach to the level of transparency created by public contract registries similar to the envisaged central digital platform, which needs to fall short of the full transparency paradigm in which it seems to have been conceived. As a functional criterion, only the information that is necessary to ensure proper oversight and the effectiveness of anti-corruption measures should be disclosed, whereas the information that can be most damaging for competition should be withheld.

Generally, what is needed is granularity in the levels of information that are made accessible to different stakeholders. A full transparency approach whereby all information was made available to everyone would fall very short from the desired balance between the transparency and competition goals of public procurement. A system based on enabling or targeted transparency, whereby each stakeholder gets access to the information it needs for a specific purpose, is clearly preferable.

In more specific terms, it is submitted that:

• The content of the central digital platform should not be fully available to the public. Access to the full registry should be restricted to public sector officials under a strong duty of confidentiality protected by appropriate sanctions in cases of illegitimate disclosure.

• Even within the public sector, full access to the central digital platform should be made available on a need-to-know basis. Oversight entities, such as the National Audit Office, the Serious Fraud Office, or the Competition and Markets Authority, as well as the new public procurement review unit (PPRU) should have full access. However, other entities or specific civil servants should only access the information they require to carry out their functions.

• Limited versions of the central digital platform that are made accessible to the public should aggregate information by contracting authority and avoid disclosing any particulars that could be traced back to specific tenders, specific contracts, or specific undertakings.

• Representative institutions, such as third sector organisations, journalists or academics should have the opportunity of seeking full access to the central digital platform on a case-by-case basis where they can justify a legitimate or research-related interest. In case of access, ethical approval shall be obtained, anonymization of data attempted, and specific confidentiality requirements duly imposed.

• Delayed full access to the central digital platform could also be allowed for, provided there are sufficient safeguards to ensure that historic information does not remain relevant for the purposes of protecting market competition, business secrets and commercial interests.

• Tenderers should have access to their own records, even if they are not publicly-available, so as to enable them to check their accuracy. This is particularly relevant if public contract registries are used for the purposes of assessing past performance under the new rules.

• Big data should be published on an anonymised basis, so that general trends can be analysed without enabling ‘reverse engineering’ of information that can be traced to specific bidders.

• The entity in charge of the central digital platform should regularly publish aggregated statistics by type of procurement procedure, object of contract, or any other items deemed relevant for the purposes of the public accountability of public buyers (such as percentages of expenditure in green procurement, etc).

• The entity in charge of the central digital platform should develop a system of red flag indicators and monitor them with a view to reporting instances of legal non-compliance to the relevant oversight entity, or potential collusion to the competition authority. In that regard, the earlier attempts (eg through the abandoned ‘Screening for Cartels’ tool) should be carefully analysed to avoid replicating past errors.

Open Contracting: Where is the UK and What to Expect?

I had the pleasure of delivering a webinar on ‘Open Contracting Data: Where Are We & What Could We Expect?‘ for the Gloucester branch of the Chartered Institute of Procurement & Supply. The webinar assessed the current state of development and implementation of open contracting data initiatives in the UK. It also considered the main principles and goals of open contracting, as well as its practical implementation, and the specific challenges posed by the disclosure of business sensitive information. The webinar also mapped potential future developments and, more generally, reflected on the relevance of an adequate procurement data infrastructure for the deployment of digital technologies and, in particular, AI. The slides are available (via dropbox) and the recording is also accessible through the image below (as well as via dropbox).

As always, feedback most welcome: a.sanchez-graells@bristol.ac.uk.

PS. For some an update on recent EBRD/EU sponsored open contracting initiatives in Greece and Poland, see here.

Interesting paper on effects of open procurement data on outcomes: Duguay, Rauter & Samuels (2019)

A recently published working paper has assessed the impact of increased availability of procurement data on competition for public contracts and on procurement outcomes in the EU context: R Duguay, T Rauter & D Samuels, ‘The Impact of Open Data on Public Procurement’ (November 22, 2019).

Duguay, Rauter & Samuels concentrate on the increased availability of TED data in a (more) user-friendly format in July 2015 (when the data started being available for a bulk download on CSV format) to assess the effects that easier access to procurement data has on the functioning of procurement markets and on procurement outcomes. The paper is very interesting and their results are quite striking.

However, it is important to stress the important caveat that their analysis is still based on TED data and, thus, potentially affected by the quality shortcomings of that data. As mentioned in other occasions, the TED database has problems because it is constructed as a result of the self-declaration of data by the contracting authorities of the Member States, which makes its content very inhomogeneous and difficult to analyse, including significant problems of under-inclusiveness, definitional fuzziness and the lack of filtering of errors—as recognised, repeatedly, in the methodology underpinning the SMSPP itself (see here and here).

With that in mind, however, it is interesting to look closely at their findings.

A seemingly striking insight derived from the paper is that ‘the new European government contracting provisions have anti-competitive effects‘ (at 17). This is in the context of an analysis of the ‘likelihood that government agencies allocate public contracts through an open procedure‘ and should thus not be surprising, given the flexibilisation in the use of procedures involving negotiations. However, even with this regulatory effect, the authors find that more open data triggers more use of open procedures, in particular in EU countries with weaker institutional frameworks (at 18-19, and see below). This could be symptomatic of the fact that more complexity in procurement subjected to higher levels of transparency pushes for a risk-averse approach to procurement compliance. The same would be supported by their finding of higher levels of award of contracts on the basis of price-only award criteria (at 25, and see below).

This tension between procurement complexity and transparency is generally strongly evidenced in the paper.

On the one hand, and in line with claims of the pro-competitive nature of more openness in procurement data (note, not of more openness or transparency of contract opportunities), the authors find that

• the likelihood of competitive bidding increases sharply for TED contracts around July 2015 and that this increase persists through the end of our sample period [ie to the end of 2018] (at 18);

• open procurement data leads government officials to implement more competitive bidding processes [ie open procedures], and that this increase in competitive bidding is driven by countries that do not have the institutions to effectively monitor public officials (at 19);

• the number of bids increases sharply for TED contracts soon after the open data initiative, and this increase persists throughout our sample period (at 20);

• public officials are 8.7 percentage points more likely to award government contracts to new vendors after the open data initiative (at 21);

• contract values fall by approximately 8% ... after the open data initiative (at 23).

On the other hand, and also in line with theoretical expectations of a degradation of procurement decisions subjected to higher levels of transparency (and the fact that this transparency does not concern contract opportunities, but more general open procurement data), the authors also find that

• [the results] are inconsistent with the idea that easier access to procurement data fosters cross-border competition throughout the European Union … open procurement data fosters local competition among vendors by reducing barriers to entry but does not promote cross-border competition across the European single market (at 22);

• after the open data initiative, the likelihood of a contract modification increases by 2.9 percentage points for contracts above TED publication thresholds (at 24);

• after the open data initiative, public officials are 38% ... more likely to award contracts above TED publication thresholds exclusively based on price (at 25);

• the performance ... is significantly worse if price was the only award criterion in the allocation decision (at 26);

• the increase in modifications is driven by contracts awarded to new government suppliers, consistent with information asymmetries contributing to the observed deterioration in contract performance. Moreover, this evidence suggests that procurement relationships before the open data initiative were not necessarily corrupt or otherwise inefficient (at 26);

• the decline in contract performance is stronger for complex procurements, consistent with project complexity exacerbating the potential allocative distortions of open procurement data (at 27).

Their overall conclusion is that

Comparing government contracts above and below EU publication thresholds, we find that increasing the public accessibility of procurement data raises the likelihood of having competitive bidding processes, increases the number of bids per contract, and facilitates market entry by new vendors. After the open data initiative, procurement prices decrease and EU government agencies are more likely to award contracts to the lowest bidder. However, the increased competition comes at the expense of lower contract performance, particularly if suppliers are new, procurement projects are complex, and contracts are awarded solely based on price.

Overall, our results suggest that open data on procurement awards facilitates competition and lowers ex-ante procurement prices, but does not necessarily increase allocative efficiency in government contracting (at 27-28, emphases added).

I find these results striking and difficult to assess from the perspective of evidence-based policy-making. There are two issues of particular concern/interest to me.

One, the finding that more availability of data does not generate more cross-border procurement, and that the push for more competitive (ie open) procedures is mostly appreciable in countries with weaker institutional frameworks. This could support the position that institutional robustness is an alternative to data transparency, which would significantly alter the prioritisation of systemic procurement reforms and take the sides of systems that favour strong institutional oversight in a context of relative opacity.

Second, that transparency exacerbates problems at execution phase, in particular in complex projects and/or projects with new suppliers. This would take the wind out of the sails of reform and policy-making approaches concentrating on perceived or apparent competition for the contract at award stage, and rather force a refocus on an analysis of procurement outcomes at the end of the relevant project. This would also side with approaches that would advocate for more robust institutional approaches to contract design and performance management, rather than relying on transparency to correct contract execution problems.

The mixed results of the paper are also interesting in the context of the long-term effect of more open procurement data on competition, as well as on cartelisation and bid rigging risks, which are not assessed in the paper.

On the round, I think that the paper offers some interesting evidence to back up that there is a need to reconsider the level of transparency given to procurement data. I do not think this should stop the development of an improved procurement data architecture in the EU. To the contrary. I think this should reignite and prioritise discussions concerning the level of disclosure or public access to that information (ie its openness), which cannot be simply assumed to be positive in what, in my view, is currently an excessively simplistic approach in leading policy-making and think tank proposals. For more (but not new) discussion, see here and here.