Initial UK guidance on pro-innovation AI regulation: Much ado about nothing?

The UK Government’s Department for Science, Innovation and Technology (DSIT) has recently published its Initial Guidance for Regulators on Implementing the UK’s AI Regulatory Principles (Feb 2024) (the ‘AI guidance’). This follows from the Government’s response to the public consultation on its ‘pro-innovation approach’ to AI regulation (see here).

The AI guidance is meant to support regulators develop tailored guidance for the implementation of the five principles underpinning the pro-innovation approach to AI regulation, that is: (i) Safety, security & robustness; (ii) Appropriate transparency and explainability; (iii) Fairness;
(iv) Accountability and governance; and (v) Contestability and redress.

Voluntary approach and timeline for implementation

A first, perhaps, surprising element of the AI guidance comes from the way in which engagement with the principles by current regulators is framed as voluntary. The white paper describing the pro-innovation approach to AI regulation (the ‘AI white paper’) had indicated that, initially, ‘the principles will be issued on a non-statutory basis and implemented by existing regulators’, with a clear expectation for regulators to make use their ‘domain-specific expertise to tailor the implementation of the principles to the specific context in which AI is used’.

The AI white paper made it clear that a failure by regulators to implement the principles would lead the government to introduce ‘a statutory duty on regulators requiring them to have due regard to the principles’, which would still ‘allow regulators the flexibility to exercise judgement when applying the principles in particular contexts, while also strengthening their mandate to implement them’. There seemed to be little room for discretion for regulators to decide whether to engage with the principles, even if they were expected to exercise discretion on how to implement them.

By contrast, the initial AI guidance indicates that it ‘is not intended to be a prescriptive guide on implementation as the principles are voluntary and how they are considered is ultimately at regulators’ discretion’. There is also a clear indication in the response to the public consultation that the introduction of a statutory duty is not in the immediate legislative horizon and the absence of a pre-determined date for the assessment of whether the principles have been ‘sufficiently implemented’ on a voluntary basis (for example, in two years’ time) will make it very hard to press for such legislative proposal (depending on the policy direction of the Government at the time).

This seems to follow from the Government’s position that ‘acknowledge[s] concerns from respondents that rushing the implementation of a duty to regard could cause disruption to responsible AI innovation. We will not rush to legislate’. At the same time, however, the response to the public consultation indicates that DSIT has asked a number of regulators to publish by 30 April 2024 updates on their strategic approaches to AI. This seems to create an expectation that regulators will in fact engage—or have defined plans for engaging—with the principles in the very short term. How this does not create a ‘rush to implement’ and how putting the duty to consider the principles on a statutory footing would alter any of this is hard to fathom, though.

An iterative, phased approach

The very tentative approach to the issuing of guidance is also clear in the fact that the Government is taking an iterative, phased approach to the production of AI regulation guidance, with three phases foreseen. A phase one consisting of the publication of the AI guidance in Feb 2024, a phase two comprising an iteration and development of the guidance in summer of 2024, and a phase three (with no timeline) involving further developments in cooperation with regulators—to eg ‘encourage multi-regulator guidance’. Given the short time between phases one and two, some questions arise as to how much practical experience will be accumulated in the coming 4-6 months and whether there is much value in the high-level guidance provided in phase one, as it only goes slightly beyond the tentative steer included in the AI white paper—which already contained some indication of ‘factors that government believes regulators may wish to consider when providing guidance/implementing each principle’ (Annex A).

Indeed, the AI guidance is still rather high-level and it does not provide much substantive interpretation of what the different principles mean. It is very much a ‘how to develop guidance’ document, rather than a document setting out core considerations and requirements for regulators to embed within their respective remits. A significant part of the document provides guidance on ‘interpreting and applying the AI regulatory framework’ (pp 7-12) but this is really ‘meta-guidance’ on issues such as potential collaboration between regulators for the issuance of joint guidance/tools, or an encouragement to benchmarking and the avoidance of duplicated guidance where relevant. General recommendations such as the value of publishing the guidance and keeping it updated seem superfluous in a context where the regulatory approach is premised on ‘the expertise of [UK] world class regulators’.

The core of the AI guidance is limited to the section on ‘applying individual principles’ (pp 13-22), which sets out a series of questions to consider in relation to each of the five principles. The guidance offers no answers and very limited steer for their formulation, which is entirely left to regulators. We will probably have to wait (at least) for the summer iteration to get some more detail of what substantive requirements relate to each of the principles. However, the AI guidance already contains some issues worthy of careful consideration, in particular in relation to the tunnelling of regulatory power and the imbalanced approach to the different principles that follows from its reliance on existing (and soon to emerge) technical standards.

technical standards and interpretation of the regulatory principles

regulatory tunnelling

As we said in our response to the public consultation on the AI white paper,

The principles-based approach to AI regulation suggested in the AI [white paper] is undeliverable, not only due to lack of detail on the meaning and regulatory implications of each of the principles, but also due to barriers to translation into enforceable requirements, and tensions with existing regulatory frameworks. The AI [white paper] indicates in Annex A that each regulator should consider issuing guidance on the interpretation of the principles within its regulatory remit, and suggests that in doing so they may want to rely on emerging technical standards (such as ISO or IEEE standards). This presumes both the adequacy of those standards and their sufficiency to translate general principles into operationalizable and enforceable requirements. This is by no means straightforward, and it is hard to see how regulators with significantly limited capabilities … can undertake that task effectively. There is a clear risk that regulators may simply rely on emerging industry-led standards. However, it has already been pointed out that this creates a privatisation of AI regulation and generates significant implicit risks (at para 27).

The AI guidance, in sticking to the same approach, confirms this risk of regulatory tunnelling. The guidance encourages regulators to explicitly and directly refer to technical standards ‘to support AI developers and AI deployers’—while at the same time stressing that ‘this guidance is not an endorsement of any specific standard. It is for regulators to consider standards and their suitability in a given situation (and/or encourage those they regulate to do so likewise).’ This does not seem to be the best approach. Leaving it to each of the regulators to assess the suitability of existing (and emerging) standards creates duplication of effort, as well as a risk of conflicting views and guidance. It would seem that it is precisely the role of centralised AI guidance to carry out that assessment and filter out technical standards that are aligned with the overarching regulatory principles for implementation by sectoral regulators. In failing to do that and pushing the responsibility down to each regulator, the AI guidance comes to abdicate responsibility for the provision of meaningful policy implementation guidelines.

Additionally, the strong steer to rely on references to technical standards creates an almost default position for regulators to follow—especially those with less capability to scrutinise the implications of those standards and to formulate complementary or alternative approaches in their guidance. It can be expected that regulators will tend to refer to those technical standards in their guidance and to take them as the baseline or starting point. This effectively transfers regulatory power to the standard setting organisations and further dilutes the regulatory approach followed in the UK, which in fact will be limited to industry self-regulation despite the appearance of regulatory intervention and oversight.

unbalanced approach

The second implication of this approach is that some principles are likely to be more developed than other in regulatory guidance, as they also are in the initial AI guidance. The series of questions and considerations are more developed in relation to principles for which there are technical standards—ie ‘safety, security & robustness’, and ‘accountability and governance’—and to some aspects of other principles for which there are standards. For example, in relation to ‘adequate transparency and explainability’, there is more of an emphasis on explainability than on transparency and there is no indication of how to gauge ‘adequacy’ in relation to either of them. Given that transparency, in the sense of publication of details on AI use, raises a few difficult questions on the interaction with freedom of information legislation and the protection of trade secrets, the passing reference to the algorithmic transparency recording standard will not be sufficient to support regulators in developing nuanced and pragmatic approaches.

Similarly, in relation to ‘fairness’, the AI guidance solely provides some reference in relation to AI ethics and bias, and in both cases in relation to existing standards. The document falls awfully short of any meaningful consideration of the implications and requirements of the (arguably) most important principle in AI regulation. The AI guidance solely indicates that

Tools and guidance could also consider relevant law, regulation, technical standards and assurance techniques. These should be applied and interpreted similarly by different regulators where possible. For example, regulators need to consider their responsibilities under the 2010 Equality Act and the 1998 Human Rights Act. Regulators may also need to understand how AI might exacerbate vulnerabilities or create new ones and provide tools and guidance accordingly.

This is unhelpful in many ways. First, ensuring that AI development and deployment complies with existing law and regulation should not be presented as a possibility, but as an absolute minimum requirement. Second, the duties of the regulators under the EA 2010 and HRA 1998 are likely to play a very small role here. What is crucial is to ensure that the development and use of the AI is compliant with them, especially where the use is by public sector entities (for which there is no general regulator—and in relation to which a passing reference to the EHRC guidance on AI use in the public sector will not be sufficient to support regulators in developing nuanced and pragmatic approaches). In failing to explicitly acknowledge the existence of approaches to the assessment of AI and algorithmic impacts on fundamental and human rights, the guidance creates obfuscation by omission.

‘Contestability and redress’ is the most underdeveloped principle in the AI guidance, perhaps because no technical standard addresses this issue.

final thoughts

In my view, the AI guidance does little to support regulators, especially those with less capability and resources, in their (voluntary? short-term?) task of issuing guidance in their respective remits. Meaningful AI guidance needs to provide much clearer explanations of what is expected and required for the correct implementation of the five regulatory principles. It needs to address in a centralised and unified manner the assessment of existing and emerging technical standards against the regulatory benchmark. It also needs to synthesise the multiple guidance documents issued (and to be issued) by regulators—which it currently simply lists in Annex 1—to avoid a multiplication of the effort required to assess their (in)comptability and duplications. By leaving all these tasks to the regulators, the AI guidance (and the centralised function from which it originates) does little to nothing to move the regulatory needle beyond industry-led self-regulation and fails to discharge regulators from the burden of issuing AI guidance.

High hopes but little movement for public sector AI use regulation through procurement in the UK Government's 'Pro-innovation Approach' response

The UK Government has recently published its official response (the ‘response’) to the public consultation of March 2023 on its ‘pro-innovation approach’ to AI regulation (for an initial discussion, see here). The response shows very little movement from the original approach and proposals and, despite claiming that significant developments have already taken place, it mainly provides a governmental self-congratulatory narrative and limited high-level details of a regulatory architecture still very much ‘under construction’. The publication of the response was coupled with that of Initial Guidance for Regulators on Implementing the UK’s AI Regulatory Principles (Feb 2024), which I will comment in a subsequent post.

A section of particular interest in the response refers to ‘Ensuring AI best practice in the public sector’ (at 21-22), which makes direct reference to the use of public procurement and the exercise of public sector buying power as a regulatory lever.

This section describes some measures being put in place or planned to seize ‘the opportunities presented by AI to deliver better public services including health, education, and transport’, such as:

  • tripling the number of technical AI engineers and developers within the Cabinet Office to create a new AI Incubator for the government’ (para 41).
    This is an interesting commitment to building in-house capability. It would however be interesting to know whether these are new or reassigned roles, as well as how the process of recruitment and retention is faring, given the massive difficulties evidenced in the recent analysis by the National Audit Office, Digital transformation in government: addressing the barriers to efficiency (10 Mar 2023, HC 2022-23, 1171).

  • The government is also using the procurement power of the public sector to drive responsible and safe AI innovation. The Central Digital and Data Office (CDDO) has published guidance on the procurement and use of generative AI for the UK government. Later this year, DSIT will launch the AI Management Essentials scheme, setting a minimum good practice standard for companies selling AI products and services. We will consult on introducing this as a mandatory requirement for public sector procurement, using purchasing power to drive responsible innovation in the broader economy’ (para 43).
    This is also an interesting aspiration, for several reasons. First, the GenAI guidance is very generic and solely highlights pre-existing (also very generic) guidance on how to carry out procurement of AI (see screenshot below). This can hardly be seen as a meaningful development of the existing regulatory framework. Second, the announcement of an ‘AI Management Essentials’ scheme seems to be mirrored on the ‘Cyber Essentials’ scheme in the area of cyber security, despite significant differences and the much higher level of complexity that can be expected from an ‘all-encompassing’ scheme for the management of the myriad risks generated by the deployment of AI.

Screenshot of the webpage https://www.gov.uk/government/publications/generative-ai-framework-for-hmg/generative-ai-framework-for-hmg-html (accessed 22 February 2024), where this information is available in accessible format.

  • This builds on the Algorithmic Transparency Recording Standard (ATRS), which established a standardised way for public sector organisations to proactively publish information about how and why they are using algorithmic methods in decision-making. Following a successful pilot of the standard, and publication of an approved cross-government version last year, we will now be making use of the ATRS a requirement for all government departments and plan to expand this across the broader public sector over time’ (para 44).
    This is also interesting in that the ‘success’ attributed to the development of the ATRS is very clearly undermined by the almost absolute lack of use other than in relation to the pilot projects (see screenshot below). It is also interesting that the ATRS allows public sector AI deployers to fill in but not publish the relevant documents, as a form of self-reflective/evaluative exercise. I wonder how many publications we will see in the coming months, even if ‘use of the ATRS’ becomes a requirement.

Screenshot of the list of published transparency disclosures at Algorithmic Transparency Reports - GOV.UK (www.gov.uk) (accessed 22 February 2024), where this information is available in accessible format.

Overall, I think the response to the ‘pro-innovation’ AI regulation consultation does little to back up the high expectations being placed in public procurement as a mechanism of regulation by contract. I will update the analysis in this UK-focused paper on the use of procurement to regulate public sector AI use before final publication, but there will be little change. The broader analysis in my recent monograph also remains applicable (phew): Digital Technologies and Public Procurement. Gatekeeping and experimentation in digital public governance (OUP 2024).

The principle of competition is dead. Long live the principle of competition (Free webinar)

Free webinar: 22 March 2024 *revised time* 1pm UK / 2pm CET / 3pm EET. Registration here.

The role of competition in public procurement regulation continues to be debated. While it is generally accepted that the proper functioning of procurement markets requires some level of competition – and the European Court of Auditors has recently pointed out that current levels of competition for public contracts in the EU are not satisfactory – the 'legal ranking' and normative weight of competition concerns are much less settled.

This has been evidenced in a recent wave of academic discussion on whether there is a general principle of competition at all in Directive 2014/24/EU, what is its normative status and how it ranks vis-à-vis sustainability and environmental considerations, and what are its practical implications for the interpretation and application of EU public procurement law.

Bringing together voices representing a wide range of views, this webinar will explore these issues and provide a space for reflective discussion on competition and public procurement. The webinar won't settle the debate, but hopefully it will allow us to take stock and outline thoughts for the next wave of discussion. It will also provide an opportunity for an interactive Q&A.

Speakers:

  • Prof Roberto Caranta, Full Professor of Administrative Law, University of Turin.

  • Mr Trygve Harlem Losnedahl, PhD researcher, University of Oslo.

  • Dr Dagne Sabockis, Senior Associate, Vinge law firm; Stockholm School of Economics.

  • Prof Albert Sanchez-Graells, Professor of Economic Law, University of Bristol.

Pre- or post-reading:

Centralised procurement for the health care sector -- bang for your pound or siphoning off scarce resources?

The National Health Service (NHS) has been running a centralised model for health care procurement in England for a few years now. The current system resulted from a redesign of the NHS supply chain that has been operational since 2019 [for details, see A Sanchez-Graells, ‘Centralisation of procurement and supply chain management in the English NHS: some governance and compliance challenges’ (2019) 70(1) NILQ 53-75.]

Given that the main driver for the implementation and redesign of the system was to obtain efficiencies (aka savings) through the exercise of the NHS’ buying power, both the UK’s National Audit Office (NAO) and the House of Commons’ Public Accounts Committee (PAC) are scrutinising the operation of the system in its first few years.

The NAO published a scathing report on 12 January 2024. Among many other concerning issues, the report highlighted how, despite the fundamental importance of measuring savings, ‘NHS Supply Chain has used different methods to report savings to different audiences, which could cause confusion.’ This triggered a clear risk of recounting (ie exaggeration) of claims of savings, as detailed below.

In my submission of written evidence to the PAC Inquiry ‘NHS Supply Chain and efficiencies in procurement’, I look in detail at the potential implications of the use of different savings reporting methods for the (mis)management of scarce NHS resources, should the recounting of savings have allowed private subcontractors to also overclaim savings in order to boost the financial return under their contracts. The full text of my submission is reproduced below, in case of interest.

nao’s findings on recounting of savings

There are three crucial findings in the NAO’s report concerning the use of different (and potentially problematic) savings reporting methods. They are as follows:

DHSC [the Department of Health and Social Care] set Supply Chain a cumulative target of making £2.4 billion savings by 2023-24. Supply Chain told us that it had exceeded this target by the end of 2022-23 although we have not validated this saving. The method for calculating this re-counted savings from each year since 2015-16. Supply Chain calculated its reported savings against the £2.4 billion target by using 2015-16 prices as its baseline. Even if prices had not reduced in any year compared with the year before, a saving was reported as long as prices were lower than that of the baseline year. This method then accumulated savings each year, by adding the difference in price as at the baseline year, for each year. This accumulation continued to re-count savings made in earlier years and did not take inflation into account. For example, if a product cost £10 in 2015-16 and reduced to £9 in 2016-17, Supply Chain would report a saving of £1. If it remained at £9 in 2017-18, Supply Chain would report a total saving of £2 (re-counting the £1 saved in 2016-17). If it then reduced to £8 in 2018-19, Supply Chain would report a total saving of £4 (re-counting the £1 saved in each of 2016-17 and 2017-18 and saving a further £2 in 2018-19) […]. DHSC could not provide us with any original sign-off or agreement that this was how Supply Chain should calculate its savings figure (para 2.4, emphasis added).

Supply Chain has used other methods for calculating savings which could cause confusion. It has used different methods for different audiences, for example, to government, trusts and suppliers (see Figure 5). When reporting progress against its £2.4 billion target it used a baseline from 2015-16 and accumulated the amount each year. To help show the savings that trusts have made individually, it also calculates in-year savings each trust has made using prices paid the previous year as the baseline. In this example, if a trust paid £10 for an item in 2015-16, and then procured it for £9 from Supply Chain in 2016-17 and 2017-18, Supply Chain would report a saving of £1 in the first year and no saving in the second year. These different methods have evolved since Supply Chain was established and there is a rationale for each. Having several methods to calculate savings has the potential to cause confusion (para 2.6, emphasis added).

When I read the report, I thought that the difference between the methods was not only problematic in itself, but also showed that the ‘main method’ for NHS Supply Chain and government to claim savings, in allowing recounting of savings, was likely to have allowed for excessive claims. This is not only a technical or political problem, but also a clear risk of siphoning off NHS scarce budgetary resources, for the reasons detailed below.

Submission to the pac inquiry

00. This brief written submission responds to the call for evidence issued by the Public Accounts Committee in relation to its Inquiry “NHS Supply Chain and efficiencies in procurement”. It focuses on the specific point of ‘Progress in delivering savings for the NHS’. This submission provides further details on the structure and functioning of NHS Supply Chain than those included in the National Audit Office’s report “NHS Supply Chain and efficiencies in procurement” (2023-24, HC 390). The purpose of this further detail is to highlight the broader implications that the potential overclaim of savings generated by NHS Supply Chain may have had in relation to payments made to private providers to whom some of the supply chain functions have been outsourced. It raises some questions that the Committee may want to explore in the context of its Inquiry.

1. NHS Supply Chain operating structure

01. The NAO report analyses the functioning and performance of NHS Supply Chain and SCCL in a holistic manner and without considering details of the complex structure of outsourced functions that underpins the model. This can obscure some of the practical impacts of some of NAO’s findings, in particular in relation with the potential overclaim of savings generated by NHS Supply Chain (paras 2.4, 2.6 and Figure 5 in the report). Approaching the analysis at a deeper level of detail on NHS Supply Chain’s operating structure can shed light on problems with the methods for calculating NHS Supply Chain savings other than the confusion caused by the use of multiple methods, and the potential overclaim of savings in relation to the original target set by DHSC.

02. NHS Supply Chain does not operate as a single entity and SCCL is not the only relevant actor in the operating structure.[1] Crucially, the operating model consists of a complex network of outsourcing contracts around what are called ‘category towers’ of products and services. SCCL coordinates a series of ‘Category Tower Service Providers’ (CTSPs), as listed in the graph below. CTSPs have an active role in developing category management strategies (that is, the ‘go to market approach’ at product level) and heavily influence the procurement strategy for the relevant category, subject to SCCL approval.

03. CTSPs are incentivised to reduce total cost in the system, not just reduce unit prices of the goods and services covered by the relevant category. They hold Guaranteed Maximum Price Target Cost (GMPTC) contracts, under which CTSPs will be paid the operational costs incurred in performing the services against an annual target set out in the contract, but will only make a profit when savings are delivered, on a gainshare basis that is capped.

Source: NHS Supply Chain - New operating model (2018).[2]

04. There are very limited public details on how the relevant targets for financial services have been set and managed throughout the operation of the system. However, it is clear that CTSPs have financial incentives tied to the generation of savings for SCCL. Given that SCCL does not carry out procurement activities without CTSP involvement, it seems plausible that SCCL’s own targets and claimed savings would (primarily) have been the result of the simple aggregation of those of CTSPs. If that is correct, the issues identified in the NAO report may have resulted in financial advantages to CTSPs if they have been allowed to overclaim savings generated.

05. NHS Supply Chain has publicly stated that[3]:

  • ‘Savings are contractual to the CTSPs. As part of the procurement, bidders were asked to provide contractual savings targets for each year. These were assessed and challenged through the process and are core to the commercial model. CTSPs cannot attain their target margins (i.e. profit) unless they are able to achieve contractual savings.’

  • ‘The CTSPs financial reward mechanism [is] based upon a gain share from the delivery of savings. The model includes savings generated across the total system, not just the price of the product. The level of gain share is directly proportional to the level of savings delivered.’

06. In view of this, if CTSPs had been allowed to use a method of savings calculation that re-counted savings in the way NAO details at para 2.4 of its report, it is likely that their financial compensation will have been higher than it should have been under alternative models of savings calculation that did not allow for such re-count. Given the volumes of savings claimed through the period covered by the report, any potential overcompensation could have been significant. As any such overcompensation would have been covered by NHS funding, the Committee may want to include its consideration within its Inquiry and in its evidence-gathering efforts.

__________________________________

[1] For a detailed account, see A Sanchez-Graells, “Centralisation of procurement and supply chain management in the English NHS: some governance and compliance challenges” (2019) 70(1) Northern Ireland Legal Quarterly 53-75.

[2] Available at https://wwwmedia.supplychain.nhs.uk/media/Customer_FAQ_November_2018.pdf (last accessed 12 January 2024).

[3] Ibid, FAQs 24 and 25.

Public Procurement of Artificial Intelligence: recent developments and remaining challenges in EU law

Now that the (more than likely) final of the EU AI Act is available, and building on the analysis of my now officially published new monograph Digital Technologies and Public Procurement (OUP 2024), I have put together my assessment of its impact for the procurement of AI under EU law and uploaded on SSRN the new paper: ‘Public Procurement of Artificial Intelligence: recent developments and remaining challenges in EU law’. The abstract is as follows:

EU Member States are increasingly experimenting with Artificial Intelligence (AI), but the acquisition and deployment of AI by the public sector is currently largely unregulated. This puts public procurement in the awkward position of a regulatory gatekeeper—a role it cannot effectively carry out. This article provides an overview of recent EU developments on the public procurement of AI. It reflects on the narrow scope of application and questionable effectiveness of tools linked to the EU AI Act, such as technical standards or model contractual clauses, and highlights broader challenges in the use of procurement law and practice to regulate the adoption and use of ‘trustworthy’ AI by the public sector. The paper stresses the need for an alternative regulatory approach.

The paper can be freely downloaded: A Sanchez-Graells, ‘Public Procurement of Artificial Intelligence: recent developments and remaining challenges in EU law’ (January 25, 2024). To be published in LTZ (Legal Tech Journal) 2/2024: https://ssrn.com/abstract=4706400.

As this will be an area of contention and continuous developments, comments most welcome!

Source: h

Implementation Challenges for the Procurement Act 2023

I have put together a consolidated overview of the primary challenges for the implementation of the Procurement Act 2023, to be included as a country report in a forthcoming issue of the European Procurement & Public Private Partnership Law Review.

It brings together developments discussed in the blog over the last year or so, including the transparency ambition, the innovation ambition, and the training offer linked to the Transforming Public Procurement project.

In case of interest, it can be downloaded from SSRN: https://ssrn.com/abstract=4692660.

It contains nothing new, though, so assiduous readers may want to skip this one!

Resh(AI)ping Good Administration: Addressing the mass effects of public sector digitalisation

Happy New Year! I hope 2024 is off to a good start for you.

My last project of last year (finished on the buzzer…) was a paper expanding the ideas first floated in the DigiCon blog post ‘Resh(AI)ping good administration: beyond systemic risks vs individual rights?’, which sparked interesting discussion at the DigiCon III conference last fall.

With a slightly different (and hopefully clearer) title, the paper is now under peer-review (and so, as always, comments welcome ahead of a final revision!).

Titled ‘Resh(AI)ping Good Administration: Addressing the mass effects of public sector digitalisation’, the paper focuses on what I think is the most distinctive feature of public sector digitalisation and the prime challenge to traditional good administration guarantees: mass effects. Its abstract is as follows:

Public sector digitalisation is transforming public governance at an accelerating rate. Digitalisation is outpacing the evolution of the legal framework. Despite several strands of international efforts to adjust good administration guarantees to new modes of digital public governance, progress has so far been slow and tepid. The increasing automation of decision-making processes puts significant pressure on traditional good administration guarantees, jeopardises individual due process rights, and risks eroding public trust. Automated decision-making has so far attracted the bulk of scholarly attention, especially in the European context. However, most analyses seek to reconcile existing duties towards individuals under the right to good administration with the challenges arising from digitalisation. Taking a critical and technology-centred doctrinal approach to developments under the law of the European Union and the Council of Europe, this paper goes beyond current debates to challenge the sufficiency of existing good administration duties. By stressing the mass effects that can derive from automated decision-making by the public sector, the paper advances the need to adapt good administration guarantees to a collective dimension through an extension and a broadening of the public sector’s good administration duties: that is, through an extended ex ante control of organisational risk-taking, and a broader ex post duty of automated redress. These legal modifications should be urgently implemented.

Sanchez-Graells, Albert, ‘Resh(AI)ping Good Administration: Addressing the mass effects of public sector digitalisation’ (December 19, 2023). Available at SSRN: https://ssrn.com/abstract=4669589.

Responsibly Buying Artificial Intelligence: A ‘Regulatory Hallucination’ -- draft paper for comment

© Matt Lowe/LinkedIn.

Following yesterday’s Current Legal Problems Lecture, I have uploaded the current full draft of the paper on SSRN. I would be very grateful for any comments in the next few weeks, as I plan to do a final revision and to submit it for peer-review in early 2024. Thanks in advance for those who take the time. As always, you can reach me at a.sanchez-graells@bristol.ac.uk.

The abstract of the paper is as follows:

Here, I focus on the UK’s approach to regulating public sector procurement and use of artificial intelligence (AI) in the context of the broader ‘pro-innovation’ approach to AI regulation. Borrowing from the description of AI ‘hallucinations’ as plausible but incorrect answers given with high confidence by AI systems, I argue that UK policymaking is trapped in a ‘regulatory hallucination.’ Despite having embraced the plausible ‘pro-innovation’ regulatory approach with high confidence, that is the incorrect answer to the challenge of regulating AI procurement and use by the public sector. I conceptualise the current strategy as one of ‘regulation by contract’ and identify two of its underpinning presumptions that make its deployment in the digital context particularly challenging. I show how neither the presumption of superiority of the public buyer over the public contractor, nor the related presumption that the public buyer is the rule-maker and the public contractor is the rule-taker, necessarily hold in this context. Public buyer superiority is undermined by the two-sided gatekeeping required to simultaneously discipline the behaviour of the public sector AI user and the tech provider. The public buyer’s rule-making role is also undermined by its reliance on industry-led standards, as well as by the tech provider’s upper hand in setting contractual benchmarks and controlling the ensuing self-assessments. In view of the ineffectiveness of regulating public sector AI use by contract, I then sketch an alternative strategy to boost the effectiveness of the goals of AI regulation and the protection of individual rights and collective interests through the creation of an independent authority.

Sanchez-Graells, Albert, ‘Responsibly Buying Artificial Intelligence: A “Regulatory Hallucination”’ (November 24, 2023). Current Legal Problems 2023-24, Available at SSRN: https://ssrn.com/abstract=4643273.

Responsibly Buying Artificial Intelligence: A Regulatory Hallucination?

I look forward to delivering the lecture ‘Responsibly Buying Artificial Intelligence: A Regulatory Hallucination?’ as part of the Current Legal Problems Lecture Series 2023-24 organised by UCL Laws. The lecture will be this Thursday 23 November 2023 at 6pm GMT and you can still register to participate (either online or in person). These are the slides I will be using, in case you want to take a sneak peek. I will post a draft version of the paper after the lecture. Comments welcome!

Public procurement (entry for an Encyclopaedia)

I was invited to provide an entry on ‘public procurement’ for the forthcoming Elgar Encyclopedia of European Law co-edited by Andrea Biondi and Oana Stefan. I must say I struggled to decide what to write about, as the entry was limited to 4,000 words and there are so many (!!) things going on in procurement. Below is my draft entry with perhaps an eclectic choice of content. Comments most welcome!

The draft entry is also available on SSRN if you prefer a pdf version: A Sanchez-Graells, ‘Public procurement’ in A Biondi and O Stefan, Elgar Encyclopedia of European Law (forthcoming) available at https://ssrn.com/abstract=4621399.

Public Procurement

I. Introduction

From up close, public procurement law can be seen as the set of mostly procedural rules controlling the way in which the public sector buys goods, services, and works from the market. Procurement would thus be a set of administrative law requirements concerned with the design and advertisement of tenders for public contracts, the decision-making process leading to the award of those contracts, and the advertisement and potential challenge of such decisions. To a more limited extent, some requirements would extend to the contract execution phase, and control in particular the modification and eventual termination of public contracts. From this narrow perspective, procurement would be primarily concerned with ensuring the integrity and probity of decision-making processes involving the management of public funds, as well as fostering the generation of value for money through effective reliance on competition for public contracts.

The importance and positive contribution of public procurement law to the adequate management of public funds may seem difficult to appreciate in ordinary times, and there are recurrent calls for a reduction of the administrative burden and bureaucracy related to procurement procedures, checks and balances. However, as the pervasive abuses of direct awards under the emergency conditions generated by the covid pandemic evidenced in virtually all jurisdictions, dispensing with those requirements, checks and balances comes with a very high price tag for taxpayers in terms of corruption, favouritism, and wastage of public funds.

Even from this relatively narrow perspective of procurement as a process-based mechanism of public governance, procurement attracts a significant amount of attention from EU legislators and from the EU Courts and is an area of crucial importance in the development of the European administrative space. As procurement regulation has been developed through successive generations of directives, and as many Member States had long traditions on the regulation of public procurement prior to the emergence of EU law on the topic, procurement offers a fertile ground for comparative public law scholarship. More recently, as EU procurement policy increasingly seeks to promote cross-border collaboration, procurement is also becoming a driver (or an irritant) for the transnational regulation of administrative processes and a living lab for experimentation and legal innovation.

From a slightly broader perspective, public procurement can be seen as a tool for the self-organisation of the State and as a primary conduit for the privatisation and outsourcing of State functions. A decision preceding procurement concerns the size and shape of the State, especially in relation to which functions and activities the State carries out in-house (including through public-public collaboration mechanisms), and which other are contracted out to the market (‘make or buy’ decisions). Procurement then controls the design and award of contracts involving the exercise of public powers, or the direct provision of public services to citizens where market agents are called upon to do so (including in the context of quasi-markets). Procurement thus heavily influences the interaction between the State’s contractual agents and citizens, and becomes a tool for the regulation of public service delivery. The more the State relies on markets for the provision of public services, the larger the potential influence (both positive and negative) of procurement mechanisms on citizens’ experience of their (indirect) interaction with the State. On this view, procurement is a tool of public governance and a conduit for public-private cooperation, as well as a regulatory mechanism for delegated public-public and public-private interactions. From this perspective, procurement is often seen as a neoliberal tool closely linked to new public management (NPM), although it should be stressed that procurement rules only activate once the decision to resort to contracting out or outsourcing has been made, as EU law does not mandate ‘going to market’.

From an even broader perspective, public procurement represents a more complex and multi-layered regulatory instrument. Given the enormous amounts of public funds channelled through public procurement, and the market-shaping effects that can follow from the exercise of such buying power, procurement regulation is often used as a lever for the promotion of policies and goals well beyond the narrower confines of procurement as a regulated administrative process. In the EU, procurement has always been an instrument of internal market regulation and sought to dismantle barriers to cross-border competition for the award of public contracts. More recently, and in line with developments in other jurisdictions, procurement has been increasingly singled out as a tool to promote environmental and sustainability goals, as well as social goals, or as a tool to foster innovation. Procurement is also increasingly identified as a tool to foster compliance with human rights along increasingly complex supply chains, or to address social inequality, such as through gender responsive procurement. In the face of the challenges posed by the mainstreaming of digital technologies, and artificial intelligence in particular, procurement is also increasingly identified as a tool of digital regulation. And, against the background of rule of law challenges within the EU, procurement conditionality has added to the fiscal control effect traditionally linked to the use of EU funds to subsidise procurement projects at Member State level. From this perspective, procurement is either an enforcement (or reinforcement) mechanism, or a self-standing regulatory tool for the pursuit of an increasingly diverse array of horizontal policies seeking to steer market activities.

Relatedly, given the importance of procurement as an economic activity, its regulation is of crucial importance in the context of industrial and trade policies. The interaction between procurement and industrial policy is not entirely straightforward, and neither is the position of procurement in the context of trade liberalisation. While there have been waves of policy efforts seeking to minimise the use of procurement for industrial policy purposes (ie the award of public contracts to national champions), in particular given the State aid implications of such uses of public contracts under EU law, and while there is a general push for the liberalisation of international trade through procurement—there are also periodic waves of protectionism where procurement is used as a tool of international economic regulation or, more broadly, geopolitics. Most recently, the EU has aggressively (re)regulated access to its procurement markets on grounds of such considerations.

It would be impossible to address all the issues that arise from the regulation of public procurement in all these (and other potential) dimensions within a single entry. Here, I will touch upon some the issues highlighted by recent developments in EU law and policy, and in relation to contemporary debates around the salient grand challenges encapsulated in the need for procurement to support the ‘twin transition’ to green and digital. I will not focus on the detail of procurement rules, which is better left to in-depth analysis (eg Arrowsmith [2014] and [2018], Steinicke and Vesterdorf [2018], or Caranta and Sanchez-Graells [2021]). There are a few common threats in the developments discussed below, especially in relation to the increasing complexity of procurement policymaking and administration, or the crucial role of expertise and capability, as well as some challenges in coordinating them in a way that generates meaningful outcomes. I will briefly return to these issues in the conclusion.

II. Procurement, Trade, and Geopolitics

A constant tension in the regulation of procurement concerns the openness of procurement markets. On the one hand, procurement can be a catalyst for trade liberalisation and there are many economic advantages stemming from increased (international) competition for public contracts—as evidenced in the context of the World Trade Organisation Government Procurement Agreement (WTO GPA) (Georgopoulos et al [2017]). In the narrower context of the EU’s internal market, public procurement openness is taken to its logical extremes and barriers to cross-border tendering are systematically dismantled through legislation, such as the most recent 2014 Public Procurement Package, and its interpretation by the Court of Justice. While there is disparity in national practice, the (complete) openness of procurement markets in the EU tends to not only benefit EU tenderers, but also those of third countries, who tend to be treated equally with EU ‘domestic’ tenderers.

On the other hand, the same (international) competition that can bring economic advantages can also put pressure on (less competitive) domestic industries or create risks of uneven playing field—especially where (foreign national champion) tenderers are propped up by their States. In some industries and in relation to some critical infrastructure, the award of oftentimes large and sensitive public contracts to foreign undertakings also generates concerns around safety and sovereignty.

A mechanism to mediate this tension is to make procurement-related trade liberalisation conditional on reciprocity, which in turn leverages multilateral instruments such as the WTO GPA. This is an area where EU law has recently generated significant developments. After protracted negotiations, EU procurement law now comprises a set of three instruments seeking to rebalance the (complete) openness of EU procurement markets.

As a starting point, under EU law, only foreign economic operators covered by an existing international agreement (such as the WTO GPA, or bilateral or multilateral trade agreements concluded with the EU that include commitments on access to public procurement) are entitled to equal treatment. However, differential treatment or outright exclusion of economic operators not covered by such equal treatment obligation tends (or has historically tended to) be rare. This can be seen to weaken the hand of the European Commission in international negotiations, as EU procurement markets are de facto almost entirely open, regardless of the much more limited legal openness resulting from those international agreements.

To nudge contracting authorities to enforce differential treatment, in 2020, the European Commission issued guidance on the participation of third country bidders and goods in EU procurement markets, stressing the several ways in which public buyers could address concerns regarding unfair competitive advantages of foreign tenderers. This should be seen as a first step towards ramping up the ‘rebalancing’ of access to EU procurement markets, though it is a soft (law) step and one that would still hinge on coordinated decision-making by a very large number of public buyers making tender-by-tender decisions.

A second and crucial step was taken in 2022 with the adoption of the EU’s International Procurement Instrument (IPI), which empowers the European Commission to carry out investigations where there are concerns about measures or practices negatively affecting the access of EU businesses, goods and services to non-EU procurement markets and, eventually, to impose (centralised) IPI measures to restrict access to EU public procurement procedures for businesses, goods and services from the non-EU countries concerned. The main effect of the IPI can be expected to be twofold. Outwardly, the IPI will lead to the European Commission having ‘a stick’ to push for reciprocity in procurement liberalisation as a complement to ‘the carrot’ used to persuade more and more countries to enter into bilateral trade deals, or for them to join the WTO GPA. Internally, the IPI will allow the Commission to mandate Member States to implement the relevant restrictions or exclusions from the EU procurement markets in relation to the jurisdictions concerned. This is expected to address the issue of de facto openness beyond existing (international) legal requirements, and therefore galvanise the ability of the Commission to control access to ‘the EU procurement market’ and thus bolster its ability to use procurement reciprocity as a tool for trade liberalisation more effectively.

A third and final crucial step came with the adoption in 2023 of the Regulation on foreign subsidies distorting the internal market, which creates a mechanism for the control of potential foreign subsidies in tenders for contracts with an estimated value above EUR 250 million, and can also result in the imposition of (centralised) measures curving access to the relevant contracts by the beneficiaries of those foreign subsidies. This comes to somehow create an international functional equivalent to the State aid control in place for domestic tenders, as well as a mechanism for the EU to enforce international anti-dumping standards within its own jurisdiction.

This trend of evolution in EU public procurement regulation evidences that public buyers are increasingly constrained by geopolitical and international economic considerations administered by the European Commission in a centralised manner (Andhov and Kania [2023]). Whether this will create friction between the Commission and Member States, perhaps in relation to particularly critical or sensitive procurement projects, remains to be seen. In any case, this line of policy and legal developments generates increased complexity in the administration of procurement processes on a day-to-day basis, and will require public buyers to develop expertise in the assessment of the relevant trade-related instruments and associated documentation, which will be a theme in common with other developments discussed below.

III. Procurement and Sustainability

It is relatively uncontroversial that public expenditure has a crucial role to play in supporting (or driving) the transition towards a more sustainable economy, and most jurisdictions explicitly consider how to harness public expenditure to decarbonise their economy and achieve net zero targets—sometimes in the broader context of efforts to achieve interlinked sustainable development goals. However, the details on the specific sustainability goals to be pursued through procurement (as compared to other means of public finances, such as subsidies or tax incentives), and on how to design and implement sustainable procurement are more contested.

Green procurement has been a primary focus of EU public procurement policy for a long time now, and it has received even further increased attention in recent years, culminating in the attribution of a prominent role for the implementation of the EU’s Green Deal. EU procurement law has been increasingly permissive and facilitative of the inclusion of environmental considerations in procurement decision-making and the European Commission has developed sets of guidance and technical documentation that are kept under permanent review and update. Overall, EU procurement law offers a diverse toolkit for public buyers to embed sustainability requirements.

However, the uptake of green procurement is much lower than would be desirable and progress is very uneven across jurisdictions and in different sectors of the economy. There is a growing realisation that facilitative or permissive approaches will not result in the quick generalisation of sustainability concerns across procurement practice required to contribute to mitigating the devastating effects of climate change in a timely fashion, or with sufficient scale. Informational and skills barriers, difficult economic assessments and competing (political) priorities necessarily slow down the uptake of sustainable procurement. In this context, it seems clear that technical complexity in the administration of procurement on a day-to-day basis, and limited technical skills in relation to sustainability assessments, are the primary obstacle in the road to mainstreaming sustainable public procurement. It is hard for public buyers to identify the relevant sustainability requirements and to embed them in their decision-making, especially where the inclusion of such requirements is bound to be checked against its suitability, proportionality, and its effect on potential competition for the relevant public contract.

To overcome this obstacle, it seems clear that a more proactive or prescriptive approach is required and that sustainability requirements must be embedded in legislation that binds public buyers—so that their role becomes one of (reinforced) compliance assessment or indirect enforcement. The question that arises, and which reopens age old discussions, is whether such legislation should solely target public procurement (Janssen and Caranta [2023]) or rather be of general application across the economy (Halonen [2021]).

This controversy evidences different understandings of the role of procurement-specific legislation and different levels of concern with the partitioning of markets. While the passing of procurement-specific legislation could be easier and politically more palatable—as it would be perceived to ultimately impose the relevant burden on economic operators seeking to gain public business (and so embed a certain element of opt-in or balanced regulatory burden against the prospect of accessing public funds), and the cost would ultimately fall on public buyers as ‘responsible (sustainable) buyers’—it would partition markets and eg potentially prevent the generation of economies of scale where public demand is not majoritarian. Moreover, such market partitioning would raise entry barriers for entities new to bidding for public contracts, as well as facilitate the emergence of anticompetitive and collusive practices in the more concentrated and partly isolated from potential competition ‘public markets’ (Sanchez-Graells [2015]) in ways that general legislation would not. More generally, advances in mandating sustainable procurement could deactivate the pressure for developments in more general sustainability mandates, as policymakers could claim to already be doing significant efforts (in the narrow setting of procurement).

A narrow sectoral approach to legislating for public procurement only would probably also over-rely on the hopes that procurement practices can become best practices and thus disseminate themselves across the economy through some understanding of mimicking, or race to the top. This relates to discussions in other areas and to the broader expectation that procurement can be a trend setter and influence industry practice and standards. However, as the discussion on digitalisation will show, the direction of influence tends to be on reverse and there are very limited mechanisms to promote or force industry adaptation to procurement standards other than in relation to direct access to procurement.

IV. Procurement and the ‘Digital Transformation’ of the State

Another area of growing consensus is that public procurement has a key role to play in the ‘digital transformation’ of the State, as the process of digitalisation is bound to rely on the acquisition of technology from market providers to a large or sole extent (depending on each jurisdiction’s make or buy decisions). This can in turn facilitate the role of procurement as a tool of digital industrial policy, especially because procurement expenditure can be a way of ensuring demand for innovation, and because public sector technology adoption can be used as a domain for experimentation with new technologies and new forms of technology-enabled governance.

The European Union has set very high expectations in its Digital Agenda 2030, and the Commission has recently stressed that achieving them would require roughly doubling the predicted level of public procurement expenditure in digital technologies, and artificial intelligence (AI) in particular. It can thus be expected that the procurement of digital technologies will quickly gain practical importance even in jurisdictions that have been lagging so far.

However, echoing some of the issues concerning sustainable procurement, in this second stream of the ‘twin transition’, the uptake of procurement of digital technologies is slowed down by the complexity of procuring unregulated immature technologies, and the (digital) skills gaps in the public sector—which are exacerbated by the absence of a toolkit of regulatory and practical resources equivalent to that of green procurement. In such a context of technological fluidity and hype, given the skills and power imbalances between technology providers and public buyers, the shortcomings of the use of public procurement as a regulatory mechanism become stark and the flaws in the logic or expectation that procurement can be an effective tool of market steering are laid bare (Sanchez-Graells [2024]).

Public buyers are expected to act as responsible AI buyers and to ensure the ‘responsible use of AI’ in the public sector. The EU AI Act will soon establish specific requirements in that regard, although solely in relation to high-risk AI uses as defined therein. Implementing the requirements of the EU AI Act—and their extension to other types of uses of digital technology or algorithms as a matter of ‘best practice’—will leverage procurement processes and, in particular, the ensuing public contracts to impose the relevant obligations on technology providers. In that connection, the European Commission has promoted the development of model contractual AI clauses that seek to regulate the technology to be procured and their future use by the relevant public sector deployer.

However, an analysis of the model clauses and broader guidance on the procurement of AI shows that public buyers will still face a very steep knowledge gap as it will be difficult to set the detail of the relevant contracts, which will tend to be highly context dependent. In other words, the model clauses are not ‘plug and play’ and implementing meaningful safeguards in the procurement and use of AI and other digital technologies will require advanced digital skills and sufficient commercial leverage—which are not to be taken as a given. Crucially, all obligations under the model clauses (and the EU AI Act itself) hinge on (self-assessment) processes controlled by the technology provider and/or refer back to technical standards or the state-of-the-art, which are driven and heavily influenced (or entirely controlled) by the technology industry. Public buyers are at a significant disadvantage not only to set, but also to monitor compliance with relevant requirements.

This shows that, in the absence of mandatory requirements and binding (general) legislation, the use of procurement for regulatory purposes has a high risk of commercial determination and regulatory tunnelling as public buyers with limited skills and capabilities struggle to impose requirements on technology providers, and where references to standards also displace regulatory decision-making. This means that public procurement can no longer be expected to ‘monitor itself’, and that new forms of institutional oversight are required to ensure that the procurement of digital technologies works in the broader public interest.

V. Conclusion

Although the issues discussed above may seem rather disparate, they share a few common threads. First, in all areas, the regulatory use of procurement generates complexity and makes the day-to-day administration of procurement processes more complex. It can be hard for a public buyer to navigate socio-political, sustainability and digitalisation concerns—and these are only some of the ‘non-strictly procurement-related’ concerns and considerations to be taken into account. Such difficulty can be compounded by limited capabilities and by gaps in the required skills. While this is particularly clear in the digital context, the issue of limited (technical) capability is also highly relevant in relation to sustainable procurement. An imbalance in skills and commercial leverage between the public buyer and technology providers undermines the logic of using procurement as a regulatory tool. Implementation issues thus require much further thought and investment than they currently receive.

Ultimately, the effectiveness of the regulatory goals underpinning the leveraging of procurement hinges on the ability of public buyers to meaningfully implement them. This raises the further question whether all goals can be achieved at the same time, especially where there can be difficult trade-offs. And there can be many of those. For example, it can well be that the offeror of the most attractive technology comes from a ‘black-listed’ jurisdiction. It can also be that the most attractive technology is also the most polluting, or one that raises significant other risks or harms from a social perspective, etc. Navigating these risks and making the (implicit) political choices may be too taxing a task for public buyers, as well as raise issues of democratic accountability more generally. Moreover, enabling public buyers to deal with these issues and to exercise judgement and discretion reopens the door to risks of eg bias, capture or corruption, as well as maladministration and error, which are some of the core concerns in the narrow approach to the regulation of procurement as an administrative procedure to being with. Those trade-offs are also pervasive and hard to assess.

It is difficult to foresee the future, but my intuition is that the trend of piling up of regulatory goals on procurement’s shoulders will need to slow down or reverse if it is meant to remain operational, and that a return to a more paired down understanding of the role of procurement will need to be enabled by the emergence of (generally applicable) legislation and external oversight mechanisms that can discharge procurement of these regulatory roles. Or, at least, that is the way I would like to see the broader regulation and policymaking around procurement to evolve.

Bibliography

Andhov, Marta and Michal Andrzej Kania, ‘Restricting Freedom of Contract – the EU Foreign Subsidies Regulation and its Consequences for Public Procurement’ (2023) Journal of Public Procurement.

Arrowsmith, Sue, The Law of Public and Utilities Procurement. Regulation in the EU and the UK, vols 1 & 2 (3rd edn, Sweet & Maxwell 2014 and 2018).

Caranta, Roberto and Albert Sanchez-Graells (eds), European Public Procurement. Commentary on Directive 2014/24/EU (Edward Elgar 2021).

Georgopoulos, Aris, Bernard Hoekman and Petros C Mavroidis (eds), The Internationalization of Government Procurement Regulation (OUP 2017).

Halonen, Kirsi-Maria, ‘Is public procurement fit for reaching sustainability goals? A law and economics approach to green public procurement’ (2021) 28(4) Maastricht Journal of European and Comparative Law 535-555.

Janssen, Willem and Roberto Caranta (eds), Mandatory Sustainability Requirements in EU Public Procurement Law. Reflections on a Paradigm Shift (Hart 2023).

Sanchez-Graells, Albert, Public Procurement and the EU Competition rules (2nd end, Hart, 2015).

Sanchez-Graells, Albert, Digital Technologies and Public Procurement. Gatekeeping and Experimentation in Digital Public Governance (OUP 2024).

Steinicke, Michael and Peter L Vesterdorf (eds), Brussels Commentary on EU Public Procurement Law (C H Beck, Hart & Nomos 2018).

Innovation procurement under the Procurement Act 2023 -- changing procurement culture on the cheap?

On 13 November 2023, the UK Government published guidance setting out its ambitions for innovation procurement under the new Procurement Act 2023 (not yet in force, of which you can read a summary here). This further expands on the ambitions underpinning the Transforming Public Procurement project that started after Brexit. The Government’s expectation is that the ‘the new legislation will allow public procurement to be done in more flexible and innovative ways’, and that this will ‘enable public sector organisations to embrace innovation more’.

The innovation procurement guidance bases its expectation that the Procurement Act will unlock more procurement of innovation and more innovative procurement on the ambition that this will be an actively supported policy by all relevant policy- and decision-makers and that there will be advocacy for the development of commercial expertise. A first hurdle here is that unless such advocacy comes with the investment of significant funds in developing skills (and this relates to both commercial and technical skills, especially where the innovation relates to digital technologies), such high-level political buy-in may not translate into any meaningful changes. The guidance itself acknowledges that the ‘overall culture, expertise and incentive structure of the public sector has led to relatively low appetite for risk and experimentation’. Therefore, that greater investment in expertise needs to be coupled with a culture change. And we know this is a process that is very difficult to push forward.

The guidance also indicates that ‘Greater transparency of procurement data will make it easier to see what approaches have been successful and encourage use of those approaches more widely across the public sector.’ This potentially points to another hurdle in unlocking this policy because generic data is not enough to support innovation procurement or the procurement of innovation. Being able to successfully replicate innovation procurement practices requires a detailed understanding of how things were done, and how they need to be adapted when replicated. However, the new transparency regime does not necessarily guarantee that such granular and detailed information will be available, especially as the practical level of transparency that will stem from the new obligations crucially hinges on the treatment of commercially sensitive information (which is exempted from disclosure in s.94 PA 2023). Unless there is clear guidance on disclosure / withholding of sensitive commercial information, it can well be that the new regime does not generate additional meaningful (publicly accessible) data to push the knowledge stock and support innovative procurement. This is an important issue that may require further discussion in a separate post.

The guidance indicates that the changes in the Procurement Act will help public buyers in three ways:

  • The new rules focus more on delivering outcomes (as opposed to ‘going through the motions’ of a rigid process). Contracting authorities will be able to design their own process, tailored to the unique circumstances of the requirement and, most importantly, those who are best placed to deliver the best solution.

  • There will be clearer rules overall and more flexibility for procurers to use their commercial skills to achieve the desired outcomes.

  • Procurers will be able to better communicate their particular problem to suppliers and work with them to come up with potential solutions. Using product demonstrations alongside written tenders will help buyers get a proper appreciation of solutions being offered by suppliers. That is particularly impactful for newer, more innovative solutions which the authority may not be familiar with.

Although the guidance document indicates that the ‘new measures include general obligations, options for preliminary market engagement, and an important new mechanism, the Competitive Flexible Procedure’, in practice, there are limited changes to what was already allowed in terms of market consultation and the general obligations— to eg publish a pipeline notice (for contracting authorities with an annual spend over £100 million), or to ‘have regard to the fact that SMEs face barriers to participation and consider whether these barriers can be removed or reduced’—are also marginal (if at all) changes from the still current regime (see regs.48 and 46 PCR 2015). Therefore, it all boils down to the new ‘innovation-friendly procurement processes’ that are enabled by the flexible (under)regulation of the competitive flexible procedure (s.20 PA 2023).

The guidance stresses that the ‘objective is that the Competitive Flexible Procedure removes some of the existing barriers to procuring new and better solutions and gives contracting authorities freedom to enable them to achieve the best fit between the specific requirement and the best the market offers.’ The example provided in the guidance provides the skeleton structure of a 3-phase procedure involving an initial ideas and feasibility phase 1, an R&D and prototype phase 2 and a final tendering leading to the award of a production/service contract (phase 3). At this level of generality, there is little to distinguish this from a competitive dialogue under the current rules (reg.30 PCR 2015). Devil will be in the detail.

Moreover, as repeatedly highlighted from the initial consultations, the under-regulation of the competitive flexible procedure will raise the information costs and risks of engaging with innovation procurement as each new approach taken by a contracting authority will require significant investment of time in its design, as well as an unavoidable risk of challenge. The incentives are not particularly geared towards facilitating risk-taking. And any more detailed guidance on ‘how to'‘ carry out an innovative competitive flexible procedure will simply replace regulation and become a de facto standard through which contracting authorities may take the same ‘going through the motions’ approach as the process detailed in teh guidance rigidifies.

The guidance acknowledges this, at least partially, when it stresses that ‘Behavioural changes will make the biggest difference’. Such behavioural changes will be supported through training, which the guidance document also describes (and there is more detail here). The training offered will consist of:

  • Knowledge drops (open to everyone): An on-demand, watchable resource up to a maximum of 45 minutes in total, providing an overview of all of the changes in legislation.

  • E-learning (for skilled practitioners within the public sector only): a learning & development self-guided course consisting of ‘10 1-hour modules and concludes with a skilled practitioner certification’.

  • Advanced course deep dives (for public sector expert practitioners only): ‘3-day, interactive, instructor-led course. It consists of virtual ‘deep dive’ webinars, which allow learners to engage with subject matter experts. This level of interaction allows a deeper insight across the full spectrum of the legislative change and support ‘hearts and minds’ change amongst the learner population (creating ‘superusers’).

  • Communities of practice (for skilled and expert practitioners only): ‘a system of collective critical inquiry and reflection into the regime changes. Supported by the central team and superusers, they will support individuals to embed what they have learned.’

As an educator and based on my experience of training expert professionals in complex procurement, I am skeptical that this amount of training can lead to meaningful changes. The 45-minute resource can hardly cover the entirety of changes in the Procurement Act, and even the 10 hour course for public buyers only will be quite limited in how far it can go. 3 days of training are also insufficient to go much further than exploring a few examples in meaningful detail. And this is relevant because that training is not only for innovation procurement, but for all types of ‘different’ procurement under the Procurement Act 2023 (ie green, social, more robustly anti-corruption, more focused on contract performance, etc). Shifting culture and practice would require a lot more than this.

It is also unclear why this (minimal) investment in public sector understanding of the procurement framework has not taken place earlier. As I already said in the consultation, all of this could have taken place years ago and a better understanding of the current regime would have led to improvements in the practice of innovative procurement in the UK.

All in all, it seems that the aspirations of more innovation procurement and more innovative procurement are pinned on a rather limited amount of training and in (largely voluntary, in addition to the day job) collaboration for super-user experienced practitioners (who will probably see their scarce skills in high demand). It is unclear to me how this will be a game changer. Especially as most of this (and in particular collaboration and voluntary knowledge exchange) could already take place. It may be that more structure and coordination will bring better outcomes, but this would require adequate and sufficient resourcing.

Whether there will be more innovation procurement then depends on whether more money will be put into procurement structures and support. From where I stand, this is by no means a given. I guess we’ll have to wait and see.

Some thoughts on the US' Executive Order on the Safe, Secure, and Trustworthy Development and Use of AI

On 30 October 2023, President Biden adopted the Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence (the ‘AI Executive Order’, see also its Factsheet). The use of AI by the US Federal Government is an important focus of the AI Executive Order. It will be subject to a new governance regime detailed in the Draft Policy on the use of AI in the Federal Government (the ‘Draft AI in Government Policy’, see also its Factsheet), which is open for comment until 5 December 2023. Here, I reflect on these documents from the perspective of AI procurement as a major plank of this governance reform.

Procurement in the AI Executive Order

Section 2 of the AI Executive Order formulates eight guiding principles and priorities in advancing and governing the development and use of AI. Section 2(g) refers to AI risk management, and states that

It is important to manage the risks from the Federal Government’s own use of AI and increase its internal capacity to regulate, govern, and support responsible use of AI to deliver better results for Americans. These efforts start with people, our Nation’s greatest asset. My Administration will take steps to attract, retain, and develop public service-oriented AI professionals, including from underserved communities, across disciplines — including technology, policy, managerial, procurement, regulatory, ethical, governance, and legal fields — and ease AI professionals’ path into the Federal Government to help harness and govern AI. The Federal Government will work to ensure that all members of its workforce receive adequate training to understand the benefits, risks, and limitations of AI for their job functions, and to modernize Federal Government information technology infrastructure, remove bureaucratic obstacles, and ensure that safe and rights-respecting AI is adopted, deployed, and used.

Section 10 then establishes specific measures to advance Federal Government use of AI. Section 10.1(b) details a set of governance reforms to be implemented in view of the Director of the Office of Management and Budget (OMB)’s guidance to strengthen the effective and appropriate use of AI, advance AI innovation, and manage risks from AI in the Federal Government. Section 10.1(b) includes the following (emphases added):

The Director of OMB’s guidance shall specify, to the extent appropriate and consistent with applicable law:

(i) the requirement to designate at each agency within 60 days of the issuance of the guidance a Chief Artificial Intelligence Officer who shall hold primary responsibility in their agency, in coordination with other responsible officials, for coordinating their agency’s use of AI, promoting AI innovation in their agency, managing risks from their agency’s use of AI …;

(ii) the Chief Artificial Intelligence Officers’ roles, responsibilities, seniority, position, and reporting structures;

(iii) for [covered] agencies […], the creation of internal Artificial Intelligence Governance Boards, or other appropriate mechanisms, at each agency within 60 days of the issuance of the guidance to coordinate and govern AI issues through relevant senior leaders from across the agency;

(iv) required minimum risk-management practices for Government uses of AI that impact people’s rights or safety, including, where appropriate, the following practices derived from OSTP’s Blueprint for an AI Bill of Rights and the NIST AI Risk Management Framework: conducting public consultation; assessing data quality; assessing and mitigating disparate impacts and algorithmic discrimination; providing notice of the use of AI; continuously monitoring and evaluating deployed AI; and granting human consideration and remedies for adverse decisions made using AI;

(v) specific Federal Government uses of AI that are presumed by default to impact rights or safety;

(vi) recommendations to agencies to reduce barriers to the responsible use of AI, including barriers related to information technology infrastructure, data, workforce, budgetary restrictions, and cybersecurity processes;

(vii) requirements that [covered] agencies […] develop AI strategies and pursue high-impact AI use cases;

(viii) in consultation with the Secretary of Commerce, the Secretary of Homeland Security, and the heads of other appropriate agencies as determined by the Director of OMB, recommendations to agencies regarding:

(A) external testing for AI, including AI red-teaming for generative AI, to be developed in coordination with the Cybersecurity and Infrastructure Security Agency;

(B) testing and safeguards against discriminatory, misleading, inflammatory, unsafe, or deceptive outputs, as well as against producing child sexual abuse material and against producing non-consensual intimate imagery of real individuals (including intimate digital depictions of the body or body parts of an identifiable individual), for generative AI;

(C) reasonable steps to watermark or otherwise label output from generative AI;

(D) application of the mandatory minimum risk-management practices defined under subsection 10.1(b)(iv) of this section to procured AI;

(E) independent evaluation of vendors’ claims concerning both the effectiveness and risk mitigation of their AI offerings;

(F) documentation and oversight of procured AI;

(G) maximizing the value to agencies when relying on contractors to use and enrich Federal Government data for the purposes of AI development and operation;

(H) provision of incentives for the continuous improvement of procured AI; and

(I) training on AI in accordance with the principles set out in this order and in other references related to AI listed herein; and

(ix) requirements for public reporting on compliance with this guidance.

Section 10.1(b) of the AI Executive Order establishes two sets or types of requirements.

First, there are internal governance requirements and these revolve around the appointment of Chief Artificial Intelligence Officers (CAIOs), AI Governance Boards, their roles, and support structures. This set of requirements seeks to strengthen the ability of Federal Agencies to understand AI and to provide effective safeguards in its governmental use. The crucial set of substantive protections from this internal perspective derives from the required minimum risk-management practices for Government uses of AI, which is directly placed under the responsibility of the relevant CAIO.

Second, there are external (or relational) governance requirements that revolve around the agency’s ability to control and challenge tech providers. This involves the transfer (back to back) of minimum risk-management practices to AI contractors, but also includes commercial considerations. The tone of the Executive Order indicates that this set of requirements is meant to neutralise risks of commercial capture and commercial determination by imposing oversight and external verification. From an AI procurement governance perspective, the requirements in Section 10.1(b)(viii) are particularly relevant. As some of those requirements will need further development with a view to their operationalisation, Section 10.1(d)(ii) of the AI Executive Order requires the Director of OMB to develop an initial means to ensure that agency contracts for the acquisition of AI systems and services align with its Section 10.1(b) guidance.

Procurement in the Draft AI in Government Policy

The guidance required by Section 10.1(b) of the AI Executive Order has been formulated in the Draft AI in Government Policy, which offers more detail on the relevant governance mechanisms and the requirements for AI procurement. Section 5 on managing risks from the use of AI is particularly relevant from an AI procurement perspective. While Section 5(d) refers explicitly to managing risks in AI procurement, given that the primary substantive obligations will arise from the need to comply with the required minimum risk-management practices for Government uses of AI, this specific guidance needs to be read in the broader context of AI risk-management within Section 5 of the Draft AI in Government Policy.

Scope

The Draft AI in Government Policy relies on a tiered approach to AI risk by imposing specific obligations in relation to safety-impacting and rights-impacting AI only. This is an important element of the policy because these two categories are defined (in Section 6) and in principle will cover pre-established lists of AI use, based on a set of presumptions (Section 5(b)(i) and (ii)). However, CAIOs will be able to waive the application of minimum requirements for specific AI uses where, ‘based upon a system-specific risk assessment, [it is shown] that fulfilling the requirement would increase risks to safety or rights overall or would create an unacceptable impediment to critical agency operations‘ (Section 5(c)(iii)). Therefore, these are not closed lists and the specific scope of coverage of the policy will vary with such determinations. There are also some exclusions from minimum requirements where the AI is used for narrow purposes (Section 5(c)(i))—notably the ‘Evaluation of a potential vendor, commercial capability, or freely available AI capability that is not otherwise used in agency operations, solely for the purpose of making a procurement or acquisition decision’; AI evaluation in the context of regulatory enforcement, law enforcement or national security action; or research and development.

This scope of the policy may be under-inclusive, or generate risks of under-inclusiveness at the boundary, in two respects. First, the way AI is defined for the purposes of the Draft AI in Government Policy, excludes ‘robotic process automation or other systems whose behavior is defined only by human-defined rules or that learn solely by repeating an observed practice exactly as it was conducted’ (Section 6). This could be under-inclusive to the extent that the minimum risk-management practices for Government uses of AI create requirements that are not otherwise applicable to Government use of (non-AI) algorithms. There is a commonality of risks (eg discrimination, data governance risks) that would be better managed if there was a joined up approach. Moreover, developing minimum practices in relation to those means of automation would serve to develop institutional capability that could then support the adoption of AI as defined in the policy. Second, the variability in coverage stemming from consideration of ‘unacceptable impediments to critical agency operations‘ opens the door to potentially problematic waivers. While these are subject to disclosure and notification to OMB, it is not entirely clear on what grounds OMB could challenge those waivers. This is thus an area where the guidance may require further development.

extensions and waivers

In relation to covered safety-impacting or rights-impacting AI (as above), Section 5(a)(i) establishes the important principle that US Federal Government agencies have until 1 August 2024 to implement the minimum practices in Section 5(c), ‘or else stop using any AI that is not compliant with the minimum practices’. This type of sunset clause concerning the currently implicit authorisation for the use of AI is a potentially powerful mechanism. However, the Draft also establishes that such obligation to discontinue non-compliant AI use must be ‘consistent with the details and caveats in that section [5(c)]’, which includes the possibility, until 1 August 2024, for agencies to

request from OMB an extension of limited and defined duration for a particular use of AI that cannot feasibly meet the minimum requirements in this section by that date. The request must be accompanied by a detailed justification for why the agency cannot achieve compliance for the use case in question and what practices the agency has in place to mitigate the risks from noncompliance, as well as a plan for how the agency will come to implement the full set of required minimum practices from this section.

Again, the guidance does not detail on what grounds OMB would grant those extensions or how long they would be for. There is a clear interaction between the extension and waiver mechanism. For example, an agency that saw its request for an extension declined could try to waive that particular AI use—or agencies could simply try to waive AI uses rather than applying for extensions, as the requirements for a waiver seem to be rather different (and potentially less demanding) than those applicable to a waiver. In that regard, it seems that waiver determinations are ‘all or nothing’, whereas the system could be more flexible (and protective) if waiver decisions not only needed to explain why meeting the minimum requirements would generate the heightened overall risks or pose such ‘unacceptable impediments to critical agency operations‘, but also had to meet the lower burden of mitigation currently expected in extension applications, concerning detailed justification for what practices the agency has in place to mitigate the risks from noncompliance where they can be partly mitigated. In other words, it would be preferable to have a more continuous spectrum of mitigation measures in the context of waivers as well.

general minimum practices

Both in relation to safety- and rights-impact AI uses, the Draft AI in Government Policy would require agencies to engage in risk management both before and while using AI.

Preventative measures include:

  • completing an AI Impact Assessment documenting the intended purpose of the AI and its expected benefit, the potential risks of using AI, and and analysis of the quality and appropriateness of the relevant data;

  • testing the AI for performance in a real-world context—that is, testing under conditions that ‘mirror as closely as possible the conditions in which the AI will be deployed’; and

  • independently evaluate the AI, with the particularly important requirement that ‘The independent reviewing authority must not have been directly involved in the system’s development.’ In my view, it would also be important for the independent reviewing authority not to be involved in the future use of the AI, as its (future) operational interest could also be a source of bias in the testing process and the analysis of its results.

In-use measures include:

  • conducting ongoing monitoring and establish thresholds for periodic human review, with a focus on monitoring ‘degradation to the AI’s functionality and to detect changes in the AI’s impact on rights or safety’—‘human review, including renewed testing for performance of the AI in a real-world context, must be conducted at least annually, and after significant modifications to the AI or to the conditions or context in which the AI is used’;

  • mitigating emerging risks to rights and safety—crucially, ‘Where the AI’s risks to rights or safety exceed an acceptable level and where mitigation is not practicable, agencies must stop using the affected AI as soon as is practicable’. In that regard, the draft indicates that ‘Agencies are responsible for determining how to safely decommission AI that was already in use at the time of this memorandum’s release without significant disruptions to essential government functions’, but it would seem that this is also a process that would benefit from close oversight by OMB as it would otherwise jeopardise the effectiveness of the extension and waiver mechanisms discussed above—in which case additional detail in the guidance would be required;

  • ensuring adequate human training and assessment;

  • providing appropriate human consideration as part of decisions that pose a high risk to rights or safety; and

  • providing public notice and plain-language documentation through the AI use case inventory—however, this is subject a large number of caveats (notice must be ‘consistent with applicable law and governmentwide guidance, including those concerning protection of privacy and of sensitive law enforcement, national security, and other protected information’) and more detailed guidance on how to assess these issues would be welcome (if it exists, a cross-reference in the draft policy would be helpful).

additional minimum practices for rights-impacting ai

In relation to rights-affecting AI only, the Draft AI in Government Policy would require agencies to take additional measures.

Preventative measures include:

  • take steps to ensure that the AI will advance equity, dignity, and fairness—including proactively identifying and removing factors contributing to algorithmic discrimination or bias; assessing and mitigating disparate impacts; and using representative data; and

  • consult and incorporate feedback from affected groups.

In-use measures include:

  • conducting ongoing monitoring and mitigation for AI-enabled discrimination;

  • notifying negatively affected individuals—this is an area where the draft guidance is rather woolly, as it also includes a set of complex caveats, as individual notice that ‘AI meaningfully influences the outcome of decisions specifically concerning them, such as the denial of benefits’ must only be given ‘[w]here practicable and consistent with applicable law and governmentwide guidance’. Moreover, the draft only indicates that ‘Agencies are also strongly encouraged to provide explanations for such decisions and actions’, but not required to. In my view, this tackles two of the most important implications for individuals in Government use of AI: the possibility to understand why decisions are made (reason giving duties) and the burden of challenging automated decisions, which is increased if there is a lack of transparency on the automation. Therefore, on this point, the guidance seems too tepid—especially bearing in mind that this requirement only applies to ‘AI whose output serves as a basis for decision or action that has a legal, material, or similarly significant effect on an individual’s’ civil rights, civil liberties, or privacy; equal opportunities; or access to critical resources or services. In these cases, it seems clear that notice and explainability requirements need to go further.

  • maintaining human consideration and remedy processes—including ‘potential remedy to the use of the AI by a fallback and escalation system in the event that an impacted individual would like to appeal or contest the AI’s negative impacts on them. In developing appropriate remedies, agencies should follow OMB guidance on calculating administrative burden and the remedy process should not place unnecessary burden on the impacted individual. When law or governmentwide guidance precludes disclosure of the use of AI or an opportunity for an individual appeal, agencies must create appropriate mechanisms for human oversight of rights-impacting AI’. This is another crucial area concerning rights not to be subjected to fully-automated decision-making where there is no meaningful remedy. This is also an area of the guidance that requires more detail, especially as to what is the adequate balance of burdens where eg the agency can automate the undoing of negative effects on individuals identified as a result of challenges by other individuals or in the context of the broader monitoring of the functioning and effects of the rights-impacting AI. In my view, this would be an opportunity to mandate automation of remediation in a meaningful way.

  • maintaining options to opt-out where practicable.

procurement related practices

In addition to the need for agencies to be able to meet the above requirements in relation to procured AI—which will in itself create the need to cascade some of the requirements down to contractors, and which will be the object of future guidance on how to ensure that AI contracts align with the requirements—the Draft AI in Government Policy also requires that agencies procuring AI manage risks by:

  • aligning to National Values and Law by ensuring ‘that procured AI exhibits due respect for our Nation’s values, is consistent with the Constitution, and complies with all other applicable laws, regulations, and policies, including those addressing privacy, confidentiality, copyright, human and civil rights, and civil liberties’;

  • taking ‘steps to ensure transparency and adequate performance for their procured AI, including by: obtaining adequate documentation of procured AI, such as through the use of model, data, and system cards; regularly evaluating AI-performance claims made by Federal contractors, including in the particular environment where the agency expects to deploy the capability; and considering contracting provisions that incentivize the continuous improvement of procured AI’;

  • taking ‘appropriate steps to ensure that Federal AI procurement practices promote opportunities for competition among contractors and do not improperly entrench incumbents. Such steps may include promoting interoperability and ensuring that vendors do not inappropriately favor their own products at the expense of competitors’ offering’;

  • maximizing the value of data for AI; and

  • responsibly procuring Generative AI.

These high level requirements are well targeted and compliance with them would go a long way to fostering ‘responsible AI procurement’ through adequate risk mitigation in ways that still allow the procurement mechanism to harness market forces to generate value for money.

However, operationalising these requirements will be complex and the further OMB guidance should be rather detailed and practical.

Final thoughts

In my view, the AI Executive Order and the Draft AI in Government Policy lay the foundations for a significant strengthening of the governance of AI procurement with a view to embedding safeguards in public sector AI use. A crucially important characteristic in the design of these governance mechanisms is that it imposes significant duties on the agencies seeking to procure and use the AI, and it explicitly seeks to address risks of commercial capture and commercial determination. Another crucially important characteristic is that, at least in principle, use of AI is made conditional on compliance with a rather comprehensive set of preventative and in-use risk mitigation measures. The general aspects of this governance approach thus offer a very valuable blueprint for other jurisdictions considering how to boost AI procurement governance.

However, as always, the devil is in the details. One of the crucial risks in this approach to AI governance concerns a lack of independence of the entities making the relevant assessments. In the Draft AI in Government Policy, there are some risks of under-inclusion and/or excessive waivers of compliance with the relevant requirements (both explicit and implicit, through protracted processes of decommissioning of non-compliant AI), as well as a risk that ‘practical considerations’ will push compliance with the risk mitigation requirements well past the (ambitious) 1 August 2024 deadline through long or rolling extensions.

To mitigate for this, the guidance should be much clearer on the role of OMB in extension, waiver and decommissioning decisions, as well as in relation to the specific criteria and limits that should form part of those decisions. Only by ensuring adequate OMB intervention can a system of governance that still does not entirely (organisationally) separate procurement, use and oversight decisions reach the levels of independent verification required not only to neutralise commercial determination, but also operational dependency and the ‘policy irresistibility’ of digital technologies.

Thoughts on the AI Safety Summit from a public sector procurement & use of AI perspective

The UK Government hosted an AI Safety Summit on 1-2 November 2023. A summary of the targeted discussions in a set of 8 roundtables has been published for Day 1, as well as a set of Chair’s statements for Day 2, including considerations around safety testing, the state of the science, and a general summary of discussions. There is also, of course, the (flagship?) Bletchley Declaration, and an introduction to the announced AI Safety Institute (UK AISI).

In this post, I collect some of my thoughts on these outputs of the AI Safety Summit from the perspective of public sector procurement and use of AI.

What was said at the AI safety Summit?

Although the summit was narrowly targeted to discussion of ‘frontier AI’ as particularly advanced AI systems, some of the discussions seem to have involved issues also applicable to less advanced (ie currently in existence) AI systems, and even to non-AI algorithms used by the public sector. As the general summary reflects, ‘There was also substantive discussion of the impact of AI upon wider societal issues, and suggestions that such risks may themselves pose an urgent threat to democracy, human rights, and equality. Participants expressed a range of views as to which risks should be prioritised, noting that addressing frontier risks is not mutually exclusive from addressing existing AI risks and harms.’ Crucially, ‘participants across both days noted a range of current AI risks and harmful impacts, and reiterated the need for them to be tackled with the same energy, cross-disciplinary expertise, and urgency as risks at the frontier.’ Hopefully, then, some of the rather far-fetched discussions of future existential risks can be conducive to taking action on current harms and risks arising from the procurement and use of less advanced systems.

There seemed to be some recognition of the need for more State intervention through regulation, for more regulatory control of standard-setting, and for more attention to be paid to testing and evaluation in the procurement context. For example, the summary of Day 1 discussions indicates that participants agreed that

  • ‘We should invest in basic research, including in governments’ own systems. Public procurement is an opportunity to put into practice how we will evaluate and use technology.’ (Roundtable 4)

  • ‘Company policies are just the baseline and don’t replace the need for governments to set standards and regulate. In particular, standardised benchmarks will be required from trusted external third parties such as the recently announced UK and US AI Safety Institutes.’ (Roundtable 5)

In Day 2, in the context of safety testing, participants agreed that

  • Governments have a responsibility for the overall framework for AI in their countries, including in relation to standard setting. Governments recognise their increasing role for seeing that external evaluations are undertaken for frontier AI models developed within their countries in accordance with their locally applicable legal frameworks, working in collaboration with other governments with aligned interests and relevant capabilities as appropriate, and taking into account, where possible, any established international standards.

  • Governments plan, depending on their circumstances, to invest in public sector capability for testing and other safety research, including advancing the science of evaluating frontier AI models, and to work in partnership with the private sector and other relevant sectors, and other governments as appropriate to this end.

  • Governments will plan to collaborate with one another and promote consistent approaches in this effort, and to share the outcomes of these evaluations, where sharing can be done safely, securely and appropriately, with other countries where the frontier AI model will be deployed.

This could be a basis on which to build an international consensus on the need for more robust and decisive regulation of AI development and testing, as well as a consensus of the sets of considerations and constraints that should be applicable to the procurement and use of AI by the public sector in a way that is compliant with individual (human) rights and social interests. The general summary reflects that ‘Participants welcomed the exchange of ideas and evidence on current and upcoming initiatives, including individual countries’ efforts to utilise AI in public service delivery and elsewhere to improve human wellbeing. They also affirmed the need for the benefits of AI to be made widely available’.

However, some statements seem at first sight contradictory or problematic. While the excerpt above stresses that ‘Governments have a responsibility for the overall framework for AI in their countries, including in relation to standard setting’ (emphasis added), the general summary also stresses that ‘The UK and others recognised the importance of a global digital standards ecosystem which is open, transparent, multi-stakeholder and consensus-based and many standards bodies were noted, including the International Standards Organisation (ISO), International Electrotechnical Commission (IEC), Institute of Electrical and Electronics Engineers (IEEE) and relevant study groups of the International Telecommunication Union (ITU).’ Quite how State responsibility for standard setting fits with industry-led standard setting by such organisations is not only difficult to fathom, but also one of the potentially most problematic issues due to the risk of regulatory tunnelling that delegation of standard setting without a verification or certification mechanism entails.

Moreover, there seemed to be insufficient agreement around crucial issues, which are summarised as ‘a set of more ambitious policies to be returned to in future sessions’, including:

‘1. Multiple participants suggested that existing voluntary commitments would need to be put on a legal or regulatory footing in due course. There was agreement about the need to set common international standards for safety, which should be scientifically measurable.

2. It was suggested that there might be certain circumstances in which governments should apply the principle that models must be proven to be safe before they are deployed, with a presumption that they are otherwise dangerous. This principle could be applied to the current generation of models, or applied when certain capability thresholds were met. This would create certain ‘gates’ that a model had to pass through before it could be deployed.

3. It was suggested that governments should have a role in testing models not just pre- and post-deployment, but earlier in the lifecycle of the model, including early in training runs. There was a discussion about the ability of governments and companies to develop new tools to forecast the capabilities of models before they are trained.

4. The approach to safety should also consider the propensity for accidents and mistakes; governments could set standards relating to how often the machine could be allowed to fail or surprise, measured in an observable and reproducible way.

5. There was a discussion about the need for safety testing not just in the development of models, but in their deployment, since some risks would be contextual. For example, any AI used in critical infrastructure, or equivalent use cases, should have an infallible off-switch.

8. Finally, the participants also discussed the question of equity, and the need to make sure that the broadest spectrum was able to benefit from AI and was shielded from its harms.’

All of these are crucial considerations in relation to the regulation of AI development, (procurement) and use. A lack of consensus around these issues already indicates that there was a generic agreement that some regulation is necessary, but much more limited agreement on what regulation is necessary. This is clearly reflected in what was actually agreed at the summit.

What was agreed at the AI Safety Summit?

Despite all the discussions, little was actually agreed at the AI Safety Summit. The Blethcley Declaration includes a lengthy (but rather uncontroversial?) description of the potential benefits and actual risks of (frontier) AI, some rather generic agreement that ‘something needs to be done’ (eg welcoming ‘the recognition that the protection of human rights, transparency and explainability, fairness, accountability, regulation, safety, appropriate human oversight, ethics, bias mitigation, privacy and data protection needs to be addressed’) and very limited and unspecific commitments.

Indeed, signatories only ‘committed’ to a joint agenda, comprising:

  • ‘identifying AI safety risks of shared concern, building a shared scientific and evidence-based understanding of these risks, and sustaining that understanding as capabilities continue to increase, in the context of a wider global approach to understanding the impact of AI in our societies.

  • building respective risk-based policies across our countries to ensure safety in light of such risks, collaborating as appropriate while recognising our approaches may differ based on national circumstances and applicable legal frameworks. This includes, alongside increased transparency by private actors developing frontier AI capabilities, appropriate evaluation metrics, tools for safety testing, and developing relevant public sector capability and scientific research’ (emphases added).

This does not amount to much that would not happen anyway and, given that one of the UK Government’s objectives for the Summit was to create mechanisms for global collaboration (‘a forward process for international collaboration on frontier AI safety, including how best to support national and international frameworks’), this agreement for each jurisdiction to do things as they see fit in accordance to their own circumstances and collaborate ‘as appropriate’ in view of those seems like a very poor ‘win’.

In reality, there seems to be little coming out of the Summit other than a plan to continue the conversations in 2024. Given what had been said in one of the roundtables (num 5) in relation to the need to put in place adequate safeguards: ‘this work is urgent, and must be put in place in months, not years’; it looks like the ‘to be continued’ approach won’t do or, at least, cannot be claimed to have made much of a difference.

What did the UK Government promise in the AI Summit?

A more specific development announced with the occasion of the Summit (and overshadowed by the earlier US announcement) is that the UK will create the AI Safety Institute (UK AISI), a ‘state-backed organisation focused on advanced AI safety for the public interest. Its mission is to minimise surprise to the UK and humanity from rapid and unexpected advances in AI. It will work towards this by developing the sociotechnical infrastructure needed to understand the risks of advanced AI and enable its governance.’

Crucially, ‘The Institute will focus on the most advanced current AI capabilities and any future developments, aiming to ensure that the UK and the world are not caught off guard by progress at the frontier of AI in a field that is highly uncertain. It will consider open-source systems as well as those deployed with various forms of access controls. Both AI safety and security are in scope’ (emphasis added). This seems to carry forward the extremely narrow focus on ‘frontier AI’ and catastrophic risks that augured a failure of the Summit. It is also in clear contrast with the much more sensible and repeated assertions/consensus in that other types of AI cause very significant risks and that there is ‘a range of current AI risks and harmful impacts, and reiterated the need for them to be tackled with the same energy, cross-disciplinary expertise, and urgency as risks at the frontier.’

Also crucially, UK AISI ‘is not a regulator and will not determine government regulation. It will collaborate with existing organisations within government, academia, civil society, and the private sector to avoid duplication, ensuring that activity is both informing and complementing the UK’s regulatory approach to AI as set out in the AI Regulation white paper’.

According to initial plans, UK AISI ‘will initially perform 3 core functions:

  • Develop and conduct evaluations on advanced AI systems, aiming to characterise safety-relevant capabilities, understand the safety and security of systems, and assess their societal impacts

  • Drive foundational AI safety research, including through launching a range of exploratory research projects and convening external researchers

  • Facilitate information exchange, including by establishing – on a voluntary basis and subject to existing privacy and data regulation – clear information-sharing channels between the Institute and other national and international actors, such as policymakers, international partners, private companies, academia, civil society, and the broader public’

It is also stated that ‘We see a key role for government in providing external evaluations independent of commercial pressures and supporting greater standardisation and promotion of best practice in evaluation more broadly.’ However, the extent to which UK AISI will be able to do that will hinge on issues that are not currently clear (or publicly disclosed), such as the membership of UK AISI or its institutional set up (as ‘state-backed organisation’ does not say much about this).

On that very point, it is somewhat problematic that the UK AISI ‘is an evolution of the UK’s Frontier AI Taskforce. The Frontier AI Taskforce was announced by the Prime Minister and Technology Secretary in April 2023’ (ahem, as ‘Foundation Model Taskforce’—so this is the second rebranding of the same initiative in half a year). As is problematic that UK AISI ‘will continue the Taskforce’s safety research and evaluations. The other core parts of the Taskforce’s mission will remain in [the Department for Science, Innovation and Technology] as policy functions: identifying new uses for AI in the public sector; and strengthening the UK’s capabilities in AI.’ I find the retention of analysis pertaining to public sector AI use within government problematic and a clear indication of the UK’s Government unwillingness to put meaningful mechanisms in place to monitor the process of public sector digitalisation. UK AISI very much sounds like a research institute with a focus on a very narrow set of AI systems and with a remit that will hardly translate into relevant policymaking in areas in dire need of regulation. Finally, it is also very problematic that funding is not locked: ‘The Institute will be backed with a continuation of the Taskforce’s 2024 to 2025 funding as an annual amount for the rest of this decade, subject to it demonstrating the continued requirement for that level of public funds.’ In reality, this means that the Institute’s continued existence will depend on the Government’s satisfaction with its work and the direction of travel of its activities and outputs. This is not at all conducive to independence, in my view.

So, all in all, there is very little new in the announcement of the creation of the UK AISI and, while there is a (theoretical) possibility for the Institute to make a positive contribution to regulating AI procurement and use (in the public sector), this seems extremely remote and potentially undermined by the Institute’s institutional set up. This is probably in stark contrast with the US approach the UK is trying to mimic (though more on the US approach in a future entry).

European Commission wants to see more AI procurement. Ok, but priorities need reordering

The European Commission recently published its 2023 State of the Digital Decade report. One of its key takeaways is that the Commission recommends Member States to step up innovation procurement investments in digital sector.

The Commission has identified that ‘While the roll-out of digital public services is progressing steadily, investment in public procurement of innovative digital solutions (e.g. based on AI or big data) is insufficient and would need to increase substantially from EUR 188 billon to EUR 295 billon in order to reach full speed adoption of innovative digital solutions in public services’ (para 4.2, original emphasis).

The Commission has thus recommended that ‘Member States should step up investment and regulatory measures to develop and make available secure, sovereign and interoperable digital solutions for online public and government services’; and that ‘Member States should develop action plans in support of innovation procurement and step up efforts to increase public procurement investments in developing, testing and deploying innovative digital solutions’.

Tucked away in a different part of the report (which, frankly, has a rather odd structure), the Commission also recommends that ‘Member States should foster the availability of legal and technical support to procure and implement trustworthy and sovereign AI solutions across sectors.’

To my mind, the priorities for investment of public money need to be further clarified. Without a significant investment in an ambitious plan to quickly expand the public sector’s digital skills and capabilities, there can be no hope that increased procurement expenditure in digital technologies will bring adequate public sector digitalisation or foster the public interest more broadly.

Without a sophisticated public buyer that can adequately cut through the process of technological innovation, there is no hope that ‘throwing money at the problem’ will bring meaningful change. In my view, the focus and priority should be on upskilling the public sector before anything else—including ahead of the also recommended mobilisation of ‘public policies, including innovative procurement to foster the scaling up of start-ups, to facilitate the creation of spinoffs from universities and research centres, and to monitor progress in this area’ (para 3.2.3). Perhaps a substantial fraction of the 100+ billion EUR the Commission expects Member States to put into public sector digitalisation could go to building up the required capability… too much to ask?

G7 Guiding Principles and Code of Conduct on Artificial Intelligence -- some comments from a UK perspective

On 30 October 2023, G7 leaders published the Hiroshima Process International Guiding Principles for Advanced AI system (the G7 AI Principles), a non-exhaustive list of guiding principles formulated as a living document that builds on the OECD AI Principles to take account of recent developments in advanced AI systems. The G7 stresses that these principles should apply to all AI actors, when and as applicable to cover the design, development, deployment and use of advanced AI systems.

The G7 AI Principles are supported by a voluntary Code of Conduct for Advanced AI Systems (the G7 AI Code of Conduct), which is meant to provide guidance to help seize the benefits and address the risks and challenges brought by these technologies.

The G7 AI Principles and Code of Conduct came just two days before the start of the UK’s AI Safety Summit 2023. Given that the UK is part of the G7 and has endorsed the G7 Hiroshima Process and its outcomes, the interaction between the G7’s documents, the UK Government’s March 2023 ‘pro-innovation’ approach to AI and its aspirations for the AI Safety Summit deserves some comment.

G7 AI Principles and Code of Conduct

The G7 AI Principles aim ‘to promote safe, secure, and trustworthy AI worldwide and will provide guidance for organizations developing and using the most advanced AI systems, including the most advanced foundation models and generative AI systems.’ The principles are meant to be cross-cutting, as they target ‘among others, entities from academia, civil society, the private sector, and the public sector.’ Importantly, also, the G7 AI Principles are meant to be a stop gap solution, as G7 leaders ‘call on organizations in consultation with other relevant stakeholders to follow these [principles], in line with a risk-based approach, while governments develop more enduring and/or detailed governance and regulatory approaches.’

The principles include the reminder that ‘[w]hile harnessing the opportunities of innovation, organizations should respect the rule of law, human rights, due process, diversity, fairness and non-discrimination, democracy, and human-centricity, in the design, development and deployment of advanced AI system’, as well as a reminder that organizations developing and deploying AI should not undermine democratic values, harm individuals or communities, ‘facilitate terrorism, enable criminal misuse, or pose substantial risks to safety, security, and human rights’. State (AI users) are reminder of their ‘obligations under international human rights law to promote that human rights are fully respected and protected’ and private sector actors are called to align their activities ‘with international frameworks such as the United Nations Guiding Principles on Business and Human Rights and the OECD Guidelines for Multinational Enterprises’.

These are all very high level declarations and aspirations that do not go much beyond pre-existing commitments and (soft) law norms, if at all.

The G7 AI Principles comprises a non-exhaustive list of 11 high-level regulatory goals that organizations should abide by ‘commensurate to the risks’—ie following the already mentioned risk-based approach—which introduces a first element of uncertainty because the document does not establish any methodology or explanation on how risks should be assessed and tiered (one of the primary, and debated, features of the proposed EU AI Act). The principles are the following, prefaced by my own labelling between square brackets:

  1. [risk identification, evaluation and mitigation] Take appropriate measures throughout the development of advanced AI systems, including prior to and throughout their deployment and placement on the market, to identify, evaluate, and mitigate risks across the AI lifecycle;

  2. [misuse monitoring] Patterns of misuse, after deployment including placement on the market;

  3. [transparency and accountability] Publicly report advanced AI systems’ capabilities, limitations and domains of appropriate and inappropriate use, to support ensuring sufficient transparency, thereby contributing to increase accountability.

  4. [incident intelligence exchange] Work towards responsible information sharing and reporting of incidents among organizations developing advanced AI systems including with industry, governments, civil society, and academia.

  5. [risk management governance] Develop, implement and disclose AI governance and risk management policies, grounded in a risk-based approach – including privacy policies, and mitigation measures, in particular for organizations developing advanced AI systems.

  6. [(cyber) security] Invest in and implement robust security controls, including physical security, cybersecurity and insider threat safeguards across the AI lifecycle.

  7. [content authentication and watermarking] Develop and deploy reliable content authentication and provenance mechanisms, where technically feasible, such as watermarking or other techniques to enable users to identify AI-generated content.

  8. [risk mitigation priority] Prioritize research to mitigate societal, safety and security risks and prioritize investment in effective mitigation measures.

  9. [grand challenges priority] Prioritize the development of advanced AI systems to address the world’s greatest challenges, notably but not limited to the climate crisis, global health and education.

  10. [technical standardisation] Advance the development of and, where appropriate, adoption of international technical standards.

  11. [personal data and IP safeguards] Implement appropriate data input measures and protections for personal data and intellectual property.

Each of the principles is accompanied by additional guidance or precision, where possible, and this is further developed in the G7 Code of Conduct.

In my view, the list is a bit of a mixed bag.

There are some very general aspirations or steers that can hardly be considered principles of AI regulation, for example principle 9 setting a grand challenges priority and, possibly, principle 8 setting a risk mitigation priority beyond the ‘requirements’ of principle 1 on risk identification, evaluation and mitigation—which thus seems to boil down to the more specific steer in the G7 Code of Conduct for (private) organisations to ‘share research and best practices on risk mitigation’.

Quite how these principles could be complied by current major AI developers seems rather difficult to foresee, especially in relation to principle 9. Most developers of generative AI or other AI applications linked to eg social media platforms will have a hard time demonstrating their engagement with this principle, unless we accept a general justification of ‘general purpose application’ or ‘dual use application’—which to me seems quite unpalatable. What is the purpose of this principle if eg it pushes organisations away from engaging with the rest of the G7 AI Principles? Or if organisations are allowed to gloss over it in any (future) disclosures linked to an eventual mechanism of commitment, chartering, or labelling associated with the principles? It seems like the sort of purely political aspiration that may have been better left aside.

Some other principles seem to push at an open door, such as principle 10 on the development of international technical standards. Again, the only meaningful detail seems to be in the G7 Code of Conduct, which specifies that ‘In particular, organizations also are encouraged to work to develop interoperable international technical standards and frameworks to help users distinguish content generated by AI from non-AI generated content.’ However, this is closely linked to principle 7 on content authentication and watermarking, so it is not clear how much that adds. Moreover, this comes to further embed the role of industry-led technical standards as a foundational element of AI regulation, with all the potential problems that arise from it (for some discussion from the perspective of regulatory tunnelling, see here and here).

Yet other principles present as relatively soft requirements or ‘noble’ commitments issues that are, in reality, legal requirements already binding on entities and States and that, in my view, should have been placed as hard obligations and a renewed commitment from G7 States to enforce them. These include principle 11 on personal data and IP safeguards, where the G7 Code of Conduct includes as an apparent after thought that ‘Organizations should also comply with applicable legal frameworks’. In my view, this should be starting point.

This reduces the list of AI Principles ‘proper’. But, even then, they can be further grouped and synthesised, in my view. For example, principles 1 and 5 are both about risk management, with the (outward-looking) governance layer of principle 5 seeking to give transparency to the (inward-looking) governance layer in principle 1. Principle 2 seems to simply seek to extend the need to engage with risk-based management post-market placement, which is also closely connected to the (inward-looking) governance layer in principle 1. All of them focus on the (undefined) risk-based approach to development and deployment of AI underpinning the G7’s AI Principles and Code of Conduct.

Some aspects of the incident intelligence exchange also relate to principle 1, while some other aspects relate to (cyber) security issues encapsulated in principle 6. However, given that this principle may be a placeholder for the development of some specific mechanisms of collaboration—either based on cyber security collaboration or other approaches, such as the much touted aviation industry’s—it may be treated separately.

Perhaps, then, the ‘core’ AI Principles arising from the G7 document could be trimmed down to:

  • Life-cycle risk-based management and governance, inclusive of principles 1, 2, and 5.

  • Transparency and accountability, principle 3.

  • Incident intelligence exchange, principle 4.

  • (Cyber) security, principle 6.

  • Content authentication and watermarking, principle 7 (though perhaps narrowly targeted to generative AI).

Most of the value in the G7 AI Principles and Code of Conduct thus arises from the pointers for collaboration, the more detailed self-regulatory measures, and the more specific potential commitments included in the latter. For example, in relation to the potential AI risks that are identified as potential targets for the risk assessments expected of AI developers (under guidance related to principle 1), or the desirable content of AI-related disclosures (under guidance related to principle 3).

It is however unclear how these principles will evolve when adopted at the national level, and to what extent they offer a sufficient blueprint to ensure international coherence in the development of the ‘more enduring and/or detailed governance and regulatory approaches’ envisaged by G7 leaders. It seems for example striking that both the EU and the UK have supported these principles, given that they have relatively opposing approaches to AI regulation—with the EU seeking to finalise the legislative negotiations on the first ‘golden standard’ of AI regulation and the UK taking an entirely deregulatory approach. Perhaps this is in itself an indication that, even at the level of detail achieved in the G7 AI Code of Conduct, the regulatory leeway is quite broad and still necessitates significant further concretisation for it to be meaningful in operational terms—as evidenced eg by the US President’s ‘Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence’, which calls for that concretisation and provides a good example of the many areas for detailed work required to translate high level principles into actionable requirements (even if it leaves enforcement still undefined).

How do the G7 Principles compare to the UK’s ‘pro-innovation’ ones?

In March 2023, the UK Government published its white paper ‘A pro-innovation approach to AI regulation’ (the ‘UK AI White Paper’; for a critique, see here). The UK AI White Paper indicated (at para 10) that its ‘framework is underpinned by five principles to guide and inform the responsible development and use of AI in all sectors of the economy:

  • Safety, security and robustness

  • Appropriate transparency and explainability

  • Fairness

  • Accountability and governance

  • Contestability and redress’.

A comparison of the UK and the G7 principles can show a few things.

First, that there are some areas where there seems to be a clear correlation—in particular concerning (cyber) security as a self-standing challenge requiring a direct regulatory focus.

Second, that it is hard to decide at which level to place incommensurable aspects of AI regulation. Notably, the G7 principles do not directly refer to fairness—while the UK does. However, the G7 Principles do spend some time in the preamble addressing the issue of fairness and unacceptable AI use (though in a woolly manner). Whether placing this type of ‘requirement’ at a level or other makes a difference (at all) is highly debatable.

Third, that there are different ways of ‘packaging’ principles or (soft) obligations. Just like some of the G7 principles are closely connected or fold into each other (as above), so do the UK’s principles in relation to the G7’s. For example, the G7 packaged together transparency and accountability (principle 3), while the UK had them separated. While the UK explicitly mentioned the issue of AI explainability, this remains implicit in the G7 principles (also in principle 3).

Finally, in line with the considerations above, that distinct regulatory approaches only emerge or become clear once the ‘principles’ become specific (so they arguably stop being principles). For example, it seems clear that the G7 Principles aspire to higher levels of incident intelligence governance and to a specific target of generative AI watermarking than the UK’s. However, whether the G7 or the UK principles are equally or more demanding on any other dimension of AI regulation is close to impossible to establish. In my view, this further supports the need for a much more detailed AI regulatory framework—else, technical standards will entirely occupy that regulatory space.

What do the G7 AI Principles tell us about the UK’s AI Safety Summit?

The Hiroshima Process that has led to the adoption of the G7 AI Principles and Code of Conduct emerged from the Ministerial Declaration of The G7 Digital and Tech Ministers’ Meeting of 30 April 2023, which explicitly stated that:

‘Given that generative AI technologies are increasingly prominent across countries and sectors, we recognise the need to take stock in the near term of the opportunities and challenges of these technologies and to continue promoting safety and trust as these technologies develop. We plan to convene future G7 discussions on generative AI which could include topics such as governance, how to safeguard intellectual property rights including copyright, promote transparency, address disinformation, including foreign information manipulation, and how to responsibly utilise these technologies’ (at para 47).

The UK Government’s ambitions for the AI Safety Summit largely focus on those same issues, albeit within the very narrow confines of ‘frontier AI’, which it has defined as ‘highly capable general-purpose AI models that can perform a wide variety of tasks and match or exceed the capabilities present in today’s most advanced models‘. While the UK Government has published specific reports to focus discussion on (1) Capabilities and risks from frontier AI and (2) Emerging Processes for Frontier AI Safety, it is unclear how the level of detail of such narrow approach could translate into broader international commitments.

The G7 AI Principles already claim to tackle ‘the most advanced AI systems, including the most advanced foundation models and generative AI systems (henceforth "advanced AI systems")’ within their scope. It seems unclear that such approach would be based on a lack of knowledge or understanding of the detail the UK has condensed in those reports. It rather seems that the G7 was not ready to move quickly to a level of detail beyond that included in the G7 AI Code of Conduct. Whether significant further developments can be expected beyond the G7 AI Principles and Code of Conduct just two days after they were published seems hard to fathom.

Moreover, although the UK Government is downplaying the fact that eg Chinese participation in the AI Safety Summit is unclear and potentially rather marginal, it seems that, at best, the UK AI Safety Summit will be an opportunity for a continued conversation between G7 countries and a few others. It is also unclear whether significant progress will be made in a forum that seems rather clearly tilted towards industry voice and influence.

Let’s wait and see what the outcomes are, but I am not optimistic for significant progress other than, worryingly, a risk of further displacement of regulatory decision-making towards industry and industry-led (future) standards.

More model contractual AI clauses -- some comments on the SCL AI Clauses

Following the launch of the final version of the model contractual AI clauses sponsored by the European Commission earlier this month, the topic of how to develop and how to use contractual model clauses for AI procurement is getting hotter. As part of its AI Action Plan, New York City has announced that it is starting work to develop its own model clauses for AI procurement (to be completed in 2025). We can expect to see a proliferation of model AI clauses as more ‘AI legislation’ imposes constraints on contractual freedom and compliance obligations, and as different model clauses are revised to (hopefully) capture the learning from current experimentation in AI procurement.

Although not (closely) focused on procurement, a new set of interesting AI contractual clauses has been released by the Society for Computers & Law (SCL) AI Group (thanks to Gisele Waters for bringing them to my attention on LinkedIn!). In this post, I reflect on some aspects of the SCL AI clauses and try to answer Gisele’s question/challenge (below).

SCL AI Clauses

The SCL AI clauses have a clear commercial orientation and are meant as a starting point for supplier-customer negotiations, which is reflected on the fact that the proposed clauses contain two options: (1) a ‘pro-supplier’ drafting based on off-the-shelf provision, and (2) a ‘pro-customer’ drafting based on a bespoke arrangement. Following that commercial logic, most of the SCL AI clauses focus on an allocation of obligations (and thus costs and liability) between the parties (eg in relation to compliance with legal requirements).

The clauses include a few substantive requirements implicit in the allocation of the respective obligations (eg on data or third party licences) but mostly refer to detailed schedules of which there is no default proposal, or to industry standards (and thus have this limitation in common with eg the EU’s model AI clauses). The SCL AI clauses do contain some drafting notes that would help identify issues needing specific regulation in the relevant schedules, although this guidance necessarily remains rather abstract or generic.

This pro-supplier/pro-customer orientation prompted Gisele’s question/challenge, which is whether ‘there is EVER an opportunity for government (customer-buyer) to be better able to negotiate the final language with clauses like these in order to weigh the trade offs between interests?’, especially bearing in mind that the outcome of the negotiations could be strongly pro-supplier, strongly pro-customer, or balanced (and something in between those). I think that answering this question requires exploring what pro-supplier or pro-customer may mean in this specific context.

From a substantive regulation perspective, the SCL AI clauses include a few interesting elements, such as an obligation to establish a circuit-breaker capable of stopping the AI (aka an ‘off button’) and a roll-back obligation (to an earlier, non-faulty version of the AI solution) where the AI is malfunctioning or this is necessary to comply with applicable law. However, most of the substantive obligations are established by reference to ‘Good Industry Practice’, which requires some further unpacking.

SCL AI Clauses and ‘Good Industry Practice’

Most of crucial proposed clauses refer to the benchmark of ‘Good Industry Practice’ as a primary qualifier for the relevant obligations. The proposed clause on explainability is a good example. The SCL AI clause (C1.15) reads as follows:

C1.15 The Supplier will ensure that the AI System is designed, developed and tested in a way which ensures that its operation is sufficiently transparent to enable the Customer to understand and use the AI System appropriately. In particular, the Supplier will produce to the Customer, on request, information which allows the Customer to understand:

C1.15.1 the logic behind an individual output from the AI System; and

C1.15.2 in respect of the AI System or any specific part thereof, which features contributed most to the output of the AI System, in each case, in accordance with Good Industry Practice.

A first observation is that the SCL AI clauses seem to presume that off-the-shelf AI solutions would not be (necessarily) explainable, as they include no clause under the ‘pro-supplier’ version.

Second, the ‘pro-customer’ version both limits the types of explanation that would be contractually owed (to a model-level or global explanation under C1.15.2 and a limited decision-level or local explanation under C1.15.1 — which leaves out eg a counterfactual explanation, as well as not setting any specific requirements on how the explanation needs to be produced, eg is a ‘post hoc’ explanation acceptable and if so how should it be produced?) and qualifies it in two important ways: (1) the overall requirement is that the AI system’s operation should be ‘sufficiently transparent’, with ‘sufficient’ creating a lot of potential issues here; and, (2) the reference to ‘Good Industry Practice’ [more on this below].

The issue of transparency is similarly problematic in its more general treatment under another specific clause (C4.6), which also only has a ‘pro-customer’ version:

C4.6 The Supplier warrants that, so far as is possible [to achieve the intended use of the AI System / comply with the Specification], the AI System is transparent and interpretable [such that its output can be traced back to the input data] .

The qualifier ‘so far as is possible’ is again potentially quite problematic here, as are the open-ended references to transparency and interpretability of the system (with a potential conflict between interpretability for the purposes of this clause and explainability under C1.15).

What I find interesting about this clause is that the drafting notes explain that:

… the purpose of this provision is to ensure that the Supplier has not used an overly-complex algorithm if this is unnecessary for the intended use of the AI System or to comply with its Specification. That said, effectiveness and accuracy are often trade-offs for transparency in AI models.

From this perspective, I think the clause should be retitled and entirely redrafted to make explicit that the purpose is to establish a principle of ‘AI minimisation’ in the sense of the supplier guaranteeing that the AI system is the least complex that can provide the desired functionality — which, of course, has the tricky issue of trade-off and the establishment of the desired functionality in itself to work around. (and which in a procurement context would have been dealt with pre-contract, eg in the context of technical specifications and/or tender evaluation). Interestingly, this issue is another one where reference could be made to ‘Good Industry Practice’ if one accepted that it should be best practice to always use the most explainable/interpretable and most simple model available for a given task.

As mentioned, reference to ‘Good Industry Practice’ is used extensively in the SCL AI clauses, including crucial issues such as: explainability (above), user manual/user training, preventing unlawful discrimination, security (which is inclusive of cyber secturity and some aspects of data protection/privacy), or quality standards. The drafting notes are clear that

… while parties often refer to ‘best practice’ or ‘good industry practice’, these standards can be difficult to apply in developing industry. Accordingly a clear Specification is required, …

Which is the reason why the SCL AI clauses foresee that ‘Good Industry Practice’ will be a defined contract term, whereby the parties will specify the relevant requirements and obligations. And here lies the catch.

Defining ‘Good Industry Practice’?

In the SCL AI clauses, all references to ‘Good Industry Practice’ are used as qualifiers in the pro-customer version of the clauses. It is possible that the same term would be of relevance to establishing whether the supplier had discharged its reasonable duties/best efforts under the pro-supplier version (where the term would be defined but not explicitly used). In both cases, the need to define ‘Good Industry Practice’ is the Achilles heel of the model clauses, as well as a potential Trojan horse for customers seeking a seemingly pro-customer contractual design,

The fact is that the extent of the substantive obligations arising from the contract will entirely depend on how the concept of ‘Good Industry Practice’ is defined and specified. This leaves even seemingly strongly ‘pro-customer’ contracts exposed to weak substantive protections. The biggest challenge for buyers/procurers of AI will be that (1) it will be hard to know how to define the term and what standards to refer to, and (2) it will be difficult to monitor compliance with the standards, especially where those establish eg mechanisms of self-asessment by the tech supplier as the primary or sole quality control mechanims.

So, my answer to Gisele’s question/challenge would be that the SCL AI clauses, much like the EU’s, do not (and cannot?) go far enough in ensuring that the contract for the procurement/purchase of AI embeds adequate substantive requirements. The model clauses are helpful in understanding who needs to do what when, and thus who shoulders the relevant cost and risk. But they do not address the all-important question of how it needs to be done. And that is the crucial issue that will determine whether the contract (and the AI solution) really is in the public buyer’s interest and, ultimately in the public interest.

In a context where tech providers (almost always) have the upper hand in negotiations, this foundational weakness is all important, as suppliers could well ‘agree to pro-customer drafting’ and then immediately deactivate it through the more challenging and technical definition (and implementation) of ‘Good Industry Practices’.

That is why I think we need to cover this regulatory tunnelling risk and this foundational shortcoming of ‘AI regulation by contract’ by creating clear and binding requirements on the how (ie the ‘good (industry) practice’ or technical standards). The emergence of model AI contract clauses to me makes it clear that the most efficient contract design is such that it needs to refer to external benchmarks. Establishing adequarte protections and an adequate balance of risks and benefits (from a social perspective) hinges on this. The contract can then deal with an apportionment of the burdens, obligations, costs and risks stemming from the already set requirements.

So I would suggest that the focus needs to be squarely on developing the regulatory architecture that will lead us to the development of such mandatory requirements and standards for the procurement and use of AI by the public sector — which may then become adequate good industry practice for strictly commercial or private contracts. My proposal in that regard is sketched out here.

Final EU model contractual AI Clauses available -- some thoughts on regulatory tunnelling

Source: https://tinyurl.com/mrx9sbz8.

The European Commission has published the final version of the EU model contractual AI clauses to pilot in procurements of AI, which have been ‘developed for pilot use in the procurement of AI with the aim to establish responsibilities for trustworthy, transparent, and accountable development of AI technologies between the supplier and the public organisation.’

The model AI clauses have been developed by reference to the (future) obligations arising from the EU AI Act currently under advanced stages of negotiation. This regulatory technique simply seeks to allow public buyers to ensure compliance with the EU AI Act by cascading the relevant obligations and requirements down to tech providers (largely on a back to back basis). By the same regulatory logic, this technique will be a conveyor belt for the shortcomings of the EU AI Act, which will be embedded in public contracts using the clauses. It is thus important to understand the shortcomings inherent to this approach and to the model AI clauses, before assuming that their use will actually ensure the ‘trustworthy, transparent, and accountable development [and deployment] of AI technologies’. Much more is needed than mere reliance on the model AI clauses.

Two sets of model AI clauses

The EU AI Act will not be applicable to all types of AI use. Remarkably, most requirements will be limited to ‘high-risk AI uses’ as defined in its Article 6. This immediately translates into the generation of two sets of model AI clauses: one for ‘high-risk’ AI procurement, which embeds the requirements expected to arise from the EU AI Act once finalised, and another ‘light version’ for non-high-risk AI procurement, which would support the voluntary extension of some of those requirements to the procurement of AI for other uses, or even to the use of other types of algorithmic solutions not meeting the regulatory definition of AI.

A first observation is that the controversy surrounding the definition of ‘high-risk’ in the EU AI Act immediately carries over to the model AI clauses and to the choice of ‘demanding’ vs light version. While the original proposal of the EU AI Act contained a numerus clausus of high-risk uses (which was already arguably too limited, see here), the trilogue negotiations could well end suppressing a pre-defined classification and leaving it to AI providers to (self)assess whether the use would be ‘high-risk’.

This has been heavily criticised in a recent open letter. If the final version of the EU AI Act ended up embedding such a self-assessment of what uses are bound to be high-risk, there would be clear risks of gaming of the self-assessment to avoid compliance with the heightened obligations under the Act (and it is unclear that the system of oversight and potential fines foreseen in the EU AI Act would suffice to prevent this). This would directly translate into a risk of gaming (or strategic opportunism) in the choice between ‘demanding’ vs light version of the model AI clauses by public buyers as well.

As things stand today, it seems that most procurement of AI will be subject to the light version of the model AI clauses, where contracting authorities will need to decide which clauses to use and which standards to refer to. Importantly, the light version does not include default options in relation to quality management, conformity assessments, corrective actions, inscription in an AI register, or compliance and audit (some of which are also optional under the ‘demanding’ model). This means that, unless public buyers are familiar with both sets of model AI clauses, taking the light version as a starting point already generates a risk of under-inclusiveness and under-regulation.

Limitations in the model AI clauses

The model AI clauses come with some additional ‘caveat emptor’ warnings. As the Commission has stressed in the press release accompanying the model AI clauses:

The EU model contractual AI clauses contain provisions specific to AI Systems and on matters covered by the proposed AI Act, thus excluding other obligations or requirements that may arise under relevant applicable legislation such as the General Data Protection Regulation. Furthermore, these EU model contractual AI clauses do not comprise a full contractual arrangement. They need to be customized to each specific contractual context. For example, EU model contractual AI clauses do not contain any conditions concerning intellectual property, acceptance, payment, delivery times, applicable law or liability. The EU model contractual AI clauses are drafted in such a way that they can be attached as a schedule to an agreement in which such matters have already been laid down.

This is an important warning, as the sole remit of the model AI clauses links back to the EU AI Act and, in the case of the light version, only partially.

the link between model AI clauses and standards

However, the most significant shortcoming of the model AI clauses is that, by design, they do not include any substantive or material constraints or requirements on the development and use of AI. All substantive obligations are meant to be incorporated by reference to the (harmonised) standards to be developed under the EU AI Act, other sets of standards or, more generally, the state-of-the-art. Plainly, there is no definition or requirement in the model AI clauses that establishes the meaning of eg trustworthiness—and there is thus no baseline safety net ensuring it. Similarly, most requirements are offloaded to (yet to emerge) standards or the technical and organisational measures devised by the parties. For example,

  • Obligations on record-keeping (Art 5 high-risk model) refer to capabilities conforming ‘to state of the art and, if available, recognised standards or common specifications. <Optional: add, if available, a specific standard>’.

  • Measures to ensure transparency (Art 6 high-risk model) are highly qualified: ‘The Supplier ensures that the AI System has been and shall be designed and developed in such a way that the operation of the AI System is sufficiently transparent to enable the Public Organisation to reasonably understand the system’s functioning’. Moreover, the detail of the technical and organisational measures that need to be implemented to reach those (qualified) goals is left entirely undefined in the relevant Annex (E) — thus leaving the option open for referral to emerging transparency standards.

  • Measures on human oversight (Art 7 high-risk model) are also highly qualified: ‘The Supplier ensures that the AI System has been and shall be designed and developed in such a way, including with appropriate human-machine interface tools, that it can be effectively overseen by natural persons as proportionate to the risks associated with the system’. Although there is some useful description of what ‘human oversight’ should mean as a minimum (Art 7(2)), the detail of the technical and organisational measures that need to be implemented to reach those (qualified) goals is also left entirely undefined in the relevant Annex (F) — thus leaving the option open for referral to emerging ‘human on the loop’ standards.

  • Measures on accuracy, robustness and cybersecurity (Art 8 high-risk model) follow the same pattern. Annexes G and H on levels of accuracy and on measures to ensure an appropriate level of robustness, safety and cybersecurity are also blank. While there can be mandatory obligations stemming from other sources of EU law (eg the NIS 2 Directive), only partial aspects of cybersecurity will be covered, and not in all cases.

  • Measures on the ‘explainability’ of the AI (Art 13 high-risk model) fall short of imposing an absolute requirement of intelligibility of the AI outputs, as the focus is on a technical explanation, rather than a contextual or intuitive explanation.

All in all, the model AI clauses are primarily an empty regulatory shell. Operationalising them will require reliance on (harmonised) standards—eg on transparency, human oversight, accuracy, explainability … — or, most likely (at least until such standards are in place) significant additional concretisation by the public buyer seeking to rely on the model AI clauses.

For the reasons identified in my previous research, I think this is likely to generate regulatory tunnelling and to give the upper hand to AI providers in making sure they can comfortably live with requirements in any specific contract. The regulatory tunnelling stems from the fact that all meaningful requirements and constraints are offloaded to the (harmonised) standards to be developed. And it is no secret that the governance of the standardisation process falls well short of ensuring that the resulting standards will embed high levels of protection of the desired regulatory goals — some of which are very hard to define in ways that can be translated into procurement or contractual requirements anyway.

Moreover, public buyers with limited capabilities will struggle to use the model AI clauses in ways that meaningfully ‘establish responsibilities for trustworthy, transparent, and accountable development [and deployment] of AI technologies’—other than in relation to those standards. My intuition is that the content of the all too relevant schedules in the model AI clauses will either simply refer to emerging standards or where there is no standard or the standard is for whatever reason considered inadequate, be left for negotiation with tech providers, or be part of the evaluation (eg tenderers will be required to detail how they propose to regulate eg accuracy). Whichever way this goes, this puts the public buyer in a position of rule-taker.

Only very few, well-resourced, highly skilled public buyers (if any) would be able to meaningfully flesh out a comprehensive set of requirements in the relevant annexes to give the model AI clauses sufficient bite. And they would not benefit much from the model AI clauses as it is unlikely that in their sophistication they would not have already come up with similar solutions. Therefore, at best, the contribution of the model AI clauses is rather marginal and, at worse, it comes with a significant risk of regulatory complacency.

final thoughts

indeed, given all of this, it is clear that the model IA clauses generate a risk if (non-sophisticated/most) public buyers think that relying on them will deal with the many and complex challenges inherent to the acquisition of AI. And an even bigger risk if we collectively think that the existence of such model AI clauses is all the regulation of AI procurement we need. This is not a criticism of the clauses in themselves, but rather of the technique of ‘regulation by contract’ that underlies it and of the broader approach followed by the European Commission and other regulators (including the UK’s)!

I have demonstrated how this is a flawed regulatory strategy in my forthcoming book Digital Technologies and Public Procurement. Gatekeeping and Experimentation in Digital Public Governance (OUP) and in many working papers resulting from the project with the same title. In my view, we need to do a lot more if we want to make sure that the public sector only procures and uses trustworthy AI technologies. We need to create a regulatory system that assigns to an independent authority both the permissioning of the procurement of AI and the certification of the standards underpinning such procurement. In the absence of such regulatory developments, we cannot meaningfully claim that the procurement of AI will be in line with the values and goals to be expected from ‘responsible’ AI use.

I will further explore these issues in a public lecture on 23 November 2023 at University College London. All welcome: Hybrid | Responsibly Buying Artificial Intelligence: A Regulatory Hallucination? | UCL Faculty of Laws - UCL – University College London.

Source: https://public-buyers-community.ec.europa....

Some thoughts on the need to rethink the right to good administration in the digital context

Colleagues at The Digital Constitutionalist have put together a really thought-provoking symposium on ‘Safeguarding the Right to Good Administration in the Age of AI’. I had the pleasure of contributing my own views on the need to extend and broaden good administration guarantees in the context of AI-assisted decision-making. I thoroughly recommend reading all contributions to the symposium, as this is an area of likely development in the EU Administrative Law space.