Testing the limits of ChatGPT’s procurement knowledge (and stubbornness) – guest post by Džeina Gaile

Following up on the discussion whether public sector use of ChatGPT should be banned, in this post, Džeina Gaile* shares an interesting (and at points unnerving) personal experiment with the tool. Džeina asked a few easy questions on the topic of her PhD research (tender clarifications).

The answers – and the ‘hallucinations’, that is, the glaring mistakes – and the tone are worth paying attention to. I find the bit of the conversation on the very existence of Article 56 and the content of Article 56(3) Directive 2014/24/EU particularly (ahem) illuminating. Happy reading!

PS. If you take Džeina up on her provocation and run your own procurement experiment on ChatGPT (or equivalent), I will be delighted to publish it here as well.

Liar, liar, pants on fire – what ChatGPT did not teach me
about my own PhD research topic

 DISCLAIMER: The views provided here are just a result of an experiment by some random procurement expert that is not a specialist in IT law or any other AI-related law field.

If we consider law as a form of art, as lawyers, words are our main instrument. Therefore, we have a special respect for language as well as the facts that our words represent. We know the liability that comes with the use of the wrong words. One problem with ChatGPT is - it doesn't. 

This brings us to an experiment that could be performed by anyone having at least basic knowledge of the internet and some in-depth knowledge in some specific field, or at least an idea of the information that could be tested on the web. What can you do? Ask ChatGPT (or equivalent) some questions you already know the answers to. It would be nice if the (expected) answers include some facts, numbers, or people you can find on Google. Just remember to double-check everything. And see how it goes.

My experiment was performed on May 3rd, 4th and 17th, 2023, mostly in the midst of yet another evening spent trying to do something PhD related. (As you may know, the status of student upgrades your procrastination skills to a level you never even knew before, despite your age. That is how this article came about).

I asked ChatGPT a few questions on my research topic for fun and possible insights. At the end of this article, you can see quite long excerpts from our conversation, where you will find that maybe you can get the right information (after being very persuasive with your questions!), but not always, as in the case of the May 4th and 17th interactions. And you can get very many apologies during that (if you are into that).[1]

However, such a need for persuasion oughtn’t be necessary if the information is relatively easy to find, since, well, we all have used Google and it already knows how to find things. Also, you can call the answers given on May 4th and 17th misleading, or even pure lies. This, consequently, casts doubt on any information that is provided by this tool (at least, at this moment), if we follow the human logic that simpler things (such as finding the right article or paragraph in law) are easier done than complex things (such as giving an opinion on difficult legal issues). As can be seen from the chat, we don’t even know what ChatGPT’s true sources are and how it actually works when it tells you something that is not true (while still presenting it as a fact). 

Maybe some magic words like “as far as I know” or “prima facie” in the answers could have provided me with more empathy regarding my chatty friend. The total certainty with which the information is provided also gives further reasons for concern. What if I am a normal human being and don’t know the real answer, have forgotten or not noticed the disclaimer at the bottom of the chat (as it happens with the small letter texts), or don’t have any persistence to check the info? I may include the answers in my homework, essay, or even in my views on the issue at work—since, as you know, we are short of time and need everything done by yesterday. The path of least resistance is one of the most tempting. (And in the case of AI we should be aware of a thing inherent to humans called “anthropomorphizing”, i.e., attributing human form or personality to things not human, so we might trust something a bit more or more easily than we should.)

The reliability of the information provided by State institutions as well as lawyers has been one of the cornerstones of people’s belief in the justice system. Therefore, it could be concluded that either I had bad luck, or one should be very careful when introducing AI in state institutions. And such use should be limited only to cases where only information about facts is provided (with the possibility to see and check the resources) until the credibility of AI opinions could be reviewed and verified. At this moment you should believe the disclaimers of its creators and use AI resources with quite (legitimate) mistrust and treat it somewhat as a child that has done something wrong but will not admit it, no matter how long you interrogate them. And don’t take it for something it is not, even if it sounds like you should listen to it.**

May 3rd, 2023

[Reminder: Article 56(3) of the Directive 2014/24/EU: Where information or documentation to be submitted by economic operators is or appears to be incomplete or erroneous or where specific documents are missing, contracting authorities may, unless otherwise provided by the national law implementing this Directive, request the economic operators concerned to submit, supplement, clarify or complete the relevant information or documentation within an appropriate time limit, provided that such requests are made in full compliance with the principles of equal treatment and transparency.]

[...]

[… a quite lengthy discussion about the discretion of the contracting authority to ask for the information ...]

[The author did not get into a discussion about the opinion of ChatGPT on this issue, because that was not the aim of the chat, however, this could be done in some other conversation.]

[…]

[… long explanation ...]

[...]

May 4th, 2023

[Editor’s note: apologies that some of the screenshots appear in a small font…].

[…]

Both links that the ChatGPT gave are correct:

https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32014L0024

https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32014L0024&from=EN

However, both citations are wrong.

May 17th, 2023

[As you will see, ChatGPT doesn’t give links anymore, so it could have learned a bit within these few weeks].

[Editor’s note: apologies again that the remainder of the screenshots appear in a small font…].

[...]

[Not to be continued.]

DŽEINA GAILE

My name is Džeina Gaile and I am a doctoral student at the University of Latvia. My research focuses on clarification of a submitted tender, but I am interested in many aspects of public procurement. Therefore, I am supplementing my knowledge as often as I can and have a Master of Laws in Public Procurement Law and Policy with Distinction from the University of Nottingham. I also have been practicing procurement and am working as a lawyer for a contracting authority. In a few words, a bit of a “procurement geek”. In my free time, I enjoy walks with my dog, concerts, and social dancing.

________________

** This article was reviewed by Grammarly. Still, I hope it will not tell anything to the ChatGPT… [Editor’s note – the draft was then further reviewed by a human, yours truly].

[1] To be fair, I must stress that at the bottom of the chat page, there is a disclaimer: “Free Research Preview. ChatGPT may produce inaccurate information about people, places, or facts. ChatGPT May 3 Version” or “Free Research Preview. ChatGPT may produce inaccurate information about people, places, or facts. ChatGPT May 12 Version” later. And, when you join the tool, there are several announcements that this is a work in progress.


ECJ opens door to remedial possibilities when contracting authorities aim to exclude on the basis of shady participation requirements (C-27/15)

In its Judgment of 2 June 2016 in Pizzo, C-27/15, EU:C:2016:404, the European Court of Justice (ECJ) has provided interpretative guidance on some aspects of the qualitative selection process that contracting authorities need to carry out prior to the award of public contracts covered by the relevant EU rules. It is worth noting that, even if the Pizzo Judgment is based on the rules of Directive 2004/18 (Arts 47 and 48), but the functional criteria it sets will be equally relevant under the new rules of Directive 2014/24 (Arts 56 to 58).

In particular, the Pizzo Judgment clarifies the scope of the discretion given to contracting authorities to interpret tender documents in a way that would unfavourably result in the exclusion of economic operators (following Manova, C‑336/12, EU:C:2013:647, see here; and Cartiera dell'Adda, C‑42/13, EU:C:2014:2345, see here), as well as the limits of the possibility given to economic operators to remedy formal shortcomings in the documentation presented in the course of their participation in the tender for a public contract in order to avoid such exclusion.

Pizzo also follows the flexible approach previously established by the ECJ regarding reliance on third party capacities (as per Swm Costruzioni 2 and Mannocchi Luigino, C-94/12, EU:C:2013:646, see here; and in a related fashion to Ostas celtnieks, C-234/14, EU:C:2016:6, see here). 

The case concerned the tender of a public service contract for the management of waste and cargo residues produced on board ships calling at ports within the contracting authority’s territorial jurisdiction (ie the Messina area, in Italy). The contracting authority received four tenders but decided to exclude three tenderers due to the lack of payment of an administrative fee that it considered a mandatory participation requirement, which led the authority to award the contract to the only remaining tenderer (Pizzo).

One of the excluded tenderers (CGRT) appealed the exclusion decision on the basis that the payment of the fee was only mandatory for works contracts, not for services contracts, and that the authority required such payment on the basis of a broad interpretation of the relevant rules and its general powers under Italian administrative law. In CGRT's submission, before proceeding to its exclusion from the tender, the authority should at least have given it the possibility to remedy the situation and pay the fee. CGRT's action was faced with a counterclaim by Pizzo whereby it challenged CGRT's compliance with the economic standing requirements for the participation in the tender due to the fact that it had relied on third party capacities (in the case, those of RIAL). 

Therefore, the case raised two issues: 1) to what extent can a contracting authority engage in a (discretionary) broad interpretation of the tender documents in a way that incorporates the requirement to pay a fee and proceed to the exclusion of those tenderers that had not paid the fee without giving them the opportunity to remedy the situation; and 2) to what extent must contracting authorities follow a flexible approach to the assessment of economic and technical standing requirements when tenderers rely on third party capacities. The first issue is more interesting and controversial than the second one. Thus, let's focus on the second issue first.

Again, on the flexible approach to reliance on third party capacities and how it carries over to new rules of directive 2014/24

In Pizzo, the ECJ revisits its consolidated case law in this area, without adding much to the already clear position that the applicable EU rules create significant flexibility for tenderers to rely on third party capacity, unless the tender documents establish reasonable and proportionate restrictions justified by the subject matter of the contract. In its own terms:

23      The Court has held that EU law does not require that, in order to be classified as an economic operator qualifying for tendering, a person wishing to enter into a contract with a contracting authority must be capable of direct performance using his own resources (see, to that effect, judgment of 23 December 2009 in CoNISMa, C‑305/08, EU:C:2009:807, paragraph 41).
24      ... Article 47(2) and Article 48(3) of Directive 2004/18 does not lay down any general prohibition regarding a candidate or tenderer’s reliance on the capacities of one or more third-party entities in addition to its own capacities in order to fulfil the criteria set by a contracting authority (see Swm Costruzioni 2, paragraph 30).
25      ... those provisions recognise the right of every economic operator to rely, for a particular contract, upon the capacities of other entities, ‘regardless of the nature of the links which it has with them’, provided that it proves to the contracting authority that it will have at its disposal the resources necessary for the performance of the contract (see Ostas celtnieks, paragraph 23).
26      It must therefore be held that Directive 2004/18 permits the combining of the capacities of more than one economic operator for the purpose of satisfying the minimum capacity requirements set by the contracting authority, provided that the candidate or tenderer relying on the capacities of one or more other entities proves to that authority that it will actually have at its disposal the resources of those entities necessary for the execution of the contract (see Swm Costruzioni 2, paragraph 33).
27      Such an interpretation is consistent with the objective pursued by the directives in this area of attaining the widest possible opening-up of public contracts to competition to the benefit not only of economic operators but also contracting authorities (see, to that effect, CoNISMa, paragraph 37 and the case-law cited). In addition, that interpretation also facilitates the involvement of small- and medium-sized undertakings in the contracts procurement market, an aim also pursued by Directive 2004/18, as stated in recital 32 thereof (see Swm Costruzioni 2, paragraph 34).
28      ... however ... there may be works the special requirements of which necessitate a certain capacity which cannot be obtained by combining the capacities of more than one operator, which, individually, would be inadequate. It has thus acknowledged that, in such circumstances, the contracting authority would be justified in requiring that the minimum capacity level concerned be achieved by a single economic operator or by relying on a limited number of economic operators as long as that requirement is related and proportionate to the subject matter of the contract at issue. The Court has, however, stated that since those circumstances represent an exception, the requirements in question cannot be made general rules under national law (see, to that effect, Swm Costruzioni 2, paragraphs 35 and 36) (C-27/15, paras 23-28, references shortened).

The only novelty to be found in Pizzo is that the ECJ anticipates the interpretation of Art 63 Dir 2014/24 by stressing that

the specific provisions ... provide that it is possible for the contracting authority to require that the entity which is relied on to satisfy the conditions laid down with regard to economic and financial standing is to be jointly liable (Article 63(1), third subparagraph, of Directive 2014/24) or to require that, with regard to certain types of contracts, certain critical tasks are to be performed directly by the tenderer (Article 63(2) of that directive). Those provisions do not therefore impose specific limits on the possibility of divided reliance on the capacities of third-party undertakings and, in any event, such limits should have been expressly set out for in the call for tenders in respect of the contract at issue, which is not the case in the main proceedings (C-27/15, para 33, emphasis added).

This is in line with the interpretation of Art 63 Dir 2014/24 I hold in Public procurement and the EU competition rules, 2nd edn (Oxford, Hart, 2015) 315-318, but it is worth stressing that the ECJ has not yet tackled some of the restrictions allowed for under Art 63(2) Dir 2014/24 (ie the requirement that certain critical tasks are to be performed directly by the tenderer) which in my view run contrary to its previous case law. Thus, in this instance, the fact that the ECJ makes obiter comments on the likely future interpretation of the rules of Dir 2014/24 is worrying because there are two possible readings of paragraph 33 in Pizzo: (a) that the ECJ is giving carte blanche to the potential restrictions created by Art 63(1) and (2), or (b), that the ECJ is simply stressing that (regardless of their substantive merit and from a prior formal perspective), for such requirements to apply, they need to be created in the applicable tender documents (which, having not happened in this case, makes them irrelevant). I strongly vouch for (b), but I am certain that there will be claims based on (a) when the issue properly arises in litigation. Thus, in this case, the probably well-intended effort by the ECJ to anticipate the interpretation of the new rules may have created more shadows than lights.

On the tricky issue of the interpretation of tender documents, the duty to seek clarification or, at least, allow for remediation of short-comings leading to exclusion of economic operators

When tackling the challenge of the contracting authority's broad interpretation of the obligation to pay an administrative fee and its decision to exclude, without possibility to remedy such short-coming, the economic operators that had failed to pay it, the rephrased the question to mean 'whether the principle of equal treatment and the obligation of transparency are to be interpreted as precluding an economic operator from being excluded from a procedure for the award of a public contract as a result of that economic operator’s non-compliance with an obligation which does not expressly arise from the documents relating to that procedure or out of the national law in force, but from an interpretation of that law and from the incorporation of provisions into those documents by the national authorities or administrative courts' (C-27/15, para 35).

In that regard, the ECJ makes the following arguments and establishes the following reasoning:

36 ... all the conditions and detailed rules of the award procedure must be drawn up in a clear, precise and unequivocal manner in the contract notice or specifications so that, first, all reasonably informed tenderers exercising ordinary care can understand their exact significance and interpret them in the same way and, second, the contracting authority is able to ascertain whether the tenders submitted satisfy the criteria applying to the contract in question (see, to that effect, Cartiera dell’Adda, paragraph 44 and the case-law cited).
37      The Court has also held that the principles of transparency and equal treatment which govern all procedures for the award of public contracts require the substantive and procedural conditions concerning participation in a contract to be clearly defined in advance and made public, in particular the obligations of tenderers, in order that those tenderers may know exactly the procedural requirements and be sure that the same requirements apply to all candidates ...
45 ... in the case in the main proceedings, the alleged obligation to pay a fee to the AVCP [the Supervisory Authority on Public Procurement] can be identified only by the interaction between the 2006 Finance Law, the AVCP’s decision-making practice and the judicial practice of the Italian administrative courts in applying and interpreting Law No 266/2005.
46      As the Advocate General points out ... a condition governing the right to participate in a public procurement procedure which arises out of the interpretation of national law and the practice of an authority ... would be particularly disadvantageous for tenderers established in other Member States, inasmuch as their level of knowledge of national law and the interpretation thereof and of the practice of the national authorities cannot be compared to that of national tenderers.
48 ... it is apparent from the order for reference that there is no possibility of rectifying non-compliance with that condition that a fee must be paid.
49      According to paragraph 46 of the judgment in Cartiera dell’Adda ..., the contracting authority may not accept any rectification of omissions which, as expressly provided for in the contract documentation, must result in the exclusion of the bid. The Court stated, in paragraph 48 of that judgment, that the obligation concerned was clearly laid down in the contract documentation, on pain of exclusion.
50      However, in a situation where ... a condition for participating in a procedure for the award of a contract, on pain of exclusion from that procedure, is not expressly laid down in the contract documentation and that condition can be identified only by a judicial interpretation of national law, the contracting authority may grant the excluded tenderer a sufficient period of time in order to rectify its omission (C-27/15, paras 36-50, some references omitted and emphasis added).

I agree with the main reasoning of the ECJ on this issue as it coincides with a possibilistic and functional approach to the management of the exclusion and qualitative selection procedure aimed at minimising exclusion for causes that can be remedied without infringing the principle of equal treatment [for discussion, see A Sanchez-Graells, 'Exclusion, Qualitative Selection and Short-listing', in F Lichère, R Caranta & S Treumer (eds), Modernising Public Procurement. The New Directive, vol. 6 European Procurement Law Series (Copenhagen, DJØF, 2014) 97-129; and ibid, 'Rejection of Abnormally Low and Non-Compliant Tenders in EU Public Procurement: A Comparative View on Selected Jurisdictions', M Comba & S Treumer (eds) Award of Contracts in EU Procurements, vol. 5 European Procurement Law Series (Copenhagen, DJØF, 2013) 267-302].

However, I am not convinced by the way the ECJ has limited the opportunity to remedy the (interpreted) shortcomings in the tender documentation (or material requirements) to a mere possibility. As phrased in the operational part of the Pizzo Judgment, the ECJ has interpreted that

the principle of equal treatment and the obligation of transparency must be interpreted as precluding an economic operator from being excluded from a procedure for the award of a public contract as a result of that economic operator’s non-compliance with an obligation which does not expressly arise from the documents relating to that procedure or out of the national law in force, but from an interpretation of that law and those documents and from the incorporation of provisions into those documents by the national authorities or administrative courts. Accordingly, the principles of equal treatment and of proportionality must be interpreted as not precluding an economic operator from being allowed to regularise its position and comply with that obligation within a period of time set by the contracting authority (C-27/45, para 51, emphasis added).

I find this approach too lenient and I would have expected the ECJ to create a mandatory vis-a-vis procedure similar to the one applicable in case the contracting authority suspects an offer to be abnormally low under Art 69 Dir 2014/24. Generally, I think that rather than focusing solely on the principle of equal treatment and non-discrimination, it is worth stressing the relevance of the principle of good administration as well. From that perspective, if the contracting authority identifies a participation requirement that was not obvious from the tender documentation, it should be subjected to a mandatory phase whereby it allows tenderers to remedy the situation. The same would go for the interpretation of Art 56(3) Dir 2014/24 in terms of the possibility (in my view, non-discretionary) to seek clarifications from tenderers and to 'take all appropriate steps to avoid the rejection of candidates on the basis of shortcomings in the available documentation that could be overcome if the contracting authority were to exercise the appropriate level of diligence' [Public procurement and the EU competition rules (2015) 321-323].

Overall, I think that this is an area where the ECJ is avoiding a much needed delineation of the limits (or at least checks and balances) to be imposed on the discretion of the contracting authorities to proceed to exclusion without exhausting the possibilities for clarification or remedy of formal shortcomings in the submission of tenders. This is likely to be an area of continued litigation, particularly as the Pizzo case opens the door to different treatment of participation requirements that directly derive from the tender documentation (where the contracting authority is likely to have its hands tied and not be able to provide any scope for remedial action beyond the very limited possibilities foreseen in Manova and Cartiera dell’Adda) or that indirectly arise from its contextual interpretation (where the Pizzo approach seems to open a rather big door to the enablement of remedial actions). Thus, the last word is certainly not yet written...