Wishful legal analysis as a trade strategy? A rebuttal to the Minister for International Trade

In the context of the Parliamentary scrutiny of the procurement chapters of the UK’s Free Trade Agreements with Australia and New Zealand, I submitted several pieces of written evidence, which I then gathered together and reformulated in A Sanchez-Graells, ‘The Growing Thicket of Multi-Layered Procurement Liberalisation between WTO GPA Parties, as Evidenced in Post-Brexit UK’ (2022) 49(3) Legal Issues of Economic Integration 247–268. I was also invited to submit oral evidence to the Public Bills Comittee for the Trade (Australia and New Zealand) Bill.

In my research, I raised some legal issues on the way the UK-AUS and UK-NZ procurement chapters would interact with the World Trade Agreement Government Procurement Agreement (GPA)—to which UK, AUS and NZ are members—and the Comprehensive and Progressive Agreement for Trans-Pacific Partnership (CPTPP)—to which the UK seeks accession and both AUS and NZ are members. I also raised issues with the rules on remedies in particular, both in relation to UK-AUS and the CPTPP.

I have now become aware of a letter from the Minister for International Trade, where the UK Government simply dismisses my legal analysis in an unconvicing manner. In this post, I try to rebut their position—although their lack of arguments makes this rather difficult—and stress some of the misunderstandings that the letter evidences. The letter seems to me to reflect a worrying strategy of ‘wishful legal analysis’ that does not bode well for post-Brexit UK trade realignment.

Interaction between the GPA, FTAs and the CPTPP

In my analysis and submissions, I stressed how deviations in the UK’s FTAs from the substantive obligations set in the GPA generate legal uncertainty and potential problems in ‘dual regulation’ situations, where one of the contracting parties (eg the UK) would be under the impossibility of complying at the same time with the obligations resulting from the GPA with tenderers from GPA countries and those arising from the FTAs with AUS or NZ with their tenderers—without either breaching GPA obligations or, what is more likely, ignoring the deviation in the FTAs to ensure GPA compliance. It would also generate issues where compliance with the more demanding standards in the FTAs would be automatically propagated to the benefit of economic operators from other jurisdictions. I also raised how the deviations can generate legal uncertainty and make it more difficult for UK tenderers to ascertain their legal position in AUS and NZ. And I also raised how this situation can get further complicated if the UK accesses CPTPP.

My concerns were discussed in Committee and the Minister had the following to say:

The [GPA] and the [CPTPP] are plurilateral agreements between twenty-one and eleven parties respectively, including in each case Australia and New Zealand. As recognised in Committee, the [GPA] in particular establishes a global baseline for international procurement. Nonetheless, neither prevents its members from entering into bilateral free trade agreements to sit alongside the [GPA] and [CPTPP] while at the same time going further in terms of the procurement commitments between members.

These Agreements with Australia and New Zealand do just that, going beyond both the [GPA] and the [CPTPP] baselines. … Although the texts of the Agreements with Australia and New Zealand are sometimes laid out differently to the way they are in the Agreement on Government Procurement, they in no way dilute or reduce the global baseline established by the [GPA]. (emphases added).

There are two points to note, here. The first one is that the fact that the GPA and the CPTPP allow for bilateral agreements between their parties does not clarify how the overlapping treaties would operate, which is exactly what I analysed. Of note, under the 1969 Vienna Convention on the Law of Treaties (Art 30), when States conclude successive treaties relating to the same subject matter, the most recent treary prevails, and the provisions of the earlier treaty/ies only apply to the extent that they are not incompatible with those of the later treaty.

This is crucial here, especially as the Minister indicates that the UK-AUS and UK-NZ go beyond not only the GPA, but also the CPTPP. This would mean that entering into CPTPP after UK-AUS and UK-NZ—as the UK is currently in train of doing—could negate some of the aspects that go beyond CPTPP in both those FTAs. Moreover, the simple assertion that the FTAs do not dilute the GPA baseline is unconvincing, as detailed analysis shows that there are significant problems with eg the interpretation of the national treatment under the different treaties.

Secondly, the explanation provided does not resolve the practical problems arising from ‘dual regulation’ that I have identified and leaves the question open as to how the obligations under the FTAs will be interpreted and complied with in triangular situations involving tenderers not from AUS or NZ. Either the UK will apply the more demanding obligations—which will then benefit all GPA parties, not only AUS and NZ—or will stick to the GPA baseline in breach of the FTAs. There is no recognition of this issue in the letter.

The Minister also indicated that:

There was also suggestion in Committee that it would be difficult for suppliers in the United Kingdom to navigate the Agreements with Australia and New Zealand, as well as the [CPTPP] in the future. I would like to reassure the Committee that when bidding for United Kingdom procurements, the only system that British suppliers need to concern themselves with is United Kingdom’s procurement regulations. (emphasis added).

The Minister has either not understood the situation, or is seeking to obscure the analysis here. The concerns about legal uncertainty do not relate to UK businesses tendering for contracts in the UK, but to UK businesses tendering for contracts in AUS or NZ—which are the ones that would be seeking to benefit from the trade liberalisation pursued by those FTAs. Nothing in the Minister’s letter addresses this issue.

Domestic review rights under the Australian procurement chapter

One of the specific deviations from the GPA baseline that I identified in my research concerns the exclusion of access to remedies on grounds of public interest. While the GPA only allows excluding interim measures on such grounds, the AUS-UK FTA and CPTPP seem to allow for public interest to also bar access to remedies such as compensation—and, if this does not limit access to remedies as I submit, at least it does cause legal uncertainty in that respect.

My submission is met with the following response by the Minister [the mentioned annex is reproduced at the end of this post]:

The Committee also considered the evidence raised by Professor Sánchez-Graells regarding domestic review procedures … The Government respectfully disagrees with the analysis presented at that session that a provision in the government procurement chapter of the [UK-AUS FTA] ‘allows for the exclusion of legal remedies completely on the basis of public interest’.

The public interest exclusion only applies to temporary measures put in place to ensure aggrieved suppliers may continue to participate in a procurement.

The Government also respectfully disagrees with the suggestion in the witness evidence that this public interest exclusion is not similarly reflected in the [GPA] or the [UK-NZ FTA]. The Government acknowledges that the specific position of the exclusion differs between these agreements and is closer to the approach adopted in the [CPTPP]. Nonetheless, the Government do not consider this alters the legal effect or gives rise to legal uncertainty. For the benefit of the Committee, the relevant provisions from each of the [FTAs], the [GPA] and the [CPTPP] are set out in an annex to this letter.

The Minister’s explanations are not supported by any arguments. There is no reasoning to explain why the order of the clauses and subclauses in the relevant provisions does not alter their legal interpretation or effects. There is also no justification whatsoever for the opinion that textual differences do not give rise to legal uncertainty. The Government seems to think that it can simply wish the legal issues away.

The table included in the annex to the letter (below) is revealing of the precise issue that raises legal uncertainty and, potentially, a restriction on access to remedies other than interim measures beyond the GPA (and thus, in breach thereof). Why would treaties that seem to replicate the same rules draft them differently? How can any legal interpreter be of the opinion that the positioning of the exception clause does not have an effect on the interpretation of its scope of application? Is the fact that these agreements post-date the GPA and still deviate from it not of legal relevance?

Of course, there are arguments that could be made to counter my analysis. They could eg focus on the use of different (undefined) terms in different sub-clauses (such as ‘measures’ and ‘corrective action’). They could also focus on any preparatory works to the treaties (especially the CPTPP and UK-AUS FTA, which I have not yet been able to locate). They could even be more creative and attempt functional or customary interpretation arguments. But the letter contains no arguments at all.

Conclusion

It is a sad state of affairs where detailed legal analysis—whether correct or not—is dismissed without offering any arguments to the contrary and simply seeking to leverage the ‘authority’ of a Minister or Department. If this is the generalised approach to assessing the legal implications of the trade agreements negotiated (at speed) by the UK post-Brexit, it does not bode well for the legal certainty required to promote international investments and commercial activities.

The reassurances in the letter are void of any weight, in my view. I can only hope that the Committee is not persuaded by the empty explanations contained in the letter either.



Creating reliable econometric models of the CJEU case law: a response to criticisms (by Arrebola, Mauricio & Jimenez)

One of the most satisfactory activities in academia is to engage in debate and discussion. Only by subjecting ideas to tough scrutiny can we advance in our knowledge. Thus, I am extremely pleased that Carlos Arrebola, Julia Mauricio and Hector Jimenez have reacted so quickly to my criticism of their recent paper (here) and come back with a thoughtful and forceful rebuttal. I am posting it below. You will see that there are important points of disagreement that will probably require two (or more) follow-up studies in the future. Seems like I need to brush up my econometrics...

Creating reliable econometric models of the CJEU case law:
a response to Sanchez-Graells’ criticisms

by Carlos Arrebola, Julia Mauricio and Hector Jimenez


In a recent study, we used econometric methodology to quantify the degree of influence of the Advocate General on the Court of Justice. Based on data collected from 20 years of actions for annulment, we concluded that the Court is 67% more likely to annul an act if the Advocate General suggests so in her opinion. In a post last Tuesday, Sanchez-Graells examined our paper. As he said, our conclusion is ‘bold [...] and controversial [for its] implications’, and as such it should be subject to ‘tough scrutiny’. We most definitely agree on both the importance of our claim and the need to test it rigorously. As we stated in our paper, if the conclusions are true, the role of the Advocate General within the Court might need to be reconsidered in order to secure judicial independence.

However, Sanchez-Graells voiced several criticisms regarding our econometric model that prevent him from accepting the validity of our results. We greatly welcome the debate, and appreciate the comments in his post, although we ultimately disagree. While we acknowledge that quantitative methodology is not perfect, we argue that our results are a reliable estimation of the influence of the Advocate General (hereinafter, “AG”) on the Court. If not in the specific number of 67% increased probability of a judicial outcome, our results are at least an indication that the influence relationship is positive, as it is shown by the six different econometric models estimated in our study. In the spirit of discussion and debate of this blog, we address Sanchez-Graells’ criticisms along with several other factors that, in our opinion, should have been taken into account when assessing our paper’s reliability.

1. The impossibility of using Randomised Controlled Trials

In his post, Sanchez-Graells suggests that we were too quick to discard the possibility of testing the hypothesis of the influence of the AG on the Court using Randomised Controlled Trials (“RCTs”). For a layperson, RCTs are the type of scientific methodology used in many areas of science to study causality. One of the main examples where RCTs are used is medicine. In order to prove the validity of a new drug, several groups of patients with similar features are randomly selected. Normally, one of those groups would be the control group. The control group would receive a placebo, instead of the actual drug. In this way, the researchers can easily infer whether the health outcome is caused only by the drug. If both the group taking the placebo and the group taking the drug had the same reaction, it would be clear that some external factor other than the drug had caused it. If, on the other hand, the group taking the drug and the placebo group reacted differently (for example, in the case of an illness, if the group taking the drug was the only one to recover), it could be said with certain confidence that the drug caused the recovery.

In our paper, we suggested that RCTs are not a possibility because it would require using the Court of Justice as a laboratory, experimenting with cases, judges and AGs. Nevertheless, Sanchez-Graells argued that we should have considered those cases in which the AG does not participate as our “control group”. This is a misconception about how RCTs are designed. A vital feature in the design of RCTs is making sure that the observations that included in the sample are randomly drawn. This is because, ideally, you would like every observation to be identical, so that the only factor that affects it is the treatment that you are examining in the experiment. In the case of medicine-related RCTs, you want patients with the same characteristics, symptoms, etc., so that whatever happens after taking the drug can only be traced back to the drug. In our study, we would need the same case to be repeated several times, with the same legal problem to be solved by the same judges, having access to the same amount of precedent, lawyers with the same ability to plead cases, etc. Only having that could we then observe what would happen if we took the element of the Advocate General out of the equation. However, cases are never the same. Unlike illnesses, where patients tend to have the same symptoms, cases are much more complex. Legal problems rarely have the same surrounding circumstances.

So, if we followed Sanchez-Graells’ suggestion, we would be ignoring a set of external factors that actually affect the outcome of a case. We would be wrongly attributing it to the Advocate General’s intervention, when actually it could be something else. That is, if we had two cases, one with an AG’s opinion, and one without, in which the Court reached different results, we could not say that the Advocate General caused that different result. It could be that the case had different facts, and that is why the Court decided differently. Or, it could well be that the judges were presented with different arguments by the parties, and it was the lawyers, and not the AGs, who persuaded the Court. Furthermore, Sanchez-Graells’ suggestion is unfeasible because there is a clear bias. As he explained, the cases in which the CJEU considers that there are not going to be problematic legal issues, they decide not to have an AG opinion. It means that from the very beginning of the case they are sensing that it might have an easy or clear legal solution. In other words, Sanchez-Graells is suggesting that we compare in our analysis a simple cold, with a more complicated condition, such as cancer, and that we can thus establish whether radiotherapy has any impact on health. The outcome to such a query would have a misleading result, because the colds would have a rate of recovery close to 100%, whether the cancer would be lower. However, that would not tell us anything about the effectiveness of radiotherapy. In the same way, if a case deals with unproblematic legal issues, the opinion of the AG will probably not do much to affect the Court, because the Court would have come to that conclusion by itself without any external influence. We cannot simply compare those two scenarios without losing information. After all, there would not be any “random” selection of groups, clearly not fulfilling the requirements to conduct a RCT.

For that reason, the only way to approximately estimate causality is to use regressions, in which you can account for as many variables as possible that may influence the Court, including the Advocate General, and including variables that will account for how easy it is to solve a case or clear a case is. That way we will know the exact magnitude of the variable AG on the Court.

2. Designing a reliable regression

Once we establish that the most accurate measure is a regression model accounting for variables that affect the outcome of the Court, the difficulty arises in deciding which variables to include and how to code them. It is in this respect that we think Sanchez-Graells raises his most valid criticism of our study. We acknowledge that our variables are not perfect. We will never be able to establish causality without a shadow of a doubt. This is simply because, as we said, we will always miss variables that affect the case that we will not be able to track, codify and insert in our database. Taking this to an extreme and absurd example, we will never be able to verify whether the judge in the deliberating room had a headache and wanted to go home soon, rushing her decision. However, the fact that we will always miss variables does not mean that our model cannot be reliable. We still include a number of important variables that can explain a substantial amount of what goes on in the courtroom. There are different ways in econometrics to determine the extent to which a model, albeit missing variables, is an accurate depiction of reality. For our study, these measures suggest that the model is indeed reliable. We will come back to this in a moment.

Another aspect of coding variables is, as Sanchez-Graells comments, the oversimplification. In our study, we used actions for annulment, where the outcomes of a case can be (i) annulment, (ii) partial annulment, (iii) dismissal of the case, or (iv) inadmissibility of the case. We decided to simplify this variable by looking only at whether the Court decided to annul (in any of its forms) or not. But, the oversimplification is necessary to make it more reliable, because in order to have a dataset capable of yielding significant results, we need to have a representative sample. In our case, we only had data for a very small number of partial annulments. Including them as a separate variable from total annulment would have only created “noise” in our model, making the results less significant, statistically speaking.

Sanchez-Graells especially criticises our grouping of dismissal and inadmissibility cases together, because he says that dismissing a case and declaring it inadmissible are very different things. However, that discussion in his post is unnecessary, because as he himself notes later on, our results ‘cannot be interpreted regarding inverse AG recommendations (ie recommendations to inadmit/dismiss)’. Our results are only relevant for decisions to annul or partially annul; we do not make any claim about other type of cases, which Sanchez-Graells also criticises.

However, the fact that we decided to look at the question in terms of what happens if the AG suggests to annul the act, rather than if she suggests to dismiss it or declare it inadmissible, does not affect the reliability of our results. In fact, the only thing that Sanchez-Graells is postulating is a new hypothesis. He is saying that, in his opinion, we would have got other results if we had constructed the model differently. That is a point that we cannot falsify without fiddling for a few more weeks with our data in the econometrics software. But, we invite people, and we ourselves may do it in the future, to carry out other studies, with the same or different data to check that the results are not affected if we look at things in a different way; by, for example, looking at what happens if the AG suggests dismissal, or what happens if we gather data from other periods of time. Nonetheless, the reliability of the results that we presented is a separate issue.

So, if we have acknowledged that we are not going to be able to include every variable, and that our data is only a sample, why are we confident in our results? In the paper we explain it more technically, but, basically, there are econometric measures that indicate that the model that we have created is accurate when the estimation that we get from the model is compared with actual data from reality. That is the reason why we know it is a fairly reliable model.

3. Final caveat

Whilst reading Sanchez-Graells’ words, we could not avoid feeling something we felt many times before. Lawyers are more comfortable sticking to arguing with words.  We feel somehow threatened by this terra incognita called econometrics. There seems to be a certain reticence to attempting to use mathematics to help us in our enquiries. It is worth saying that we are not accusing Sanchez-Graells of not wanting to engage with quantitative methodology. In fact, we know that he has used some statistics previously, and we would not expect a “more economic approach” type of person to disregard this evidence-based methodology.

We want to end this post with a final note about quantitative methodology. We want to say that although judicial proceedings and legal arguments cannot always be equated to numbers, and other methodologies are extremely valuable to legal research questions, quantitative analysis can help elucidate complex legal questions. As many other subjects in social sciences did before us, statistics can become a tool at the service of legal researchers. In this sense, it is worth reminding the readers that, a few centuries ago, economics was equally a merely discursive subject, and anyone who has read the Wealth of Nations can be a witness to that.  But, now, economics and mathematics cannot be separated. Therefore, we would encourage researchers to embrace statistics and econometrics, and see how they can help with their enquiries. Quantitative analysis tries to be evidence-based and objective. Therefore, anyone who believes in the benefits of science will prefer a claim based on quantitative methodology to a hypothesis made, to follow the words of Sanchez-Graells, on the basis of ‘anecdotal impression’.