Code of Practice on Disinformation. A Comparative Analysis: Methodological Limitations

There has been much ado in the press over the last month or so concerning the Twitter platform (Twitter X) and its alleged inadequacy in tackling the phenomena of disinformation/misinformation. As discussed in my previous post Potency of the European Union's Code of Practice on Disinformation the European Commission’s (EC) has put into place the Digital Service Act (DSA) which came into effect on August 25, 2023.

On September 26, 2023, Lisa O’Carroll wrote an article published in The Guardian with the following headline: EU warns Elon Musk after Twitter found to have highest rate of disinformation. Ms. O’Carroll reported:

“The EU has issued a warning to Elon Musk to comply with sweeping new laws on fake news and Russian propaganda, after X – formerly known as Twitter – was found to have the highest ratio of disinformation posts of all large social media platforms.”

The laws Ms. O’Carroll referred to are those enshrined in the DSA. Ms. O’Carroll made reference to a report that underpins the EU’s warning issued to Mr. Musk. This September 2023 Semi-Annual Report was compiled by an organisation called TrustLab. The Title of the report is: Code of Practice on Disinformation hereafter referred to as the Code.

Ms. O’Carroll related that the TrustLab report goes about “laying bare for the first time the scale of fake news on social media across the EU”.

Chris Vallance wrote for the BBC on September 26, 2023 reported, an article entitled Disinformation mostactive on X, formerly known as Twitter, EU says. Concerning TrustLab’s report Mr. Vallance wrote:

“It examined over 6,000 unique social media posts across Facebook, Instagram, LinkedIn, TikTok, X, and YouTube.”

Mr. Vallance quotes the EU’s Values and Transparency Commissioner, Vera Jourova’s warning:

"My message for [X] is: you have to comply with the hard law. We'll be watching what you're doing,"

In the light of these remarks, it is worthwhile having a closer look at the details of the TrustLab Report, something, by the way Ms. O’Carroll and Mr. Vallance and the mainstream media in general, in reporting have not tended to offer, in their commentaries much elaboration on. This lack of expansion can lead to a misleading impression that glosses over the challenges involved in identifying disinformation/misinformation in practice.

Neither Ms. O’Carroll and Mr. Vallance clearly report what the authors of the TrustLab report wrote about the report’s methodology and the caveat’s these authors have regarding their conclusions. This is why it is worth looking at what the TrustLab in their report say about their methodology.

The key points made in the opening remarks of the TrustLab Report’s Executive Summary relate to the “ambiguous and fast-changing” nature of “online disinformation” making the measurement of this phenomenon “challenging”.  The authors of the report also mention that their Report forms the “the first empirical application of the Code”. The purpose of this investigation was to “evaluate the prevalence and sources of disinformation across six major social media platforms” operating in Poland, Slovakia and Spain. The platforms covered by the study being Facebook, Instagram, LinkedIn, X (formally Twitter) and YouTube.

The authors, in the Executive Summary outline the “key metrics” examined by the Report. These are:

“…. discoverability, relative post engagement, absolute post engagement, and properties about disinformation actors, including ratio of disinformation actors, their account activities, and engagement with other users.”

In the concluding remarks of their Executive Summary the TrustLab authors write that the “study establishes an initial benchmark for the implementation of the Code” that opens up the opportunity “for further discussion and advances in the measurement of disinformation.” In saying this the authors acknowledge that the measurement of disinformation is challenging “and accordingly our metrics and methodology can, and should be, improved.”

The TrustLab authors in one of the concluding sections of their report entitled Limitations and Implications layout cautionary remarks concerning the methodology and data collection approach.

In terms of the data collection underpinning the report the TrustLab authors admit that when interpreting the data any conclusion drawn comes with “…. the caveat that budget and time constraints have led to small sample sizes and imprecise estimates for some metrics.”

Another limitation highlighted, by the TrustLab authors relates to the “…. lack of direct access to platform data in the current study [which] limits its capacity to measure broader dimensions of online disinformation.”

A further limitation that the TrustLab authors mention is that “…. the manual labelling of mis/disinformation content and disinformation actors may be subject to human error.”  In saying this they acknowledge that even with sufficient “pre-training” and “ongoing feedback” in the data collection process “… a small number of labelling errors are possible.”  They hope to further reduce such errors in future projects.

Another limiting factor, according to the TrustLab authors is the fact that “… platform heterogeneity can still present challenges.”  They elaborate on this point as follows:

“Not all types of engagement are accessible across all platforms, and even the same type of engagement may carry different meanings and implications depending on the specific content being engaged with and the functionality and established norms of the platform.”

As well as these limitations the TrustLab authors point to how their methodology can be improved through more access to platform data coupled with more time and budgetary resources. They point out that more resources would enable them to augment their methodology with a “feed-based approach”, a tool that was not used in their study due to budgetary constraints; given the resources such an approach could be included in future studies.

Finally, the TrustLab authors describe a limitation of their study revolving around trend analysis:

“The current study provided three biweekly measurements of disinformation, which does not inform meaningful time trends.”

They point out that for future analyses “Longitudinal analysis that tracks the same metrics over a long period of time can yield more robust platform measures and better capture time trends.”

The TrustLab authors to their credit disclose that:

“… ‘foreign interference in the information space’ is outside the scope of the current study. Better access to platform data and alignment on operational definitions can enable future measurements to address this limitation.”

The TrustLab authors admit that the science behind their study is still in its infancy requiring further discussion and refinement. So, we can see that the caution with which the TrustLab authors approach their work is not reflected in the mass media’s commentary, including Ms. O’Carroll’s in the Guardian and Mr. Vallance’s in the BBC which give their readers the impression, without mentioning any of the caveats that the science behind the measurement of online disinformation is based on well-established practices, rather than an evolving methodology and data collection processes.

So given the fact that the TrustLab report methodology and practice is still in its early stages misleading virtue signalling, sensationalism and finger pointing of online platforms such as Twitter X by news media needs to be avoided lest we lose sight of the science and allow politics to drive the public’s understanding of what constitutes disinformation/misinformation and what does not.

The Fundamental point, one that needs to be addressed first and before any discussion of methodology or data collection approaches that is not really discussed, by the press in any great depth is what is it that constitutes misinformation or disinformation? The assumption, in the mass media seems to be that these concepts are well defined and beyond doubt. As mentioned above this assumption is not shared by the TrustLab authors who see that our understanding of these terms is still evolving. What these terms mean is still open to debate.  The TrustLab authors accept the EU’s definitions, under the Code of Practice. They write that the basis of their definitions is:

“…. based on an amalgamation of peer-reviewed studies that TrustLab considers to be broadly aligned with industry standards that were then adopted for TrustLab’s policies. These definitions are aligned with the European Union’s 2022 Code of Practice on Disinformation.”

They go out of their way to stress “…. that consistency of methodology and metrics will further improve as the discussions among stakeholders about detailed definitions of misinformation, disinformation, engagement, and other factors continue.” 

One cannot stress enough how important it is that the terms: misinformation, disinformation and Mis/Disinformation be defined precisely because there is much at stake. We need to ensure that in a quest to rid ourselves of misinformation and disinformation we don’t impact on the free flow of ideas. The fear of being fined would likely force online platforms such as Twitter X and the others to erroneously, on the basis of imprecise definitions and political pressure censor content.  The risk in this situation is that the overzealous curtailing of news and opinions on the internet could very well impede the flow of the truth.

The Australian Human Rights Commission in a submission about the Australian Federal Government’s proposed Communications Legislation Amendment (Combating Misinformation and Disinformation) Bill has concerns regarding this draft bill, which, I contend can be extended to all attempts to define misinformation expressed in the following warning:

“Drawing a clear line between truth and falsehood is not always simple, and there may be legitimate differences in opinion as to how content should be characterised. The broad definitions used here risk enabling unpopular or controversial opinions or beliefs to be subjectively labelled as misinformation or disinformation, and censored as a result.”