Back in December I was quite critical of a paper that correlated IAS, vessels characteristics and labour abuses. Of course, my criticism was based on my understanding and experience and the fact that did din not seem to have ground-truthed their assumptions and indicators with fisherman, as they could have immediately picked up the issues I picked up.… but then I’m just an ex fisherman and a couple of masters, while the authors are all PhDs and academics… so my criticism was just on this blog.
Not bad for vessels with a big engine
Now interestingly a letter to the authors was sent by some heavyweights scientists and published over a month ago. The authors say is way more elegant English (and without needing too many details) things around the line of what I wrote, further adding that it fails a “proof of concept”
Not that it really matters at the end, but it made me feel well. Maybe one day I’ll give PhD a go.
In the meantime, I paste the letter below, original here.
McDonald et al. argue that labor conditions in fisheries can be discerned from the movement and characteristics of fishing vessels. We recognize the authors’ effort, yet have strong reservations regarding their 1) limited dataset, 2) assumptions, and 3) model validation. Forced labor is a serious human rights violation, and any scientific claims potentially informing policy must be considered with particular care to best promote efforts toward ending human terror and supporting survivors. We, therefore, urge that the authors consider how their efforts may misinform policy.
The authors create profiles from vessels reported to have committed labor abuses, search for similar profiles in a database of 16,000 vessels with available positioning (automatic identification system [AIS]) and associated data, and conclude that up to 4,200 vessels in “global” fishing fleets are “high risk” for labor abuse. First, of 193 labor abuse vessels identified by McDonald et al., only 58 used AIS and just 21 yield AIS profiles suitable for their analysis, implying that nearly 90% of their known cases are undetectable by AIS-based profiling as proposed.
Second, the 27 features used in their profiles are not causally linked to labor conditions onboard. The four features identified as most predictive of labor abuse “risk” (engine power, maximum distance from port, yearly voyages, average daily fishing hours), describe large distant-water vessels and are therefore likely correlated. Third, there is no testing of vessels with similar profiles that have not engaged in labor abuse. Existing evaluation methods for positive-undefined learning systems require some positive/negative-class knowledge of unlabeled cases in the training or test datasets, and their performance given unbalanced data (with many more “undefined” than “positive” cases) is questionable (2⇓–4). The authors fail to accomplish the former and display problems of the latter. The use of the 21 positive cases to both train and validate the model violates the golden rule of machine learning in that the test data cannot influence the training phase. Additionally, risk as presented is not likelihood or probability of labor abuse and statistical findings are not ground-truthed; thus, the data offer no evidence that forced labor is more prevalent in vessels identified as high risk.
Given the suggestion that their model provides “new opportunities for unique market, enforcement, and policy interventions,” we believe the authors must address the potential ethical implications of their approach. Statistical profiling risks being reductive, especially given limited understanding of the modeled system, and there are real concerns over its use leading to unfair decisions or discriminatory practices (5⇓–7). McDonald et al. emphasize that model limitations may underestimate high-risk cases; given their limited sample and lack of model validation on data not used for training, even this statement remains speculative. We appreciate that their model is exploratory, but it remains largely untested and these crucial limitations should have been at the fore of the analysis. In conclusion, McDonald et al. falls short as a “proof of concept,” and therefore, the capacity of remote sensing as a tool for detecting fishing vessels engaged in labor abuses remains unsubstantiated.
And to be correct, I also acknowledge the response to the above letter by the authors of the original here. Yet for me personally their answer does not really address the obvious problems with the paper