YT YouTube2Text

YouTube2Text → Transcripts → Inside Paper Mills: Detection, Evidence, and What Editors Should Look For | ACSE Webinar Series 2026

Inside Paper Mills: Detection, Evidence, and What Editors Should Look For | ACSE Webinar Series 2026 — Transcript

by The ACSE · 7,831 words · 1,216 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:04I would like also to take a moment to
0:08mention that AS ACE
0:12will be holding the 12th annual
0:15conference in Dubai. So we warmly invite
0:19you as a to be a scholarly publishing
0:23community to join us for this important
0:27gathering of editors, publishers and
0:31professionals dedicated to advancing
0:35integrity in academic publishing. So we
0:39are delighted to have all of you with us
0:43today and we hope this session will
0:47provide valuable insights and practical
0:50guidance. So without further delay, it
0:54is now my pleasure also to introduce our
0:58facilitator for today's webinar. So it
1:02is now uh my pleasure to introduce him
1:06is Dr. uh Adam Daily, the CEO and
1:11founder of Clear Skies, the company
1:16behind the paper mill alarm, one of the
1:20publishing industry, a leading h service
1:25dedicated to detecting paper mill
1:28activity.
1:30So launched in 2021,
1:34paper mills Adam was deferred service
1:38spec specifically designed to identify
1:43organized paper mill operations and is
1:47now integrated with STM integrity hub
1:52and under Adam's leadership clear uh
1:56skies received the 2024 for ALPS
2:02award for introduction in publishing and
2:06its work uh has been featured in nature
2:11sciences and the wall street journal. So
2:16Adam brings more than 15 years
2:20experience of in editorial practice and
2:23data science having previously worked
2:26with Sage Publishing. It is a well-known
2:29also publisher uh on large scale uh
2:33misconduct detection, peer-reviewed
2:36innovation and high impact editorial in
2:40um initiatives. So we are very pleased
2:43to have him here with us today and share
2:47his insight and how editors can better
2:51recognize the respond and respond to
2:54paper mail activity. Uh, Mr. Adam, the
2:58the floor is yours.
3:04>> Well, thank you very much for the
3:05introduction. Um, much appreciated and
3:08uh, thank you so much for having me here
3:10to speak. Um, let me just share my
3:13screen.
3:15>> And also, can you uh, the voice a little
3:18bit? It's very low.
3:19>> Oh, am I am I too quiet?
3:22>> Yeah.
3:24Um, I do something about that. How about
3:27now? Is that better?
3:29>> Yeah, it's it's okay. Thank you.
3:32>> Okay.
3:37Now, can we see this? Okay, here we go.
3:41[snorts] H. So, I thought one of the
3:45first things to
3:48talk about would have been the the
3:50actual definition of what we mean when
3:52we say paper mill because there are
3:56large organizations that will put out um
4:01problematic content at scale. We'll
4:03often call that a paper mill. But
4:06actually there's a lot of different
4:08things that we see that are problematic
4:10and often get called paper mills. So for
4:12example
4:14um an individual or a research group
4:16might put out uh problematic content in
4:19a similar sort of way and we could call
4:22that a paper mill but it's quite a
4:23different thing.
4:25Then there's a lot of the same practices
4:27that we see with paper mills that
4:30actually we'll see in other places as
4:32well and those could come from
4:33individual authors.
4:36So it's important that we sort of break
4:37down different things that we might look
4:39for and uh
4:43consider them perhaps individually not
4:46necessarily assigning all of them to
4:48paper mills.
4:50So
4:54the
4:56other thing that I think is really
4:57important to think about here, so we're
4:58mostly working in um the space of doing
5:02analytics that will uh identify
5:04anomalies. So we're saying there's lots
5:07of good papers, they're green, and then
5:08there's problematic ones which are red
5:10and they're very rare by comparison.
5:13So the way that um
5:17oversight works is that we'll often
5:19think in terms of
5:22screening and detection. And this is
5:26very different to investigation, which
5:30is actually what we're often um
5:33uh doing when when we're doing research
5:35integrity. We're doing a lot of
5:36investigation where we're going into a
5:37lot of depth, spending a lot of time on
5:39articles. And really the purpose of
5:43um the services that we provide and uh I
5:47think a lot of what editors do in this
5:50space the purpose is to try and avoid
5:52investigation as much as possible is to
5:54avoid having to spend that huge amount
5:55of time on something where a decision
5:58can be made much earlier much further
6:00upstream.
6:02So we have two sides to this. One side
6:04is we look at the publications. So we
6:07actually have every publication, every
6:09article that has been published in the
6:11last decade has been analyzed and um
6:14loaded into oversight.
6:16This means that we've got a clear
6:18picture of the whole um
6:22uh the whole thing the whole research
6:24research ecosystem.
6:27And in
6:31this case, we can see long-term trends
6:34over time in in what's happened there.
6:37With um
6:41this publications view, we can then go
6:43into different levels and see different
6:45things. Uh for example, we have every
6:47institution in there, every um funer
6:51that appears as an institution in the
6:53data. We have every publisher. This is a
6:56publishing house. And we can look at the
6:57data in different ways and explore how
7:00trends have changed over time. Then we
7:03can go to the journal level and um at
7:05the journal level we can say okay this
7:08is uh a journal that's very clearly been
7:11targeted
7:13and it's got a huge spike in the data
7:15which tells us okay this is one that we
7:17want to prioritize. Actually this one uh
7:20publisher did um deal with this problem
7:23very decisively and was able to bring
7:25that um alert rate down.
7:29Then if we go on to the articles
7:33themselves then we can go into a huge
7:35amount of detail. Now usually the kinds
7:38of analysis that we're doing here we're
7:40not going to be looking at the um a
7:44published article. So seeing the
7:45publications can give us a good overview
7:48of which artic which journals are most
7:50at risk which journals do we need to
7:54>> sorry is there a question
7:59>> excuse me can you mute yourself please
8:02can you mute yourself
8:07>> yes go ahead
8:08>> because it's some people that
8:11>> so so on the publication site we can see
8:13any article we can see any um uh any
8:18journal we can understand this uh the
8:20problem at a a high level. So it gives
8:22us oversight as uh that's why we called
8:25it that um gives us oversight of the
8:27whole problem. But when we want to get
8:29into uh screening and we want to
8:32identify what are the certain things
8:34that we should be looking for then we're
8:36going to get down to the article level.
8:37This is usually going to happen with
8:39submissions rather than with
8:41publications. We're going to be seeing
8:42this at the moment. A article is
8:44submitted to a journal. That's where
8:45we're going to pick this up. So, here um
8:50we have a report and this shows the
8:52findings part of our report. So, it's
8:55saying that um
8:58uh we have several different things that
9:01we're picking up here. We have um let's
9:05see
9:08templates. So particular templates get
9:11used by um paper mills and what it's
9:15saying is that there's a number of these
9:17in the reference list. So it's picked
9:19them up there. Um and then it's saying
9:23um
9:26that we have an external data source.
9:29This is the problematic paper screener
9:30where we're also finding three
9:33references uh which have been picked up
9:35by that source. This is quite important
9:37because one thing that we're very keen
9:39on is that we have different methods
9:41that will agree on an article and where
9:45there's agreement then we um we we know
9:47that there's let's say one method has
9:50some imperfections we can be quite sure
9:52that because we have two methods working
9:56in different ways finding the same
9:57article that we can expect to have a
9:59very low false positive rate on this
10:01kind of a a prediction.
10:03We also have some problems, some some
10:05concerns about the self- citation rate.
10:07Self- citation is quite an interesting
10:09thing and we'll we'll come back to that.
10:11But this gives us a bit of a an overview
10:14of what those issue might be. We can
10:16then click on these um uh these links
10:19here for for some more information about
10:22exactly what we're we're finding there.
10:26Then we have this uh network view. Um so
10:30this is a little bit low res. We
10:31probably got to um probably give you a
10:35better image would have been nice. But
10:36what this is telling us is it's also
10:38giving us a a highle overview of the
10:41article. So in this case we have an
10:42article with a set of references that we
10:45are concerned about. Um one concerning
10:49reference might not be too big of a
10:52problem. In this case we have nine. So
10:54we're very concerned. Um on top of that
10:56these dotted lines tell us that there
10:58are shared authorships amongst these
10:59articles. This uses orchid data to work
11:02out that these have uh these articles
11:04have shared authorships.
11:06So that tells us that what we have here
11:08is a group of authors who have um a
11:12history of
11:14um creating problematic articles and of
11:18citing those articles as well. So this
11:20gives us a very quick overview of an
11:23article. Now at this point I think an
11:26experienced user would look at this and
11:28say okay this this is enough information
11:30to make a decision. Um if this comes
11:32into desk review
11:34some editors would I think be quite
11:37comfortable to simply reject the paper
11:38at this point. But we want to have a lot
11:42of transparent detail here which means
11:43that we can go into more depth and say
11:46exactly why is this article being
11:48flagged.
11:50So
11:54we can say a bit about that template
11:56recognition bit. Um
11:59I want to be very clear that almost
12:01every article in the world is written
12:04with some sort of a template. Templating
12:06is fine. Um
12:11it's also very important um particularly
12:13in cases where for example um English
12:16isn't someone's first language then I
12:18think templating is really critically
12:20valuable to be able to put things into a
12:22rigorous form
12:25where we're picking this up. We're
12:28looking at templates that have um two
12:32main features. One is that there's a
12:34template that is um has a very high rate
12:37of production. A rate of production that
12:39is not consistent with real science.
12:42Real science is slow.
12:45And then we have a template that has a
12:46high rate of retractions or other clear
12:50verifiable flags. Those are the
12:52templates that we're flagging. And it
12:54happens that um
12:58flagging the template itself has turned
13:00out to be um uh very effective and very
13:04powerful at identifying paper mills.
13:07It's slightly different to saying that
13:09we're necessarily flagging the um the
13:12retractions or the the other things
13:14we're finding is really the template
13:15that we're looking for, but it's a very
13:17particular kind of template that meets
13:19certain criteria.
13:21Um and this led to us having the uh best
13:24estimate for the scale of paper milling
13:26globally. Actually this is a couple
13:28years old now and we uh can probably put
13:30out a better estimate at this point. But
13:32this [snorts]
13:34has proven to be a very scalable way to
13:36to find paper mills. But again we want
13:38to go into more depth. Uh template on
13:41its own might not be enough to say this
13:42is definitely or definitely not a paper
13:44mill.
13:46So one other thing that I did, this is a
13:49very very um long time ago uh 2019 I
13:54think I built this tool which was
13:56intended for tracking rejected articles.
13:59And the idea there was that we would
14:03um
14:05put in a we search for a title and we
14:08search for the published a published
14:09article. Um
14:14and
14:17so so we search for this title of
14:19published uh article sorry search for a
14:22title of a um submission
14:25but then if we find that it's been
14:27published then we can say okay we can
14:29track this rejected article and see
14:30where it got published after it was
14:32rejected. So that's a useful piece of
14:33analytics for a publisher.
14:36um that piece of analysis turned out to
14:38be a really good way to find the DOI for
14:41a reference. So we have incomplete uh we
14:44have some reference data. We have maybe
14:45just the title and we have the uh names
14:47of the authors. We can use the same tool
14:49to match that up and work out um
14:54uh which reference exactly is this in a
14:58reference list.
14:59So that became a really um
15:03well-developed tool but really quite
15:05advanced for the purpose of tracking
15:07rejected articles. But then we started
15:08using it for um finding references and
15:11it turned out it worked really well for
15:12that. What happened in recent years is
15:16we started to see more of these cases
15:18where the
15:21DOI
15:22um that was there
15:25couldn't be
15:27converted into uh so so one thing was
15:30that we wouldn't be able to find the DOI
15:31because the reference didn't exist. The
15:33other case was where the DOI didn't
15:36resolve. So you click on that DOI and it
15:38won't take you to the article because it
15:40doesn't exist.
15:42So we ended up with this tool that we
15:44had built to track rejected articles and
15:46it turned out that accidentally it was
15:48detecting these um hallucinated
15:50references which come up quite a lot. So
15:52they're very easy to find in oversight.
15:55We just go and look at this list and we
15:56can see these articles that sorry these
15:58deals that do not resolve.
16:01Um
16:03there's another really nice feature that
16:04this has which is that it helps us to
16:06find duplicate submissions and again
16:08that's another feature of uh paper mills
16:10but that's something we'll do usually um
16:12after uh an article has been decisioned.
16:18So
16:20what other features would we look for?
16:21Well if we're talking about fabrication
16:23we're talking about papers that have
16:24been put together um with the intention
16:27of making them look like a real paper.
16:30There's a couple of things that go in
16:31there. One is that we see paper mills
16:34selling uh citations.
16:38So, we know that they have a job to do
16:40there, which is that they have to get
16:42these citations somehow into the papers
16:44that they are creating and publishing.
16:46Secondly, we know they're using
16:47templates. So, it's got to be someone's
16:49job to put references into that
16:50template. that might be not be done with
16:52as much care as getting a reference into
16:56um
16:58uh um
17:01a real paper. So so a real author with a
17:04real reference they they they've chosen
17:06that quite carefully.
17:09So what we have is this tool which we um
17:13ran over uh
17:16a huge data set which we trained and it
17:22will recognize when a reference does not
17:26belong when it's statistically
17:28um
17:30unlikely for it to appear in a reference
17:32list. This is not necessarily by the way
17:34a sign of misconduct. Sometimes a
17:36reference can be um uh very unusual for
17:40all sorts of reasons. But what we see
17:42with paper mills, and actually this
17:43isn't the best example because there's
17:44only one example here, but what we see
17:46with paper mills is sometimes huge
17:48numbers of these irrelevant references
17:51in a reference list. And that again is
17:53very easy to spot with this tool. Um,
17:56and it's what it's saying when it picks
17:58something up is that over
18:00um a training data set of uh at least 10
18:05million examples, this is unusual. This
18:08is a really strange thing to happen.
18:11So then we have um author histories. So
18:15where we have um any submission we have
18:19these different lists. We have a list of
18:20author uh articles in an author's
18:22history which is built with Orchid and
18:24we have a list of um references which is
18:27the articles in the reference list. So
18:28we've already seen quite a lot comes
18:30from that reference list. But the author
18:32history is also really important and
18:34actually it's the first place I would
18:35start with any kind of um an
18:37investigation or or even just a
18:38screening check.
18:41So one thing that is quite strange and
18:44it seems to be something that comes up a
18:45lot more when we look at um articles
18:47that are produced with generative AI
18:50is that we will see these lists of
18:53articles in an author's history issue
18:54that are are very low relevance. So
18:58in this case we won't flag them
19:00explicitly. Um but we can see from these
19:03numbers when the number goes uh negative
19:06that's when we have very low relevance
19:09and
19:12this is something we see increasingly
19:15it's high rates of production from
19:17particular authors and also these uh
19:20high numbers of these low relevance
19:22articles in the history and like it does
19:24make some sense where if someone is
19:28producing work with genai and it's not
19:30really something that they have studied
19:32or spent a lot of time on. You can see a
19:34lot more topic drift there where they
19:36can um produce an article on anything
19:39because they are um
19:42uh
19:45they could produce an article on
19:46anything because they they're not
19:47actually doing a a piece of research
19:50there. They're they're simply asking
19:52chat GPT to produce the work for them.
19:55Um,
19:58one thing we've spoken a bit about,
19:59we've spoken a lot about um, machine
20:02learning, talked about training data and
20:04having large data sets that we use to
20:05work out when something is unusual and
20:07when it isn't. Um, and again that goes
20:09back to what we said at the start about
20:11anomaly detection. It's about we're
20:13finding things that are um, not normal
20:16in in the data.
20:18So,
20:21one other thing that we do is network
20:23analysis and that's quite different. So
20:25rather than training on a big load of
20:27data, what we're doing is we're taking
20:28um a lot of data and we're saying how
20:30does it all connect to each other and
20:33this kind of data is perfect for that
20:35because of course we have um citations
20:37there's connections between people. We
20:39have co-authorships those are
20:40connections between people. Um [snorts]
20:44what we've done with that is we've said
20:45well let's look at all these connections
20:47to all these people um and all these uh
20:51um identifiers that we have to identify
20:53individuals like like orchids and then
20:55we say which of these orchids are most
20:58connected to where we can find these
21:01kinds of problems.
21:04In doing that we find again anomalies.
21:07you find occasional individuals who are
21:09highly connected to these kinds of
21:11problems. Again, we wouldn't even go as
21:14far as to say that because somebody has
21:16a high degree of connection to
21:18problematic articles that necessarily
21:20that person is problematic. There could
21:22be other reasons for that. Um, but in
21:25terms of
21:27where it's very useful to um to flag
21:30that to an editor, what we're saying
21:32with the keystone analysis is we're
21:35saying this particular article, it's
21:38really well worth looking through the
21:39author history. And um if for nothing
21:42else than just to uh confirm that
21:44there's there is nothing wrong with this
21:46individual. We're not saying that
21:47somebody has done something wrong. Been
21:48very clear about that.
21:50So
21:52um
21:54network analysis is something that goes
21:56goes through what we do um
22:01to a great extent. Um the
22:05self- citation analysis that's network
22:08analysis
22:10um and I think it's really quite
22:12fundamental that ultimately what we're
22:14looking at here is connections between
22:15things and particularly between
22:16individuals.
22:22Um yes so I I think I mentioned self-
22:25citation
22:27already but there are a couple of things
22:29that we have here. So again it's anomaly
22:31detection. So what we do is we take the
22:33last year of citation behavior um using
22:37data from open Alex and then we say what
22:41self-citation behavior is unusual
22:44and there are some quite interesting
22:45things there one being that uh sometimes
22:47somebody will just put a ton of
22:48citations themselves in their own paper
22:51that's actually quite rare though um we
22:53do see it but um
22:57a lot of the time that's not something
22:58that we see and with paper mills we
23:00don't really see that at
23:02What we do see though are very high
23:05rates of uh citation to individuals
23:06across a large number of articles. What
23:10we also see is um
23:13some things that are a bit more subtle
23:15um like total citation benefit. So let's
23:18imagine that there are 10 authors on an
23:20article and they site one other article.
23:24That same article has all 10 authors on
23:26it. The citation benefit from that one
23:29citation is 10 because there are uh that
23:3410 increments for those individuals.
23:38They all get something.
23:41When you add that up across some
23:44articles, it becomes very interesting
23:46because you start to see that where you
23:47have large author lists that total
23:50benefit from one article can be really
23:52high. Um and
23:56that's one thing that we will measure
23:58[snorts] and we will say well when is
23:59that um anomalous? When is that
24:03statistically anomalous? When is that in
24:04say the the 99th percentile a one in a
24:07100 paper? When is it in the 99.9th
24:10percentile? A one in a thousand paper.
24:12And that's what we really want to see
24:14where something is unusual.
24:16Another feature which we're really
24:17careful about is where we see very low
24:21rates of cell citation. So often with
24:25paper mills if you can imagine
24:28somebody's buying a paper well the
24:30reference list might already exist when
24:31they buy it. So self- citation is quite
24:34normal to some extent to some small
24:36extent.
24:38So zero cell citations is strange. It's
24:41even stranger when there's a large
24:42author list and it's even stranger when
24:44that author list has a lot of history
24:47that could be cited by that reference
24:49list.
24:51We are very careful about this though
24:53because one of the things that we've
24:55seen is that um paper and mail behavior
24:58often is found in particular
25:00geographies.
25:02Those
25:03areas of the world tend to be places
25:05where we have rapidly growing scientific
25:07communities, which means we have a lot
25:09of firsttime authors, early career
25:11researchers. And where we see zero self-
25:14citations, obviously anyone writing
25:16their first ever paper cannot self-site
25:19because they have no papers to
25:21self-site. So, we're very careful to
25:23control for that. And so we're only
25:25going to flag self uh missing self-
25:27citation when it is statistically
25:31um unlikely that um
25:35uh
25:37authors with a particular history would
25:39not self-sight. So the history has got
25:42to be there. But we do find that we find
25:45these cases where we have um authors who
25:48have long histories
25:50um and a large number of them all
25:53appearing on a paper where they do not
25:54site that history and and that is very
25:56strange. That's a very strange thing to
25:58see. Um and it can be a symptom of uh
26:02paper milling. Um and so it's something
26:04that we will point out when we see it.
26:08So
26:11taking a bit of a step back um we
26:15spoke a bit about networks um
26:18there's a blog post about this which um
26:23uh you can take a look at
26:27but one of the things that
26:29if we if we take a step back and we look
26:32at the whole industry
26:34we take those networks and we say Okay,
26:38we could treat a citation a bit like a a
26:42vote. It's like a vote saying that this
26:44paper is really good.
26:48Um, and that's kind of the way that we
26:49interpret citation in um in in the
26:54research literature usually.
26:57But what we can do is we can say, well,
26:58let's imagine that like let's imagine
27:00I'm a I'm a fake paper and I know I'm a
27:03fake paper and someone cites me.
27:08Well, then I say, well, there's
27:09something wrong with that because I know
27:11I'm a fake paper and I know that you
27:12shouldn't have cited me because I'm I'm
27:14not real. Um, so then you can make a
27:18vote and say, I'm voting that there's
27:20something wrong with this paper.
27:23And
27:24what happens is that if you take that um
27:27and make an algorithm out of it, you
27:28say, "Let's make all of the papers vote
27:30that way." So we have all these
27:31problematic papers. We make them all
27:32vote for the um articles that site them.
27:37The
27:40interesting thing about that is that
27:43it's extremely resistant to noise. So if
27:46somebody accidentally cites a
27:48problematic paper, that that doesn't
27:50matter. That's okay. that's not going to
27:53give them a lot of votes.
27:55But where there's a pattern of citing
27:57these problematic articles, that adds up
28:00and it all adds up to right now. And it
28:02says you have all these votes in the
28:04past and they all vote towards all the
28:06things that are citing that literature
28:08now. And that could be um articles that
28:12are from paper mills that are part of an
28:14organized pattern of citation. It could
28:16also be articles that are simply citing
28:19the wrong stuff. um that that could
28:22cause the same thing where we add that
28:24up at the journal level. It's incredibly
28:26powerful because what it does is it
28:28shows us where we have nent risk. Um and
28:32that's something that is a a feature of
28:34um
28:36uh oversight that it will show where
28:38that nent risk is in a journal. And in
28:41some cases, we've been able to catch
28:43journals very early on shortly after
28:47they've been targeted by a paper mill.
28:51We also use GI detection. One of the
28:56things that when this started was um
29:01when people started writing articles
29:02with GI,
29:05one of the things that we
29:12were often asked was, you know, when you
29:14see a Geni article, you obviously that's
29:17just that's just flatout fraud and that
29:19has to get rejected. But the reality is
29:22that it's much more nuanced than that.
29:24Geni is a really useful tool for writing
29:27and this particular piece of analysis
29:30it's based on quite a small data set. So
29:32I would say that you know there's
29:34potentially more work to do to get this
29:36um uh as accurate as possible. What it's
29:39showing though is that the false
29:41positive rate is extremely low on this
29:44method of finding um uh generative AI.
29:49the
29:51um
29:53this is saying we're looking at papers
29:54from 2010 and it's saying it hardly
29:56detected anything. So this is this is
29:58tiny false positive rate. Then we look
30:01at 2025 and this is just a random sample
30:03of 2025.
30:05These are abstracts and 35% of them were
30:09picked up. So we know the false positive
30:11rate is low.
30:1335% were picked up in 2025.
30:16So the rate at which Gen AI is used is
30:19huge. Um I think it's reasonable based
30:22on that to say at least a third of the
30:23literature is in some way using
30:26uh Gen AI. And I think for something
30:29like making an abstract completely fine.
30:31Um that seems like a perfectly sensible
30:34use for Gen AI.
30:38where we're interested is where is the
30:40use of genai
30:42excessive or where does it seem that it
30:44might um undermine the paper and
30:47actually knowing simply that something
30:49is geni or isn't I don't think is enough
30:52I think it's valuable context it will
30:54tell us maybe this is something that we
30:56should look at look at with more care um
30:59if you recall I said that when we said
31:00when we see geni articles we often see
31:03that irrelevant author history and again
31:07that that is strange in context. Um
31:11that author history is is very strange
31:13and very interesting when we can see
31:15that something is uh using genai.
31:20So genai um for a long time I think
31:24people thought that we were never going
31:25to have a good detection method for it
31:27but we do very good detection method for
31:30it. Um the reality is though that geni
31:34on its own is not problematic. It has to
31:37be in context.
31:41So
31:43another thing here again I talked about
31:45having orthogonality having uh different
31:48methods that find problems with the same
31:51thing. Having sleuths I think is
31:55incredibly valuable. We have um a
31:57community of individuals who are highly
31:59talented, highly motivated and do a lot
32:01of work um in their own time on finding
32:06um problematic research.
32:10And often the sleuths will uh donate
32:12data to us that we can use to flag
32:15articles and we will always give them um
32:18appropriate attribution depending on
32:19what they have asked for. Um,
32:23and occasionally we'll do receive lists
32:26that are we aren't asked for
32:28attribution. Uh, we're asked to keep it
32:32um private, but we we will also respect
32:34that.
32:36But again, this gives us another string
32:38to our bow. It's another um uh way of
32:42looking at the data and another thing
32:43that can corroborate what we're finding
32:45with automated methods.
32:49Um,
32:52we'll also flag where we see retractions
32:55in a reference list. Actually, one thing
32:57we'll do at the moment is also flag
32:58corenda. Um, we're going to separate
33:01that out because I think it could be a
33:02bit more clear.
33:04But one really important thing with
33:06corenda is that with paper mills the
33:08rate of uh coragenda is much higher than
33:10the rate of retraction.
33:12So they can provide some really useful
33:14um insight in that we could see let's
33:16say an article that is uh problematic
33:21for particular reasons and then we find
33:25corenda on it a corendum on it and we
33:28can go and look at that corendum and it
33:30might say something like um there was a
33:32problem with one of the images so we've
33:33changed the image and that on its own it
33:36seems perfectly reasonable of course a
33:37publisher would change the image if the
33:39author said there was a problem with it
33:41but then when we get to um
33:45uh
33:49looking at say pub here, we might find
33:52that there was a comment on that image
33:54and that that possibly was the cause of
33:57um the author asking to change it. So
34:00that the story can sort of come together
34:02around that corendum. So it can be quite
34:04a useful thing to have.
34:07Um
34:10let me just have sorry targeted journal
34:12again sorry duplicated slide. Um so yeah
34:15that's essentially how oversight works
34:18and how submission screening works.
34:20There's a lot there for
34:22editors to be mindful of in that there
34:25are these patterns. The patterns do
34:27change over time.
34:30Um,
34:32and partly the value is in being able to
34:38monitor the submissions, look at these
34:40changing patterns and know what to look
34:42for in these reports. but also being
34:44able to take a big step back and look at
34:46the higher level patterns and say um
34:51what are the
34:54uh what are the trends which journals
34:57perhaps are being targeted which um
35:00subject areas which authors appear to be
35:02doing the targeting and
35:06being able to make those kind of
35:07strategic judgments is a really big part
35:10I think of um what what to look for It's
35:13not just what to look for, it's also
35:15knowing where to look.
35:18So, um, any questions?
35:25>> Thank you so much, Mr. Adam, for this
35:28insightful and uh, highly relevant and
35:32uh, and informative presentation.
35:36So before opening the floor for
35:39questions from the participants, I would
35:42like also to begin with few discussion
35:44points from my side. So uh
35:49what are some of the most reliable
35:52indicators that that the manuscript may
35:56be linked to paper mill? How we can know
36:00that? What are the indicators for that?
36:05So I I would say anything that
36:10um represents
36:13that sort of industrial production and
36:18things which show fabrication, things
36:20which show um the reference list has
36:23been put together in a particular way.
36:25Um, I think those are particularly
36:26important because if we're saying we've
36:29got a bunch of articles that we think
36:30are um mil that come from paper mills
36:36and then we're saying, okay, this
36:37article looks like it's been fabricated
36:39and that could mean that the author
36:40fabricated it, but it also has a huge
36:42number of citations to those uh articles
36:46that we flagged in the past. Well, that
36:48shows a long-term pattern that shows
36:50organization
36:51and that suggests that uh this is from
36:55an organized paper mill and not just an
36:57individual who is putting things
36:59together.
37:00>> Mhm. Okay. The other question for you,
37:03if I am an editor uh in a journal, how I
37:07can differentiate between poor quality
37:10submission and organized paper mill
37:14content? how I can differentiate between
37:17them.
37:18>> So, I think a lot of what we're showing
37:20in the app does show uh where to draw
37:24that line. Um
37:26almost
37:28nothing that we're flagging would be on
37:31its own just a sign of lowquality um
37:34work. It's it's signs of things that are
37:37unusual and and we we have different
37:38levels of things. you have things which
37:40are
37:42very high certainty um low false
37:45positive rate um you can hang your hat
37:47on this you know um sort of things but
37:50we also have some things which are just
37:51there for context like for example the
37:53um relevance in the author history so um
37:57an author might switch fields that would
37:59be a reason why you would see irrelevant
38:00content in their history which is
38:02completely fine um
38:06but I think that's it what we're looking
38:07at here is we're we're saying most of
38:09what we're looking at here is things
38:11that are
38:13um a sign of
38:17uh
38:20things being very much not as they
38:22should be in any research paper
38:24irrespective of quality.
38:26>> Yes. Yeah. My last question from my side
38:29you what role because you mentioned the
38:32machine learning also as well
38:34>> so and the [clears throat] AI and the
38:36technology. So what role can technology
38:39and data analysis play in strengthening
38:42the detection effort across journals and
38:45publishers?
38:48Um,
38:51I I think it's really important, but
38:52it's also something that's a continually
38:54developing situation because the
38:59kinds of tools that we were
39:03um as an industry, the kinds of tools
39:05that were around even 5 years ago
39:08compared to now are enormously
39:10different. Um, the priorities of those
39:14tools are also enormously different. Um
39:17there's one thing that I think is quite
39:20is quite interesting about the sort of
39:22we talk about a cat and mouse game in
39:24terms of how
39:26um uh uh fraud detection works which is
39:30that
39:32often you'll uh say come up with um new
39:36ways of finding um misconduct but then
39:39also there will be new ways of making
39:40misconduct. And so you'll always have to
39:42be chasing
39:43>> uh to try and find those new ways and
39:44and find ways to
39:46>> adapt to them. And then the question is
39:48well does
39:49>> does does that mean that all those old
39:51methods that we used used to use even
39:53just a few years ago
39:55>> do we stop using those? Well no because
39:58there's still a new generation of people
39:59coming in who will try the old methods
40:02and and get caught by by those those
40:04older methods. So
40:05>> yeah,
40:07>> the the importance of different methods
40:09will change over time, but also the um
40:13uh
40:16ju sorry just going to the question
40:17about machine learning um I I think
40:20that's always going to be there. It's
40:21always going to be part of how this is
40:23done. And we can also look to other
40:25industries as well like for example
40:26finance cyber security they've been
40:28doing fraud detection a lot longer than
40:31this industry has. And you look at the
40:34solutions they have now very much
40:36focused on um
40:39uh the same kinds of principles machine
40:40learning network data and um
40:45uh building large
40:48data resources which is another part
40:50that we didn't really touch on here but
40:51we have to build huge and quite
40:55well-maintained databases to be able to
40:56do a lot of this stuff.
40:58>> Yeah, thank you so much for all your
41:01answers. Now let's open the floor also
41:04for our participants to ask the
41:06questions and we have in the chat box I
41:08think some questions that need to be
41:11answered. If you look at the uh
41:15question, I have question. How often do
41:19you recalibrate
41:21uh retrain the characteristics of
41:24problems and the quality of uh gene AI
41:28output are changing very rapidly. So
41:32what if some new fraud tech um technique
41:37comes also in and how long before you
41:40feel confident that you would defect
41:44detect?
41:45>> Few few questions there. So um oh also
41:48hello Marie I know Marie well. Um
41:53uh in terms of how often things could be
41:56reccalibrated, retrained as soon as new
41:58data comes in uh the setup we have is
42:01that we can retrain that uh immediately.
42:03Actually the last few months we have
42:05built a complete rebuild from the ground
42:09up of our entire training pipeline. Um,
42:13over that time we haven't put out so
42:15many updates of the older one, but over
42:18that period of time, six months, it's
42:19not really going to make a huge uh a
42:21huge difference to what we pick up. Um,
42:24but some of the newer updates, the way
42:27that those have been um
42:30built, the the the purpose is to be able
42:33to make um much smaller incremental
42:37updates. So where we have actually just
42:39a very small number of examples of uh a
42:42paper mill,
42:44we can train on that much better now
42:45with these new updates. Um so that's one
42:48thing that's quite important is being
42:50able to work with small amounts of data
42:52as well as very large amounts. Um
42:57uh and that sort of goes to the other
42:58question about you know if there's a new
43:00technique something new in the in the
43:02data um
43:04>> this is much more geared towards
43:06handling those changes.
43:09>> Yeah.
43:10>> Yeah. The second question can paper
43:12mills use gen AI to generate
43:17fake papers and how can editors also
43:20detect them?
43:21>> Yeah. So the integration that we have
43:24with panggram I think is um a very good
43:29way to get that information in that we
43:31can say not just has geni been used but
43:34where has it been used and to what
43:36extent so that is very important to have
43:40but as I was stressing in the uh
43:43presentation we see that as context we
43:46don't see that as misconduct
43:48um where
43:51the the line between misconduct and um
43:55uh mistakes is is intent. So, it's
43:58saying, did somebody mean to do
44:00something that they shouldn't have done?
44:02And we're never going to make a judgment
44:04about that. We're never going to
44:06automate a service that um
44:10accuses people of intentional
44:12wrongdoing.
44:14So what we are going to provide is we're
44:16going to say well here is an article
44:18which has a very high rate of geni and
44:20we think that's potentially problematic.
44:22So it's worth checking but actually it's
44:24the other stuff. It's the other context
44:26that really tells us um if that is
44:29problematic. What is it referencing? Um
44:33are there issues with the references
44:35like we saw here where say there's a a
44:38fake reference in there? Well, you know,
44:40if the paper is um a real paper written
44:43with Genai, that's that's fine. But the
44:46if the referencing is is um not reliable
44:49because we can see that I I think that
44:52suggests that um uh other references
44:56there might not be reliable as well. And
44:58if we're seeing referencing into known
45:00mil content and retracted content,
45:03again, that's much more solid. we can
45:07say that that is not something we should
45:09see in a real paper irrespective of
45:12whether it was written with geni or not.
45:14So
45:16I I
45:18I'm a big fan of the
45:23as I say those network methods. It's
45:25looking at how things connect because
45:29genai um controls the content but it
45:33doesn't control the connections. that
45:35that's something that is very much um
45:38>> a human part of this that we can we can
45:41look at and and Jenny has no
45:43>> no no bearing on it.
45:46>> Yeah. Yeah. There's uh other two
45:48questions. The first one also from
45:50Finley Overland. Can oversight be used
45:54as an implemented screening tool in a
45:58journal publishers workflow bare paper
46:01or it's more see uh so an external
46:05consulting tool to tackle trends? What
46:08do you think?
46:09>> It's it's both. Um so we screen
46:13um I think close to 100,000 articles a
46:17month. Um these are new submissions and
46:19they'll come in through uh peerreview
46:22systems and they also come in through um
46:26the publishers themselves. Some
46:27publishers set up their own integration.
46:28It's a very simple API to integrate
46:30with.
46:32Submissions come in at the moment that
46:33they are submitted to the publisher. Um
46:37we screen them to analysis produce the
46:39exact reports that we just looked at in
46:41the presentation. And that means that
46:43the publisher has got a lot of um
46:48scope in how they how they use that. So
46:50some publishers um they won't use the
46:53reports a lot, but they'll base their um
46:56decisions on the alerts and they move
46:58very fast because of that. That doesn't
47:00mean that necessarily an alert means an
47:01article is getting rejected. It might
47:03mean it's going to go to the best
47:04possible referees and get very good
47:06quality um uh care but it also means
47:10that the publisher has a lot of
47:11confidence in their decision on that
47:12article. Um primarily that is how the
47:16service is used. It's I would say it
47:19does vary but maybe uh at least 80% of
47:24usage is looking at those submissions
47:27and it's a smaller percentage of usage
47:29that is looking at the historic
47:30published content even though there's a
47:32lot more of that.
47:34>> Um
47:35it makes more sense that the priority is
47:37always to be working with the the the
47:39submissions and um screening those
47:42before they go into peer review. Um, and
47:45again, it's really about having a fast
47:48way to make a triage or a decision on
47:50those submissions as soon as they come
47:52in and avoiding a situation where we
47:55have to do a huge investigation later.
47:57It's really the the key point.
48:00>> Yeah. And I can take the uh um the
48:03second part of the question of betan
48:06holdex. Uh is there any verification
48:09[clears throat]
48:10process prior to integration into your
48:13system?
48:15>> Um yes and no. in the
48:20where we have
48:22um
48:24uh information coming in from a sleuth.
48:26We will we will look at that and we will
48:28decide whether we believe that is um
48:32sound or not.
48:35There are
48:37certainly nuances certainly nuances
48:39there. So I can think of one sleuth who
48:41provides data publicly but not all of
48:44that is meant to be um misconduct. They
48:48they do lots of investigations of
48:49different kinds of things. So we have to
48:51be very careful about that. Um but also
48:54we we'll uh
48:58generally assume that when a data set is
48:59given to us and we've spot checked it,
49:01we've looked at it and we also got some
49:03confirmation through things like um have
49:05these articles been retracted? some of
49:06them not all of them then we can have a
49:09lot of confidence in the data. Um so no
49:12we wouldn't ever blindly um just put a
49:16list of articles in but where we are
49:19doing that it's
49:21um individuals who've got a clear track
49:24record of identifying problems
49:27and also um
49:32uh
49:36one thing is talking to sleuths and
49:41hearing their kind of attitude to how
49:42they will flag things. And um I can
49:45think of one in particular who is very
49:49despite having flagged thousands of
49:50articles, he said that he would flag
49:53maybe one in 10 of the articles he
49:55thought were problematic
49:57um because he was very very careful to
49:59not allow false positives. And that's
50:01been a key part of what we do as well.
50:04We're very very strict on um avoiding
50:07false positives and that works really
50:10well in practice as well because we know
50:12that if this is the first stage in peer
50:14review and uh we allow it to
50:17occasionally miss things that's actually
50:19okay because the rejection rate on this
50:21stuff is really really high in peer
50:23review. So we know that peer review um
50:27would catch anything that we miss. So
50:28it's okay to miss things occasionally.
50:30Um but false positives those cost time
50:34and so we avoid them uh very very
50:37carefully.
50:39[clears throat]
50:39>> Yeah. Okay. So our last question maybe
50:44how safe is to sorry this is we already
50:48done that but what are some early
50:51warning signals editors can detect even
50:55before peer review begins? H um maybe
51:00just answer the first question quickly.
51:01We have um for every publisher we have
51:05completely siloed data. So it's not
51:07possible for anyone else to ever see uh
51:10your submission data.
51:14We also have um uh a manual upload
51:16feature. So you can just upload a single
51:18article and check it. But the way that
51:20our system works, we will screen 100% of
51:23publications and 100% of submissions. So
51:26it's pretty rare that anyone ever needs
51:27to do that. Um the
51:31um sorry the the question was about uh
51:35yeah the early warning things. So I'd
51:37say the the alert is intended to be an
51:41early warning. I mean we we've called it
51:43the early warning API
51:46and
51:47that can be as simple as just a traffic
51:49light alert system. Red, orange, green.
51:53>> Yeah. And like I say, some publishers
51:55can work very efficiently and quickly
51:57with just that. Um, but where more
52:01information is needed, these reports
52:03have always been built around how can we
52:05get to a decision quickly? Um,
52:10without having to put a huge amount of
52:11time in. Um, and that's really the
52:16the logic here with screening. Um and we
52:19treat that as being a bit distinct to
52:21investigation which is when we do need
52:22to put time in.
52:25>> Yeah. As we come to the end of this
52:28great webinar, yeah, I would like to
52:30thank you very much Mr. Adam Day for the
52:34sharing your experience and for this
52:37great presentation and understanding how
52:42paper mills operate and how editors can
52:45detect and uh respond to them is
52:49essential for protecting the integrity
52:53and credibility of the research record.
52:57And thank you

About this transcript

This page contains the full transcript of Inside Paper Mills: Detection, Evidence, and What Editors Should Look For | ACSE Webinar Series 2026 by The ACSE, generated from the public captions YouTube serves with the video. The transcript has 7,831 words across 1,216 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.