Title & Abstract Screening (PRISMA) and AI Assist Overview | HubMeta Tutorial #8 — Transcript

Full transcript

1. 0:15Welcome back. Now that we have imported
2. 0:19our deduplicated articles into HubMeta,
3. 0:22we are ready to start the selection
4. 0:26process.
5. 0:27As you know, in the Prisma selection, we
6. 0:30have two main stages here. One is to go
7. 0:34through your articles and just the title
8. 0:37and abstract of them.
9. 0:39And then the second one is to go through
10. 0:42it as look at the whole full text of the
11. 0:45article. So, we have named them title
12. 0:48scan and deep scan in our process, but
13. 0:52honestly, we are thinking that with AI
14. 0:56included in the process, it might not be
15. 0:59as clear cut as a start from this then
16. 1:01go to that. It might be a little bit of
17. 1:04a back and forth between them or you
18. 1:06just might skip one of it completely or
19. 1:10you might just find your own pace with
20. 1:12this, but the ultimate goal here in
21. 1:15these multiple steps is to have a final
22. 1:19includable set of studies with all of
23. 1:23their PDFs, all of their sources ready
24. 1:27to extract data from. Okay, so pick is
25. 1:31the second stage which helps us get
26. 1:33through that. Let's look at the first
27. 1:35part which is looking at the title scan.
28. 1:39This is how it looks. We will see the
29. 1:43articles that we have included
30. 1:46a an abstract from them and then we can
31. 1:50use like some highlight keywords like,
32. 1:52for example, I want to see if they have
33. 1:54mentioned the word multinationality
34. 1:57in the abstract or let's just say
35. 2:00performance.
36. 2:02We see this one as
37. 2:05being highlighted in some of them, so
38. 2:07they are mentioning performance. This
39. 2:09just makes my job easier in reading the
40. 2:12abstracts and making a decision. And I
41. 2:14can submit my decision. One thing to to
42. 2:17mention here is that we have built a
43. 2:20metaphor collaboration. So, if you are
44. 2:23having a team of people to go over this,
45. 2:26like multiple people looking at your
46. 2:29articles, they will also have access to
47. 2:31this. They will not see each other's
48. 2:33decision. They will not see your
49. 2:34decision. They can submit their
50. 2:37suggestions as to inclusion or exclusion
51. 2:40of the article. And then here in your
52. 2:42project, you will see that. I'm going to
53. 2:44show this to you through an example of
54. 2:46one of my published projects. I will go
55. 2:50to my title scan. This will show you the
56. 2:54decisions that my team of RAs have made
57. 2:58and that helps me make my decision about
58. 3:01that. You might see this message again
59. 3:03and again that your articles are being
60. 3:06loaded. Please be patient with us on
61. 3:08that. Believe me, this was a much needed
62. 3:11step to make the next steps of the
63. 3:14search
64. 3:15the next steps of navigating through the
65. 3:18platform much faster. For example, if I
66. 3:20go to page three, all my papers
67. 3:23immediately load. So, I don't have to
68. 3:25wait every time, just one time wait. So,
69. 3:28now you can see that I have the
70. 3:31decisions of my RAs shown here together
71. 3:35with the reason they have decided, for
72. 3:37example, to exclude these papers. And
73. 3:41then based on that, I can, for example,
74. 3:44filter based on their decisions and
75. 3:47then, you know, submit my own decision
76. 3:49there. So, let's go back to the original
77. 3:52project that we are working on right
78. 3:54now. So, again, I can click accept or
79. 3:59reject for each of these articles.
80. 4:02Up here, we have multiple ways to filter
81. 4:08and show or hide some of the articles
82. 4:11down here, which are very important. The
83. 4:14most important one is the filters, where
84. 4:17you can filter based on, for example,
85. 4:20the title or the author or, you know,
86. 4:24let's say the years. Let's say you only
87. 4:27want the ones from after 2015, you can
88. 4:30see that
89. 4:32out of
90. 4:3320,000,
91. 4:35around 15,000 of them were published
92. 4:38after 2015, which is, you know, part of
93. 4:41the reason we have to do meta-analysis
94. 4:43and systematic reviews because
95. 4:45a whole lot of research is being done
96. 4:48every single year. There has to be some
97. 4:50way to integrate all of this. So, you
98. 4:52can play around with these filters. This
99. 4:54AI part is very important. I will talk
100. 4:57more about it when we explain the AI
101. 4:59assist function, but let's skip it for
102. 5:02now.
103. 5:04And the actions that you see here are
104. 5:07multiple things that you would need.
105. 5:09First of all, you might need to export
106. 5:12the papers that you have in this stage
107. 5:15along with the decisions that your team
108. 5:17has submitted. This will give you a CSV
109. 5:20file with all the papers, all the
110. 5:22decisions on them, the reasons and
111. 5:24everything. The other one is
112. 5:27for different reasons, you might still
113. 5:29have duplicates in this stage. You might
114. 5:32have added another file later in the
115. 5:36project or have updated your search
116. 5:38later on and you might not have had a
117. 5:41chance to run the duplication on that.
118. 5:44So, you can remove duplicates when you
119. 5:46are in title scan stage. What this does
120. 5:49is that it looks through all of these
121. 5:51and just adds a tag as duplicate. And
122. 5:55then with this button, you can show with
123. 5:58this filter, you can show or hide them
124. 6:00and then make decisions about them. This
125. 6:03enrich metadata, what this one does is
126. 6:06that it goes through a long process of
127. 6:10going through all of your articles and
128. 6:12trying to use Crossref
129. 6:15and OpenAlex, like all of these
130. 6:18platforms that provide information for
131. 6:20our articles, to see for example, if one
132. 6:23of study is missing an abstract, let's
133. 6:25go and find it. If maybe there is some
134. 6:27ambiguity in the name or in the name of
135. 6:31the authors, the bibliographic
136. 6:33information, that just enriches the
137. 6:36metadata of our articles. The other one
138. 6:39is detect language. It will detect the
139. 6:42language of
140. 6:44the articles that you have included.
141. 6:46Then for example, you can say, you know,
142. 6:49after you have run that, just say for
143. 6:51example, show me only French ones or
144. 6:53Spanish ones. And this is an important
145. 6:55point because
146. 6:56with AI being present, it is no more
147. 7:00acceptable to say in your inclusion
148. 7:03criteria that we only looked at English
149. 7:06language articles. We have to select all
150. 7:10languages because AI can very easily
151. 7:14read and translate from all the
152. 7:17languages that our articles have been
153. 7:19published. And so, you can't just not
154. 7:22look at one part of the evidence. You
155. 7:24have to include that. So, this
156. 7:26detect language is also important. After
157. 7:29you run it, you can filter and see like
158. 7:31for example, how many papers in other
159. 7:33languages you have. This last one, which
160. 7:36is interrater reliability, I will show
161. 7:38it in in an older project of mine where
162. 7:41I have uh reviewers submitted their
163. 7:43votes. So, this is basically the
164. 7:46reliability
165. 7:48of their decision. So, it calculates
166. 7:50Cohen's kappa, the level of agreement
167. 7:54between that. This is something usually
168. 7:57uh required in reporting. Like, what was
169. 8:00the Cohen's kappa of uh you know, the
170. 8:02decisions that you know, about
171. 8:04inclusion. This gives you the numbers
172. 8:06that you need to report. The other one
173. 8:09is ML-Rank. So, what this one does is uh
174. 8:14this one is the AI rank function that we
175. 8:16had in the legacy HubMed 2. So, what
176. 8:19this one does is it learns from the
177. 8:22decisions that you have submitted so far
178. 8:25and rank orders the rest of your body of
179. 8:30research, all the other papers based on
180. 8:34similarity to your inclusion decisions.
181. 8:37So, if I run ML-Rank, it will run it on
182. 8:40all the thousands of articles that I
183. 8:42have. So, basically, the idea here is
184. 8:45that
185. 8:46it will bring up in this list of
186. 8:50thousands of articles
187. 8:52those articles that are closer in
188. 8:55wording and in meaning to these ones
189. 8:59that I have already included. So, you
190. 9:01see all the inclusion one come on top.
191. 9:05And then, the rest of them are what AI
192. 9:09or machine learning has detected to be
193. 9:12very close in meaning and in mostly
194. 9:16wording, close to these ones that I have
195. 9:20said yes to. So, this makes my job
196. 9:22easier. So, the idea here was instead of
197. 9:25going through all the 2,000 articles
198. 9:28that I have go through 5,000 of them and
199. 9:32then when you keep doing the ML-Rank
200. 9:35every time the more relevant ones come
201. 9:37up and come up, so what ends up being at
202. 9:41the bottom of the pool would be all the
203. 9:43irrelevant ones. So, after, you know,
204. 9:46some point
205. 9:48you will see that you if you keep going
206. 9:50down, you are just rejecting one after
207. 9:53the other after the other. And then you
208. 9:55have kept rejecting for like 100 papers
209. 9:59in a row, that means you are not finding
210. 10:01much relevant articles in the remaining,
211. 10:03and that's where
212. 10:05that becomes your stopping point. So,
213. 10:07you only look at part of this 22,000.
214. 10:11That was a big time-saver at the time,
215. 10:13but of course, in the new age, we have a
216. 10:17better tool, which is called AI Assist.
217. 10:21I'm going to talk about that in the next
218. 10:23week.
219. 10:33>> [music]