AI-Assisted Screening: Inclusion & Exclusion Criteria | HubMeta Tutorial #9 — Transcript

Full transcript

1. 0:15Welcome back. Now, this is where things
2. 0:18get very exciting. Why? Because remember
3. 0:22in the old days we had to go through all
4. 0:24these thousands of articles and make a
5. 0:26decision about is this relevant or is
6. 0:29this not? And then say yay or nay to
7. 0:31them, right? Now, what if we could train
8. 0:35a bunch of AI bots to go through all of
9. 0:38these articles, not just part of them,
10. 0:41to go through all of these articles and
11. 0:44then put in their vote, and then we will
12. 0:48make our decision based on the level of
13. 0:50agreement between these AI bots. That's
14. 0:53exactly what we have built in this
15. 0:55estate, which is pick and assess. Now,
16. 1:00what we do here is we define the
17. 1:04inclusion exclusion criteria
18. 1:07for our project in this estate, and
19. 1:11based on this, we will train the bots
20. 1:15and then ask them to go through our
21. 1:17articles. Again, for inclusion
22. 1:19exclusion, it is a very good idea to
23. 1:22start not from a scratch because this is
24. 1:26something that matters a lot in you
25. 1:29being able to defend your inclusion
26. 1:31exclusion criteria against reviewers and
27. 1:34editors, and it really matters what
28. 1:36comes out of the selection process. So,
29. 1:39here you can obviously spend more time
30. 1:43defining it. Right now, I will just do
31. 1:46one of the actual steps that I normally
32. 1:48do, which is just copy the inclusion
33. 1:51exclusion criteria from, you know, one
34. 1:53of the articles in the field, and then
35. 1:55based on that, I will ask it to generate
36. 1:59inclusion criteria. And again, don't
37. 2:01worry about the format or typos or
38. 2:03whatever that you type in here. There is
39. 2:05an AI in the back. It will create it for
40. 2:07you. This part is extremely important.
41. 2:10As a human researcher, it is our duty to
42. 2:14spend the time to review these criteria
43. 2:18that AI has created for us. Now, of
44. 2:21course, in building up Meta, we have
45. 2:23done our best in training the AI to make
46. 2:26sure it doesn't produce hallucinations,
47. 2:29bad stuff here, but still, this is your
48. 2:32research. This is your job to carefully
49. 2:36review all of these criteria. So, here,
50. 2:39for example, it is suggesting a studies
51. 2:41must be empirical and reports of a
52. 2:44sufficient sample size and outcome
53. 2:46statistics. Studies must investigate the
54. 2:49relationship between one or more
55. 2:51operationalization of multinationality
56. 2:53and financial performance. These are all
57. 2:55very good. Sometimes, although we have
58. 2:58kept instructing it to not have a
59. 3:01language criteria here, sometimes you
60. 3:04would just see, "Oh, it says studies
61. 3:06have to be in English language."
62. 3:08Usually, if anything like that happens
63. 3:11in the exclusion criteria, I remove it
64. 3:14because obviously, we can't only have
65. 3:16English languages. But anyways, after we
66. 3:19are happy with our inclusion and
67. 3:21exclusion criteria, we can now, of
68. 3:23course, edit all of these, add one, or
69. 3:26remove one from this set. After we are
70. 3:30happy with this inclusion exclusion
71. 3:32criteria, it is a it's a very important
72. 3:35step to take a screenshot of this or
73. 3:37keep it for your records because later
74. 3:39on, in the actual project, you would
75. 3:42have to report this inclusion exclusion
76. 3:45criteria. Then, the next step is to set
77. 3:48up the bots. So, these are the AI bots.
78. 3:52So, think of them as tiny agents where
79. 3:55you give them these instructions for
80. 3:58inclusion and exclusion, and then one by
81. 4:01one they go through the papers and look
82. 4:04at the title and the abstract of the
83. 4:06paper, compare it against this inclusion
84. 4:08exclusion criteria. The only output they
85. 4:12produce is a yes or a no vote. That's
86. 4:16the only thing this AI bot does. Now,
87. 4:20think of it like, do we need only one?
88. 4:22Of course not. We want more of them. So,
89. 4:25we have multiple language models to
90. 4:27select from. This is a, you know,
91. 4:29dynamic list. It will keep getting
92. 4:31updated as new models come along. We
93. 4:33will add them here. The ones that have
94. 4:36been tested and produced good results.
95. 4:37So far, I think, you know, Mistral Nemo
96. 4:40is a good one. I usually have one from
97. 4:43that. I usually try to have one from
98. 4:45Quen, and one from GPT. And then here we
99. 4:50add these bots. I can, you know, remove
100. 4:53this one. Here is if you want to provide
101. 4:56some extra instructions. Like, let's
102. 4:59say, you know, you want this person to
103. 5:02pay special attention to sample size
104. 5:05being present. Make sure this article is
105. 5:09not using that specific data set that
106. 5:13you want. Like, something like that.
107. 5:15Whatever the case is, you can add as
108. 5:17many bots
109. 5:19as you want. But, usually, you know, you
110. 5:22want an odd number here. So, an even
111. 5:26number because, you know, it just makes
112. 5:28it easier to make supermajority when it
113. 5:31is an odd number. After you set this up,
114. 5:34you go to next, and then here it will
115. 5:37show you which of your articles you want
116. 5:39to include. You will say, is it a, you
117. 5:42know, you're just looking at the title,
118. 5:45or do you want the full text too? That
119. 5:48part is assuming that you have already
120. 5:51downloaded the PDFs and you have them
121. 5:53here. Are you looking only at the
122. 5:55on-screen papers, all of them? You know,
123. 5:58different options are available here.
124. 6:00And then when you click on start a
125. 6:02screening, it will show you how many
126. 6:06credits, [snorts]
127. 6:07how many AI credits will be used in
128. 6:11doing this review for you. So, this is
129. 6:14where you might actually be incurring
130. 6:17costs. So, each of these credits is
131. 6:19worth half a dollar in our current
132. 6:22pricing model. And this is basically per
133. 6:26each bot and per 1,000 articles that is
134. 6:29going to run you down one credit. And
135. 6:33that's, as I explained at the start,
136. 6:36this is what we have to pay for external
137. 6:39large language model providers. And you
138. 6:42are only paying for what AI you actually
139. 6:45use. After you confirm this, the three
140. 6:48bots will start working on this. This is
141. 6:52like hiring multiple research assistants
142. 6:55going through your literature and
143. 6:57submitting their decisions. In the next
144. 7:00video, I will show you after these
145. 7:03decisions are made, how I use them to
146. 7:07make my inclusion exclusion decisions.
147. 7:14>> [music]