Making Inclusion Decisions from AI Consensus Votes | HubMeta Tutorial #10 — Transcript

Full transcript

1. 0:15Welcome back.
2. 0:16Now, we have initiated our AI bots to go
3. 0:21through all of our articles, and as you
4. 0:24can see in the screen, it's going one by
5. 0:27one.
6. 0:29And it says like three bots are active.
7. 0:32That's how we set this up. Uh depending
8. 0:35on the size of your library, it of
9. 0:36course takes some time uh for it to
10. 0:39finish. Think about it this way, for
11. 0:41each article, it is sending that request
12. 0:44to three different AI, and those AI will
13. 0:48have to come back with a response, and
14. 0:50that has to be recorded. But you can
15. 0:53continue working on other parts of the
16. 0:55platform. This will always go on in the
17. 0:57background, and then the decisions will
18. 0:59be saved in database, so you can come to
19. 1:02back to it at any time. But while it is
20. 1:04doing that, uh I'm going to show you how
21. 1:07we are going to actually use this.
22. 1:10Back to pick title scan page, uh you can
23. 1:14see that now that this one is running,
24. 1:18the AI assist is running, some of my
25. 1:20papers have now this thing showing the
26. 1:26AI bots votes here. Let's see what this
27. 1:30actually means by looking at these two
28. 1:32papers. The first one, the first paper
29. 1:35says that three bots, this number three
30. 1:38means the number of bots that have
31. 1:40reviewed this, have looked at this and
32. 1:43compared it against our inclusion and
33. 1:45exclusion criteria,
34. 1:47and then this 0% number means none of
35. 1:51them
36. 1:52found this article to be includable
37. 1:56according to inclusion exclusion
38. 1:57criteria.
39. 1:59Similarly, this one This is how I read
40. 2:02this. So, three bots have looked at it.
41. 2:05Only one of them, 33%,
42. 2:08found it includable. And then we can
43. 2:11look at the rest of them. You can see
44. 2:13that only a few of them are showing. The
45. 2:15rest of them are empty. This is because
46. 2:17my process is still, you know, working.
47. 2:20It has a long way to go.
48. 2:22But let's see how we are going to use
49. 2:25these numbers. Remember when I was
50. 2:28explaining the filters, I skipped over
51. 2:31this AI. Now is the time to look at
52. 2:34that. Here you can filter by this
53. 2:38average vote, by this consensus number
54. 2:42of your AI bots, these articles. Like
55. 2:44for example, if I want to
56. 2:47If I want this to show me the ones that
57. 2:51all of them, like 100% consensus that
58. 2:54this article should be included, I can
59. 2:57set it up for like show me all the ones
60. 3:00where AI vote is above 0.9 for example,
61. 3:04right? And, you know, in this case
62. 3:07because we have three bots, that just
63. 3:09means 100%. So, you can see it is
64. 3:12showing me all the papers that all the
65. 3:15three bots that were running it are
66. 3:17saying, "Yeah, include that."
67. 3:20And then, because this is now filtered,
68. 3:22I can hit accept all, and this will
69. 3:26accept all of these articles. Similarly,
70. 3:30I can say
71. 3:31>> [clears throat]
72. 3:32>> show me the ones that all of them are
73. 3:34saying exclude. So, all the ones that
74. 3:38are below, for example, 10%, 0.1. And
75. 3:43you see all of these are 0% vote. So,
76. 3:47669.
77. 3:49Now,
78. 3:51>> [clears throat]
79. 3:51>> and then again, of course, know, reject
80. 3:53all of this all at the same time. Now,
81. 3:56here is the important part is that this
82. 3:59is not a written process as yet. This is
83. 4:03not something that has been as
84. 4:06established as the Prisma. This is a new
85. 4:10way of doing things. So, we have to test
86. 4:12it again and again in different projects
87. 4:15and we have to see how it actually
88. 4:17works. Currently, the way I personally
89. 4:19work with this is I exclude the top
90. 4:23parts like I exclude and include just
91. 4:26like I showed you. exclude the ones that
92. 4:28are everybody agrees, all the bots agree
93. 4:31that it should be removed, include all
94. 4:34the ones that everyone says has to be
95. 4:38included, and then for all the ones in
96. 4:41between, they're like in this case 75,
97. 4:44these would be marginal cases, right?
98. 4:48This will become things that you have to
99. 4:50make a decision about. For these, I will
100. 4:53run two other bots, right? Just more AI
101. 4:57bots. Maybe I will change my inclusion
102. 4:59exclusion criteria, make it tighter, I
103. 5:01will add more specification, and then
104. 5:04run it again. With the addition of new
105. 5:06votes, I have more data to work with. I
106. 5:09will continue this process until this
107. 5:12marginal part becomes low enough, and
108. 5:15then, honestly, the best thing to do
109. 5:18when this is like the part that is
110. 5:21marginal and AI has not come to a
111. 5:23consensus is manageable enough, it just
112. 5:27needs a pot of coffee and some brute
113. 5:30force going one by one manually, and
114. 5:33usually it's best if the lead researcher
115. 5:35does that so that, you know, you just
116. 5:36don't keep going a whole lot of back and
117. 5:39forth and a lot of decision-making. What
118. 5:41I have found is that when you get to a
119. 5:43speed, you can do in 2 hours anywhere
120. 5:46between 500 to 1,000 articles manually.
121. 5:49That's a good number to have in between,
122. 5:51then that's the marginal place. And for
123. 5:54those we will do them manually and be
124. 5:57done with the title scan stage. In the
125. 5:59next video, we will talk about what we
126. 6:02will do with the articles that we have
127. 6:06included in [music] the title scan
128. 6:08phase.
129. 6:18>> [music]