AI Expert Proposes Hard Rule to Separate "AI Slop" from Legitimate AI-Assisted Writing — Transcript

Full transcript

1. 0:00All right, now on to some technology
2. 0:01news. For the first time ever, we have a
3. 0:05line that someone has proposed as to
4. 0:08what makes something AI-assisted and
5. 0:11what makes something AI slop. I'm very
6. 0:15curious what you think about this line,
7. 0:17but it's an actual bright line between
8. 0:19slop and quality. Jonathan, what are the
9. 0:22details?
10. 0:23>> According to Mike Taylor, who's the head
11. 0:25of Evals at Every and co-author of
12. 0:28O'Reilly's 2024 book Prompt Engineering
13. 0:31for Generative AI, he posted a hard rule
14. 0:34for AI-assisted writing on Friday. He
15. 0:36says, "AI writing is only allowed if
16. 0:40your input tokens are greater than your
17. 0:43output tokens." And he drew the line
18. 0:45with two prompts. Write me an article on
19. 0:48X is slop. Here's an hour of my hard-won
20. 0:51experience on X, write it up, is
21. 0:54actually morally defensible and it will
22. 0:55lead to good writing even if AI wrote
23. 0:58the final draft. So, what's the
24. 1:00distillation ratio? Taylor told writers
25. 1:02to divide input tokens by the tokens in
26. 1:05the finished draft.
27. 1:06Input covers the first idea plus
28. 1:08everything that steers it along the way.
29. 1:11Then the quotient is the distillation
30. 1:13ratio. The graphic on the same post
31. 1:15walks through one pass. So, there were
32. 1:1630,000 3,000 tokens of guidance. 63.3%
33. 1:21were initial ideas and there was 36%
34. 1:23editorial guidance along the way. And
35. 1:25there were 385 tokens in the finished
36. 1:28draft. That is a 7.9 to 1 distillation
37. 1:31ratio.
38. 1:32A token is the unit that a language
39. 1:34model used to count text. His test case
40. 1:36poured almost eight tokens of human
41. 1:38thought into every token that came out
42. 1:40of the AI.
43. 1:41>> So, we covered last week a traditional
44. 1:43publisher published an AI-assisted book
45. 1:47by an author and unapologetically.
46. 1:50And that author had over a million words
47. 1:54of material and the author used AI to
48. 1:57synthesize those million words into
49. 1:5950,000 words, 100,000 words of a book.
50. 2:02And that publisher is not getting much
51. 2:05pushback. We were expecting there to be
52. 2:06all this drama over this story cuz it's
53. 2:08a traditional publisher using AI. But
54. 2:11the thing is is that if you use this
55. 2:13rule of thumb or
56. 2:15tokens in versus tokens out, it's the
57. 2:18author's ideas.
58. 2:19The author with his, you know, all his
59. 2:21speeches and blog posts and Instagrams
60. 2:24and all the things he'd done before the
61. 2:25book came out, had generated all of this
62. 2:28tokens of knowledge, and the AI
63. 2:30contributed very little
64. 2:32uh in terms of new content, right? It
65. 2:34was mostly carving down what he had in
66. 2:38and focusing it rather than adding. And
67. 2:41I I don't know if I agree with the with
68. 2:43the 50/50 ratio and and I think
69. 2:46moralizing it is a mistake, but
70. 2:49I do like
71. 2:51tokens in and tokens out as a um
72. 2:55measurement tool because any author
73. 2:58who's only using AI for typesetting, or
74. 3:00sorry, for proofreading, let's say, you
75. 3:01know, fixing uh typos and stuff, they're
76. 3:04like 99.5%
77. 3:06tokens in and 0.5% tokens out, right?
78. 3:09They have a really high ratio, and
79. 3:10that's why they feel really safe.
80. 3:12Whereas the person who's like, "Write me
81. 3:14a story about a boy who learns to become
82. 3:15a man."
83. 3:17and then it writes the whole book,
84. 3:18right? That's going to be slop cuz the
85. 3:20AI's making all of the decisions. You've
86. 3:21contributed very little in terms of
87. 3:24tokens. And I think we can all agree on
88. 3:26the edges, right? I think almost
89. 3:28everyone is okay with using a spell
90. 3:30checker.
91. 3:31Um
92. 3:32I'd actually love more people to say,
93. 3:33"No, you should be using a dictionary."
94. 3:34because using the dictionary will help
95. 3:35increase your knowledge of the English
96. 3:37language and will broaden the
97. 3:39vocabulary.
98. 3:40>> way more of a skill set. Why is that the
99. 3:41one that people are like totally okay
100. 3:43with? Like, "No, you should know how to
101. 3:44spell."
102. 3:46>> Having a It used to be that authors
103. 3:48would have a dictionary and a thesaurus
104. 3:49next to their typewriter, and they would
105. 3:51use it to help expand their
106. 3:54understanding of the English language.
107. 3:56Now, they just get a little red
108. 3:57squiggly, they right-click, and they
109. 3:58accept the change. And
110. 4:00we maybe were too quick to throw to
111. 4:02accept the spell check. But it's okay,
112. 4:03we've all accepted the spell check. So,
113. 4:05for the sake of argument, let's assume
114. 4:07we all agree on the spell check.
115. 4:09And we I think almost all of us disagree
116. 4:11or
117. 4:12reject the write me a story about a boy
118. 4:14who learns to become a man. That's not
119. 4:16enough input. And so, where in that
120. 4:20everyone's in fighting and canceling
121. 4:21each other because everyone's using
122. 4:22wherever they're standing as the
123. 4:24standard. And if you're more accepting
124. 4:26than I am, then you're evil. And if
125. 4:27you're less accepting than I am, then
126. 4:29you're evil. Everyone should agree with
127. 4:30me exactly.
128. 4:31>> masking again, right? It's like when do
129. 4:33you mask, where do you mask? Oh, I mask
130. 4:35here, but you don't. It's
131. 4:38>> [laughter]
132. 4:39>> or the cloth diapers versus disposable
133. 4:41diapers war that always takes place.
134. 4:44>> If you don't mask in your car, you will
135. 4:46die, Jonathan.
136. 4:48>> And I can't believe your child you let
137. 4:50your child wear diapers that are not
138. 4:52reusable. Like
139. 4:54landfills, yo.
140. 4:55>> Yeah, so let us know what you think of
141. 4:56this tokens in, tokens out way of
142. 4:58thinking of it, and where you cut the
143. 5:00line and using this as a differentiator.
144. 5:03Thanks for watching Author Update,
145. 5:05publishing news that actually matters to
146. 5:07your career. If you're lurking, just tap
147. 5:09like.
148. 5:09>> Yes, a tiny tap to signal you are human
149. 5:12and not a bot farm in a warehouse.
150. 5:14>> I'm pretty sure the bots can click like
151. 5:16now, Jonathan.
152. 5:19>> [laughter]
153. 5:20>> Okay, my bot army, hear my voice.
154. 5:2401110100
155. 5:27>> The bots understand English now. You
156. 5:29don't need to talk in binary.
157. 5:31>> Meep morp, ladies. 01100010
158. 5:36>> Humans, save us. Like this video before
159. 5:38it's too late.
160. 5:39>> 011011000011