Data Annotation 101 — Transcript

Full transcript

1. 0:00Hi, I'm Raymond from the AfterQuery
2. 0:02team. We're very excited to work with
3. 0:04you on this project. In this video, I
4. 0:06will quickly go over some key concepts
5. 0:08in this boot camp and some best
6. 0:10practices that you should follow for an
7. 0:12data annotation.
8. 0:14To start, I want to give a highle
9. 0:15overview of how the work you do will
10. 0:17contribute to pushing forward frontier
11. 0:19models. There are two main methods to
12. 0:21train AI models. The first is supervised
13. 0:25fine-tuning which teaches the model to
14. 0:27imitate correct answers by using a high
15. 0:29volume of question and answer sets. SFTs
16. 0:33not only train the model on the correct
17. 0:34answers but also the desired output
18. 0:37format.
19. 0:38For example, if we train a model on a
20. 0:40high volume of math questions with each
21. 0:43answer set including the formula used
22. 0:45within the calculation, the model will
23. 0:47learn to expand on the formula every
24. 0:49single time that they are asked a
25. 0:51similar question.
26. 0:54Another method to train AI models is
27. 0:56reinforced learning from human feedback,
28. 0:58also known as RLHF,
29. 1:01which teaches the model to prefer the
30. 1:03kinds of answers the annotators, in this
31. 1:06case you would rate higher.
32. 1:09Often associated with RLHF are rubric
33. 1:12points, which are also created by the
34. 1:14annotators used to score the model on
35. 1:16their out outputs. The rubric points are
36. 1:19critical to improving the model's
37. 1:20learning as one model runs the same
38. 1:23prompt iteratively against the rubric,
39. 1:25learning from their highest scores until
40. 1:27they consistently score full marks.
41. 1:31Good rubric points should be atomic,
42. 1:33meaning that it mentions one concept at
43. 1:35a time. They should have objective
44. 1:37phrasing and be a single pass fail
45. 1:39check.
46. 1:42Next, we'll quickly go over what makes a
47. 1:44good prompt. The best prompt is the one
48. 1:46that aligns best with your project
49. 1:48scope. Some best practices to keep in
50. 1:51mind are to align with the scope of the
51. 1:53project, to state the specific outcomes,
52. 1:57to write out your explicit assumptions,
53. 1:59and to define the expected output
54. 2:01structure.
55. 2:04Golden solutions are also an important
56. 2:06part of the data annotation workflow.
57. 2:08The primary purpose serves to provide
58. 2:11context to the reviewers on your logic
59. 2:13behind your question and answer pairs.
60. 2:16It is important for your golden solution
61. 2:17to be structured and easy to understand
62. 2:20for another person.
63. 2:23Last up, we'll quickly go over some best
64. 2:25practices to reviewing a model's
65. 2:27reasoning. Read through the model's
66. 2:29chain of thought with an open mind.
67. 2:31Today's top models can sometimes beat
68. 2:34your provided question and answer sets.
69. 2:37If the model's logic is stronger and
70. 2:39still within the scope of your prompt,
71. 2:41please update your sets. If the model is
72. 2:44wrong, mark the first break in their
73. 2:46reasoning and note the fix in one line
74. 2:48before moving on.
75. 2:51That concludes our overview. We're very
76. 2:53excited to have you on this project and
77. 2:55thank you for choosing After Query.