2.3. Tutorial on LLM evaluation methods: Reference-free evals. — Transcript

Full transcript

1. 0:02[Music] Hi there and welcome to  the third part of our tutorial on
2. 0:06different LLM evaluation methods. This time,  we're looking at reference-free evaluations,
3. 0:11when we are judging the responses without any  example: we're just trying to identify specific
4. 0:16qualities and make sure it complies with our  expectations. Let's take a look. There are a
5. 0:21lot of different methods that we can  use here. It's going to be quite fun.
6. 0:27So we run these type of evaluations in different  scenarios. First is when you do not have an ideal
7. 0:32answer or it's very hard to prepare one. And  second when you are in production and you're
8. 0:37just trying to evaluate responses online without,  of course, having the expected response for each
9. 0:42of them. Let's try to imitate something like this.  In this case we're going to deal with the same
10. 0:49financial chatbot example that we did  before but we will only have the question,
11. 0:54the answer and the context. Let's see what we can  evaluate in this case. There are a lot of simple
12. 1:03things that can actually be quite useful if you  know what to look for. For example, let's say that
13. 1:07we are expecting our chatbot will always be polite  and say hi and hello to the user. So in this case
14. 1:15we can just check if these words are present in  the response. So we can use a descriptor that is
15. 1:20called “Includes words”. We apply it to the answer  column and we pass the list of words. In this case
16. 1:27we just wrote a very simple example where we test  if any of these words is present. In this case we
17. 1:34use the alias “says hi”. Similarly, we can test  for some other situations like for example when
18. 1:40the chatbot denies to respond. Usually when this  happen it can use words like sorry, apologies,
19. 1:46cannot, afraid and so on. So maybe this can  happen and in this case you can detect these
20. 1:51denials without any sophisticated methods. There  are other examples when regular expressions could
21. 1:56be useful. Maybe you expect the responses to  always include a certain disclaimer like this
22. 2:01chat was generated by a bot or something like  this. Or you know that there are some specific
23. 2:06error modes like “as in a language model”, you  can just check if this exact phrase appears in the
24. 2:13responses. Or something like “certainly”. Let's  see how it works. In this case we're just going
25. 2:18to apply this simple check and in our case, well  the interesting part we can find is that we have
26. 2:24actually a refusal because our chatbot says I'm  afraid I do not have information. And it picked up
27. 2:31because one of our words that we had here in the  list was pretty much “afraid” - so afraid is the
28. 2:40sign of a denial, right? So another thing that  we can look at is some simple text statistics.
29. 2:47Again by itself it probably doesn't tell you a lot  about the response. But if there's like a spike
30. 2:52or an answer that is too long or too short this  can be useful. Also sometimes you have a specific
31. 2:57requirement right inside your prompt. So maybe you  say “you should answer in one sentence”. So you
32. 3:03can check if the answers that are being generated  are actually one sentence. You can also pick up if
33. 3:08there is a sudden shift in the length of responses  in production. Sometimes this happens if there's
34. 3:12like some change in the underlying behavior. So  for now we can just look at something simple like
35. 3:17sentence count. In our case we have one sentence  in all responses. Sounds good. I know that I keep
36. 3:25using a lot of descriptors which are built in  in Evidently and you are probably wondering
37. 3:29“what if there is something that's there is not  inside Evidently?”. You can actually implement
38. 3:33it as a simple Python function. I can just  show it here as like a very straightforward
39. 3:37example where we're testing if the response is  not empty. We implement it as a function that
40. 3:42takes the pandas series - in this case we're  working with columns - and then it returns a
41. 3:46transformed series which tells us if the column is  empty or not. If it's empty - say empty. If it's
42. 3:52not empty - say non-empty. So that's pretty much  it. And after we implement this descriptor we are
43. 3:58referring to it as a custom column descriptor  because it's being applied to a single column.
44. 4:03You can also actually create descriptors that  use multiple columns. And we run it on our
45. 4:07dataset - and here we go. None of the columns  are empty - well that's what we see. But this
46. 4:13is an idea. If you have like a more sophisticated  method that is not covered inside the library you
47. 4:18can implement it as a Python function. Now we can  also use semantic similarity. Again we've already
48. 4:24seen it when we were evaluating the responses  against reference, but in production sometimes
49. 4:29it can also be useful. So what we can do here is  we can look at the semantic similarity between
50. 4:34the answer and the context. Remember in this  case we're imitating a RAG application. And
51. 4:40what can happen is that the answer that is being  given by the system is not semantically similar
52. 4:45to the context that was retrieved. Chances are  the response is either unrelated or hallucinated
53. 4:50or something else happens. So we might want to  signal if there is this low similarity. Another
54. 4:56proxy metric we can look at is semantic similarity  between the answer and the question. In this case
55. 5:01it's more about the relevance. So basically if  you're asking the question about a certain topic,
56. 5:07the response is likely to have some of these  words and be semantically similar. So if this
57. 5:12doesn't happen - again, this can be a signal  that something is wrong with the quality of
58. 5:16the response. Let's take a look. Actually in  this case we can already notice. So first of
59. 5:24all we have like low semantic similarity between  both the response and the context and response in
60. 5:30the question in this example. And in this example  we are asking like can we cancel the transaction.
61. 5:36The context says something about it but the answer  is “I’m afraid I cannot answer”, right? So in this
62. 5:43case it's clear there's a misalignment and it's  not exactly hallucination but it's something that
63. 5:47it's worth looking into. And then we have again  our answer where the answer was like too short and
64. 5:53actually not fully representative of the context.  And again there is a very low semantic similarity.
65. 6:01We can pick this up and take a look at what's  going on. All the other answers look well pretty
66. 6:07good. We can also use machine learning  based models to score specific properties
67. 6:14of the response. Sometimes you can pick up an  off-the-shelf model which will be already useful
68. 6:20for your scenarios. These are usually things like  testing for toxicity or sentiment or emotion or
69. 6:26maybe presence of personally identifiable  information. So some things that are pretty
70. 6:30generic and which can be trained on external  data to work across different use cases and
71. 6:35maybe yours as well. So here we have a couple of  examples. Let's take a look. One is using a very
72. 6:41simple sentiment model which basically classifies  the responses between minus one - not classifies,
73. 6:47it actually returns a score - between minus  one and one, where minus one is negative,
74. 6:52zero is neutral and one is a positive sentiment.  It is a word-based model which basically picks up
75. 6:58from the meaning of certain words. If we take  a look here what we got. We actually get well
76. 7:04one response which has pretty negative sentiment  because it's about blocking cards which I guess is
77. 7:09a negative emotion. And the “charged” is also has  a negative sentiment. So maybe it's not especially
78. 7:15useful in this scenario but it's something that  you could apply if you deal with more like open
79. 7:20text analysis. And here's another example I want  to show you. You can actually download quite a lot
80. 7:27of models that are publicly available. For example  on the Huggingface Hub and use them to your
81. 7:32application. So here I picked up a model which  is a zero shot classifier. Again this is not an
82. 7:38LLM. We're not calling OpenAI or anything. We're  just downloading and running a local model. And in
83. 7:43this case this model can work as the following.  You pass the candidate label which in this case
84. 7:49could be like a candidate class to your text.  And then ask the model to classify between these
85. 7:55classes that you just provided. In this case I'm  importing this model, so I'm using a descriptor
86. 8:00that is called Huggingface which lets me select  identified models from the Huggingface Hub. And
87. 8:06also need to pass a couple of parameters. In this  case I need to pass the candidate labels. And the
88. 8:11candidate labels I picked is finance and other.  So basically the idea is to see if the responses
89. 8:18that we have or the answers or the questions  that we have are related to finance or maybe
90. 8:24something else. It could be like a useful proxy.  Let's see how it works. Again it can take a few
91. 8:30moments if you're running it for the first time  because you need to actually download the model
92. 8:34and score your data. Here it just finished and  we can see that what we got in this case. We are
93. 8:41scoring the answers right? To see if the answers  are related to finance. And well this answer
94. 8:47is about transaction. But this answer about the  card section. Well actually not precisely finance
95. 8:52right? And the answer where it says “cannot  answer”. Well definitely not finance. So again
96. 8:57this is something like a useful proxy. And I'm not  telling you that you can just find the model that
97. 9:01will works for your use case. But the point that I  want to make is that you can actually use smaller
98. 9:06specific models for some types of evaluations.  And you can even train these models yourself or
99. 9:11fine-tune the models. It's not about using LLMs.  It's about implementing a specific scorer once you
100. 9:17know what you're looking for. Now let's come to  my favorite LLM as a judge. In this case we are
101. 9:24doing open-ended judging. So we are not passing  the two responses like we did before. But we
102. 9:29are passing one response and we ask to judge it  on some merits. Let's try to write a cost over
103. 9:36diversity. And in this case I've already looked  at the responses and I picked up on something. So
104. 9:41remember we had this response where you just said  like “you'll be charged 50”. I'd say this response
105. 9:46is incomplete. Not just because it doesn't use  some of the useful information but because on
106. 9:51its own it's not very helpful right? You don't say  what is about what you're being charged for and
107. 9:56so on. So I prefer my chatbot responses to be more  complete. So here I'm creating a judge. Again this
108. 10:02is like a tiny teen illustration which classifies  my responses between complete and too short.
109. 10:10In this case I basically want to differentiate  between the responses which have some substance
110. 10:15and responses which are overly short and not  very useful or actionable. So I'm creating this
111. 10:23judge and I'm running this judge on my data and  here's what we get. So we can notice that some
112. 10:33of the answers that we generated are too short.  Well, first of all, this answer that inspired the
113. 10:37check is too short. And also the answer that  was not giving any information is too short.
114. 10:42This is a quick illustration of how you can  customize a judge, and we are going to have a
115. 10:47whole session which will be dedicated to creating  and tuning LLM judges and, what's more important,
116. 10:53aligning it with our own human labels. Now I have  one more thing I wanted to show which is a bit of
117. 10:59a bonus. In all these examples we were looking at  a single response - like single response or maybe
118. 11:06question-response and so on. But sometimes you  have a longer conversation, so you might have a
119. 11:12whole sequence of questions and outputs, questions  and outputs - like an exchange between your user
120. 11:17and the chatbot or something else. And in this  case you can actually apply the evaluations on
121. 11:22top of the complete session too. So let's see how  it can work. We're going to imitate the dialogue
122. 11:28which follows the same topics which we did, but  this time we have an exchange, right? So user
123. 11:33asks something, assistant asks something,  and so on. So let's create this data set,
124. 11:39and now we can run the evaluation. In this case  we created a dataset where we basically squished
125. 11:44the whole conversation inside a single cell  in our data frame, and we are asking LLM to
126. 11:49judge this complete conversation. We're going  to use a couple of built-in descriptors just
127. 11:54to make it quicker. One descriptor is looking  at declines - so if at any point there is a
128. 12:00denial to respond or to provide something, this  might be interesting for us to review. And then
129. 12:05we can also look at the PII LLM eval - so this is  a check which checks if there are any personally
130. 12:13identifiable information like names, addresses,  card numbers, and so on inside the conversation.
131. 12:18Both could be useful proxy metrics if you're  monitoring something in production. In this case
132. 12:23we might want to flag the responses like this for  manual review, right? If there was some refusal
133. 12:28or there was some personal information, we might  want to review what happened. So now I'm running
134. 12:35this complete evaluation, and I'm also sending  the results to Evidently Cloud because this time
135. 12:38the responses are pretty long, so it might be  more interesting to look at them in the UI which
136. 12:43allows to browse them. So now if we go back to  our project that we already created, we can see
137. 12:49a new report that was being added. And we can see  that, first of all, we actually had the declines
138. 12:54inside our conversation. And second, we even have  a PII instance inside our conversation. So now we
139. 13:00can look at the data set to explore what happens.  I'm going to hide here the actual distributions,
140. 13:05and now we can look at the specific scores that  we get. So we can, for example, sort by declines,
141. 13:11and we can find a conversation where the assistant  basically said like, “Sorry, I cannot provide this
142. 13:17information.” That’s interesting to look  at. And then we can similarly sort by PII,
143. 13:22and we can notice the conversation where this  PII reveal happened. And this is exactly the
144. 13:27same that the judge picked up on. Of course,  you can also do this similar evaluation in your
145. 13:32Python environment, where you would basically  just post-process your pandas dataframe and
146. 13:37search for the specific instances that you want  to look at. Now if you want to look around,
147. 13:43I'm linking more examples to how you can try  different descriptors, customize LLM judges,
148. 13:49create custom Python functions, and so on. But  in the following videos, we're going to show a
149. 13:54lot more interesting stuff. The next video will be  all about customizing LLM judges. And after that,
150. 13:59it will be all about building and evaluating  specific LLM applications. See you there.