YT YouTube2Text

YouTube2Text → Transcripts → From checklist to judgment: See the new Ivo Review in action

From checklist to judgment: See the new Ivo Review in action — Transcript

by Ivo · 10,294 words · 1,622 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:01Cool, okay. It's a few minutes past the
0:03clock and so um I think it's good to get
0:05started. Um so wanted to do some
0:09introductions. So I'm Ling. I lead the
0:13team of legal engineers here at Ivor
0:16um and I'll be moderating the session
0:17today.
0:19Um I'm also joined by my colleagues here
0:21and I'll pass the mic uh onto them to
0:23introduce themselves.
0:27Hi everyone. Uh very nice to to see uh
0:31so many people here. I believe uh many
0:33of you might I I might have met already,
0:35but for those of you who haven't, I'm
0:37Min Kyu I'm the co-founder here at Ivor
0:39uh and extremely excited to show you all
0:41the Review 2.0 today. So thanks for
0:44attending and I hope you find it
0:46interesting.
0:48Hey everyone. My name's Tom. Very glad
0:50to be here. So at Ivor I sit at the
0:52intersection of legal engineering and
0:54in-house counsel. Thank you.
1:00Cool. So um a few housekeeping matters.
1:03We are recording today's session and
1:05we'll share the slides and the
1:06recordings afterwards.
1:08We have about 60 minutes together um
1:11with the last 15 minutes reserved for
1:13Q&A. But feel free to drop your
1:15questions in the chat at any time. Um
1:17we'll make sure to get to them.
1:20If you're not actively speaking today,
1:22um uh please stay on mute.
1:25Um yeah, and in terms of agenda, here's
1:28uh how we'll spend uh
1:30our together. Um
1:32Tom's going to start us off with a
1:34question that I think maybe all most all
1:36of you share. What makes a good contract
1:38redline? Um and then we'll show you some
1:41real examples from our recent
1:42benchmarking study.
1:44And then Min Kyu will then introduce
1:45Review 2.0, um what it is, what is new,
1:48and how it's different from anything
1:50else uh on the market. And then we'll
1:53dive uh into a live demo and close out
1:56with a open Q&A.
1:58All right, Tom. Over to you.
2:01Sounds good. Thank you, Elaine.
2:03So,
2:05before diving into Review 2.0, I want to
2:08ponder on this very, very basic
2:10question, but it's very important to get
2:12this right. That is, what makes a good
2:14redline? And um nowadays, almost all the
2:18legal AI tools or almost all the AI
2:20tools in the market claim they have some
2:22kind of redlining capabilities. But,
2:25what is the difference between all the
2:27redlines they produce, and how do we
2:29evaluate what is a good redline? And
2:32that's the whole purpose of our
2:33benchmark study.
2:36Um so, we did this benchmarking study to
2:39compare Evisort, which is representing a
2:42purpose-built legal AI tool, and Clause,
2:46um that's representing a generic generic
2:49AI tool, and human attorneys. We have 19
2:52real-world anonymized contracts,
2:54including NDAs, MSAs, DPAs, terms and
2:58conditions, ranging from five pages to
3:01more than 30 pages. And we have three
3:03playbooks. And we gave all the materials
3:06to the three participants with minimal
3:08instruction and prompting. And then we
3:11have three judges coming from big law
3:13and in-house background with their rich
3:16experience with commercial agreements.
3:18And then we scored the outputs in a
3:20scale of 1 to 10.
3:26And
3:29and in terms of answering the question,
3:31what is a good redline? This is our
3:34rubric, and this is what we think is the
3:36most important. Um first, issue
3:38spotting. Did the reviewer catch all the
3:41issues that the playbook flags? And just
3:44as importantly, is it under issue issue
3:47flagging? Did they avoid raising things
3:49that I don't apply to the contract.
3:52Um second, surgical editing. When you do
3:55make a change, are you making the
3:57minimum precise edits needed? If the
4:00change only requires to insert five
4:02words, are you instead deleting the
4:05whole paragraph and reinserting the
4:06paragraph?
4:08And formatting retention. Um does the
4:10redline respect the numbering,
4:12cross-references, defined terms in the
4:14document still?
4:16Commenting. If the playbook calls out
4:19for a default external comment, did the
4:21reviewer attach the right comment in the
4:24right place?
4:25And finally, which I think is the most
4:27interesting and the most challenging
4:29one, is called judgment. When the
4:31playbook is ambiguous, or if the
4:33playbook contains conflicting rules, did
4:36the reviewer make the right judgment
4:38call?
4:39So this is our rubric, and this is what
4:41we think will make a good redline, and
4:44we will evaluate the output based on
4:46those five rubrics.
4:50And here's the results. So um in a
4:53summary, um I fell scored at 4.52
4:57in average across all five categories.
5:00And the human attorney scored 4.56.
5:04And Claude scored 3.50.
5:07And just as important as the score is
5:09the speed. Um for I fell, I fell spends
5:13about 2 minutes and 45 seconds in
5:15average on each contract. That's
5:18including the really short contract and
5:20the relatively longer ones.
5:23And the human attorney spent 10 hours in
5:26total for all 19 contracts. So that's
5:29about 30 minutes in average for each
5:31one.
5:32And Claude spent about 5 minutes each
5:35contract.
5:36So I think there are a few things super
5:38interesting about this. One is that we
5:41all know legal AIs and AI tools in
5:43general are fast. But when combining the
5:47speed and the quality together, it's
5:49really um satisfying and surprising to
5:52see that um I thought it's basically
5:54indistinguishable with human attorneys.
5:57And this is a blind test.
5:59In second, the point of the study is not
6:02to say Claude is bad. We generally think
6:05Claude is a very capable model and it's
6:07the best model behind a lot of the legal
6:09AI tools in the market. But, the point
6:11is that when a user does not want to
6:15learn how to develop a skill or how to
6:18write a really good and long and
6:20complicated prompt, can the tool still
6:22perform and give a good redline in that
6:24case? And the answer from our study is
6:28that the purpose-built
6:30the purpose-built legal AI tools already
6:32have a lot of hardcoding and harness in
6:35the backend that's kind of already think
6:38about what makes a good redline. And
6:40therefore, we have already taken all
6:42those um aspects into consideration. And
6:45when a user does not want to do a prompt
6:49each time or develop a skill each time,
6:51they can trust a purpose-built AI tool
6:55compared to a general LLM.
6:58So, going back to the five categories,
7:01um if you remember then issues spotting,
7:03judgment, comments, form retention, and
7:06surgical editing. Um in summary, Ivo
7:10outperformed Claude in each of the five
7:12category. And Ivo um
7:15is the ranked number one for surgical
7:17editing and also judgment. So, I want to
7:21show you some examples.
7:24And all the results and all the
7:26examples, you can see and download from
7:29our website.
7:30Um part of the core principle of the
7:33whole the whole um study is
7:34transparency.
7:37And the first example is about surgical
7:39editing.
7:40And um what you see on the screen here
7:43is I thought performance. So, there are
7:46two things that I thought added to this
7:48assignment section. One is an affiliate
7:51carve out, and the other is um,
7:55the other is a competitor and the
7:57competitor
7:59restriction and a notice requirement.
8:02So, you can see here I thought it the
8:04two very precise and clean insertion to
8:07the existing sentence without really
8:09deleting or restructuring any of the
8:11original text.
8:13And the next one is Claude's output.
8:16So,
8:18Claude's completely deleted the existing
8:21M&A carve out and replaced with a new
8:24M&A carve out that includes competitor
8:27restriction and a 30-day notice.
8:30Um, it also missed the affiliate carve
8:32out, but just on the surgical editing
8:35ground. Um, the whole deleting and
8:37reinserting is confusing and might waste
8:41a lot of the redlining um, capital.
8:46And third, this is the human attorney's
8:48output. So, it is definitely cleaner
8:51than the Claude's output. However, um,
8:54if you just judge the human attorney's
8:57output uh, comparing to I thought's
8:59output, you'll probably notice that the
9:02human attorney's output might be a
9:04little bit more redundant. Um, and in
9:07addition, it missed the affiliate carve
9:09out required by the playbook.
9:13And this is, I think, the most
9:15interesting one. It's about judgment.
9:18And for this example, the playbook have
9:21some ambiguous and conflicting
9:23requirements. So, the playbook prefers
9:26Delaware or California as governing law.
9:28But, if the the counterparty has a nexus
9:32with any reasonable US state, then we
9:35can also accept that as the governing
9:38law. So, in this agreement, the
9:41counterparty is actually a Wisconsin
9:43company, and therefore Apple recognize
9:46that connection to Wisconsin and
9:49accepted the governing law of state of
9:51Wisconsin.
9:54In Cloud example, Cloud followed the
9:56first part of the requirement, which is
9:59preferring Delaware or California as the
10:01governing law, and it deleted Wisconsin
10:03at all. And in that case, it's actually
10:07um not following the playbook
10:09instruction and did not make the judge
10:11make the right judgment call.
10:14And for the human attorney, um the same
10:17he they deleted um the
10:20they deleted the uh governing law of
10:22Wisconsin and replaced it with
10:24California, which is the preferred one,
10:27but they ignored the the second part
10:30that our playbook actually accept any
10:32reasonable US state where the
10:34counterparty has an access. And they
10:36also included a lot of the arbitration
10:39clause, but what what's in the playbook
10:42is that arbitration is acceptable if the
10:46counterparty proposed it, but we do not
10:48want to um proactively include it in the
10:51um in the contract.
10:54So, in a nutshell, um we we're committed
10:58to do more evaluation and um
11:00benchmarking in the future, and this is
11:02just our first study, and um we will
11:05welcome any feedback you have.
11:08Thank you, and over to Ming Q.
11:12All right. Um thank you very much, Tom.
11:16Uh so, we're going to do a couple of
11:17things. We're going to spend most of the
11:18time in uh a demo of the tool, but just
11:21before we get into a demo, I wanted to
11:23spend a little bit of time explaining
11:25why we released this feature in the
11:27first place, uh what was wrong with
11:29review 1.0, um why did we uh uh
11:33uh decide to spend a bunch of time and
11:35energy,
11:36uh, into into building out this
11:38completely new,
11:39um, architecture? And perhaps the way to
11:42think about it is when we started, uh,
11:44with this tool,
11:46uh,
11:48we built the original version of the
11:49product a very long time ago. We were
11:50probably the the very first, uh, AI
11:53redlining solution. I would at least
11:56first generative AI AI redlining
11:58solution. And this chart over here is a
12:00famous chart many of you might be
12:02familiar with. Uh, it shows the
12:05capabilities of the models over time,
12:07uh, as measured by the time horizon of
12:09software tasks that different LLMs can
12:11complete 50% of the time.
12:14And at the time that that we, uh,
12:17built the original contract review
12:18product, it was somewhere between GPT
12:203.5 and GPT-4.
12:23And, uh, the the progress of the models
12:25has been incredibly
12:27explosive. You can see over here, uh,
12:29over this 4.6. Uh, the the green dot
12:32dotted line here is going to where you
12:33might expect progress to be and you can
12:35see that, uh, especially just over the
12:37last year or so, um, progress, uh, with
12:40these models has, uh, outpaced where
12:43expectations were.
12:44And, um, the this is reflected in the
12:46way we built the product originally.
12:48When we built Review 1.0, uh, we were
12:50working under a lot of constraints. So,
12:52the state of the art model at the time,
12:54GPT-4, had, uh,
12:56a quote,
12:57context window of 8,000 tokens. Only
13:00about half of those were actually
13:01usable. It had this problem of skim
13:03reading. So, it turns out on contracts
13:05that, uh, words really matter. So, if
13:08you skim read past a word like not, uh,
13:10that that that can be pretty important
13:12even if it's a single word. And you'd
13:14also have this problem with compounding
13:15errors where every single, um, error you
13:18made somewhere in this long chain of,
13:21uh,
13:21queries across your pipeline, that would
13:23compound across every other step. So,
13:26even if, uh, sensibly you have a 99%
13:28accuracy rating, when you compound that
13:301% failure rate across 10 or 20 steps,
13:33uh you can end up with an unusable uh
13:35product.
13:36So, the way that we try to get around
13:38this problem was we tried to use the
13:40LLMs as little as possible. And the way
13:42we did that is we created very uh rigid
13:44tight uh pipelines that uh gave kind of
13:47uh very minimal instructions uh that
13:49that we were there was very little room
13:51for the LLM to deviate from that
13:53instruction.
13:54For me, the analogy is imagine you had a
13:57an intern join your company,
14:00and the intern is very prone to making
14:01mistakes, but you still want the intern
14:03to be productive at creating some sort
14:05of reliable work product. Well, the way
14:08you might work with them is giving them
14:09a very prescriptive set of instructions.
14:12And instructions are formulated in such
14:13a way that even if the intern has a
14:15tendency to make a lot of mistakes, as
14:17long as they follow the instructions, um
14:19uh uh you know, they kind of exercise
14:21their creativity, they kind of exercise
14:22their judgment, uh they they there's
14:24very little they can do, but at least
14:26you are going to get a usable output. Um
14:29uh so, the way we built the architecture
14:31was that we had uh constant playbooks
14:33that represent your preferred positions
14:35that were encoded into what we called
14:37checklist items. And we very rigidly and
14:39independently go through each checklist
14:41item one by one. So, you might have a
14:43checklist item for insurance, or you
14:45might have 10 different checklist items
14:47for insurance, you might have a
14:48checklist item for your warranty
14:50provision, and so on and so forth. You
14:51might have a hundred different checklist
14:52items. And even though the analogy was,
14:55"Hey, I was kind of like a a junior
14:57lawyer." It might be more accurate to
14:59think of it as I was like an army of a
15:02hundred junior lawyers who are all
15:04independently reviewing uh each item in
15:07your playbook uh without any knowledge
15:09of what the other uh lawyers are doing.
15:12Um so, you know, there there are a few
15:14problems with this. Uh one problem is
15:16that creating and maintaining checklist
15:17items is really tedious. Um checklist
15:19items are rigid. Uh in practice, uh
15:22playbooks tend to be pretty fluid. You
15:23have fallback positions, and under
15:26certain circumstances, you want to
15:27surface those fallback positions. Um I
15:29have no way of doing that. You'd have to
15:31manually decide which fallback position
15:33you wanted to use. Um progress on
15:35different checklist items wasn't shared
15:37between the agents. Um and there were
15:39also a number of other issues as well.
15:40And maybe the the core issue that we saw
15:43is that when you when you negotiate with
15:45a counterparty, there's a variety of
15:47context you want to take into
15:48consideration. It's very rare that
15:50you're just rigidly
15:51uh uh
15:52you're prescribing your your playbook
15:54one item at a time or your standard
15:56preferred position. You usually have to
15:57you have to think about what happened in
15:59previous turns and negotiations with the
16:00counterparty. What are the time
16:02constraints we have? What is the
16:03relative negotiating leverage we have?
16:06You know, what time is it in the year?
16:07Maybe it's the end of the quarter and
16:08your sales team are really pushing you
16:09to get these deals done. Uh what is the
16:12market standard for this particular
16:13position?
16:14Uh so, all of this context is really
16:16important. And when you think about the
16:18bottleneck for uh a lot of our um
16:22a lot of our customers when they think
16:24about having good contract review, the
16:26bottleneck isn't intelligence.
16:27Intelligence is part of it, but the
16:29other bottleneck is
16:30uh is the context about your business
16:33and informs the kind of recommendations
16:34that we want to generate.
16:36Uh Mary asked a Mary Karan asked a
16:38question, "Can you use Review without a
16:40playbook?" Uh yes, you can. Uh and we
16:42we'll talk a little bit about that in a
16:45in in a moment. Uh but the short version
16:47is because we're looking at a variety of
16:48different context sources, we're not
16:50just looking at your playbook. We're
16:51looking at other sources as well to
16:53inform the recommendations.
16:56Um so,
16:57if if previously we were hundreds of
16:59smart junior lawyers, each responsible
17:01for small tasks, now you can think of
17:03Review 2.0 kind of like a team of
17:05specialist lawyers, uh specialist
17:07experienced lawyers who have broad uh
17:10responsibilities. And we took
17:11inspiration from bureaucracy,
17:14and we created this architecture where
17:16you have an orchestrator agent who
17:18understands the playbook, understands
17:20the variety of different context sources
17:21you have, and has a high-level overview
17:24of the document. We created our own data
17:25structure that the orchestrator agent
17:27passes through.
17:28Um and then there are a variety of sub
17:31agents that are responsible for
17:33specialized
17:35areas of responsibility. So we have the
17:36playbook agent, investigator agent, and
17:38instructor agent. And the most important
17:40element here is every agent always has
17:43visibility at any given time of what the
17:45other agent is working on. And that
17:47allows us to to surface
17:49uh you know, very holistic
17:50recommendations.
17:53Uh so, what does that mean? So, when it
17:55comes to context sources, we're looking
17:57at a variety of places. We're looking at
17:58your benchmarks. So, we're looking at
18:00your contract history, the way you've
18:01redlined agreements in the past. Uh
18:04we're looking at deal context, specific
18:06context that's relevant to the deal that
18:07you have in front of you. Uh we're
18:09looking at your playbooks, and you can
18:10run multiple playbooks simultaneously if
18:12you want to. And then finally, we're
18:14looking at uh external benchmarks. So,
18:17we're looking at anything that your
18:18playbook uh didn't cover, or anything
18:20that your benchmarks didn't cover uh
18:22against external market standards.
18:25So, with that in mind, I'd like to share
18:27my screen. Um so, I'm going to
18:30uh
18:30share my screen and get into an example
18:33uh inside Microsoft Word. So, I will The
18:36review tool lives inside Microsoft Word.
18:38Uh we also have viewer doc support uh if
18:40you prefer that as well.
18:42But the example in front of us is we
18:43have an inbound services agreement. Uh
18:46it's a medium-sized contract, 14 pages.
18:48Of course, we can accommodate any type
18:49of agreement uh and and any size. Uh
18:52that's fine. But here in this screen
18:54that we have in front of us, we have
18:55access to a variety of features. Uh and
18:58then and then and the main feature I
18:59want to focus on is the the review
19:00feature, of course. Um we do have of
19:02course the AI agent, and we have the
19:04ability to to pin uh these custom skills
19:07here as well. But the primary feature
19:10for today is the review tool. And the
19:12review tool is our kind of specialized
19:14flagship feature intended for reviewing
19:16and redlining contracts.
19:18And when I click this button, there'll
19:20be a few things. I'll just walk you
19:21through the anatomy of what you're
19:22looking at here. So, first, we will
19:24identify which party you are
19:25representing in the agreement. In the
19:27background here, we have a feature
19:29called company profile, where we're
19:30taking into consideration a variety of
19:32context about your
19:34the way your company operates. For
19:36example, we'll take into consideration
19:38any drafting guidance guidance that
19:41you've provided. If you have a style
19:42guide, maybe you follow the Ken Adams
19:44style across your firm, we can
19:46accommodate that as well. But the other
19:48other information we we keep track of
19:50includes details about your your
19:51companies
19:53your company and the various entities
19:56within your organization, and we can
19:58automatically detect the party that
20:00you're representing.
20:01The other thing you have the ability to
20:02do here is select which review source
20:05you want to use. So, you have three
20:06options. You have playbook,
20:08you can select which playbook you want
20:09to run the review against.
20:11You can select benchmarks, and this is
20:13where we benchmark against your
20:15historical contracts. I can select which
20:17room I think is appropriate as a
20:19comparison set. And then we also have
20:21risk flags. So, to Mary Caron's question
20:24earlier, this is where we have the
20:26option if we didn't have any playbooks,
20:27we can simply turn it off. Or the one
20:29thing actually if we do have time later,
20:31I'd like to show you how you can create
20:33playbooks automatically using the
20:35benchmarks functionality as well.
20:38I can also select between review a
20:40standard review and redline review.
20:42And I can also add additional documents
20:44if I want to. But then once I'm done, I
20:46click continue to deal context. This is
20:48an optional screen. If I want to, I can
20:50click the run review button here.
20:52But there are there are a number of
20:53things going on here. So, first of all,
20:55I can type in any information about my
20:57deal that could be relevant to this
20:59particular review. Again, this is
21:00completely optional, but I have the
21:02ability
21:03you know, think of this as being kind of
21:05analogous to working with a human
21:07colleague, where if you were to ask him
21:08to review a contract for you, maybe
21:10there's some information about that
21:11contract they should be aware of or
21:13about that deal they should be aware of
21:16uh before they start the review.
21:18The other thing we do is then for the
21:19benchmarks and the compared to market,
21:21you can decide what are the relevant
21:23parameters that you want to assess
21:25against. Um so you can decide uh across
21:28a number of different parameters. Uh
21:30obviously we we when we draw the
21:31comparison set
21:33uh against historical contracts or
21:34against the market, we want to make sure
21:36it's relevant to the contract you have
21:38in front of you. Um but otherwise, we'll
21:39just default some recommendations for
21:41you. And then the other thing here is
21:43you'll see here it says negotiation
21:45context. We'll give you a uh a number of
21:47questions for you to answer that will
21:48also inform the way we make our
21:50recommendations.
21:51This feeds into uh the the benchmarking
21:54and the playbooks as well, because when
21:56you have playbooks, often you have
21:57fallback positions and you'll have
21:59certain instructions for how those
22:01fallback uh uh positions should be
22:02accommodated. So for example, uh if the
22:05counterparty is based in the United
22:07Kingdom or the EU, maybe you want to
22:09service a certain uh position that is
22:11only relevant um uh
22:13if if if the counterparty is based in
22:15the UK or EU uh for GDPR reasons or
22:18whatever it is. Uh you see another
22:20question here, does the counterparty
22:21handle company's proprietary
22:23confidential personal data? Um if yes,
22:25that again will impact the kind of
22:26recommendations that we want to service
22:28for you as well. Uh is this a
22:30cross-border transaction? Let's just say
22:31no. Um does this agreement bundle uh
22:34different services? Let's say yes. Uh so
22:36based on these parameters, I'm going to
22:38click the run review button
22:40and we're going to review the agreement.
22:42Now, for those of you who are familiar
22:44with Review 1.0 and really any tool on
22:46the market, you'll remember that the way
22:48it works is we start streaming the
22:50results in for you one by one, so we
22:51start giving you recommendations as we
22:53go along. We don't do that with Review
22:552.0. With Review 2.0, we wait until the
22:59entire set of analysis the entire
23:02analysis is completed before we surface
23:04agreement recommendations to the user.
23:06And the reason we do that is because I
23:09mentioned earlier that we want to have
23:10take a holistic look at your entire
23:12contract and we want to have an
23:13understanding of what are the downstream
23:15implications that certain
23:17issues will have on other parts of the
23:19agreement. That that is only possible if
23:22we wait for the entire review to
23:24complete before we surface the results
23:26to the user.
23:27Um so I want to flag that that this is
23:29really important. If you want uh your
23:31tool to be able to give you a holistic
23:33set of recommendations,
23:34um you want it to wait until all of the
23:37reviews are done because you know,
23:39sometimes there's a very basic change, a
23:40change to a limitation of liability
23:42might have downstream implications to
23:44your indemnity provision or a change to
23:47a definition or a cross-reference
23:48somewhere in your agreement might then
23:50also have downstream implications
23:52elsewhere in your document as well. So,
23:54over here I have an example that I've uh
23:56preloaded.
23:58Um
23:58I'll just answer a couple of questions
24:00in the meantime. Am I disadvantaged at
24:02all by creating a playbook outside of
24:04the Icertis playbook tool? Does it
24:06create any integration issues when I
24:07upload my playbook? Uh not at all. Um so
24:10again, one of the one of the changes
24:11from review 1.0 to review 2.0 is with
24:14review 1.0, you had to be very
24:16prescriptive with conforming to our
24:19structure for what what what a playbook
24:20looks like. Uh with review 2.0, you can
24:23be much more flexible. Um
24:25you can you can upload your document in
24:26whatever form it is.
24:28Icertis will turn it into a markdown
24:29file uh and we can read it that way. Uh
24:32there's another question, do all old
24:34playbooks in Icertis still work with the
24:35new version? Not by default, um but if
24:39you go into your playbook settings,
24:40there's a button you can click to
24:41migrate them to review 2.0 and it will
24:43happen automatically.
24:45Uh thank you. We also have a question
24:47from Lou. Where is the benchmark
24:48information being pulled from? Can
24:50Icertis now connect to an existing
24:52contract repository? Exactly. Yes,
24:54exactly. And I'll I'll I'll I'll get to
24:56the I'll I'll uh come back to that
24:58question in a moment cuz I'm going to
24:59spend quite a bit of time
25:01on that question
25:03as we go through the demo.
25:04All right. So, as you can see in front
25:06of you, we surface a number of
25:07recommendations. And these
25:10recommendations, to be clear, these are
25:11not directly tied to your playbook. So,
25:13if I go into my sources, I can see my
25:15playbook here. So, those who are
25:16familiar with Review 1.0 or really any
25:19other tool in our market, they'll go
25:22through each of the checklist items one
25:23by one and pass and say pass or fail.
25:25With Ivor, we we kind of call this these
25:28recommendations into buckets. We call
25:30them positions. And the reason we do
25:32that is let's say you have 10 different
25:34requirements for how you negotiate
25:36insurance provisions. We don't want to
25:38give you 10 different recommendations.
25:39We want to give you a single graceful
25:41recommendation that encompasses all of
25:42those different sources. The other thing
25:46you'll notice is that we have these
25:47different badges here. We have the
25:48badges that say playbook, benchmark, and
25:51risk flag. And the badges denote what
25:54sources are we using to derive these
25:56recommendations. And often we'll be
25:58using a combination of the three. But
26:00sometimes, like in this fees and payment
26:03issue, we're just looking at your
26:04playbook, we're just looking at your
26:05risk flag, or just looking at your
26:06benchmark.
26:08Okay. So, let's click confidentiality
26:10and proprietary rights as a starting
26:12point. You can see we made a number of
26:14recommendations.
26:16And the explanation for the first
26:17recommendation is per playbook position
26:19P3, the permitted disclosures must
26:22include employees, affiliates, agents,
26:24professional advisers, etc. And number
26:26two, since the governing law is now
26:28England and Wales, confidentiality
26:30obligations must expressly extend to
26:32former employees and contractors who
26:33have since left the organization.
26:36Nice little touch here, by the way,
26:37because this is a UK government
26:39agreement. We've said we've used S for
26:43organization to use the the British
26:45spelling rather than the the American
26:46one.
26:48So, we now generate the recommendation.
26:50I want to flag here, by the way, this is
26:52an example of where the deal context is
26:53helpful, because we know, because of the
26:55user input at the beginning, because we
26:57know that
26:58um uh the counterparty uh is uh based in
27:02the UK or Europe, uh and because we know
27:04that the governing law is England and
27:05Wales, that impacts the way we generate
27:07the recommendations.
27:09Um so, when we generate the
27:11recommendations, uh you can see that uh
27:13we surface the redline. I can come in
27:15here and make changes if I want to.
27:17I can type in an instruction to to
27:19update the redline as well. I can also
27:21add a comment to justify the change to
27:23the counterparty.
27:24Uh this is This should all be familiar
27:27to you if you uh familiar with Review
27:291.0. Uh that part of the functionality
27:31is exactly the same.
27:32But the the difference here is if you go
27:34to the sources, you'll see that there
27:35are now a number of different sources.
27:37So, we've derived this recommendation,
27:38first of all, from our playbook. So,
27:40these are the two playbook requirements
27:41that weren't met. And then also here
27:44from our benchmarks.
27:46And in a moment uh I'll come back later
27:48to this. I'll show you how I can really
27:50drill down very deeply into each of
27:52these
27:53uh recommendations. Toby Toby asked a
27:55question, is risk flag a flag from
27:57Market Standards as opposed to our
27:59contracts or playbooks? Correct. Yeah,
28:01correct. Um the way I think about it is
28:04sometimes you won't have a playbook or
28:05your playbook uh won't contemplate every
28:07single thing that could appear in a
28:09contract. If the counterparty says, "You
28:12must deliver the services on the on the
28:13back of a fire-breathing dragon." We'll
28:15have no way of capturing that in a
28:17playbook. Uh so, we have uh the risk
28:19flags uh as a backup. Uh and also
28:22benchmarks, of course, as well, but uh
28:24your historical contracts probably also
28:26haven't contemplated that particular
28:28scenario as well.
28:31All right. So, uh this is the way we
28:32generate the recommendations, and you
28:34can see we've generated a number of
28:35recommendations. And the way we do it
28:37this way, again, is because we want to
28:40review the entire
28:41uh confidentiality provision. And uh all
28:45all these uh recommendations we surface
28:47touch the other recommendations we
28:49surface in this in this set of
28:51recommendations. So, I'm going to go
28:53ahead and click apply and then we'll
28:55apply all of these redlines into into
28:57the agreement one by one like this.
29:00One little detail you'll notice as we do
29:02this is Review 2.0 is uh surprisingly
29:05good at maintaining the formatting uh of
29:08the document. Uh we're still not
29:10perfect. Uh we this was actually the
29:12main area where the human lawyer
29:13significantly outperformed our rival,
29:15but we significantly outperformed of
29:17course the generic AI legal tool. I I
29:20always joke that the last frontier for
29:22AGI is if AI can figure out how to
29:24format Microsoft Word documents. I'm
29:26going to actually not 100% sure if it's
29:28a joke cuz it's an incredibly uh
29:29challenging uh technical problem.
29:31Microsoft Word is a uh is it is a
29:38It's an interesting piece of software.
29:39At this point it's more like a living
29:40organism than it is uh software, but you
29:42can see that uh a lot of the work that
29:44we've done on top of the model is uh
29:46just details like this. We want to
29:48preserve the formatting here of the
29:49heading. We want to include the the
29:51period at the end. When we insert the
29:53the publicity provision, we want to
29:55preserve the the formatting. Uh one
29:57little I guess thing I don't like here
29:58is we didn't uh increment the the
30:00number. Usually we're pretty good at
30:02this, uh but the reason for that is this
30:044.6 isn't a list. Uh so, if it isn't
30:08formatted as a list, sometimes I will
30:10miss that. But, still quite good from a
30:13formatting perspective.
30:15Anyway, so we can keep In this way we
30:16can keep going through all of the
30:17recommendations one by one. Um maybe
30:20I'll just give you one more example for
30:21the sake of uh example. So, if I go to
30:23Fees and Payment, this is an example of
30:26a recommendation that's purely derived
30:28from our playbook. So, in this case
30:30we've said that our playbook uh position
30:32requires itemized invoices submitted to
30:34client's online portal, uh requires
30:36monthly invoicing, uh send uh and we
30:39need payment terms to be uh 60 you know,
30:4160 rather than get 30. So, we made a
30:43number of recommendations to accommodate
30:45that. And again, you can see this is a
30:48lot of the work we did did through our
30:50call listing algorithm is we want to
30:51call this a redline in such a way that
30:53they look very surgical and precise to
30:57the counterparty so that they don't get
30:59mad at us for redlining the agreement
31:02too much.
31:04All right. So, that is the core
31:05functionality here. In this way, we can
31:06go through each of the positions one by
31:08one inside applying the the redlines.
31:11Now, I think
31:12what I want to do is I want to dive a
31:14little bit deeper into the sources
31:16cuz part of what we're trying to do here
31:18is we're trying to ground all of our
31:19recommendations on on real data. And one
31:22of the ways we do this is if you go into
31:24benchmarks, you'll see that we show you
31:26a distribution of all of the agreements
31:28you've agreed to that we consider to be
31:29analogous to this current agreement. And
31:32then we'll we'll benchmark where your
31:34current agreement falls or the
31:35counterparty's agreement falls relative
31:37to the other agreements you negotiated.
31:38So, in this case, this agreement is
31:39already an 80th percentile agreement, so
31:41it's actually already pretty favorable
31:42to us. So, this you know, should
31:44actually impact probably the way that
31:46you want to negotiate this contract. If
31:48this is already favorable compared to
31:49your other SaaS agreements, and this is
31:52the sixth turn of negotiations, you
31:53know, maybe you just want to let it go
31:55and you want to accept the document
31:57because the counterparty has been pretty
31:59reasonable.
32:00Um
32:01I In fact, click overall market
32:02position, you can see the comparisons
32:04that that we are comparing the document
32:05against.
32:06The next thing we can do is we can drill
32:08down deeper. So, every single issue that
32:10we raise for you from a benchmarking
32:12perspective, we'll show you why we've
32:14raised it as a
32:15as a recommendation. So, for example, if
32:17I go down here to this particular issue
32:19that I've always flagged, we said,
32:21"Well, the return destruction period for
32:23counterparty materials in days
32:26should be 30 days. 30 days is our most
32:28commonly held position."
32:30And because it isn't 30 days, we are
32:31going to make a recommendation. If I go
32:33to the next item, you can see another
32:35another issue, which is is reverse
32:37engineering prohibited? In most cases,
32:39yes.
32:41Otherwise, it's not specified. And this
32:43requirement actually is already met.
32:46So, that we haven't made
32:49recommendation over here. What about our
32:50security breach notification period in
32:52days? Here's another recommendation. Or
32:54what about
32:55standard of care applicable to
32:57counterparty's performance? Most of the
32:58time it isn't specified. So, we seem to
33:00be okay with this position. And the
33:02other other interesting thing about this
33:04is with all of these recommendations, if
33:06I click the view in repository button,
33:10it'll take me to the specific clause
33:11language as well. Sorry, let me let me
33:14come back to that. I just realized
33:16I lost my page. So, I'll open we do we
33:18retain pre-existing IP? Yes or no. If I
33:21click view in repository, you won't be
33:23able to see my screen now because it's
33:24opened a new window. So, I'm going to
33:26swap to the contract intelligence tool
33:29over here.
33:30So, you can see here this is what it
33:31looks like. This is a list of all of the
33:33agreements that we're using as the basis
33:35of comparison. And you can see that I
33:37have always added this column here, do
33:39we retain pre-existing IP? Yes or no or
33:42not applicable. If I click into any of
33:44these contracts, and remember these are
33:46our historical contracts, and I open
33:48this particular source and I click on
33:50this button, it will take me to the
33:51relevant passage in the agreement. And
33:52you can see here it says very clearly,
33:54company retains all rights, title, and
33:56interest in and to the platform and all
33:58improvements, including those resulting
33:59from professional services. So, we have
34:01the ability to drill down all the way to
34:03the specific contract language that
34:06we're using to justify this particular
34:08recommendation to you when we surface
34:10the recommendation.
34:12Now, my guess is the vast majority of
34:13the time you don't want to do this cuz
34:15it's it takes a lot of time to be
34:16drilling in every single for every
34:18single recommendation. But if you're
34:20particularly scrupulous, particularly
34:21careful, you might want to do that. Or
34:23maybe you see a recommendation that I
34:25was in a race and you're like, "Huh,
34:26that that seems off. It seems really
34:29interesting to me that
34:30the data retention post termination of
34:34counterparty is 30 days. I thought it
34:37was more like 60 days, so let me just
34:38double check myself. And then if I go
34:41back to the repository view that's just
34:44opened up for me,
34:45you'll see that I can see a list of all
34:47of these agreements and I can actually
34:49just go into them and just double check
34:51if my intuition is correct or not. And
34:53like, okay, I mean this seems
34:54reasonable. Customer may request
34:55deletion within 30 days after
34:56termination.
34:58And if I just look through all of these
35:00agreements, actually this does look like
35:02a pretty reasonable recommendation for
35:04us to surface.
35:08All right. So, that is the way the
35:10benchmarking feature works. So, the
35:12benchmarking feature does require access
35:15to the intelligence tool.
35:16But one of the things we want to do,
35:19I'm not sure if I'm allowed to say this,
35:20but one of the
35:21one of the things that we want to do is
35:24just give every Iuvo customer access to
35:26a set of contracts,
35:29something like around 200 documents or
35:31so that they can use just as part of
35:32their Iuvo license.
35:34What that will allow you to do is you
35:35can upload a set of contracts that you
35:38want to use for benchmarking purposes.
35:41If you want to try it out, talk to your
35:44customer success manager and I'm sure
35:45they'll
35:47see if they can accommodate the request.
35:50But this in our opinion is a very
35:52powerful feature. Now, we do something
35:54similar for risk flags.
35:56One thing we don't do at the moment is
35:58drill into the specific publicly
36:01available contract or set of publicly
36:03available contracts that we're using to
36:05source the recommendation, but that is
36:07coming next on our road map. So, they'll
36:08work exactly the same as what I showed
36:10you in the benchmarks feature. We'll
36:12give you a breakdown of what we've seen
36:13across the publicly available contracts
36:16that that we are benchmarking against.
36:19For example, if it's the re-retain
36:21existing IP issue, we'll show you how
36:24your contracts and how the
36:25counterparty's contract compares against
36:28the set of historical agreements we
36:29have, we'll give you a similar
36:31breakdown, and then again, you'll have
36:33the option to view all of the specific
36:35agreements. So, let's say for example, I
36:37don't know, um
36:38McDonald's has agreed to uh something
36:41that seems very different to what you'd
36:44agree to in the past, um you can click
36:46into this button and in the same way
36:47that I showed you earlier, you can see
36:49the specific contract language that they
36:50used
36:51uh as part of uh whatever uh public
36:55disclosure filing they had to make uh in
36:57the last year or two or whatever set of
36:59parameters you want to assign
37:02uh for the basis of that comparison.
37:06All right, so that covers most of what I
37:08what I wanted to go through. Um I
37:11I
37:12I I I I I I guess in summary, Review 2.0
37:14uh powerful for a few reasons. Number
37:17one is that it can express judgment, and
37:19the reason it can express judgment is
37:20because it has access to a variety of
37:22different uh context sources uh across
37:25your organization. Uh number two is that
37:28when it surfaces recommendations, you
37:29can drill down into the exact source
37:31language uh that we used to derive the
37:33recommendations, whether or not you have
37:35a playbook.
37:36Um and then number three, uh when we
37:38make the recommendations, we're looking
37:40at your entire agreement holistically
37:42rather than looking at each issue one at
37:44a time.
37:45One note on the playbook side is I do
37:47really want to emphasize that playbooks
37:49are just markdown files, so that we can
37:51accommodate literally whatever format
37:53you want for your playbooks. Um as long
37:55as they stipulate your positions in some
37:57way, we can capture that. Uh we can also
37:59create playbooks from your historical
38:01contracts as well. So, just to give you
38:03one example of this,
38:05um if I go into
38:08uh my web application,
38:10uh you can see that I've run a query
38:12over here. I've said, you know,
38:13"Benchmark all of my MSAs, give me a
38:15breakdown on my most commonly agreed
38:16positions and typical fallbacks." You
38:18can see that I've
38:19through the benchmarking functionality
38:21that I showed you earlier, it's capable
38:23of giving uh a very granular breakdown
38:25of everything that you've agreed to.
38:27Based on this information, we can
38:29generate
38:31a summary of your positions and your
38:33typical fallbacks. And then based on
38:35this, we can also generate a playbook
38:36for you in Microsoft Word format. And
38:38then it's simply a matter of uploading
38:40that playbook back into your
38:43Icertis
38:44for us to create a playbook for you
38:46based on your historical agreements as
38:47well. One note I'll make on that briefly
38:50is that the difference between playbooks
38:52and benchmarks is that playbooks are
38:54where you as an organization have a
38:55point of view on how you want to
38:57negotiate contracts going forward. So
38:58there there is an aspirational element
39:00to it. Whereas benchmarks reflect what
39:02you've actually agreed to.
39:04And those two things are highly
39:05correlated but are not
39:08they're not exactly the same thing.
39:12Fantastic. So I'll I'll I'll I'll pause
39:13there.
39:14Christian from Ryan, does Icertis
39:16preserve context across multiple rounds
39:18of redline exchanges?
39:21Yes and no. So one of the things that
39:25you can do with So so there there are
39:27two things. Number one is if you're
39:28preserving the redlines back and forth
39:30with a counterparty, Icertis
39:32Icertis can look at the date that those
39:35redlines were added. So therefore we
39:36have knowledge of how those redlines are
39:38evolving over time and we can use that
39:40to inform our recommendations.
39:42The other note here as well is you can
39:44upload additional documents for context
39:46when you run the review and so you can
39:48upload your previous contract previous
39:51contract versions as well.
39:54The reason I say no is we have another
39:56feature that we're working on.
39:59I I I can't share too much details about
40:00it just just yet.
40:02But part of what that feature will do is
40:04it'll automatically track versions back
40:07and forth with a counterparty but also
40:09versions internally.
40:10And when we have access to your versions
40:12automatically, that will mean there's no
40:14manual effort been
40:16We'll would automatically understand
40:17that, "Hey, you're in version six of
40:18this agreement. Um in in the second turn
40:21of negotiations, you already agreed to
40:23this governing law position, so we're
40:24we're not going to surface it for you
40:25again uh because we know that it's a
40:27settled issue already. Or maybe we know
40:29that because you're on the sixth round
40:30of negotiations, and typically your
40:32negotiations are done in two rounds,
40:34maybe you have um
40:36uh maybe you want to get this
40:37negotiation over with quickly. Uh so we
40:39will take that into consideration. Or
40:40maybe um you have a playbook rule that
40:43there are certain fallback positions
40:45you'll accommodate only if you're on the
40:46fourth or fifth or sixth uh round of
40:48negotiations. So we're we're better uh
40:51so we have kind of a
40:52basic primitive version of what you
40:53described today, uh but it'll get a lot
40:55more sophisticated uh over the over the
40:57coming months.
41:00Amazing. Thank you, Minkyu.
41:02Um so we've got about 15 minutes left
41:05for uh questions, so please um drop them
41:08in the chat or feel free to come off
41:09mute, and uh yeah, we'll work through as
41:12many as we can.
41:16Um while we're just waiting for
41:18questions to trickle in, um maybe
41:21something you can help us cover, Minkyu,
41:23is um
41:25do we need intelligence in uh order to
41:28use Review 2.0?
41:31Uh so uh
41:33a good question. So for people who are
41:34unaware, Intelligence is our
41:35post-signature solution where we extract
41:37intelligence at scale across your
41:39historical contracts,
41:40um and we can plug into your existing
41:42systems, or we can
41:44uh become the system of record if you if
41:46if if you prefer that option. Uh so
41:49Intelligence is completely optional. Uh
41:51Intelligence is necessary for benchmarks
41:53uh because otherwise we don't have a way
41:55of accessing your historical contracts,
41:57but the external benchmarking and the
41:59playbook functionality uh doesn't
42:01require the Intelligence tool. Uh and as
42:03I mentioned earlier, uh we do want to
42:05make a set of contracts via Intelligence
42:08available for every customer regardless
42:10of whether you have an Intelligence
42:11subscription or not. Uh so you can still
42:13take advantage of the benchmarking
42:15functionality.
42:16So again, if you do have an interest in
42:17that, uh please feel free to ping me or
42:20talk to your CSMs, and uh we'll see what
42:23we can do to accommodate.
42:32Uh so there was a question that just
42:33came in. How effective are assistant
42:36reviews without any playbooks?
42:38Um so I think the uh just for everyone
42:41else's benefit who who isn't familiar,
42:43assistant is our our agent uh
42:46functionality
42:47uh that is uh that is more kind of a
42:50chat interface where you can type in
42:52whatever questions you want. Uh
42:54assistant reviews without playbooks are
42:56good. Um
42:57it takes advantage of a lot of the the
42:59work that our team has done to to make
43:01Ivor effective at uh contract review
43:03problems. Uh it isn't as good as review
43:052.0. Um so the advantage of assistant uh
43:09you can you So to be clear, you can use
43:11review 2.0 without playbooks as well. So
43:13you can use it just for generic review.
43:15Um it doesn't perform as well as review
43:18in our benchmarks. Um
43:20where assistant is really useful is
43:22sometimes you want to review the
43:23agreement in a way that's just much more
43:25open-ended flexible, and sometimes you
43:27want to be able to go back and forth
43:28with the agent to iterate on the
43:30responses that we're we're surfacing for
43:31you.
43:32Uh and that uh
43:34uh
43:34assistant is a better fit for that
43:36particular use case.
43:39Amazing. Uh question from David. Does
43:41Ivor maintain its own file system of
43:43contract database structure
43:45uh
43:46or a contract database structure from
43:48which our company's previous agreements
43:50exist for use in new edits?
43:52Uh so the answer to that question is
43:54both. Uh you can just upload your
43:56contracts into Ivor if you want to. If
43:57you're only using Ivor for the purpose
43:59of benchmarks, and you only need say 200
44:01contracts, I would recommend you just do
44:03that cuz it's a more straightforward. Um
44:05but we can also plug into your existing
44:07system. So, if you have a Google Drive
44:09or or a SharePoint or a or a CLM that
44:11you're using to store and manage your
44:13contracts today, we can plug into those
44:15systems and we can act as the
44:16intelligence layer on top of them.
44:24Well, maybe something else we can touch
44:25on is uh the playbook implementation
44:28time for Review 2.0. How long does it
44:30take to set up?
44:33Uh so, it's much faster than Review 1.0.
44:36Uh so, Review 2.0 really all you need to
44:38do is upload your playbook uh and we
44:41will
44:42automatically process that into a
44:43playbook for you.
44:45Uh if you do uh want to create a
44:47playbook more manually, uh it's it's
44:49just a markdown file. So, you can just
44:51type in what your instructions are in
44:52plain plain English. You can say uh
44:56just the way that you would to a human
44:57colleague uh and IBO will be able to
44:59interpret that and pass that into uh a a
45:03IBO playbook for you.
45:05And as I mentioned earlier, we can also
45:07create the playbooks from your
45:08historical contracts as well because we
45:10have knowledge of your uh standard
45:11positions that you've previously agreed
45:13to and we have knowledge of common
45:14fallback positions as well.
45:22>> [clears throat]
45:22>> Uh Toby, can you please talk a little
45:24bit about IBO's AI solution and whether
45:27it's based on current market solutions,
45:30uh your own proprietary solution, or a
45:32combination of both?
45:33A combination. Uh we you know, the we
45:37uh rely on the foundation models for in
45:40large parts of our product. Um, there's
45:43also a lot of contract specific uh
45:45engineering work that has gone into to
45:47making the tool effective. Uh one of the
45:50things that we were trying to establish
45:52with the report that uh Tong went
45:54through earlier on this call is that the
45:56variation in the quality of the outputs
45:59between these different tools and maybe
46:01uh
46:02from from from our internal testing
46:04maybe 45% attributable attributable
46:07attributable to the underlying model
46:08itself. Um and actually the vast
46:11majority of the the quality of the the
46:13outputs and and the accuracy against our
46:15benchmarks
46:18decided by factors outside of the
46:21underlying model.
46:23Uh which is which is why we were we are
46:25able to achieve results that far surpass
46:28the underlying model providers
46:31including quality and including chat GPT
46:33etc.
46:35Um this is you know this is one of those
46:37things where I'd encourage you if you
46:38have access to Iver I just try running
46:40those side by side. That'll probably be
46:43more compelling to you than us telling
46:44you that and showing you our
46:47studies.
46:49But this is consistently what we've seen
46:50over and over again with our
46:52with the market. Uh there's a question
46:54is Ajarov a CLM Iver can plug into
46:57access contract database? Correct. We do
46:59have an Ajarov integration. We have
47:01integrations with most of the CLM
47:03providers. If if if if not today we will
47:07certainly very soon.
47:09And we also support indirect
47:10integration. So if you have a
47:12integration from your CLM into a cloud
47:15storage provider we can integrate into
47:17the cloud storage provider
47:19instead and that functions effectively
47:21the same way.
47:29Yes I Maybe to elaborate on Toby's
47:32question a little bit further as well.
47:33So
47:34there there are kind of like a three
47:36layers of things that you can do to make
47:39these products work really well for
47:42actually maybe four things that you make
47:44products work really well. Like the
47:45first one is you can do post training on
47:47top of the model. So there's a variety
47:50of approaches you can take to that but
47:53you can you can fine tune models to make
47:55them more effective at certain tasks.
47:58You know, what we're seeing in in
47:59actually in in a lot of domains outside
48:01of legal and I expect this will happen
48:02more commonly in legal as well is you
48:04see uh companies like Curvature, for
48:06example, in the dev tool space
48:08uh fine-tuning open-source Chinese
48:10models and they're able to achieve uh
48:12accuracy that's
48:14comparable or surpasses the the frontier
48:16models in accuracy across the domain
48:18that they care about. Um but in addition
48:21to that, there's also incredible latency
48:23benefits. So, um you know, uh Composer
48:252.5, which is Curvature's uh model of uh
48:29uh one of the uh open-source models,
48:31uh is much faster than Opus 4.7 uh while
48:35being comparable in performance. So,
48:36that's number one. Uh number two is
48:38there's a lot of algorithmic work you
48:39can do. So, there's algorithmic work you
48:41can do to make the redlines more
48:42surgical, to to handle all sorts of
48:45corner cases that appear when you're
48:46trying to redline an agreement
48:48accurately. Um uh it's it's uh
48:52uh
48:53there's just a very long tail of work
48:54that needs to happen to to make uh
48:56contracts give good or give make these
48:59tools give uh high-quality uh redlining
49:01outputs. The third thing you can do is
49:03you can create your own data structures
49:05that allow you to pass these agreements
49:07in such a way that they're more likely
49:09to give you good outputs when you pass
49:10them through an LLM.
49:12And in the fourth uh thing you can do is
49:14you can make sure you're passing the
49:15correct context into the model uh so
49:17that the model is um uh has access to
49:19the information that it needs to make
49:21good decisions. Um and part of what
49:22we've tried to do with number four uh in
49:24addition to all of that kind of the the
49:26basic basic engineering work around uh
49:28context engineering is also making sure
49:30we create that context for you. So,
49:32creating that context via benchmarks,
49:34via the the risk flags, via the uh deal
49:37context functionality we showed you
49:38earlier, uh and via your playbooks.
49:46Oh, Ryan's asked, um can I will
49:48facilitate internal delegation for
49:50escalated approvals? Not yet, uh but
49:52very soon. This is This is part of the
49:55functionality I was talking about
49:56earlier with preserving context across
49:58multiple rounds of redline exchanges,
50:00which
50:01you also asked that question, it looks
50:02like.
50:03We will be supporting
50:06that particular release will also
50:07support
50:08internal delegation for escalated
50:10approvals.
50:12I actually variety of other
50:15approval and workflow functionality
50:16beyond that as well.
50:25Excellent.
50:27One more question from Kim. How does 2.0
50:31handle significantly inconsistent
50:33historical positions?
50:36So, you know, that's a good question
50:38that comes up quite a bit. The The way
50:41we think about this is is in two ways.
50:43Number one is if you have a strong point
50:46of view on what
50:49how how I version handles certain cases
50:52or how lawyers within your organization
50:54or anybody within your organization
50:55should deal with certain scenarios.
50:57The way you should think about it is
51:00that that rule should be encoded in your
51:01playbook cuz a playbook always overrides
51:04the the benchmarks.
51:06The second comment I'll make is by
51:08definition benchmarks is looking at an
51:11aggregate of many of your contracts,
51:13which means that if there are outlier
51:14positions that you normally wouldn't
51:16agree to, but you agree to because
51:19it was really important for whatever
51:20whatever reason that's specific to that
51:22transaction, that won't be captured in
51:24the set of recommendations we surface
51:25because definitionally it is an outlier
51:28position. Now, if it happens to be the
51:29case there's one wrinkle to this, which
51:31is sometimes you'll have situations
51:33where our customers may have had a
51:35standard position in the past, and if
51:37you look at the totality of all of the
51:38agreements that previous position
51:41represents the most common position, but
51:43they've decided to move away from that.
51:46The the you'd handle that is within the
51:47set of parameters that you use to
51:49compare the benchmarks against, you'd
51:50want to limit it to agreements within a
51:52certain time period. Uh, I and that
51:55would handle the evolving business needs
51:57uh concern that you have.
51:59The way other customers handle this as
52:00well is they have a set of golden terms
52:03uh or golden uh documents that they use
52:05to inform the benchmarks. In my opinion,
52:08that's a little bit unnecessary. Uh, you
52:10already have playbooks. Uh, if you want
52:12to be really prescriptive, the purpose
52:13of the benchmarks is just to give you
52:15some visibility into what you've agreed
52:16to in the past and hopefully the people
52:18in your team will express good judgment
52:20in terms of how they apply it.
52:25Oh, I think we have time for maybe one
52:27more question. Um, so David, this whole
52:30idea of understanding deal context, how
52:32does I know how the urgency,
52:34profitability, or other non-legal
52:36aspects of an agreement relationship is
52:38for it to then introduce deal-oriented
52:40approaches to its context changes?
52:43Yeah, so again, I'm going to give two
52:44answers to this question, which is how
52:46we do this today and how we
52:48uh would do this in the future. And this
52:50actually ties into the uh the the
52:52roadmap item that Ryan's been talking
52:53about. So, for the first part of your
52:56question uh for for my first uh I I
52:58guess response to your question is
53:00at the moment, I would doesn't know by
53:02default. So, we assume that the user
53:04needs to provide that context for I
53:06would. So, the user uh if you remember
53:09that we went through that deal context
53:10demo where we showed you how there was a
53:12number of triggers that that would
53:13appear. Um, so the user would have the
53:15the ability to to choose between a
53:17variety of different options. And the
53:19user also has the ability to type in an
53:21instruction, the same way you would to a
53:23human colleague uh that um
53:25that would impact the negotiation. So,
53:28um you know, it's the end of the
53:29quarter. This is really urgent. We need
53:31to make sure that we get this deal
53:32across the line. Let's just make sure
53:34that we um
53:35uh as permissive as possible and we only
53:37uh flag really high-risk items. That's
53:39fine and that will impact the the
53:40recommendations. Now, the roadmap
53:42version of this is I I mentioned earlier
53:44that one of the things we'll do is we'll
53:46keep track of versions as agreements
53:47move through the negotiation life cycle.
53:49Part of the reason we're really excited
53:51about that is once we have an
53:52understanding of how agreements are
53:53moving through your draft versions, that
53:56data will also allow us to surface what
53:57we call position intelligence. So, with
53:59position intelligence, we're going to
54:01tell you things like, "Okay, so you have
54:02a position for not accepting order in
54:05your provisions in
54:07sales vendor agreements, well, we can
54:09show you that each time you negotiate
54:11this position, you're adding on average
54:122.4 days to your your your your deal
54:15cycle or 2.4 turns to your to your
54:18deals, and that turns out to be about
54:208.4 days,
54:22that and therefore we consider that to
54:24be a high friction position."
54:27When we surface these insights for you,
54:29we can then use those insights to more
54:31proactively make recommendations based
54:34on a deal from a deal context position.
54:36So, we can say,
54:38"Hey, we've we've noticed that you're on
54:40the sixth turn of negotiations with this
54:42counterparty, and now you experience
54:45this particular issue that you're
54:47negotiating can take 12 turns to
54:49negotiate because the counterparty is
54:51really don't like it. Are you sure you
54:53want to continue with this
54:56continue insisting on this particular
54:58position
54:59if it adds another six turns to your
55:01negotiation cycles?"
55:03I did a very poor job explaining that,
55:05but
55:07maybe maybe maybe the point I'm trying
55:08to get at is
55:09uh uh I will have a better sense of
55:12understanding deal context proactively
55:13without user input in the future.
55:17Uh
55:19Oh, sorry.
55:20Just maybe one last question from
55:22Marcelo. For newer regulations, how does
55:25Benchmark take these into account, and
55:26what information is used to make
55:28provision changes recommendations? Um
55:31So, Benchmark, I assume by this you're
55:33referring to the external benchmarks
55:34rather than the internal benchmarks cuz
55:36internal benchmarks are just looking at
55:38your historical documents. Um so, to the
55:40extent that regulations would change the
55:42way you should think about these
55:43negotiations, you should do two things.
55:45One is you should encode it into your
55:46playbook. And number two is you should
55:48limit the parameters for the comparison
55:50set to only documents after the date
55:53that new uh regulation came into effect.
55:56Um uh but uh
55:59uh from an external benchmark
56:00perspective, we just continuously up
56:02update uh we we continuously update uh
56:05the the data set. Uh so, it's always
56:08fresh.
56:09Um
56:12Cool. I think we are at time. Um and if
56:15there's any uh unanswered questions, we
56:17will um uh send out a follow-up email.
56:20But, thank you so much for uh everyone
56:23who's joined today. Uh thank you, Tom.
56:25Thank you, MunKee, for being our
56:26speakers. Um if you want to dig deeper
56:29on Review 2.0 specifically, please reach
56:31out to your uh CSM or um uh reach out to
56:35one of us, and we will um show you the
56:39Review 2.0 um and the product more in
56:42depth.
56:43Thank you so much, everyone. Really
56:44appreciate it, and please feel free to
56:46reach out to me uh
56:47directly if there's anything you want to
56:48ask uh or or share with me. Thank you.
56:51Take care. everyone.

About this transcript

This page contains the full transcript of From checklist to judgment: See the new Ivo Review in action by Ivo, generated from the public captions YouTube serves with the video. The transcript has 10,294 words across 1,622 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.