YT YouTube2Text

YouTube2Text → Transcripts → Image Decision Models for RPA: Forms, Scans and Screenshots

Image Decision Models for RPA: Forms, Scans and Screenshots — Transcript

by Sam Witteveen · 3,169 words · 433 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Okay, if you draw any business process
0:02as a flowchart, you get two main shapes.
0:05The rectangles are the steps. They're
0:08things like copy this, click that, file
0:10this. And the diamonds are the
0:13decisions. Does this look right? Is this
0:15an invoice? Is this a payment? For
0:18years, RPA tools have been automating
0:20the rectangles. That's robotic process
0:22automation. Software bots that click
0:25through screens and copy data between
0:27systems the way that a person would.
0:29Usually following fixed rules. But the
0:32diamonds still often go to a person
0:35because a rule can't look at a page and
0:37tell you whether it's got a signature or
0:38not. Now, people have had a lot of
0:40success at using deep learning to do
0:42this, but generally the challenge there
0:44has been that people have to use custom
0:46models for each particular company and
0:49for each particular business use case.
0:52But it's turning out that this is
0:53another place that a very specific kind
0:56of Jev model can really have a huge
0:58impact. So, what I'm going to show in
1:00this video is that I built an inspector
1:02that basically looks at forms, whether
1:04they're PDFs or images, and then runs
1:07them through two small open models that
1:10make decisions straight from images.
1:13These are literally open Jev models. And
1:16both of the models are really high up on
1:18the image Jev benchmark. In fact, the
1:20one that's at the top here was built by
1:22one person in 15 days for just $1,200 of
1:27rented GPUs. So, I know I make quite a
1:29number of videos about the Jev decision
1:31model concepts, and I've made videos of
1:33showing you how you can use it for
1:34agents and how you can use it for text
1:37decision making. In this video, I want
1:39to focus purely on images and purely on
1:42the image Jev decision models. Remember,
1:45this is something at the time that I'm
1:47recording this that even the Jev model
1:49itself can't handle images. So, in this
1:52case, we've really only got open models
1:54to try these things out. All right, so
1:56what's the problem that this is actually
1:58solving? If you've automated anything in
2:00business, you've hit some kind of
2:02version of this. You've got some kind of
2:04bot that can fill in a web form, can
2:07click and submit, but often you don't
2:09know if it actually worked until someone
2:10checks the actual screenshot. You've
2:12also got examples of where a customer
2:14uploads a photo for a refund claim. Does
2:17that photo actually show what they're
2:19claiming in it? Well, someone has to
2:21look at it. If you're looking at
2:23invoices or contracts or forms, you've
2:27often actually got to go and look at
2:29that to see was it actually signed? And
2:31every one of these is one of those
2:33diamonds in that flowchart. It's a yes
2:35or no call that a person or a deep
2:38learning model will actually make after
2:40looking at something. Now, automating
2:42all of these kinds of things has been a
2:44huge industry. It's got its own name,
2:47RPA, robotic process automation. If you
2:51look at some of the biggest companies in
2:52the space, you see ones like UiPath.
2:55And so I first came across UiPath in
2:572018. And at the time what they were
3:00doing looked really impressive. They
3:02were certainly raising lots of money.
3:04And sure enough by 2021, they had gone
3:06public with a valuation around 30-40
3:09billion dollars. Since then though, it's
3:12kind of collapsed. If you look at their
3:14stock price has dropped from the high
3:1670s right down to sort of 13-14 dollars
3:19a share now. That market cap nowadays is
3:22under 6 and 1/2 billion dollars. So the
3:25biggest question here is going to be
3:26what went actually wrong here? And the
3:29thing that I found from talking to
3:30people who actually worked with UiPath
3:33was that while the promise of full
3:35automation was really awesome, the
3:37models never actually worked often like
3:39they were promised. And to be fair, I
3:41don't think it was necessarily that
3:42UiPath was making a really bad product.
3:45It's just that the type of deep learning
3:47that people were using for these kind of
3:49models really required a lot of
3:53for the specific use case. They just
3:55didn't have the general universal
3:58classifiers to be able to do a lot of
4:01these tasks with a high level of
4:03accuracy. So, as I started to see people
4:05make these Image Jev models, I was
4:07really curious to see, is this something
4:09that could be applied to RPA? So, for
4:12the demo here, I basically picked forms
4:14because pretty much every office has
4:16piles of forms that they're either
4:19scanning in or they've got electronic
4:21versions of as PDFs, etc. And what I was
4:24curious to test out was could we make a
4:26demo where you drop a PDF of a form on
4:30there, or even do it in batch mode?
4:32Could you drop a PDF of a form and then
4:35have it run very quickly to generate
4:38answers back about that form? So, for
4:42things like did the person fill out
4:43their date of birth properly? Is the
4:45form actually signed? Does it have an
4:48address for the person on it? Is it a
4:50tax form? Is it an invoice? All these
4:53things are sort of stock-standard RPA
4:55tasks, and they were all things that I
4:57was curious to see, can we take a model
4:59that hasn't been fine-tuned for this
5:01specific use case, and see how it
5:03actually handles this. All right, so
5:05this is Image Jev, a new benchmark for
5:08systems that make decisions from images.
5:11At the top there, we've got Image Jev
5:134B. As you see, that that one's
5:15definitely scoring really well here.
5:17Below that, we've got Jev Omni, below
5:20that, we've got Neo Horse and a number
5:22of other models in here. So,
5:24interestingly, the person behind Image
5:25Jev actually wrote a Reddit post all
5:28about not only how they made it, but why
5:30they made it. So, they're a process
5:32consultant who worked on refunds,
5:34returns, and customer support. And his
5:37point was that in those process maps,
5:39almost every decision node ended up
5:41being made by a human because just the
5:44ambiguity is really hard to deal with
5:46with code alone. So, I was curious to
5:48try out his model and some of the other
5:50models for my particular demo here. So,
5:52I decided to actually make a demo with
5:54the first two top models here and see
5:56how it actually works out. Now, I know
5:59I've made a bunch of videos about Jeff
6:01already, but those were mostly focused
6:03on text. The image Jeffs open up a whole
6:05set of new opportunities because a lot
6:07of the decisions in business aren't
6:09sitting in text. They're in screenshots.
6:12They're in photos. They're in scans.
6:14They're in PDF files. And that's what I
6:16really want to dig into in this video.
6:18So, the whole idea of these models is
6:20that they're basically the diamond that
6:22gets the evidence, a screenshot, a
6:25photo, a page. It can really just go
6:27through and answer a lot of different
6:29yes no questions based on that image as
6:32the context. So, in a normal Jeff model,
6:34we basically pass in text as the state
6:37and then we pass in our typed questions.
6:39Here, we're passing in the image as part
6:41of that state, but we're still also
6:42passing in our typed questions just like
6:44before. Now, it's important that I point
6:46out here, we're not doing any kind of
6:48OCR here, right? It's not like I'm
6:50OCRing the form first and then passing
6:53that in. I'm just passing in a raw image
6:55to the model. So, the model itself is
6:57deciding what's actually represented on
6:59that image and then using it to
7:01basically answer questions like is there
7:03a handwritten signature? Is a particular
7:06field empty? Is something filled out
7:08with a date there or not? And the cool
7:10thing here is that it gives us back a
7:12probability for every option. An image
7:15Jeff has also been trained to give you
7:17back the probability of unknown. So, the
7:20good thing about having that in there is
7:21it just helps the model so that
7:22hopefully the model is not guessing,
7:24it's basically either sure that it's
7:26something or it's going to answer
7:28unknown. Now, with the responses that we
7:31get back, we still got the whole sort of
7:33same smart if statement that we had with
7:36the text models. It's just now the smart
7:39if statements are things that you can
7:41actually see. And one of the cool things
7:43about having those if statements is that
7:46you can actually set a confidence score
7:49that if it's returning below that, then
7:51you can basically say right, I don't
7:53trust the model. I'm going to send it to
7:55either another model to either confirm
7:57the prediction or I'm going to actually
7:59send it to a human to look at. Okay, so
8:01if we jump into the demo here, this is
8:04basically the form decision inspector.
8:07So I'll show you just quickly what it
8:09can do. We can basically drag in a PDF
8:13and just showing you what this is. This
8:16is basically a US tax form and the text
8:18is pretty tiny in there, but you can see
8:20at least that okay, it's got a name,
8:22it's got a bunch of details in there.
8:24And in this case you can see that the
8:26signature is actually missing in here.
8:28The date that person signing it is
8:30there, the date of birth is there, but
8:33signature in this case is missing. So
8:35now if we come over to the actual
8:37questions, you can see we've got a bunch
8:39of questions in here. We've got things
8:42like is the date next to the the
8:43signature filled in, right? The postal
8:45address is filled in, date of birth is
8:48filled in. And then we've got fields for
8:50missing, we've got fields for what kind
8:52of form is this in here. And then we've
8:55got some other things in there about
8:57things like how legible is the
8:59handwriting on this page. So I'm going
9:02to run this. I can also change things
9:04like the DPI, so basically the
9:07resolution of the images and what's
9:09going into the actual model. I'm
9:10[snorts] running two models here. I'm
9:12running the Image of 4B and the Jev Omni
9:15in here. Now I'm going to show you some
9:17of the issues you run into. So this goes
9:20through, it basically runs
9:22and you can see sure enough we can see
9:24that this model has done a pretty good
9:27job, right? It's basically said hey, is
9:28it signed? No, it's false, it's not
9:31signed in this case. Now up here we can
9:33basically set how confident does the
9:36model have to be before we actually
9:38accept the result or or send it back to
9:40a human? Yeah. Because it was basically
9:43lower than the 90% that I had set there,
9:45it's going to say that it should go to a
9:47human cuz I had that set up there. More
9:50importantly, this one is wrong. So, let
9:52me come back to that in a second. Dated,
9:55they're both in agreement. They're both
9:57getting it right there. Address, that's
9:59looks right. We can see that this one
10:02one said that the date of birth is
10:03there, one said that it's not in this
10:05case. And we can see that this one now
10:08when we ask what is missing, the
10:10signature is missing here, right? Yet up
10:13here when we asked the signature is
10:15there true or false, it said that it was
10:18there.
10:18>> [snorts]
10:19>> And this kind of highlights the problem
10:22that you're going to face with the
10:24different models at the moment is that
10:26they tend to be better at different
10:28kinds of questions. So, you can see here
10:31this model when we ask it what's
10:33missing, it's saying that nothing is
10:36missing, right? So, that's a choice
10:38question. That's a classification
10:39question. The actual sort of true or
10:41false question in this case, it gets it
10:44right. We can see that the response that
10:46it got back is going to be like that and
10:48we can see the image if we want to
10:50there. In this case, it gets it wrong.
10:53Now, this is where you have to be
10:54careful about what model you use and not
10:56necessarily that one model is great and
10:59one model sucks. They tend to be better
11:01at different things. So, I'll show you
11:04just quickly here with this exact same
11:06form. If we basically turn signature to
11:10be a choice, right? And I run it now,
11:14you can see that this one is now saying,
11:16"Okay, that it's typed name only.
11:18There's no handwritten signature." And
11:20also this one now is basically saying
11:22typed name only, no signature. So, it
11:26shows that especially in this model
11:28here, it's probably better at the
11:31classifications than it is at the true
11:34or false in relation to this task,
11:36right? I'm not going to say that it's
11:37better for every single task at this,
11:40but in that relation to this kind of
11:42task, I need to be calibrating for that
11:44early on. And we can see that both of
11:46them got it right for a tax form. Both
11:49of them said that it was either clear or
11:51readable in there. In some ways probably
11:53clear and readable almost the the same
11:55kind of thing going through this. The
11:57other thing we can do, too, is we can
11:59actually have a follow-up email. So,
12:01this is not something is being sent to
12:04an LLM to write the email. This is
12:06literally just conditional logic. If the
12:10signature is not there, then we
12:13basically say, "Thanks for sending your"
12:15and we fill in tax form. Before we can
12:18process it, could you please sign the
12:20form in the signature field. So, that's
12:22not using a normal LLM in any way, but
12:25it's important to understand that
12:26there's no OCR going on here. This is
12:30literally just passing in an image and
12:32then running it through for this kind of
12:34thing. And the cool thing here is that
12:36all of these we can basically test
12:39different questions in here. So, if I
12:41want to edit something is missing could
12:43be the date of birth, I can just come in
12:46here and you can see that each of these
12:48is basically just a string field that
12:50goes in there. And now I've got a new
12:53class that it can predict as well. I'll
12:55run it again. Obviously, the date of
12:57birth was here for this one. And you can
12:59see for this model it wasn't phased by
13:01me adding that at all. For this model it
13:03was. We've now got a little bit of
13:05things where it's saying nothing missing
13:06to okay, possibly the date of birth is
13:08missing. So, you do want to test these
13:11and work out okay, one, what are the
13:12best kinds of questions for what you're
13:15doing? And then also, which model is the
13:18best one for those types of questions.
13:21Okay, so if I give it a new form now,
13:23this is a form where the date of birth
13:25is actually missing, the signature is
13:27there that you can see in there. So,
13:29let's try and run this one and see what
13:31we get. And you see, they're running
13:33very quickly. So, you can see now we're
13:35getting handwritten signature is there.
13:37Both of the models concur, that's the
13:40key thing there. If we go through and we
13:43get to date of birth, we can see now
13:45both of them are saying that date of
13:47birth is missing. This one's less sure
13:50about it than this one in here. And then
13:52again here when we ask what's missing,
13:54this one can clearly see now that the
13:56date of birth is missing. And this model
13:58just doesn't seem to be doing great at
14:01this type of question. So, I'd probably
14:03think about, okay, am I going to
14:05rephrase this for this model? Am I going
14:08to only go for the results with this
14:10model here? But the key point that I
14:11want to show you here is that you need
14:14to be aware of this, right? Don't just
14:16go and think, oh yeah, this model is the
14:18best model, therefore it's going to work
14:20for all my tasks really well in here.
14:22So, this is just a really simple sort of
14:24thing. You can see again, we can
14:26basically have it do a follow-up email,
14:29which is conditional logic. Here it's
14:31basically saying fill in your date of
14:32birth, it's worked that out really
14:34nicely. And we can set this up to
14:36actually just use a sort of confidence
14:39score to decide if we throw it back to a
14:42human or not. And that's something that
14:44we could set. Now, with a little bit
14:46more work, we could actually set that
14:48confidence score to be different for
14:51each of the questions that we've got in
14:52here. We can add in questions quite
14:55easily, right? You can build something
14:56where you could put in a choice
14:58question, you could put in a scoring
15:00question, you could put in a null
15:01question in there. So, hopefully you can
15:04see here that this is something that is
15:05pretty easy to do with these kinds of
15:09models, right? You're not having to
15:10train up a lot of models or do anything
15:13like that. Now, that said, if people are
15:16interested, maybe I will make a video
15:17about actually fine-tuning some of the
15:19Open Jaws models, cuz that's the next
15:21stage that you take it to. If you're
15:23starting to get some data where you've
15:25actually got some decent data, you could
15:28actually take some of these models,
15:30whether it's the image one or whether
15:32it's one of the text ones, and actually
15:34fine-tune it with a little Laura adapter
15:37in there to basically make it work for
15:39what it is that you're trying to do. So,
15:42anyway, hopefully that shows basically
15:44how these Jev image models can be really
15:47powerful for things like RPA, for things
15:51like very quick image classification,
15:54and dealing with large amounts of images
15:56in here. We can run a whole bunch of
15:58different questions through it, and
16:00basically customize the questions and
16:02the model for our specific use case in
16:04here. So, anyway, let me know in the
16:05comments if you've got any questions
16:07about this. Hopefully again, this shows
16:09you a real-world example of how this
16:11kind of thing can be used so that you've
16:13got a sense of what you can actually do
16:15with this. And as always, if you found
16:17the video useful, click like and
16:18subscribe, and I will talk to you in the
16:20next video. Bye for now.

About this transcript

This page contains the full transcript of Image Decision Models for RPA: Forms, Scans and Screenshots by Sam Witteveen, generated from the public captions YouTube serves with the video. The transcript has 3,169 words across 433 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.