YT YouTube2Text

YouTube2Text → Transcripts → Make Your AI Agents Predictable With Graph Engineering - Zachary Blumenfeld, Neo4j

Make Your AI Agents Predictable With Graph Engineering - Zachary Blumenfeld, Neo4j — Transcript

by AI Engineer · 3,797 words · 569 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:01[music]
0:12>> Uh we're going to talk about graph
0:13engineering today. How many people have
0:15heard about that term graph engineering?
0:18Okay, so so fair amount of the room. And
0:19then just so I understand, how many
0:21people here are familiar with Neo4j and
0:23have used Neo4j?
0:26Okay, so we got a few people. So that's
0:27good for uh
0:29just letting me know
0:32who we're talking to today.
0:36Um so just some history on some of the
0:39verbage that's being used. Uh there was
0:42a tweet that went out uh by Peter
0:44Steinberger, I think uh earlier this
0:46year in the summer,
0:48uh that talked about graph engineering
0:50and it was almost like is this sort of a
0:51new discipline that's happening as this
0:53evolution from prompt engineering and
0:55context engineering. Uh and there's a
0:58lot of hype around this.
1:00Um and the name for this is new, but the
1:02mechanisms are not really new. They
1:04we've always known in computer science
1:06generally about state machines and
1:08workflow graphs and knowledge graphs and
1:11all these different things. What's
1:13changing is really how we think about
1:15them in reference to agents.
1:18Um and what I'm going to talk to you
1:20about today is just how we think about
1:22graph engineering. I'm going to offer
1:24you a little bit of a new perspective on
1:26this. So if we actually try to define
1:28what graph engineering is, I think a
1:30good definition is it's just the
1:32practice of designing agent work as an
1:35explicit graph. So where nodes do work
1:38and edges effectively route state
1:41between them. And I tend to think of it
1:44in terms of three different types of
1:46graphs. We have our general execution
1:50graphs, which you're probably very
1:51familiar with. So these are the workflow
1:53graphs with routers and agents as actual
1:56nodes uh that happen within side of an
1:59agent harness. But, there's other ways
2:01that we can think about graph
2:02engineering, too, that overlap with
2:04this. So, that includes context graphs.
2:07These are decision traces, instances of
2:10what's happened, and how it connects to
2:12other data inside of your domain. And
2:15then there's also something that I'd
2:16call more procedural graphs. So, this is
2:19almost like having a skill or procedural
2:22knowledge as a graph, where the steps
2:25might not be actual agents, but they
2:27might be steps inside of uh
2:30uh a way of doing something, basically a
2:32procedure. So, different calls, maybe a
2:34step for a decision model, steps for
2:37different agents, whether to escalate,
2:39validation steps, and these sorts of
2:41things.
2:42And I'm really going to focus on these
2:44last two, because I think we don't talk
2:46about them enough, and I think by
2:48understanding these, you can actually
2:50get a little bit at what the first graph
2:52is trying to do as well, but in a very
2:54portable and efficient way.
2:57So, graph engineering lets agents really
3:00take on bigger, more complex work, and
3:02that's why we care about it. And really,
3:05if you were to split it up, I think in
3:07terms of the benefits that a graph
3:08offers agents in this uh context, it
3:11would be predictability, so being able
3:13to constrain a path that an agent takes,
3:16making sure that the the answer is
3:18basically accurate from a task that it's
3:20doing, um and having an explicit graph.
3:24The ability to govern, control, and
3:26guide what the agent does, and audit it,
3:28so you can understand what it's doing
3:30and make it better next time. And then
3:32the third one, which is going to be
3:34interesting that we'll talk about, is
3:35how do we actually make graph
3:36engineering portable, and instead of
3:39thinking about a graph as something that
3:40happens in the runtime, it could be
3:43something that you can store and
3:44actually move between different runtimes
3:47and have inside of a knowledge layer,
3:49which gives you more interoperability
3:51and allows you to have a very simple
3:52stack.
3:54Um and in the runtime, when we think
3:56about the graph, it's really like a
3:57piece of orchestration. So, we're
3:58probably all of us or a large portion of
4:01us are probably familiar with LangGraph,
4:03ADK,
4:05um and all these other agent frameworks.
4:08Um even to a certain extent Anthropic's
4:10multi-agent research and and some other
4:12stuff fits inside of graph engineering.
4:14Um and what I'm going to talk about
4:16today is actually what happens when we
4:18take that sort of graph and we put it
4:20next to our data as context graphs and
4:23some other things that I was talking
4:24about before.
4:26Um and really when you do this, as I was
4:29saying, you add this portability layer,
4:32you add the ability to take all of this
4:34knowledge and make it work basically
4:37with any type of runtime. So, that's
4:39what I'm going to go over today.
4:41Um and this is what we're doing at Neo4j
4:43is we're defining this knowledge layer
4:46and this knowledge layer sits between
4:48systems of record that you would
4:50ordinarily retrieve data from. So,
4:52DataStax, BigQuery, Snowflake, these
4:55types of places. It could be
4:56documentation, it could be Obsidian,
4:58these these markdown files. Um and then
5:01can we create standard protocols and
5:03artifacts that live inside of that layer
5:05that then you can take a graph
5:07engineering process and make it
5:09available to any runtime or any agent.
5:14Um and the way our product works, just
5:16so you know at Neo4j, uh is we have our
5:20core database and then we have a ton of
5:22other AI stuff on top of it. Uh so, the
5:25database and the analytics layer, um I'm
5:27going to be showing some of this
5:29throughout my talk, uh but what I'm
5:31really going to be focusing on is this
5:33top layer where we talk about sort of
5:34this agentic brain or these other memory
5:37tools that wrap around the database. So,
5:40just know that as I go through, I'm
5:42going to be showing you data being
5:43loaded into the database. That's going
5:45to constitute our knowledge layer. It's
5:46going to come from multiple different
5:48sources. Um but a lot of the sort of
5:51extensions of that tooling and some of
5:53the experimental things that I'm going
5:55to show you exist as sort of this graph
5:57AI that hooks into that and loads data
5:59in and out of that thing.
6:00>> [snorts]
6:01>> Um so, what we're doing at Neo4j and
6:04kind of how we're thinking about this,
6:06we start with this concept of a context
6:09graph, which has different types of
6:11memory from short-term memory, which is
6:13going to be like your conversation
6:15history, your long-term memory being
6:18entities that are extracted from that or
6:20from your data, and then reasoning or
6:23decision trees inside of that.
6:25And one of the interesting things that
6:27we've been looking at from a research
6:29perspective, and this really started out
6:31not so much at that graph, but actually
6:34at individual skill level. There was a
6:36paper that we wrote earlier this year
6:39that we published in a workshop, and we
6:41were really trying to ask the question,
6:43can we take procedural knowledge,
6:45whether it be from past decisions or
6:48even something like a skill like we have
6:50here, and compile it down to a graph?
6:53And what we came up with was a process
6:55of taking written material, often a
6:57skill MD file, and the scripts and the
6:59references and assets, and moving that
7:02into an actual graph structure, which is
7:04what you see on that other side.
7:07And the result of doing this, and giving
7:09the agent the access to that graph
7:11structure, which has different nodes
7:13again representing
7:15inside of the skill decisions, different
7:17routing points, different scripts that
7:19have to be executed,
7:22taking pros and compiling them down to
7:24to determine the six scripts.
7:26When we do that and we force the agent
7:28to sort of run that graph, what we see
7:31is increases in uh task reward and
7:33performance. So, this was uh from a
7:36while ago. I I'm still [clears throat]
7:37working on updating it to the latest
7:39models,
7:40um
7:41but we were able to use a smaller model.
7:43In this case, I think it was Sonnet, um
7:45and now we're we're on to Haiku, as a
7:48matter of fact, but uh we were seeing
7:50even for really challenging tasks over
7:52diverse fields in physics and computer
7:55science and um various security, uh
7:59physical training, a whole a whole ton
8:01of things,
8:02um is this overall increase in task
8:05reward
8:06um performance.
8:08Then we were able to switch over to
8:09this.
8:10And our PMs, uh, Will Lein, who works in
8:13our Labs team, actually took this idea
8:15and said, "Well, if we can do that with
8:17skills, maybe we can do it, too, with
8:20these memory graphs." So, you can
8:22actually take that context graph that I
8:24was showing you earlier and distill a
8:27skill from that and have it still be
8:29connected in the same graph over to all
8:33of your past decision traces and data.
8:35And so, then you get those performance
8:37benefits, but you also have all this
8:39provenance that goes back from every
8:40step. It's versioned, it's tracked, and
8:43you have governance over time where you
8:45can assess drift, you can understand
8:47where certain tool calls are coming from
8:49and their history. A human can go
8:51through actually inside of NAMS, Neo4j
8:54Agent Memory Service, which is our
8:55reference architecture for this. There's
8:57a QR code there, and there will be
8:58another one at the end, um, and be able
9:00to understand, like, exactly how this
9:03works, edit it, change it, do all sorts
9:06of cool stuff with it. Um, so, again,
9:08what we're talking about now are these
9:10artifacts that you can move around. I
9:12mean, this thing makes a skill, right?
9:14And so, any agent harness that can take
9:16that skill can now be running these
9:18graphs and doing all of this logic.
9:21Um, and we have various customers that
9:24are using us for graph engineering types
9:28of tasks. I'll go quickly because we're
9:30we're starting to run out of time.
9:33Um, and even though they're not may not
9:35be using exactly the Labs project that
9:37they that we have, uh, we have
9:39consultancies which are working with
9:41other asset managers, uh, for tough
9:43problems in compliance. Uh, so, being
9:45able to write audit reports, for
9:47example, where you have data that might
9:49come from multiple sources. How do you
9:51automate that, but do it in a way that's
9:53governed and trustworthy? So, you have
9:55compliance documentation. You might have
9:57entities with different ticker symbols
9:59and names from different databases, and
10:01those all need to be resolved. And so,
10:03giving your agent these graphs that
10:05express the logic that you need to do on
10:08besides the data basically allows you to
10:10automate those things in a trustworthy
10:12way.
10:13Um and we have other people uh such as
10:15Uber that do this with configuration
10:17graphs. Uh so basically, if you have um
10:21you can imagine if you have drivers
10:22across the country, uh compliance and
10:24regulations for different areas
10:26geographically vary greatly,
10:29um and they come from this sprawl of
10:30microservices that you need to pull data
10:32from to figure out like, "Okay, well
10:33what is it for each area, for each
10:36component of a car, for example?" Um and
10:39so being able to express that logic and
10:41have it sit aside the data um also helps
10:43them operationalize some of these things
10:45effectively.
10:47So, I'm going to jump to a demo now. It
10:50I'm going to warn you in advance, it's
10:52going to be quite a lot, and I'm going
10:53to jump through it kind of quickly, um
10:55but it's very interesting, and what I'm
10:57going to show you is how we can use that
10:59AIP library that I was talking about
11:01before
11:03uh to compile a skill and actually get a
11:04task that was not governed very well and
11:07took very long to run, basically cut its
11:10cost by like 100% and then run it a lot
11:13lot faster. Um and of course, it's a
11:15it's a demo, it's not a formal research
11:17proposal, but you can go there and look
11:19at the code, it's all it's all very
11:20transparent.
11:22Uh so,
11:24this is going to be an example for a
11:26loan um a loan approval process. So, in
11:30a side of a loan approval process,
11:31there's lots of policies that have to be
11:33followed, obviously. I'm not a financial
11:36expert, but many of you in the room are.
11:38Um and so there's probably going to be a
11:40policy doc, there's going to be a
11:42systems of steps that have to be
11:43followed exactly, and then you need your
11:45agent to be able to explain those steps
11:47very clearly, articulate them, and
11:49really get the answer right as if it was
11:51a human reviewing. Um and the first
11:53thing you can do, like I was saying
11:55before, and I don't have time to
11:56probably run this all live, and the
11:58internet's going to be slow. So, luckily
12:01I I have this preloaded. Um and
12:03basically, the question that was asked
12:05was, "Should we approve this new loan
12:07for this Rosa's Bakery business?" And in
12:10this case, you know, we have a business
12:12plan that's in a PDF that was uploaded.
12:16Um and that is, let's see,
12:20um the business plan is the orange data.
12:22So, if I was to scroll out here, you can
12:24see it here in the graph. That connects
12:27also to data that we pulled from
12:28DataBricks from past transactions,
12:31lending policy data, and past decisions
12:34that were actually made. And with that,
12:37the agent can crawl over all that
12:38information and basically realize that
12:42this person either had a loan before or
12:44there was a mistake before, something
12:46happened before, and come up with an
12:48audit trail for that and basically post
12:50a solution uh very very quickly.
12:53>> [snorts]
12:53>> Um and this is a great start. Just in
12:56terms of like resolving data and being
12:58able to follow policies and understand
13:00how things connect, uh this is great.
13:02One of the issues with this is, and I
13:05can show you a graph kind of of what
13:06this looks like. This is an example of
13:08me running more of these policies.
13:11Um and if I zoom in, I have a
13:13conversation
13:15here. And then I have my my reasoning
13:18trace that exists inside of this graph.
13:19And all these steps are time-stamped and
13:21they have tool calls.
13:23Uh what can happen
13:25is if I
13:27uh
13:29look at this happening twice. I think
13:30this is the right query.
13:32Um this is this is two decision chains.
13:35It's going to be a little bit hard to
13:36see. Let me zoom in.
13:38Um they they're of different lengths,
13:40you'll notice. So, one is longer than
13:42the other. Um in this case it was cuz it
13:45found a precedent in one, um basically
13:47something else that had happened before.
13:50But the the problem with this is that if
13:53you run these over and over again,
13:54there's nothing actually like enforcing
13:57that the agent follows an exact path.
13:58You might have a skill or something that
14:00will say like do these things.
14:02Uh but then there's other problems where
14:04it's like there's these decision steps
14:05where it's approve or deny is a simple
14:07one or approve it or send to a human to
14:09review.
14:10And those decision steps as well come
14:12with confidence scores.
14:14Uh or you say like if you're confident
14:16about it, if you're not confident about
14:18something, but those decisions aren't
14:19calibrated. So, it would be great to
14:21like even take some of this and send it
14:23to something like a type safe, you know,
14:25Jev type of model as well, to be able to
14:28do some of those things.
14:30So, what you can do is you can scrape
14:31all of this and you can compile it into
14:34a graph shaped skill, which is what we
14:36did here.
14:37And that graph shaped skill, if I go
14:39here, you're not going to be able to see
14:41it. But that's okay that you don't see a
14:43lot of the detail here.
14:45The important part is that we sort of
14:47compiled all the information down from
14:49those successful runs that we did, even
14:51though they were a little bit
14:52inconsistent, and we put them into this
14:55graph YAML shape inside of this skill.
14:58And this skill, even though you probably
15:00have trouble seeing, has scripts, it has
15:04assets here, it has sometimes other
15:06reference resources, it doesn't for this
15:08one.
15:09It also has source documentation around
15:11the policy that you're supposed to do
15:14for the loan. So, there's So, there's
15:15basically a ton of stuff in here for
15:17traceability, but then also telling the
15:19agent how to run the skill. But the
15:21great thing is, once the skill is in
15:23this YAML format, we can port it into
15:26all this different tooling, including
15:28inside of a graph, cuz it's all sort of
15:29this deterministic format.
15:32And in a graph, it looks kind of like
15:35this thing here. So, let me Actually,
15:38let me just run this Run this sucker
15:39again. Um
15:42This is kind of what it looks like. So,
15:45you have all of these steps out here,
15:47and then basically what I just did is I
15:50wrapped the small cli around this, so
15:51the agent can say, "Hey, run the skill."
15:54And instead of the agent deciding all
15:56the routing steps, it's just done
15:57programmatically.
15:59We took a lot of that policy down, and
16:01we put a lot of it into these execution
16:03scripts.
16:05Basically, what's happening here, if if
16:07I was just to sum it up,
16:09there's an execution that basically
16:11looks at, "Hey, do do we meet certain
16:14hard rules around policies?"
16:16After that, it's routed if it's not auto
16:19approved or denied based on that, it's
16:21routed to a decision model like Jave or
16:24TypeSafe. Um, and I can even probably
16:27show you a little bit of a clearer view
16:30here.
16:31Um, and it's given that that model is
16:33given like a risk tier choice.
16:36Um, let me make that a little bigger.
16:38Hold on.
16:41Um, whether or not there's there's some
16:44exposure material and then like an
16:46outcome of whether or not you're
16:47approving or denying.
16:49And then the rest of this um, will
16:51basically follow down into writing a
16:53report. Whether or not you need to
16:55escalate to a human review as this
16:57router. And then if not, basically the
16:59agent's going to write a report and
17:00there's a circulation here to sort of
17:02check and lint that report. Um, so once
17:06you do this and you sort of get the
17:08model going where okay, well now we have
17:10this cli that we can just run this thing
17:11in.
17:12Um, well now
17:14I can go in here and then maybe I can
17:18probably don't have time to run this
17:19whole thing again, but I will give it a
17:21shot. Um,
17:24I'll see if it goes and picks this up.
17:26It takes about 2 minutes to run.
17:28Uh, but the idea I guess while this is
17:31running and it should pick up the AIP
17:33scale which which it is here.
17:35Um,
17:37once you have it in this format, not
17:38only can you have the agent run this
17:40very quickly.
17:42Uh, you can track the runs. So for
17:44example, um,
17:47if I was to look at a single run,
17:50I can run this again. I can basically
17:52persist the runs inside of the graph.
17:55Um, so this is a run of that scale with
17:57like individual steps that it ran. Um,
18:00and then uh, I can just persist these
18:02over time. I can examine how well they
18:04work. So from a governance perspective,
18:07um, I have a little inspector here that
18:09I built really quickly. You can look at
18:11the status of all these. You can look at
18:13how a specific one ran, what was
18:15approved, what was denied inside of the
18:17process, what branches it took.
18:20Um, and then once you have a graph, you
18:21can even do these cool observability
18:23things where you can ask well like what
18:24branches were taken and which branches
18:27weren't taken,
18:29um, which is very interesting from a
18:30governance perspective. It failed ever,
18:32[snorts]
18:33um, all these cool different things and
18:34all of that is stored besides your data.
18:38Um, and so in none of this, we're not
18:40like creating a loop, right, inside of
18:42our agent. All we're doing here is
18:44basically giving it a cli to run this
18:47like programmatic skill that we compiled
18:49from documentation or in this case from
18:51our memory graph.
18:53Um, and this might take a while to run
18:55just because of the the internet.
18:59But what I will show you is I we ran
19:01this a few times.
19:04Um, and what we saw is just running, I
19:06was using Fable, so my organization
19:09already yells at me for this enough so
19:10I'm spending a bunch of money.
19:12Um, but like just using Fable alone on
19:15the graph and it was correct, you know,
19:17pretty much every time, but it took 2.7
19:19minutes. When we had the procedure,
19:22it almost cut in half just using AIP.
19:24And then because we had the procedure,
19:26we can use a simpler model like Haiku
19:28and then we can cut the model time down
19:29even more. And then of course, you know,
19:31the cost just goes way down. So cuz
19:34you're taking all of that load, right,
19:36of having to figure out all the routing,
19:37all the stuff you're doing, you're
19:38putting into these execution scripts,
19:40these Java models and everything,
19:42um, which makes everything just much
19:44much more efficient.
19:46Um,
19:47and this is going to take too long to
19:48run.
19:49Uh,
19:50it's mostly internet.
19:52You can see here about half a minute or
19:542 minutes wall clock on average down
19:56from like 7 and 1/2.
19:58All right, so anyway, um, I'll end it
20:01here. So I want to give you some
20:03resources to check out if you're
20:04interested in this type of stuff. Aura
20:07is our cloud service offering where you
20:09can get the database that I was showing
20:10you before. You get $100 of free
20:12credits. We have nodes coming up which
20:15are is speakers event. There's a startup
20:17program. And then all the code that I
20:19was referencing in the other projects
20:21that I was using to hook up to your
20:23agent memory, all that sort of stuff is
20:25down there. So, I'll leave this up for a
20:27second until someone kicks me off, and I
20:29guess that's it for me for now.

About this transcript

This page contains the full transcript of Make Your AI Agents Predictable With Graph Engineering - Zachary Blumenfeld, Neo4j by AI Engineer, generated from the public captions YouTube serves with the video. The transcript has 3,797 words across 569 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.