AI NPCs you can actually talk to — Unreal Engine 5 + Pipecat C++ SDK — Transcript

Full transcript

1. 0:01Hey, Orby
2. 0:02Hey, John
3. 0:03You know, I could really use a Pipecat C++ client SDK.
4. 0:07Is there anything you can do about that?
5. 0:09Sure, John, let's make some magic
6. 0:17Pipecat C++ Client SDK has landed on Linux, macOS, and Windows.
7. 0:22It'll also cross-compile for Aarch64 too, so that includes ARM boards.
8. 0:26You can now build native compiled experiences that connect straight to
9. 0:30your Pipecat agents with transports for WebSockets and Daily WebRTC.
10. 0:39A Pipecat client that works outside the browser or a phone opens up a
11. 0:42whole class of things: game engines, robotics, embedded devices and kiosks,
12. 0:48native desktop apps, anything where you need low-level control of audio
13. 0:52and a web view just isn't enough.
14. 0:54And these new clients are the perfect excuse to put the Pipecat agent
15. 0:57somewhere a bit more interesting.
16. 0:58We could, for example, drop it into the Unreal Engine.
17. 1:01Let's take a look at that
18. 1:03I am a small spirit here to light your way.
19. 1:06If you wish, I can bring the sun and shape the world around us.
20. 1:10Uh, hello.
21. 1:11What's your name?
22. 1:13Hello.
23. 1:14I do not have a name, but I am here to help you shape this empty place.
24. 1:18What would you like to see first?
25. 1:20Uh, okay.
26. 1:21Well, I'll call you Orby.
27. 1:23Uh, how about we add some light?
28. 1:26I brought the sun up.
29. 1:28Now the world is bright around us.
30. 1:30And what about some clouds?
31. 1:34I added some clouds to the sky above us.
32. 1:39Looking good.
33. 1:39It's a bit flat though.
34. 1:40Can we, um, add some hills?
35. 1:44I shaped the ground into rolling hills for us.
36. 1:48And how about some mountains?
37. 1:52Mountains now rise up around this valley
38. 1:57Uh, and a bit of fog maybe
39. 2:01A light fog now drifts through the mountain valley.
40. 2:05But it's looking really good, Orby.
41. 2:07Um, oh, I suppose we need some grass.
42. 2:12I covered the ground with fresh grass.
43. 2:16I added a few trees around us in the valley
44. 2:21Looking good.
45. 2:22Can I see what it looks like at dawn?
46. 2:27I set the sky to dawn.
47. 2:29The sun is just rising over the mountains
48. 2:34How about a few more trees?
49. 2:37Now a forest surrounds us, thick with trees
50. 2:42I suppose we needed some water.
51. 2:44Um, can you add a lake?
52. 2:47I created a lake nearby, its water reflecting the dawn light.
53. 2:51Whoa, you almost drowned me there, Ruby.
54. 2:54Sorry about that.
55. 2:55The lake is big, but you are safe on dry ground.
56. 2:59in this demo, our little floating orb friend can shape the world around us.
57. 3:03The NPCs hear you through spatial audio, so proximity and direction
58. 3:07decide who's actually listening.
59. 3:10A stone figure now stands nearby, silent and still in the moonless night.
60. 3:16He's not very chatty.
61. 3:19He will not speak while he is stone.
62. 3:21If you wish, I can bring him to life.
63. 3:24Uh, yeah, let's do that
64. 3:27I touched the statue and brought him to life.
65. 3:30He is waking up now.
66. 3:33Where am I?
67. 3:34Who are you?
68. 3:35Everything feels strange
69. 3:37Uh, hi.
70. 3:38What's your name?
71. 3:40I, I do not know.
72. 3:42I do not remember any name.
73. 3:45Um, well, how about we call you Frank?
74. 3:48Frank.
75. 3:49That sounds good.
76. 3:50Thank you for giving me a name.
77. 3:52Uh, this is my friend Orby.
78. 3:54Uh, he's kind of cool.
79. 3:55He makes stuff happen here.
80. 3:58Hello, Orb.
81. 3:59You are glowing.
82. 4:01How do you make things happen?
83. 4:03Hello, Frank.
84. 4:04I shape the world when someone wishes for something new.
85. 4:07If there is anything you would like to see, just ask.
86. 4:11The game state and events flow both ways over RTVI, and those are
87. 4:14the same messages you're already using in the web and mobile SDKs.
88. 4:18So the bot knows what's happening in the world, and the world
89. 4:21can react to the bot too.
90. 4:23I brought the sunrise, filling the valley with warm morning light
91. 4:29How do you like that, Frank?
92. 4:31It is wonderful.
93. 4:33The colors are gentle and bright.
94. 4:36I feel warm inside.
95. 4:37Thank you.
96. 4:39Frank, why don't you ask Orbi for something else?
97. 4:42I am not sure what else to ask for.
98. 4:45The sunrise was perfect.
99. 4:47What do people usually wish for?
100. 4:49Frank, wind is the air moving across the land and through the trees.
101. 4:54It can be a soft breeze or strong and wild.
102. 4:57Would you like to feel it?
103. 4:59Yes, Orbi, I would like to feel the wind.
104. 5:02I think it might make me smile.
105. 5:05A gentle breeze now stirs the trees and brushes against your face, Frank
106. 5:12And how about we add a river?
107. 5:17I brought a river flowing
108. 5:20through the valley, sparkling in the sunlight
109. 5:23Well, you almost drowned Frank
110. 5:28I am all right.
111. 5:29The river looks beautiful, but I am glad I am not.
112. 5:33Thank you for thinking of me.
113. 5:34Well, you kind of are.
114. 5:37Oh, I did not notice.
115. 5:39The water feels cool and fresh.
116. 5:41It is a strange, pleasant feeling.
117. 5:44now, recently I've been experimenting with multiple character pipelines.
118. 5:48That's multiple voices, multiple personalities, and
119. 5:50just one agent orchestrating the conversation between them
120. 5:56Yeah, guys, I'm trying to record a video.
121. 5:59Thanks
122. 6:00Uh,
123. 6:00The trick to making that work is a fast decision model that front-runs
124. 6:03the LLM and works out who you're actually talking to when there are
125. 6:07several characters in the scene.
126. 6:08but you could also feed it spatial data from the Unreal Engine so it
127. 6:12knows what you're looking at or how close you are to an entity to
128. 6:15decide who or what you're addressing.
129. 6:18Nice to meet you, John.
130. 6:19Welcome to my conservatory.
131. 6:21Ooh,
132. 6:21what are those?
133. 6:24Those are my flowers.
134. 6:25Red, yellow- Could I get one, please?
135. 6:27… pink, white.
136. 6:29Of course.
137. 6:30I'll pick you a lovely one.
138. 6:36There you are, a bright red flower freshly picked for you.
139. 6:40And what's that?
140. 6:43That's my wishing fountain.
141. 6:44Toss in a coin and make a wish.
142. 6:46Hey, where's that music coming from?
143. 6:50That's Juno playing in the music room.
144. 6:52Sounds like she's having fun.
145. 6:54Sounds like she's having fun, but it's not very good, is it?
146. 6:57So how does all this work?
147. 6:59Oh, oh, I know.
148. 7:01It works with magic
149. 7:03No, Orbi, it isn't quite magic.
150. 7:05Under the hood, this is the C++ client inside Unreal talking
151. 7:09to a Pipecat bot over WebRTC.
152. 7:12That matters here because WebRTC gives us multiple custom tracks.
153. 7:16Every character gets its own audio track routed to its own actor in
154. 7:20the scene so they can talk over one another just like people do
155. 7:23Now, this demo is using hosted services, so that's Deepgram for speech-to-text,
156. 7:28Pipecat PhoneLLM, Cartesia for the voices, and Jev for the decision model.
157. 7:32All hosted is great if your target hardware isn't cutting edge since the
158. 7:36heavy lifting happens elsewhere, even deployed, such as on Pipecat Cloud.
159. 7:40But an interesting alternative to that is the opposite, an all local pipeline
160. 7:44compiled and bundled as one executable.
161. 7:47And that's what makes the C++ libraries exciting.
162. 7:50Your bot file doesn't change.
163. 7:51It's the same Python, same Pipecat, written the way you always have.
164. 7:55So interruptions, VAD, turn detection, the model integrations,
165. 7:59all of it should just keep working.
166. 8:01Nothing to rewrite.
167. 8:03The client does the native bits, the agent stays in Python, and together with the
168. 8:07local models, you get one executable.. No server, no network hopping.
169. 8:12The Pipecat C++ client SDK is of course open source and can be found on
170. 8:16GitHub along with all the Unreal Engine demos that we've shown in this video.
171. 8:20If you build something with it, whether that's a game, a robot, or a wacky art
172. 8:24installation or something, please come and show us in the Pipecat Discord