YT YouTube2Text

YouTube2Text → Transcripts → iPad drawing into StreamDiffusion - Touchdesigner Tutorial

iPad drawing into StreamDiffusion - Touchdesigner Tutorial — Transcript

by Jascha Süss · 4,269 words · 606 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00hey guys so I recently posted a video on
0:03Instagram where I was using my iPad with
0:06touch designer running into the stream
0:08diffusion uh Talks by simulate to create
0:11this kind of surreal uh AI animations
0:14with some life drawing input and a lot
0:17of people asked me on how I did that and
0:19I thought that's actually a good topic
0:20for a tutorial so that's what I want to
0:23cover today now this is my first
0:24tutorial in touch designer and if you've
0:27never worked with touch designer before
0:29I'm going to try try to keep it as
0:30beginner friendly as possible uh but
0:32also I think it's good if you have some
0:35knowledge about it already and if you've
0:37used it before now if you haven't used
0:39touch designer at all I can highly
0:41recommend checking out this page on
0:43their website now if you go to the touch
0:45designer website
0:46derivative uh here you have this learn
0:49Tab and then when you go in here you
0:52have this touch designer curriculum
0:55which is basically a free set of courses
1:00that you can do with the touch designer
1:02fundamentals right here for
1:04example uh just taking you through all
1:07of the core principles all of the
1:09elements the different operators and all
1:12of that so this is just a very well
1:14structured kind of course to get through
1:17all of the basics which I can highly
1:19recommend and then the other core thing
1:22is that this is based heavily on do
1:25simulates stream diffusion talks that's
1:27kind of the core element here doing all
1:29of the AI magic and you can get this by
1:33subscribing to his patreon uh he spent a
1:36lot of time developing this so you know
1:38it's definitely worth uh
1:41supporting what what he had done here
1:44and uh it's $5 a month in the tier where
1:47you can get the talkx file um but you
1:50can also always you know um spend a bit
1:53more for some more support and some more
1:55access you will also get access to his
1:59discard Discord server to get some uh
2:02support if something is not working
2:04which is which was very helpful to me at
2:06the beginning as well uh so yeah that's
2:09that's the second thing and then also he
2:12has created a full Deep dive uh
2:1610-minute video on the installation of
2:18all of this and so I'm not going to go
2:21too deep into that and I'm not really
2:23going to cover the whole installation
2:25process just because that's covered very
2:27well in this video and I'm going to link
2:29that in the description below uh cuz I'm
2:32just want to I would just want to focus
2:33like on the more interactive part uh and
2:37the touch designer setup for that going
2:39into touch designer I have an empty
2:41project file open right here and I can
2:44load in my talks the stream diffusion
2:48talks which I have saved here in the
2:50components now if you don't know you can
2:52actually save custom components uh in
2:55the pallet next to all of those that
2:58exist under the my components tab you
3:01can do a right click say add folder and
3:04then call this like whatever
3:07folder and then this will be a folder
3:09where you can put in any uh anything
3:12that you saved along the way so uh any
3:16like base comp with I don't know like
3:19let's put a noise inside and an output
3:23so an
3:25out uh now the Bas you can drag into
3:29here and uh you can save this as a sort
3:32of preset which is super handy and uh I
3:36do that a lot I just have to uh I don't
3:38have that much right here now cuz that's
3:40a different computer than the one that
3:42I'm usually using just a quick side note
3:45uh very handy to save stuff as presets
3:48right here I have the stream diffusion
3:50TD and that's the same as the talk file
3:53as if you would download it from the
3:54internet so I also still have it in the
3:56download files here if I just import
3:59this
4:01uh that's how it would look if you
4:03download the file
4:05freshly and if you haven't used it
4:08before then you will need to go here to
4:11the install tab open this installation
4:14guide and then uh do all of these steps
4:17again this is all covered in the video
4:21um from do simulate so I recommend
4:23checking that out and I will just start
4:26from the point where this is already
4:28running so
4:30if you haven't set this up uh just pause
4:32the video check out his video and I'll
4:36see you back when this is working now
4:39you can see that this is running for me
4:41now with around 10 FPS and um you can do
4:44some tweaking to uh to get this even
4:48faster and this will of course depend on
4:49your graphics card but uh to be honest
4:53something around this range like 10 to
4:5415 is something I really enjoy for these
4:57kind of AI animations cuz it gets this
4:59kind of stop motion effect as soon as
5:02get uh as soon as it gets like too fast
5:04I find it a bit irritating and like um
5:08less organic in some way so so that's
5:11cool uh now the way that this works
5:15is it's uh video to well it's image to
5:18image so it's feeding in uh an image
5:22here and then you get the output which
5:25is um you know um Changed by The Prompt
5:29in that you have so uh to start out
5:33let's just take a circle for
5:36now just so we can see something and in
5:40the circle I'm going to set a border
5:42width and then I'm going to also set the
5:44Border color to uh green for
5:49now and uh slightly thinner maybe and
5:52then I'm going
5:53to set the fill to black or actually
5:57maybe even just turn down the alpha of
5:59the fill so we just have this
6:02outline uh I'm going to type take this
6:05into a switch just so we
6:08can change this later when we take our
6:13iPad
6:14input uh but I just want to first set up
6:16the whole touch designer setup and uh
6:19and then go to the iPad so uh so here's
6:22our switch and then what I want to do
6:26with this with our input like with the
6:28drawing later as well is to kind of
6:31animate uh the whole thing by a
6:35displacement so I'm going to take a
6:37displace
6:42node not used to this
6:45keyboard and uh oh this automatically
6:48connected cuz I had this connected
6:49before so the displays can already go
6:51into the
6:55input
6:57and I also want to give if this a second
7:00input which is going to be the source
7:01for the displacement and that's going to
7:03be a
7:06noise now if I take this into here you
7:10can see that this is displacing it quite
7:12a
7:13lot uh I am getting a very interesting
7:16input already because I have the prompt
7:17set to realistic plans so in here in the
7:21prompt block you have your prompt which
7:22is going to be something like banana for
7:24default uh you can change that for
7:27whatever you like I have it to to plans
7:30right now I'm going to come to the
7:32settings of this in a moment but for now
7:36uh let's get back to our
7:39displacement and I want to turn this
7:42down by a lot so the displace weight is
7:45set to one and I want to have this at
7:47like
7:490.05 or something like that uh and I'm
7:52changing these values by using the
7:54middle Mouse button and just um just
7:57dragging it to the left and you can
7:59change the increments by which you uh by
8:02which you can change
8:03them so now you can see that
8:06our uh noise here is affecting the
8:10displacement of the ring and a good way
8:13to demonstrate this is by changing the
8:14amplitude so if I have it at zero then
8:17we have a gray image which is not
8:19affecting it at all and as soon we get
8:22uh some black and white values the
8:27respective colors will uh displace the
8:31image so the stronger the contrast the
8:34stronger the displacement will
8:36be and I now want to animate this
8:41noise which if you've used touch
8:43designer you know that it's in the
8:45transform section here and under the set
8:49translation now I'm going to use the
8:52apps
8:54timeframe um expression and divide this
8:57by 500 cuz I know it's going to be too
9:00fast um by
9:02default and this will give me a nice
9:07subtle displacement
9:09animation which already Works quite
9:11nicely as my input here so you can see
9:13that this is affecting my uh my
9:17image and now we can have a look at
9:19what's going on here uh you have your D
9:23noise
9:23schedule the fewer steps basically the
9:26closer it will be to your prompt
9:29and the higher the step value the more
9:32you will um be closer to your input
9:36image so you can find a balance
9:40here and I found something between 20
9:42and 30 is a is a quite nice balance you
9:45can also add steps but uh the server has
9:49to be stopped for that so you can add
9:50multiple iterations and theoretically it
9:54will get a bit um more concrete and a
9:58bit better the more steps you have here
10:00but I find that actually one step is
10:02totally enough so I could even delete
10:05this one restart my
10:07stream and I find that for this example
10:10and for what I'm trying to do here uh
10:13one like one
10:15step is totally fine the guidance scale
10:19and the Delta I don't really mess too
10:21much with in in this example uh also you
10:24can now see that having only one step I
10:27immediately have this uh performance
10:29boost to 15 FPS which is super nice so
10:34uh yeah I definitely will just stick to
10:37that and you can see that uh again
10:40changing this will affect the image the
10:43other IM the the other settings that you
10:45have right here if you have a strong
10:47graphics card you can enable this video
10:50to video which will take the previous
10:52frames into consideration and do more of
10:55like Smooth
10:56animations uh so if you want a bit more
10:58consistency this is definitely what you
11:00want to use for that uh I'm going to
11:02leave that turned off for
11:04now uh because I'm fine with with that
11:08uh the rest is is fine so really like
11:11the most important step here are the
11:12steps and uh and the prompts obviously
11:16so um maybe I'll work this out a bit
11:18more I love to use uh anle as a
11:22reference if you don't know this book
11:26um the art of nature
11:30is uh super
11:33inspiring and it's this kind of um yeah
11:37it's kind of very
11:41artistic representations of plants and
11:46um and this was kind of a main
11:48inspiration for for this approach
11:53so um what else we could use St the
11:56field
11:58maybe and and uh maybe also something
12:03like
12:07ornaments and yeah I mean right now it's
12:09not too exciting but even that you know
12:11I find very interesting I even love to
12:13use it like at a very low setting just
12:17to get this kind of organic sort of
12:20motion I think that just gives it um
12:23yeah a very interesting touch in terms
12:25of postprocessing I like to use an
12:27upscaler if the GPU allows it so if you
12:30have a strong GPU can try the
12:33Nvidia uh
12:35upscaler and the default output of the
12:39stream diffusion uh is 512 x
12:43512 uh in fact actually now that I think
12:46of it these Circle and the noise are
12:49still at the standard resolution of
12:51256 uh we can change that by selecting
12:54both and set it to 512 so now we're
12:58working at the same resolution up until
13:02here the output here is also 512 and
13:04then I'm running into the
13:07upscaler and uh if you right click here
13:09you can go to info and check your
13:11resolution it's 1024 in this case now uh
13:16that's because it's using upscale two so
13:20it's basically multiplying it by two you
13:22could also upscale
13:24three go to the info and now I have 1280
13:28x 1280 and that's the maximum resolution
13:30that you can use in the non-commercial
13:32version of touch designer so I will
13:34stick for that uh stick with that for
13:36now which is cool uh another thing that
13:39I like to do with these kind of AI
13:41outputs is to give it a little bit of
13:43noise to have some
13:45texture and to do that we can pipe it
13:48into a composite
13:52note um I'm going to disable this one
13:55with the D
13:57shortcut uh I'm going to also create a
13:59null just so I have something at my at
14:01the end uh so I can always you know
14:05insert something before that and that's
14:08going to be my
14:09viewer
14:11now in this composite node I'm going to
14:14add in a
14:15noise another noise in uh in fact and
14:20this one we're going to set the uh
14:22period all the way down to zero so it's
14:25really like the smallest um
14:29possible so it's really just like U this
14:32uh grain sort of thing
14:36and uh after that I'm going to use a
14:38level so I can change the brightness and
14:41the contrast and this is going to go
14:43into the composite now as a default this
14:46is set to multiply I'm going to change
14:48that to
14:49Overlay and I will also have to change
14:52the noise resolution as you can say in
14:54that case it's actually uh downscaling
14:57it because it's using using the
14:59resolution from the noise and I'll also
15:02set this to 1280 by
15:061280 and uh this looks quite cool
15:09already I'm going to also animate this
15:11noise so we have some animated grain
15:14basically I'm going to go into the
15:16transform tab for that and again use my
15:19expression apps time. frame in this case
15:22I
15:23can leave it at without any division cuz
15:27you can just uh basically generate a new
15:30noise every frame which is basically
15:33what I want with grain so um now if I go
15:37into my levels tab in fact I'm going to
15:39go into the post section
15:44here and um and play with the
15:47opacity so if I take this down to like I
15:51don't know something
15:53like
15:56Point Let's do five I like round number
15:59numbers
16:01um maybe a
16:03less4 something like that now of course
16:06this is all uh you know
16:10optional but I think this is kind of you
16:13know a bit more organic and nicer in
16:16comparison to to the input that we had
16:19before so yeah this is kind of it in
16:22terms of my setup inside of touch
16:25designer and the rest is really coming
16:27from the iPad
16:29so uh let's also have a look at that and
16:32see how we can actually input the live
16:36drawing so here on my iPad I have the
16:40app installed called tag tool so the
16:43interface looks like this you have your
16:45sketch pad right here where you can just
16:47start drawing you can select the
16:50colors so it's quite intuitive in a way
16:53and then uh you can change the size here
16:55of your pen tool and then you also have
16:58a an
17:00razor uh you have some more advanced
17:02animations and you can start like
17:03animating the stuff but for now let's
17:05just use the the input just uh by that
17:07you can start exploring the the app in
17:09more detail on your own but here in the
17:12menu up to the left you can set it to
17:15start ndi server and this is going to uh
17:19basically send the video of just the
17:21output over ndi to uh the computer as
17:26long as you hook them up so uh two ways
17:28to hook them up either you can use Wi-Fi
17:31so if they're connected to the same
17:32router you should be able to just see it
17:35directly I'm going to use uh network
17:38cable just for a more stable connection
17:41uh and that's something that's great uh
17:43in General on the iPad is that you can
17:45just use any of those um USBC to
17:49ethernet adapters that uh you know I
17:52also use them for the for the MacBook uh
17:55you can see use the same ones here uh
17:57and even for the new iPhone with USBC
17:59you can also just use uh network uh
18:02adapters which is great so basically
18:04this is just um hooked up directly into
18:08the ethernet port of my PC and then here
18:11on the iPad if I go into the
18:14settings uh let's see
18:19settings there will be a new like
18:21ethernet
18:23um menu popping up and here you can
18:26select your interface
18:29and then if you go into the interface
18:31here you want to configure your IP
18:35manually instead of automatic and then
18:37you're going to set some IP address I
18:40set it to uh
18:431050 with the subnet mask 255 255 255 Z
18:48and then on the computer you do the same
18:50thing um in this case let's check I used
18:55so you go here into the
18:59change adapter options and then ndi I
19:02actually renamed it right click it go to
19:05properties and then here in the
19:10um ip4 settings you want to instead of
19:15automatically um using the automatic
19:18configuration you want to manually set
19:20it to something similar like the same
19:24range basically so 10 Z 0 and then I'll
19:27give it 55
19:29uh
19:30255 255
19:33Z that's good okay okay close that um
19:39play that and so now in touch designer
19:42if you create a new
19:45node with the ndi
19:47in and then here under Source name
19:50select the Local Host tag tool then here
19:52you can see that we have our live input
19:55from our iPad which is nice and then I
19:59can plug this in to my
20:01switch as the second
20:04input and then in the switch change the
20:06index to one so I'm switching to the
20:09second
20:10input and yeah this is
20:14already much more interesting than what
20:16we had before now I can start drawing
20:20here and see how my input changes
20:23accordingly and then play around with
20:25the prompt settings so if I if I take
20:28that down it will you know become a bit
20:31more realistic maybe but it will have
20:34less to do with the actual input and so
20:36I really like to keep kind of this
20:39balance so you can still recognize the
20:43the input just enough that you can see
20:47what you're
20:51affecting but yeah this is
20:54um this is super nice and that's
20:57basically it
20:59now I mean of course the fun thing about
21:02all of this is that you can use any
21:04video input this doesn't have to be
21:06drawing this can also be like your uh
21:09webcam for example so I have my video
21:11device in right here going to plug that
21:14into the switch and use my uh Logitech
21:18webcam here and so now you can see this
21:22is also
21:24wild what's going on here
21:28um so and you could even you know like
21:31uh make a composite combine the two of
21:34these there's just endless variations of
21:37what you can do with
21:39this and
21:42um
21:45and yeah the kind of core
21:48principle for me is
21:51like with the still image using this
21:53displacement in between just so we have
21:55like a just SL um just some slide
22:00animation and also if you don't have an
22:04iPad I created this drawing
22:09tool so let me just delete this and plug
22:13that into
22:15here uh I created this drawing tool that
22:18reacts to your mouse cursor and then uh
22:23at first I wanted to do it like so you
22:25can click and draw but that doesn't make
22:27sense because you would move around the
22:29interface so I made it with with
22:31keyboard shortcuts and like with W you
22:34can draw and then with e you can erase
22:37and with r you can reset the whole
22:40thing and uh in here in the settings you
22:43can also change like stuff like the like
22:46the
22:47color
22:50and add in some uh some nice colors and
22:54stuff and then also have like the size
22:57of the circle
22:58uh and it's fun cuz you can see like the
23:00live uh cursor I madean one where you
23:03see the cursor which is the output and
23:06then one that says no cursor if you use
23:09this then you won't see the cursor in
23:12this output you will only see it
23:14here uh
23:17and yeah so if you don't have an iPad or
23:19like a tablet if you want you can get
23:21this uh I put it on my patreon that I
23:23just created and you can get it as a
23:26talks file there so that's a neat little
23:30setup uh as like a sort of
23:34drawing drawing tool which is fun as
23:39well uh so let's have a look at how we
23:42could also connect that to a projector
23:46now to do that uh we want to resize this
23:51to the native resolution that we are
23:53going to send out which is going to be
23:55in most cases some sort of 16x9 format
23:58either full HD or 4K or something like
24:01that uh I'm going to set it to full HD
24:04now to do that I'm going to plug that
24:07into a
24:08fit
24:10node and then in the comments tab set
24:12the resolution
24:14to uh 1,920 by
24:201080 I guess here you also have the
24:22presets for all of these resolutions and
24:26um well yeah it's going to say you that
24:29the resolution of the non-commercial key
24:31is limited to 12880 X 1280 uh it doesn't
24:34really matter cuz it's going to downsize
24:36it to whatever is the maximum resolution
24:39that is possible in with this license um
24:42but in the correct aspect ratio so in
24:44that case it's going to be 1280 x 720
24:48now that's gone but it's it's basically
24:50the same thing so yeah we can leave it
24:52at
24:53that um and then just also cuz it's
24:58bugging me let's remove the alpha
25:00channel here I'm going to plug that into
25:03a transform
25:06node uh where's that going oh let's just
25:10disconnect that um so transform node and
25:14then here in the background color set
25:17the alpha value to one and then comp
25:20over background color to on and so now
25:23we are
25:25actually uh compositing over black black
25:28so we have a black
25:30background uh let me just quickly change
25:34or just draw a little bit here so we can
25:36see something
25:42again all
25:45right that's cool and now we have this
25:49transform and uh at the end I'm going to
25:52plug that into an out
25:56node and to send this CH the projector
25:59we will need an a window comp so I'm
26:03going to go into my comps select the
26:08window and now it needs some reference
26:12of what operator it should display so I
26:14will drag my out into the window
26:16operator here and as you can see this is
26:20now popping up as um as a background
26:23here on the comp now you can set this as
26:26a perform
26:28uh window or the easiest case is
26:30actually just to open it on a separate
26:33window and if you open it then you have
26:36it here and you can display It On full
26:39screen you can
26:43um disable the bord so if you set
26:46borders to off then the borders won't
26:49show and then you can also specify the
26:52monitor so yeah that's where that's the
26:54part where we will have to plug in
26:56another projector uh okay so here you
26:59can see I have now plugged in a
27:01projector as a separate monitor uh be
27:04sure to uh set it to
27:07extend and then here in the Windows
27:10settings uh again you want to set the
27:13borders to off and then if you open it
27:15as a separate window then that will open
27:18it on your projector
27:21so uh yeah this is you know a lot of fun
27:24now if you start actually projecting
27:27this onto some more you know irregular
27:28surfaces or stuff like that this could
27:30be really interesting to do some kind of
27:32real time projection
27:34mapping um or as a sort of you know
27:39installation there's just a lot of
27:40possibilities with this whole kind of
27:42setup and uh it's super fun to play with
27:44and to mess around so that's just like a
27:46super fun workflow to explore you know
27:49and this has so many possibilities if
27:51you think of maybe like projecting this
27:53onto some more irregular surfaces and
27:56doing some kind of like real-time
27:57projection map or stuff like that um
28:00there's just like a thousand ways and
28:03directions this can go I hope I could
28:05inspire you a bit with this sort of
28:06setup and uh and that everything was
28:09clear so far if you have any questions
28:11leave them in the comments and I will
28:13see you in the next one

About this transcript

This page contains the full transcript of iPad drawing into StreamDiffusion - Touchdesigner Tutorial by Jascha Süss, generated from the public captions YouTube serves with the video. The transcript has 4,269 words across 606 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.