YT YouTube2Text

YouTube2Text → Transcripts → ARCHER2 - Containers for Reproducible Research: Introduction to Podman and Singularity - Session 2

ARCHER2 - Containers for Reproducible Research: Introduction to Podman and Singularity - Session 2 — Transcript

by ARCHER2 HPC · 11,863 words · 1,889 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:08Okay, I think we can start. So, uh
0:11before uh the launch break, we have seen
0:14how to uh use uh containers, how to run
0:18them on on our laptop,
0:22uh find them, download them and run
0:25them. Um
0:29now what we will see is how to how we
0:32can create why I say containers but yeah
0:34container images. Uh now we will see how
0:39we can create a new uh image. uh we will
0:43use
0:45a a sample that will take a image uh and
0:50we'll install pi python on top of it and
0:54see how uh how that will work in case
0:58that we want to uh
1:01uh to create our own uh container.
1:07Uh so yeah we are now in this lesson
1:14the good practice before writing the
1:18let's say the recipe docker file uh will
1:21be to uh test the commands that we want
1:24to uh use when build when in the new
1:28image. So what I will do is I will uh
1:33uh create a new container
1:39using the
1:41Alpine image.
1:47So we're here and uh I will check if a
1:50Python is available there. That is it's
1:55not. So I will need to uh update install
2:00uh Python there. If I
2:05one thing that I need to be familiar
2:07when I work with the different Linux
2:09flavors is how to install a software. Uh
2:14if I use Ubuntu will be AP.
2:17Uh I think Fedora use uh or uh Santos
2:24use
2:25jam I think it's yeah each uh Linux
2:29flavor we have a different way to in of
2:32installing software in this case for
2:36Alpine APK APK app
2:41um well it's like a minus update would
2:46be Python
2:48And I want also to install py3.p
2:54and python
2:573 that
3:00this is we are following. I will copy
3:02that on the chat.
3:05We're following this uh lesson here.
3:10So if I execute that I check that uh
3:14what I want to uh yeah I the
3:18installation works. Now I check that the
3:21Python 3.
3:24Yeah I check the version. This is the
3:26version that I want. I check that the
3:29command works that there are no typos
3:32nothing.
3:34So uh what I will do now is I will uh
3:38yeah I can go exit and uh this uh
3:42command I will add it to a to the docker
3:47a docker file
3:49uh in uh I'm in the desktop directory
3:52and I download this uh
3:56I will copy that in the chat the
3:58template to create a container
4:02that is The template is here. Copy link.
4:10So I uh downloaded that file and I uh uh
4:15unse that file. So I can do uh cd
4:20uh
4:23man intro. So if I do I see that I have
4:27two directories basic and sum. So I if I
4:30go to basic
4:34dos I see that I have a docker file
4:37there uh that I can uh edit.
4:42So if I do edit uh docker file you can
4:45either use edit or use u notepad or uh
4:50whatever you feel more confident editing
4:53the the file.
4:55You see that this is uh
4:59the docker file is the recipe that I
5:02will use or let's say kind of source
5:05code / recipe that I will use to build a
5:09container
5:14here in from I need to specify the
5:16existing image in in our case we'll use
5:20uh alpine
5:23but if you want you can use Wu or you
5:26can use any other image
5:30run will contains uh the commands that
5:34uh you executed on the cell. In our case
5:37we will install Python
5:41and uh cmd is the command that will run
5:45by default. That means that when you uh
5:48when you uh create the container and you
5:50don't run it, this will be the command
5:53that will uh run by default.
5:58So if we try this will be
6:03a time
6:10drum will be
6:15apk add
6:20minus minus update
6:22python 3.
6:25See if I remember the
6:36uh
6:37py3
6:41and the python
6:443 there
6:48and the command that you want to use by
6:50default uh I will need to uh this will
6:54be like a list of uh
6:57uh
7:00a list of the with a command and options
7:03that and this will be separated by a
7:08comma
7:09or
7:21let's see if so if will be by if I want
7:25to execute if I want to uh if I
7:28introduce a has h has a comment. So I
7:32want to use Python 3 minus minus
7:35version.
7:38I will need to do a Python 3
7:43minus minus version.
7:47This will be the command that I will run
7:49by default.
7:53Now to save is like a alt f
7:59save alt f
8:02x.
8:06Now if I do um
8:14yeah if I do this I see that yeah the
8:17the light has changed. We have a docker
8:19file.
8:21So now we I can start building my my new
8:25container. How to do that? Sorry, my new
8:28image
8:30map
8:32image build
8:36minus t. And here I will need to specify
8:42the
8:44the repository or let's see the URL that
8:48I want to uh where where I want to you
8:50to save the
8:52uh the image my username and the name of
8:56the image. So this will be
8:59uh I mean I can't type here just
9:05so it will be URL
9:08username
9:12and the name let's see here
9:21so URL will be docker
9:25io.
9:28My username on docker is fr
9:33and the image name will be alpine
9:38uh python. I I will call it
9:45um
9:47don't forget that I need to specify
9:49where the directory where the docker
9:51file is. In our case, I will specify it
9:55with a yeah with a dot. But that that
9:58means that I'm the current is the docker
10:02the
10:05recipe of the docker file is in the same
10:07directory where I'm running this
10:09command.
10:11Let's see if uh that works.
10:16So you see that it's executing the
10:23installation
10:26and the
10:28our by default use the target latest
10:33and we have our new image. So if I we do
10:37mount
10:41uh image ls
10:44well I have a lot of them but this uh
10:47this is the the one that I have uh I
10:52made 20 seconds ago.
11:02Actually, I will delete that one to make
11:04it some space here.
11:08This is called
11:10image.
11:27So this is the So now I can try to run
11:30this uh image here. So if I do uh no not
11:35docker all mount
11:38container run
11:42and I can specify the whole uh
11:46thing but I don't know if it will work
11:49if I specify only this.
11:55Well, okay, it works because there are
11:57no no other images that uh has the same
12:01that uh name.
12:03And you see that the output is the
12:06command that I have uh specified
12:10before.
12:24That's a good question. There is a
12:26question if the in the if uh for the
12:29docker file if uh we build an image
12:33completely from a scratch.
12:35Actually,
12:41I was trying to think that uh
12:46I'm I mean I would need to double check,
12:48but I'm not sure if uh you can build
12:51something I would say entirely from a
12:54scratch. You will need let's say the
12:57minimum that you need is a operating
13:00system to build on top of. So uh the
13:05lowest that would be like a alpine wu.
13:15So yeah I'm not I'm not sure if that
13:17would be po that's a good question. I
13:19was I don't know if that would be
13:20possible.
13:22Uh you will need a operating system to
13:25build on top of uh
13:28that. I was thinking you need
13:33yeah I I can double check my answer
13:35later but uh
13:38yeah from you will specify the image
13:41that in this case for instance if uh uh
13:46or software will be
13:48our software will be Python the my
13:52recommendation the recommendation is to
13:53use a image that is already
13:56built. So uh rather than starting with
13:59Alpi, you will start with the Python
14:04Python 3 the version that you want to
14:07use minus Al PI
14:11and you will get the
14:15uh
14:19there is we can do a
14:24Uh
14:30yeah, I was thinking we can do maybe a
14:32small exercise of like of
14:37well let me we can see how to create the
14:41image
14:43or the next. So maybe we can do a small
14:46exercise. Rather than installing Python,
14:48we can install R and build the build
14:54uh the image and see
14:58how uh
15:00how it looks like.
15:05I will give you uh five minutes. So the
15:09yeah the to try that is like
15:18This one.
15:35I will give you five minutes and you can
15:37see how to edit the docker file and
15:41install.
15:44R.
15:51Yeah, I think we can solve the exercise
15:54together. I uh
15:58so what we can do is we can uh
16:07let's see if I see this available here.
16:13Okay.
16:15So, uh I will see if I can copy.
16:31Okay. So, yeah, I can do a copy of the
16:33whole directory. So I I created the
16:36directory called exercise size and uh in
16:39that case we don't destroy the rest the
16:41basic uh uh docker file that we had
16:44before I edit this docker file actually
16:48what first first thing because we need
16:52to update uh
16:54the docker file that will be
17:02we need to figure out
17:05how I I I will install R and also the
17:09how to probably are but yeah I need to
17:13double check first.
17:15So um
17:19no this is not bad.
17:23So I will uh let's run a container.
17:26choice
17:31container
17:34sh
17:55Okay. Also,
17:58so I think there is a AP for APK. I
18:02think it's APK search. But uh that's the
18:05that's the thing that each operating
18:06system will have a different
18:09uh different way of searching that. So
18:13uh I think let me see if I that works.
18:21Yeah, seems that is not working on the I
18:26know what but you can also check that on
18:29Google that install R. I think the way
18:32is in uh yeah since that APK searcher
18:36for some reason is not working
18:39because this is not
18:43yeah I'm not sure what
18:50but yeah checking on uh
18:54uh
18:56if I search on Google how to install R
18:59on Alpine It would tell me that this APK
19:03other R that will be
19:07warning opening.
19:09Yeah, I need to check why it's not uh
19:12because if I do a APK search,
19:16it's not uh working.
19:20Oh, well, okay. a APK search will list
19:24you all packages that are but yeah that
19:27would be a bit of a
19:32so if I do
19:35ah okay apk
19:39ak
19:41search this is to find a
19:46uh actually I don't know if that is
19:50APK
19:56help.
19:58Okay. Fine.
20:01So here I can get the some help
20:06in pattern the bracket is and I think
20:08the minus E
20:11is not here.
20:21Oh yeah, match pack package names
20:23exactly
20:26of for search to a pattern in the
20:29description.
20:31Yeah, there are different options. So
20:33the option that uh I use it worked is AP
20:37search minus er
20:40and that tells you tells you the the
20:43package and the version that I will use
20:45that is the 4.6.
20:48So now we know that to install R is APK
20:52R
20:53that I guess yeah it's okay it's already
20:56installed
21:01and but yeah I it's a bit uh yeah bit uh
21:05I mean tricky depending on the operating
21:08system you will find different ways of
21:10uh so that's why I recommend like before
21:12building to do this in an interactive
21:15way to know power and to check the
21:19version of R. I think it's an R minus
21:22minus version. Let's see if that works.
21:26I print
21:28the version. Yeah, it works. So now that
21:31we know how to uh
21:35the commands that we need to run, I will
21:37use edit uh docker file.
21:43So I will uh do replace this
21:48use R and here in Python I will use R
21:51version.
21:54Oops.
21:56Okay. The way to is
22:02Oh, okay. I'm doing something is alt f
22:13x.
22:16Okay. So uh now to build the it will be
22:21mal image build
22:27minus t and docker.io IO
22:32a username
22:34and this will be
22:38R
22:42and where where the docker file is is
22:45that is this here
22:52okay we cannot use upper case
22:55so what applying R in a lower case
22:59Sometimes when we get an error is good I
23:01mean good
23:05sometimes if it's a wrong error is
23:07something that is we could turn to okay
23:10what is that what is happening to panic
23:13let's say but uh some in some some cases
23:17or well we see most of the cases but
23:19yeah sometimes it's not the the case
23:22that reading the the solution is ling
23:25the error message so here you get error
23:29that the tag docker io invalid reference
23:32format repository name must must be
23:35lower case. So I change it and yeah see
23:39if that works
23:47and yeah it works. Now do podman
23:52uh image uh ls
23:56and yeah I have now alpine r and alpine
24:00python.
24:03So if I run pol container run
24:12yeah it will show the
24:15so this is the way of uh creating an
24:17image once that we are happy with the
24:20image and we test the image and we see
24:22that everything work etc what I can do
24:26is I can upload that image to docker
24:30let's see because this might work or or
24:34not.
24:36Uh we'll um
24:42uh we'll see how to do that. Uh
24:46well actually yeah the command is easy
24:49but yeah if you get any errors let me
24:53know. So this will be pm
24:56image push
24:58and the well this will be docker.io
25:04and username
25:06and I will use a pi. Python I can upload
25:11the other one but
25:14so if I do that is a creating images or
25:17signature. Um, now it's a copying the
25:22the image that has been built to a
25:25Docker hub.
25:35Takes uh some time to upload.
25:45In the meantime, I will open
25:49this one.
25:53Okay. So, yeah, now it's done. And uh I
25:58will open this one.
26:00Let's see. This will be good enough.
26:10No, you said the address is hard.
26:14docker.com
26:20I will now stop seeing
26:24s
26:26repositories.
26:34So if I type this
26:38uh copy I will paste that in the chat
26:48again. This I mean maybe not but reminds
26:52me of
26:55a bit of like a GitHub uh in the sense
26:59that you have like some repositories you
27:01have. So here I have my image that the
27:05last push was a minute ago. If I click
27:09here
27:11in my case for instance I don't have
27:15eco. This is like just an example and I
27:19I would try to make this.
27:26I can add here a description that
27:29yeah doesn't show so well
27:32or add a category of uh
27:36and say
27:38for instance language and frameworks for
27:40in
27:44so I have a category here description is
27:46uh
27:48I can say I'll
27:51find
27:56Python install
28:04and what the tag that I have here is the
28:06latest the operating system is yeah
28:08Linux image that's
28:12I can also add an overview
28:15that this is yeah this is this
28:17incomplete and this will be like a kind
28:19of the red me of
28:23I can check the tax.
28:29I don't know what is image management.
28:37Aware this is like are the previous
28:39versions of this uh image
28:43collaborator. I can add some people that
28:46will give be given push and pull access
28:48to the repository. But what a repository
28:51is public. So everyone can have access
28:53to the
28:58I can do a static scanning. Yeah, they
29:02have like some options here also. I can
29:04archive the repository or delete the
29:06repository that the repositories that
29:09were my username and the name of the
29:12image.
29:14Um yeah what the public view if I click
29:16here this will be the public view of the
29:21yeah doesn't have any overview but yeah
29:23I have here that is like a language and
29:25frameworks and the description here
29:29tags
29:34I think if I click here
29:38yeah is it's possible to see the
29:44the layers each layer is like the
29:46commands that we have.
29:49So this is to install Python. Yeah, the
29:52cmd.
29:59Okay. So now we know how to create a
30:02image uh how to upload it to the if
30:05someone wants to use this uh image uh
30:08the only need that they need to do is
30:15to do that what will be pm image pool uh
30:20one frcker.io
30:25IO that this is there
30:28and we see I don't know
30:31how what this uh image has been
30:34downloaded 351 times
30:38not sure who who wanted to use it but
30:42and has been star zero times that that
30:46makes more sense but this is a
30:51if I click here yeah if to my profile
30:55file community user repositories. Yeah,
30:57I have like a one repository
31:01that was updated a few minutes ago.
31:08I think that's it. Let me check if I
31:11missing something. Uh
31:14well, some uh general advice on how to
31:17build a image.
31:20Uh
31:24the idea here is also not to kind of
31:27reinventing the wheel or check if uh we
31:30have already on the docker repository an
31:33image that features Python, I will use
31:37that image rather than creating uh my
31:39own unless that for some reason I want
31:44something very specific on the image
31:46that is not there. But rather than
31:49starting with Alpine, I will start with
31:52the Python Alpine. Now I will not uh
31:57let's say create something that already
32:00assist.
32:02It's important as well to check the
32:04which uh operating system which Linux
32:08flavor I want to use. Here I'm using
32:11Alpine because uh let's say that it
32:13works.
32:16is good because it doesn't take that
32:18much space but uh you could find the
32:20case that uh you will need wu or deavian
32:24to uh don't know maybe your sho has some
32:29dependencies that are only available in
32:32deavian let's deviant let's say so you
32:35can use that one uh also check that
32:38don't install things that you will not
32:41need I mean on an image you can install
32:44python r you can install other
32:47programming languages. But if you only
32:50need Python, just yeah, keep uh that um
32:58um I think that's the
33:03yeah that's it. So kind of the procedure
33:05is that what we have seen uh to test
33:08that inter in an interactive way to see
33:11that the commands work and then write
33:17the docker file build it and if you want
33:20you can upload it and you will have it
33:23on your on the docker repository.
33:28if you have a any questions.
33:34If not, we can continue. Let me see
33:42that I'm not forgetting anything.
33:54Okay. So uh stop
33:58again my
34:00terminal
34:19actually that's yeah
34:21I would say it's an important question
34:24to have like uh when because yeah okay
34:27that's a let me sure
34:33and that's something that we should be
34:34careful when we uh go to a when we use
34:38an image okay and here let's go to
34:42explore so uh I said that well the
34:46question is about I want to use python
34:48but uh uh how do I know that the let's
34:53say an image with python is this is the
34:56right one is good. So when I search here
34:59I have a Python
35:05did I search?
35:15Oh well okay here you see explore
35:19official im images images images and
35:21python you should look for the docker
35:24official image that mean that means that
35:28is uh has been let's say per review or
35:31is what per review that yeah that is um
35:37yes an official image so uh
35:41and is maintained by the docker
35:42community and but you have here So uh
35:47also another uh good uh is there like
35:50the number of do you see that this image
35:53this week has been uh download
35:5821 million times.
36:02So for instance, this uh image makes
36:05sense that looks uh
36:08I mean I will I will use this one or for
36:10instance one that they have a verified
36:12publisher
36:14could be the case that for some reason
36:16you download an image that is from that
36:20maybe don't have this uh uh docker
36:22official image or it's like a maybe less
36:26u difficult but you can find that uh I
36:31could find later an example But uh again
36:34when you when you read a paper on for
36:37instance when you use uh uh these
36:41containers and images are quite uh used
36:43in uh machine learning deep learning
36:47when when you uh run a model. So uh
36:52most of the models they provide a
36:55container and uh you should check for
36:59instance that the the the
37:03name match with the author. So this kind
37:06of a bit well common sense. Yeah that
37:09are like some ways to uh check.
37:12I'm trying to see how can I search here
37:18because I type Python.
37:22Ah, see our search result. So I search
37:25for Python
37:27but probably well you see that this is
37:29sorted by the official image uh sponsor
37:35or open software.
37:38But here for insta
37:51well they have like a 1 million loss but
37:54for instance this one is uh what
37:57repositive for Python jobs
38:00that was last updated.
38:02Well, they have a few, but for instance,
38:06this one, you don't know what is uh
38:12well, okay, they have a website
38:17look. So, that's the thing that you need
38:19to be care because you are looking for
38:21Python and maybe you have like a plenty
38:24of
38:26images here that uh but maybe you
38:28download something that is not what
38:30you're looking for.
38:36So for instance this one the docker
38:38official image that gives you like some
38:41certainty that is not sure that
38:44your question also for instance we were
38:47searching before for r
38:51our base bas is the
38:53here again we have is a docker official
38:55image or uh
39:00I think was pytor watch
39:04that is quite useful.
39:13Oh well, actually this one is hard.
39:18Well, if I manage to type it well,
39:37you see for instance a Pytho that is
39:39like a
39:41doesn't have this but it's like a
39:45have been downloaded more than 10
39:47million times. uh tags. So
39:51for instance, if I want to use PyTorch,
39:53so yeah, I could have a look at this one
39:55that we have a
39:59since that is a reliable you have all of
40:01these tags that uh what this is for
40:04CUDA.
40:06I want you see this image is 8.6 GB.
40:12So yeah, Py torto is quite of a big uh
40:16and also I think provides the tens or no
40:20for you see that this image for instance
40:22is quite big is 8.6 six.
40:25But what this is like for PyTorch. So in
40:28if I want to use run PyTorch, I will
40:31download this image. I will not deal the
40:34uh with CUDA. I will I will need to
40:36check the right version. But if it's
40:39already here,
40:45this seems that it's like a kind of
40:48reliable
40:56Yeah.
40:58Okay. So, yeah, this is like a regarding
41:01the So, what I will uh do now is because
41:05we have seen how to create a container.
41:08But what about if I want to uh
41:15use a file that is or like a use a files
41:18that are on my uh on my laptop. Let's uh
41:22for instance because now we have Python
41:24but I want to run a script in uh that is
41:27in my laptop that use a Python without
41:30installing it. So let's uh see how we
41:33can do that. Uh
41:43Okay. So we are here and uh if we go to
41:47the
41:50to this repos
41:53folder one uh directory back if I do I
41:58see that I have a folder that is called
42:01s
42:03and here I have a docker what docker
42:05file and this we have also this uh
42:09python script this editor somewhere I
42:13and have a look and see
42:17the Python script what it does.
42:21Uh, let me see if I can exit.
42:27I think I I don't think that they have
42:29Python here installed on. No, I don't
42:32have Python installed.
42:34Python 3. No, it's not working either.
42:39uh
42:43what it does is uh we can provide
42:47several numbers and we can get the
42:49addition of these uh numbers but because
42:52I don't have Python on my machine I will
42:55uh I will try to I can use my the
43:00uh the container the the image that I
43:04have created and run this uh file file
43:08or you yeah run this file. So this will
43:11be p
43:13container
43:17run the name that will be I will use the
43:21shorter one
43:24I python
43:26and I can use python 3.
43:34If I use that I will get an error here.
43:37So uh yeah Python 3 works but they will
43:41say they can't open file that not not
43:44such a file of directory.
43:48This is because uh
43:51if you remember the well you remember
43:55the
43:56the file system on the
44:00on the container is different of the
44:03file system that I have on my uh on my
44:07laptop. So if um we do this, if we do ls
44:11here,
44:17you will see all of these. You will see
44:20that the
44:25on the container you have all of these
44:27uh directories. But if I do ls on my
44:31laptop, I have like a different uh
44:34output.
44:36I have this uh output here because five
44:39systems are different.
44:42Uh one way to uh to uh solve that issue
44:50is I I will need it to mount
44:53this directory
44:56on the on the container and uh it will
44:59uh run.
45:02What is uh the way to do that? This will
45:04work complicated. This will uh yeah be
45:07the command will be a little bit longer.
45:09So that will be this will be o mount
45:13container run and here I will need to
45:16use the option mount
45:20mount has a three different uh parts. So
45:25they have a type
45:29equal we will use in in this lesson uh
45:33bind that are two two other types but uh
45:38we are not well we we don't need them we
45:40are not covering in this lesson. So
45:43remember that uh to mount we would use
45:46type will be pine
45:49uh source
45:52that uh we can specify the
45:58we have an environment variable that is
46:00pwd that is like a
46:04personal personal working directory is
46:06the the directory where I am. So uh this
46:10should work like this
46:13also you see that that change of color
46:16this because it's a
46:18uh environment environment variable
46:22comma and the third part will be uh the
46:27target where this will be mount. So this
46:30will be mounting slash 10.
46:37Now the name of the m the image alpine
46:41alpine python
46:44and again python 3
46:49sound py
46:53but uh as you can imagine this fails
46:57again because uh notice that I executed
47:01this file but this file again is not in
47:04uh
47:11is not there is in a is in a SL tag. So
47:15if I do that
47:20you will see let's go step by step mount
47:24container run this is what I have added
47:29on the alpine python and the ls
47:3310
47:35you see so just to
47:40I will need to do 10
47:44if I And this
47:47this must work well the
47:51it's a zero but I can add 11 12 or 13 15
47:59and now now it works
48:03but uh I will need to specify all of all
48:07of these auto continue well all of these
48:10mount
48:11the source that will be made My current
48:15def
48:20is this. So this directory will be in /
48:2510
48:27and then I can execute Python 3 or where
48:30the repository is.
48:38Is that uh
48:42yeah hopefully it's well clear is I know
48:45that yeah that makes things a bit uh
48:48yeah longer but this is the way of
48:51mounting a repository that uh sorry
48:55repository
48:56a directory that I have in my laptop on
49:00the on the container
49:09Uh
49:15I've done before ls. This is like a
49:18command that maybe you may find useful
49:21on my container.
49:30Uh
49:32pine python
49:42well actually no let's use let's use the
49:45previous we have executed this
49:50so here if I do ls
49:56you will see that the listing by default
49:58goes to uh to uh uh root
50:03but uh
50:09I can specify minus w that this will be
50:13the
50:15this will be my let's see the working
50:17directory the current directory on the
50:19container so here I can specify last
50:21time for instance
50:24and if I do ls
50:27Now
50:29you see that now I'm in the SL 10
50:34because I've added this uh the working
50:37directory will be SL 10.
50:44Another option that uh may be useful is
50:47to specify the username. I said before
50:49that uh we have a a
50:57man root.
51:00Uh but I can
51:03use the following. So I can
51:06say
51:09minus u um choose the username. I mean I
51:13will choose the same username. For
51:15instance, my machine is a Jig 6.
51:20So if you skip that
51:25Oh, that will uh
51:32Yeah, that will uh fail because it's not
51:36in the
51:38Yeah, we will be needed to use a Yeah.
51:42Okay, that case is not
51:48okay. So there is a question that
51:55okay I the the
51:59minus w is to specify the working
52:01directory of uh so um
52:07let's go
52:10let's forget about this amount
52:21So here if I do so or let's say pwd
52:26you see that I'm in uh let's see the
52:28working directory of the contain of the
52:30container is uh root but I can specify
52:35here
52:39uh minus w for as I can say that this
52:42will be uh
52:45I can specify that will be in this
52:48directory.
52:50So now if I execute the container you
52:54will see that now I'm in that uh
52:56directory rather than so what you
52:59specify with the minus w is the
53:03directory that uh the container will uh
53:06when you run the container the direct
53:08the direct directory that it will be you
53:11can also specify /tmp
53:16media empty
53:19exactly Yeah. Yeah. Enters the container
53:23in the specified working directory.
53:27The thing if I do here I can use TMP.
53:34Now will be
53:39you see that in this case for instance a
53:41tmp now what we have used before
53:46it is oh no it's mountains last time
53:59we can also get this error uh preparing
54:02container error the work DS system does
54:06not does not exist. So it will fail. For
54:08instance, if we choose a directory that
54:10is not uh there before it was working
54:14because we were creating a new with
54:17mount we were creating the new uh
54:20directory. This is one option that yeah
54:23could be used or not. Uh I need to
54:27investigate.
54:28I think this the option that I have
54:30explained before
54:32the minus u I think that one works if
54:36the if we are using uh maybe Linux or
54:41Mac rather than
54:44uh Windows in Windows it seems that this
54:46this is not working. I can check it
54:50actually the documentation will be
54:54container.
55:02[snorts] These are all the options that
55:04we have. Maybe that is a
55:19username of
55:24Yeah, that was failing before
55:29because I think it only works on uh
55:31maybe when you're on Linux or
55:34I need to investigate that.
55:51Okay. Next. Where you see next step. Uh
55:59also one question is should I continue?
56:02Do you need a small break or you are
56:05okay
56:07to continue with the
56:10we can have a break later of yeah not
56:16sure if you needed five minutes or
56:36Is that okay? salary. Continue with the
56:44Okay, let me Well, let me check what I
56:53Well, let's continue with this and uh we
56:55can see later maybe because it's
57:00Yeah, let's continue. is that we can
57:02have like a a quick break [laughter]
57:04later but please if you need a break
57:07just to say that in the chat and I happy
57:09to stop for five 10 minutes or what the
57:12time that you need. Uh
57:17okay. So uh we were talking about um
57:25what we have seen
57:28yeah to use the because we have a image
57:31that has Python and we have a script a
57:34script but we can make the and actually
57:37that's the maybe the idea of the
57:40what container of to we can the image we
57:44can include that script in our uh image
57:50and uh we will not need to mount. So
57:54every time that we execute the the the
57:57image we create a container we will have
58:00the script uh there.
58:03Uh this has the advantage that uh again
58:07yeah [clears throat] you don't need a
58:08script on your laptop. disadvantage that
58:11if you update the script, every time
58:13that you update the script, you will
58:15need to create a new image. There are
58:18workarounds that could be that uh you
58:21use uh
58:23uh the image when it's built to use a
58:26GitHub.
58:30Yeah, there is a way that using a GitHub
58:32repository, but again every time that
58:35the image is built will take the the
58:39latest version of the GitHub repository.
58:42So if you update the GitHub repository,
58:45you will need to update the image as
58:48well.
58:51Okay. So uh I think the option that we
58:54have is safe. So we open
58:57this docker file.
59:04This is what we have. We can do uh we
59:07can use copy
59:10and will be the script some data py and
59:13we can copy in the /home.
59:17So if we include that when we build the
59:21image we will uh copy that script there.
59:26Let's uh
59:29see if yeah if that uh if that works.
59:37Yeah. Okay. So I need to save that. This
59:39is like a alt f say alt f exit
59:46and to build there will be both mount
59:50image
59:52build
59:54minus t
59:56and will be my use
1:00:03a username and there will be a pine
1:00:10So
1:00:12let's see and where where I am.
1:00:21You may notice here that this in the
1:00:24steps for building the I use the step
1:00:26104 from Alpine fine. The second step
1:00:31you will notice that is using cache
1:00:34because we have already
1:00:37built the
1:00:40we have already done that previously.
1:00:42This is cash. So this will not run
1:00:45again.
1:00:47This part yeah this is new. So this will
1:00:50be done. And now the step four. Yeah
1:00:53done as well.
1:00:55So you see that all of the layers each
1:00:59layer is a entry of the docker or
1:01:03container file.
1:01:08So
1:01:12if I now
1:01:14what can I do?
1:01:17What do ls port
1:01:20image
1:01:23ls
1:01:28we will see that we have a some yeah
1:01:31this one we have this new
1:01:36image here
1:01:38and if I run the I will run it in an
1:01:41interactive way to bang
1:01:46contain container.
1:01:53For instance, here I can use minus
1:01:55w/home.
1:02:01Oops.
1:02:24Oh, because I forgot the sh.
1:02:28Yeah. No.
1:02:39Yeah. See a question. I'm not sure if I
1:02:40am I will fail if you try to pull the
1:02:43same image from some other directory.
1:02:48The
1:02:56the thing is when I create the when I
1:02:58build the image, this is like in
1:03:01save on the on the laptop as
1:03:05oh well uh yeah the
1:03:11yeah exactly the that
1:03:16yeah okay know what I mean the
1:03:19in the recipe at the recipe assumes that
1:03:23the that file will be in the same
1:03:26directory as the docker file. If there
1:03:29are in different directories, it will
1:03:31fail.
1:03:33So that's why when you build the image,
1:03:35you specify the directory. You don't
1:03:38specify. If you see it here
1:03:43when I build the image this is like a it
1:03:47means the current directory. So in the
1:03:50current directory I have the docker file
1:03:53and I have the python script. If the
1:03:55Python script is not here, it will fail
1:03:58this in
1:04:00or for instance, if I have a newer
1:04:02version or something, this will uh fail.
1:04:09So I'm running this
1:04:12and now if I do well, I can move to home
1:04:18and if I do ls, I have here my script.
1:04:22So I can do Python 3
1:04:28and
1:04:32I will get the
1:04:36and the option that I have tried before
1:04:41that you can say minus W home
1:04:46and you see that now rather than uh
1:04:50starting the container I'm going to to
1:04:53the to root directory I will go to /
1:04:56home directory and here I will have some
1:04:59and I will I don't know if vi will be
1:05:02available here but also I can edit the
1:05:06file if I want
1:05:12and
1:05:15an advantage of use containers for
1:05:17instance if I oops
1:05:20if I modify the script
1:05:24and uh I see that for some reason
1:05:28I don't know the the version is not
1:05:30working or something is uh if I create a
1:05:33new if I run the image again it will uh
1:05:38it will uh take the the original uh the
1:05:42original file will not contain the
1:05:46so if I now do python
1:05:493
1:05:52Oops.
1:05:57There you see that what I modify is I
1:05:59added this.
1:06:03But if I exit and run the container
1:06:06again
1:06:08and do a
1:06:10Python
1:06:123
1:06:20you see that now I I I will always get
1:06:23so this uh script will be like with a
1:06:27container. So
1:06:29if you publish a paper or something and
1:06:32you include that in your uh container
1:06:34this will be there I mean forever. The
1:06:38this is what we are talking about the
1:06:40make something make research or make the
1:06:44experiments reproducible
1:06:47that you will have there the
1:06:50uh the script that you have used. You
1:06:52have the environment that you have the
1:06:54Python version that you have used.
1:06:59You will have the script that you have
1:07:02used that is this one. So when you
1:07:06provide you don't provide only the when
1:07:09you provide let's say the container you
1:07:11don't prov you you don't provide the
1:07:13only the script you also provide the
1:07:16software that is uh the conditions to
1:07:19where the script has been executed
1:07:21because it could be the case that for
1:07:23this version of Python
1:07:26maybe the
1:07:28library or something is different that
1:07:30other versions of Python and uh the
1:07:33resource could be different. This
1:07:35happens a lot when again we use like in
1:07:38deep learning, machine learning or we
1:07:40use a cluster that has a Kubernetes.
1:07:44uh Kubernetes what it does is you create
1:07:47the you need to specify the container
1:07:50the image that you want to use to run
1:07:52the job and uh
1:07:56yeah so that's also like one of the
1:08:00characteristics that why uh this is a
1:08:05containers are useful
1:08:26Okay, that's a no.
1:08:30This is a good point
1:08:33if you notice here. No. Okay. Okay. That
1:08:37that that's interesting. Then
1:08:43yeah I will exit this container. Okay
1:08:45that's a thank you for the questions
1:08:47that we have docker file. If we open
1:08:51docker file,
1:08:56you see that I'm copying I copy the
1:08:59script to the container.
1:09:06And when I run the container, I run it
1:09:08in this way. But they say this is only
1:09:11the
1:09:13the working directory on the container.
1:09:15We can remove this to make things
1:09:18easier.
1:09:19So you don't notice here I'm I'm not uh
1:09:23mounting anything. So uh the script
1:09:26actually I can delete the or I can
1:09:29rename this script and uh it will work
1:09:33the same because the the image
1:09:36already contains this script.
1:09:40But if we
1:09:42mount that script here, let's see if I
1:09:47find the
1:09:54I do the little of this to make things
1:09:56easier.
1:09:58My container run. You see that I
1:10:01mounting the the the folder here.
1:10:06I will need to do minus 80.
1:10:12So here if I go to SL
1:10:16I can modify the script here and if I
1:10:20modify the script
1:10:24it will uh the changes will be when the
1:10:27container stops the changes will remain
1:10:31because I modifying the file on the
1:10:40Exactly. So yeah that that's the point
1:10:43that if I mount the directory that is in
1:10:47my laptop on the container the
1:10:50modifications will remain but if I
1:10:53modify something on the that that this
1:10:56is like one of the advantages of the
1:10:59uh containers and that's why we are
1:11:02mentioning that we have a the image and
1:11:04we have a like from one image we can
1:11:07create
1:11:09different copies let's
1:11:11will be different containers. So let's
1:11:14do here the sample. If I modify this
1:11:16here
1:11:21and now I exit
1:11:24if I I'm not sure if cut will work here.
1:11:28You see that the that this has been
1:11:30modified and actually the last time last
1:11:34right time is now it's uh 21 past uh 2.
1:11:40for 20 2021 past two.
1:11:45So this is also something that you need
1:11:47to yeah be be careful that yeah that
1:11:51so yeah it's a bit yeah I mean makes
1:11:53sense the I think was a bit yeah I mean
1:11:56confusing yeah
1:11:59I think I explain like at the same time
1:12:02the one thing is mount that is like a
1:12:05one further of the because so far not uh
1:12:11if we run the container itself the it
1:12:15will preserve or will not do any changes
1:12:18to the erh files that are in the laptop.
1:12:22But at the moment that they mount a
1:12:25directory from the laptop this will this
1:12:28can suffer changes or if we create new
1:12:32files this will for instance we are
1:12:35using a container or yeah we are running
1:12:38a container to create uh some data files
1:12:41or to create a plot or to
1:12:45uh even for latt to uh generate a PDF
1:12:49out of tech file.
1:12:52Uh we can use a container and uh mount
1:12:57one of the forces of that uh of that uh
1:13:01of the laptop.
1:13:07Let me see if I'm forgetting something.
1:13:18Okay, I haven't explained that.
1:13:21I will
1:13:30scroll here.
1:13:35You remember when we have a compile this
1:13:38uh
1:13:41uh yeah this part
1:13:46you see that for this uh run up ak and
1:13:50then I added a copy you see that here
1:13:52they said using cash so it's not running
1:13:55this uh is not executing this this layer
1:13:58again
1:14:01this will be different if we
1:14:08If uh we put the copy here in instead
1:14:22will make things you see that the
1:14:24container is the uh
1:14:28same I mean the same recipe let's say
1:14:30the order is different but I will not
1:14:32make any let's see changes but when we
1:14:36uh compile the
1:14:39oh this is
1:14:42I don't know if I do exit okay they will
1:14:45ask me to say yeah fine if now I do
1:14:51image build
1:14:57again docker io
1:15:00my username
1:15:08If I do this now, you will see. Well,
1:15:11let's see if
1:15:18you see that it's no longer using the
1:15:20cash because the the the order is uh uh
1:15:25different. So it is important that when
1:15:29you do some uh by it's recommended
1:15:34that uh let's see to keep this in order
1:15:37if you do a modification do it on top of
1:15:40so it's like a one layer is a built on
1:15:44top of the previous ones. So at the
1:15:46moment that you change the order of the
1:15:48layers it will uh I mean be less
1:15:52efficient because it will run this step
1:15:54again. Let's say this is a
1:15:58tiny sample so it's not a big deal but
1:16:01if uh
1:16:03uh we are building like a larger
1:16:06container and that will make things and
1:16:10I think it's a
1:16:14I don't remember I think there are like
1:16:16some uh guidance uh to use like the when
1:16:20we open the
1:16:24there are some guidance saying that uh
1:16:27well it depends like a copy must be at
1:16:29the
1:16:33I mean try to do like a as much as
1:16:36possible like to build on top of layers
1:16:38that you have already used
1:16:43because otherwise if you change the
1:16:45order
1:16:47uh yeah that uh the previous layer will
1:16:52be like wrong again. I mean build again.
1:16:56I think
1:17:01something I think this
1:17:05exit say. Yeah, that's to keep the
1:17:08original.
1:17:24Okay. So, uh
1:17:29maybe we can have a quick uh
1:17:33break.
1:17:37[clears throat]
1:17:38Well, any questions before
1:17:43I just trying to
1:17:50I want this is interesting this part.
1:18:00Okay. So maybe we can have a it's a 15
1:18:03minute break. Okay. And we'll be back at
1:18:06uh quarter 2 three. I think for the the
1:18:10lesson well we need to like explain some
1:18:12other things but I I I think that we can
1:18:16finish like uh uh we will not we can
1:18:19finish before four.
1:18:23So uh maybe we can have a yeah
1:18:28like a 15 minute break and be back at
1:18:31quarter 2 three and finish the lesson.
1:18:34Maybe the the remaining will be maybe
1:18:37half an hour or so. Is that uh okay?
1:18:42>> Yeah. All good.
1:18:44>> Good. Thank you. So I will uh Yeah. So
1:18:48we'll be back. I will write that.
1:18:59Yeah. So, this will be the new time.
1:19:01Yeah. I will share the again the
1:19:03counter. Yeah. See you in a bit. Again,
1:19:06if you have any questions or any
1:19:07comments, put them in the chat. I can
1:19:10solve them after the break. See you in a
1:19:13in a bit.
1:19:22Okay. So, uh let's uh continue. I
1:19:25noticed that I put a large break about
1:19:27that afternoon break, but it's
1:19:32okay. So uh what we will see now is uh
1:19:37keep working with well the containers
1:19:40how to make them uh because we have seen
1:19:43that uh how to build a container and how
1:19:45to include a script. So there are other
1:19:49ways that we can run the script on the
1:19:52container. So we will see them and uh we
1:19:56will see a little bit of part theory
1:19:59about the containers. Also there is a
1:20:02interesting uh article that explains the
1:20:05like the
1:20:07uh 10 points that you should have a look
1:20:11for
1:20:12um
1:20:15you should take into account when
1:20:17building a a container. Um yeah, that
1:20:21will be uh us for uh for today and we
1:20:25will be ready for uh Wednesday because
1:20:27on Wednesday we will work on archer 2
1:20:29and we will see
1:20:32uh how to use uh containers on a let's
1:20:36see on a superco computer or in a
1:20:38different uh uh computer that is not us
1:20:43and we will use a singularity for that.
1:20:49Okay. So uh I'll
1:20:53yeah share my time
1:21:00uh let me okay so what we have what we
1:21:04have done is we have included let's
1:21:11yeah we have included uh the script uh
1:21:14here fine
1:21:17uh there are two But I will there are
1:21:21two options
1:21:23uh because what we are doing if we want
1:21:26to use the script
1:21:30okay we have the script there. So one of
1:21:32the options will be
1:21:35um
1:21:40yeah so this will be podma
1:21:44container
1:21:46run align
1:21:50iPhone.
1:22:01No, it's uh home
1:22:06dy
1:22:08will be 10
1:22:1020
1:22:13H.
1:22:15Oh, because it's not alpine Python. is
1:22:17alpine. So, so I'm using the wrong
1:22:21image.
1:22:25Okay. So, this is one option that we
1:22:27have to run the container. So, podman
1:22:31container run the image and then I need
1:22:34to remember
1:22:37to execute Python 3 where the script is
1:22:40and the
1:22:43so one of the what we can do is we can
1:22:46include the that
1:22:50we can include that on the
1:22:55on the command that is always used and
1:22:58see Oh that uh so if I do edit docker
1:23:02file
1:23:05here rather than python 3 minus minus
1:23:07version I can use uh /home
1:23:13py
1:23:18and the the numbers will be uh okay so
1:23:21let's do that
1:23:25exit save
1:23:27so we'll be now OM
1:23:31uh no will be image build
1:23:41this will be docker do IO
1:23:48I
1:23:50s
1:23:52I can specify for instance a version
1:23:54version one I can I will choose that tag
1:23:58and
1:24:00and the directory.
1:24:04So this cast
1:24:07okay so now we I have this and I will
1:24:09try.
1:24:12So I don't specify this.
1:24:16I specify the version version one.
1:24:20But uh if I execute this uh this will
1:24:25not be taken as uh the arguments of the
1:24:29uh
1:24:31the 1020 will be considered as a
1:24:34command. That's why the they said
1:24:37executable file 10 not found in path. So
1:24:41we need to do uh something else and
1:24:43actually we will learn something new
1:24:45here
1:24:48that what we are learning is we need to
1:24:50use u
1:24:53we will use what is called an entry
1:24:57entry point.
1:25:00So uh when we type something on the when
1:25:04we run the container this will consider
1:25:06arguments rather than
1:25:10uh rather than the
1:25:14I will say it one day uh
1:25:20yeah arguments rather than a command. Uh
1:25:25so
1:25:27this yeah so what I need to change here
1:25:31and as I said is important to add if we
1:25:35can to add some comments. So we can do a
1:25:39rung
1:25:47for instance something like that. Uh so
1:25:51the new entry that we will use is entry
1:25:55point.
1:25:57Actually I can
1:26:02use entry point.
1:26:09So if I use that
1:26:11this will be the now when when I type
1:26:15something on the when I execute the
1:26:16container this will be taking as
1:26:19arguments rather than the uh a command.
1:26:26I will uh give a default argument
1:26:36and I will use cmd.
1:26:40Uh you can specify for instance a zero
1:26:47zero for instance. See if that
1:26:52uh
1:26:58okay I think this must work. So I will
1:27:01uh
1:27:04save.
1:27:09So just to show you the the the change
1:27:13that we have done is that we are using
1:27:15entry point rather than the cmd.
1:27:19So if we do that uh let's build the
1:27:22container. So uh for image B time
1:27:47yeah well let's use let's keep the
1:27:50version one because the previous one was
1:27:52not working.
1:27:57And now we can see
1:28:13Yeah, now you see that it work. So this
1:28:16would be like a maybe a more elegant way
1:28:18to uh provide the container because in
1:28:21that case you see that we not we don't
1:28:24need to specify Python 3. We don't need
1:28:26to specify the name of the cont the the
1:28:28name of the script. So that will be a
1:28:33an elegant uh way. What if for instance
1:28:37because in this case I have a shift any
1:28:41questions
1:28:43so far? If not I can proceed and explain
1:28:47the let's see the other way to make this
1:28:50uh the squid like a more uh
1:28:55eleant.
1:29:05Yeah.
1:29:10So this is one way. Another way that for
1:29:13instance if imagine that rather than one
1:29:15script we have a several scripts. We
1:29:18have a sum, we have a multiply, we have
1:29:21a
1:29:25uh
1:29:28that that would be gooder to specify the
1:29:32the script that I want to run on. uh on
1:29:36the
1:29:39on the container. So, one of the ways
1:29:42that I can do is uh
1:29:48yeah, let's edit that. Uh
1:29:53oops.
1:29:56I know because I don't have the user
1:29:58edit.
1:30:05So well this is like one of the
1:30:07solutions that we but another solution I
1:30:11will delete that here.
1:30:16Another solution that I can do is uh I
1:30:19can uh do uh
1:30:22change the the permissions of the
1:30:25script.
1:30:32So I will make this script will be a
1:30:36executable. So I will not need Python to
1:30:40execute it using Python 3 s. So we will
1:30:43run iter
1:30:45and I will uh create an environment well
1:30:48I will uh add to the path /h home and
1:30:53this will be the path.
1:30:57So if you are not familiar with the
1:30:59Linux here I'm I'm providing like a
1:31:03executable rights to this script and
1:31:07when I run this script uh
1:31:10by itself it will execute Python uh now
1:31:13I will explain why uh what I'm doing
1:31:16here is I'm defining the environment
1:31:18variable path that uh when I execute
1:31:23this
1:31:26uh will be searched here in /home.
1:31:31So I will uh save this.
1:31:34Not sure if that makes sense. Oh, what I
1:31:37wanted to if I do edit some
1:31:44this part that is at the beginning of
1:31:46the script means that if the executable
1:31:50file sorry this file is executed as a XF
1:31:55file let's say
1:31:57along uh will use Python 3.
1:32:04So if I oops I need to
1:32:09exit.
1:32:12So if I execute something like a
1:32:15py
1:32:17will use uh python 3 I will not need to
1:32:20specify python 3.
1:32:25I will not need to do this. Just by
1:32:28doing this
1:32:31uh the operating system will know that
1:32:33we'll need to use Python 3 because of uh
1:32:37this uh first line.
1:32:45Hopefully it makes sense.
1:32:47Oops. Uh this
1:32:51I will save I will exit without saving.
1:32:57Now if I compile the PMA
1:33:03will be PMA image
1:33:05B
1:33:08again. Yeah.
1:33:20And this will be version two.
1:33:27And let's see if that uh so this other
1:33:30way will be map container
1:33:34run I can use some here I specify the
1:33:39script this is useful for instance when
1:33:41again we have several scripts and I can
1:33:44put all of them in the same container
1:33:48and here I can specify 10
1:33:52um
1:33:58Oops.
1:34:01Ah, because I forgot to specify the name
1:34:04of the
1:34:07image that is sam version two
1:34:14and you see that yeah this is like
1:34:16another way. So we have different ways
1:34:18to do the same. Uh depending uh I will
1:34:21advise this way when we have several
1:34:23scripts. If we have only one
1:34:27probably
1:34:29this way will be nicer.
1:34:37This is all on the documentation but I
1:34:39know that it's like a bit of um a lot of
1:34:43load. Any questions uh about this or
1:34:48anything
1:35:18So, uh,
1:35:30yeah, I think okay, that that's a I need
1:35:33to but yeah, that uh when
1:35:39no that's an interesting question when
1:35:41when we run a container
1:35:43uh and the container finish uh for uh if
1:35:48something is going wrong just to double
1:35:50check I think we can uh I need to I I
1:35:54I'm not sure if I remember the syntax
1:35:56but if I let let's try this
1:36:01container
1:36:04ls
1:36:07Well, you will see that. Yeah, I have
1:36:08like a lot of
1:36:12But if I want to
1:36:20Yeah, let's see if I can I
1:36:25actually Let's check the
1:36:27mount container.
1:36:42They make this.
1:36:48There is a way to
1:36:52uh
1:36:57Okay, here
1:37:02I know this is a sandbox image.
1:37:06There's also a command in a new oh no a
1:37:08new container even
1:37:11there is a way to uh for instance a
1:37:13containers that I have already run to uh
1:37:16access them again the thing is I don't
1:37:19remember now the
1:37:21the way I will check I'll let you know
1:37:24but uh that that is a way of uh uh
1:37:30yeah like a connecting there is a way so
1:37:33uh actually
1:37:40So we have oh well for in yeah you can
1:37:43attach to a running container
1:37:46clone an existing container
1:37:50you can create but not start a container
1:37:55run a pro. Yeah you have all of these
1:37:57options.
1:38:02You can keep the running container list
1:38:06or
1:38:15oh could be a start maybe
1:38:25don't see that works but
1:38:31now.
1:38:34Yeah, I need to check. But yeah, that
1:38:37must be like a one of these options that
1:38:38are here in the I think that rather than
1:38:41pman container run would be pman
1:38:43container and something else.
1:38:47But yeah, it's possible. This is more
1:38:49for debugging to see what was wrong. And
1:38:52yes, see if uh
1:38:55uh I think let me check this P.
1:39:03Well, this is another way that uh if
1:39:05you're using inspect, you will see all
1:39:07of the
1:39:09all of the configuration of uh that the
1:39:12container that I run
1:39:16the arguments the path was created
1:39:20state. Yeah, maybe that could be I mean
1:39:23useful if something happened or
1:39:26something was not
1:39:31cool. So uh what I wanted to show you
1:39:34that yeah now we said this is what let's
1:39:36see what there are more things that uh
1:39:41well could be done with the containers
1:39:43but uh this is like let's see the
1:39:45practice now we will see a little bit of
1:39:48what theory and yeah finish uh for today
1:39:52I will stop shading and I will share my
1:39:58this Oh,
1:40:03so uh the article is quite long. Uh it's
1:40:07better if I mean you have the time and
1:40:09you go. But uh is I I I find it quite uh
1:40:12useful to uh uh to read and to learn how
1:40:17to uh these are 10 simple rules for
1:40:20writing docker files for reproducible
1:40:23data science.
1:40:25So what you have here the all of the ana
1:40:28open this article is open access so you
1:40:31can get access.
1:40:34So these 10 rules are this one. Use
1:40:36available tools. Build upon existing
1:40:39images that this is the I mean we can go
1:40:43probably
1:40:44well yeah build upon existing images
1:40:47format for clarity document within the
1:40:50docker file specify the software
1:40:51versions that this is something to be
1:40:54yeah that let's see we haven't done use
1:40:57version control mount data set and
1:41:00runtime. Well, we have done that. Make
1:41:03the image one click one one click
1:41:05runable. That is yeah what we have done
1:41:07at the end of order the instructions. So
1:41:10well use and rebuild the containers can
1:41:14go like a step up. Well this image is a
1:41:19we have a docker file that we build and
1:41:21we have the image with the different
1:41:23layers and then we run the container
1:41:25with this. So the key valent if I mean
1:41:29it's not the same but kind of if we have
1:41:33a source code here we compile the source
1:41:36code so we have an executable binary and
1:41:40every time that we run the binary this
1:41:42will create a process on the operating
1:41:44system. So every time that we run it,
1:41:46this process will be different. The same
1:41:49as this. Every time that we run the
1:41:51container,
1:41:53sorry, that we run the image, we will
1:41:55have a different container.
1:42:02Uh here there is more. So let's jump uh
1:42:05to the rules. uh the use available tools
1:42:08is uh
1:42:11yeah there are like uh if we go to
1:42:15docker hub or we go to documentation we
1:42:18have already some templates with a
1:42:20docker file so that recommends that uh
1:42:24uh yeah use a docker file that is
1:42:27already there in case that uh you want
1:42:29to make a more complex uh uh workflow
1:42:34uh build upon existing images again. If
1:42:38you want to uh what we have said before,
1:42:42okay, we can get Ubuntu Alpine build
1:42:47Python build. But if we have already an
1:42:50image that has a Python, use that image
1:42:54rather than
1:42:56doing all of
1:43:00uh format for clarity to make the let's
1:43:04say the docker file to make it readable
1:43:07because this uh will be readed by
1:43:12now if yeah by uh people so uh Don't
1:43:18yeah give like enough space actually
1:43:22this comes for the next uh well here is
1:43:25like a way more but yeah I'm going like
1:43:27a quickly to yeah it's better to have a
1:43:30a better read with more time but uh this
1:43:34is aligned with this like a document
1:43:36within the docker file as I said you can
1:43:39you can have a comment using the hash
1:43:43key
1:43:44so use it.
1:43:48Oh yeah, you have a
1:43:50uh you can also add metadata. We have we
1:43:53haven't seen that but you have a label
1:43:55instruction that you can say
1:43:59who is the maintainer.
1:44:02Yeah, you can add some lay labels there
1:44:05here on on the docker file.
1:44:09Define the versions define the
1:44:12environment available. So yeah this is
1:44:14like a
1:44:17a specified software versions uh if you
1:44:20remember we used we have used Python so
1:44:23we have used well we have specified the
1:44:26version that is Python 3 so we do APK
1:44:30other Python 3 but uh what if we want uh
1:44:333.8 eight. What if we want uh
1:44:38for a library the specific version that
1:44:41when we use Python we have a file that
1:44:43is a requirements that normally we
1:44:46specify the version there to make this
1:44:49more
1:44:52consistent is important to specify the
1:44:55version of Python like a version and the
1:44:58for the libraries as well because every
1:45:01time that you build the container maybe
1:45:03you build a different version of Python.
1:45:05But if you specify it, you will always
1:45:08build the same version of Python.
1:45:11Hopefully that makes sense of when you
1:45:13use the version control. This is what I
1:45:16I mentioned before that uh our code that
1:45:19we have added to the to the image we can
1:45:25add it using Git clone.
1:45:27But here I use the branch I use this uh
1:45:30version. So of the check out I use
1:45:33exactly this specific version. In case
1:45:36that the repository gets a newer
1:45:40versions
1:45:41uh
1:45:43it will always take the same uh
1:45:46version of the script.
1:45:52And yeah here like when I run the pep
1:45:54install I specify the versions here.
1:45:58However, another way is to add this to
1:46:00the container. The requirements txt and
1:46:04run deep install minus r requirements.
1:46:07And again, requirements will contain the
1:46:10versions of the Python libraries that we
1:46:13will use.
1:46:15Rule six, use version control.
1:46:19uh
1:46:20Docker file you can save it on a you can
1:46:24save it on a GitHub repository and you
1:46:26can see the versions of uh different
1:46:29versions of the docker file. Actually we
1:46:32have seen in the here when we use a
1:46:35docker the docker hub you can click on
1:46:38the version and you can go to the you
1:46:41can go to the repository to the github
1:46:44repository. I mention all the time
1:46:46GitHub but we have a g lab I think there
1:46:49is another one that is a bit bit bucket
1:46:53I mean it's like a g repository
1:46:59we have done this amount data sets at a
1:47:02runtime so uh
1:47:10yeah because that will make the you see
1:47:14here that is mount type bid source that
1:47:18will make it's better to uh mount a
1:47:22folder rather than let's say copying the
1:47:23whole content or that will make the the
1:47:26let's see the container independent from
1:47:28the files that we want to analyze I mean
1:47:32I will never include the files on the
1:47:34that they want to uh analyze on the on
1:47:37the container one thing that is not uh
1:47:41well was included but I haven't
1:47:42mentioned important.
1:47:45Do not include the passwords. Do not
1:47:49include the We have seen a lot of uh
1:47:52forgot the name. It's not a password.
1:47:54there is uh some uh keys or some uh yeah
1:47:58keys that are included on the container
1:48:00and you upload them to docker or uh
1:48:06GitHub and this uh I think it's secrets
1:48:10this secret will no longer secret so I
1:48:14mean will keep see the word but everyone
1:48:19will see the password everyone will see
1:48:21the or if you use SSH keys
1:48:24Never include the private part of the
1:48:28key on a container.
1:48:32This is what we have done that make the
1:48:34image a one click runnable. So we use
1:48:37using entry point and cmd instructions.
1:48:41Here we have some examples.
1:48:45We have seen already that the rule name
1:48:47all the instructions here is what is the
1:48:50recommendation. So system libraries uh
1:48:54language specific libraries
1:48:56installation of our own software copying
1:48:59data configuration files labors and
1:49:02entry point. So if I you do a change
1:49:04here you will not need to uh install of
1:49:08go through all of these. So this will
1:49:11build in top of so this is like the
1:49:14recommended
1:49:15order.
1:49:17Uh this is a regularly use and build
1:49:20containers. This is to check if uh uh to
1:49:23check if the container the docker file
1:49:26is still like relevant or still works
1:49:29because for instance imagine that you
1:49:31haven't specified the Python version and
1:49:34you build the container in three weeks
1:49:36time and maybe picks another version of
1:49:39Python the script could stop working and
1:49:43so it's like a good yeah way to check
1:49:46that the uh the docker file is still
1:49:50relevant.
1:49:51But what this like a so you may as you
1:49:55see there are some examples it's like a
1:49:57quite long
1:49:59would be good to read the article like
1:50:01uh in your own time with and uh because
1:50:06this is quite a I found it I find it
1:50:09quite useful.
1:50:13Oh to go to the this is the lesson. This
1:50:17is the end of
1:50:22a samples of using container images in
1:50:24practice. We have a we don't see like a
1:50:27GitHub actions. This is normally when we
1:50:30uh for instance when you submit a code
1:50:34you can have an action to either compile
1:50:37the code and check if uh it works. Okay.
1:50:40You can check the syntax grammar. uh you
1:50:45can build an HTML
1:50:48site. So uh in that case normally you
1:50:52you uh use uh containers. So uh here if
1:50:57we go here you have like a an example of
1:51:00how to do that uh use for instance build
1:51:03a GitHub pages using pandoc
1:51:07but well this is like a you can go
1:51:08through this sample but yeah this is
1:51:12like a more related with the GitHub
1:51:15uh yeah a bit uh
1:51:18yeah we will not cover that but you have
1:51:21that on the on the the documentation.
1:51:24This is what we will see on Wednesday
1:51:26using containers on an HPC cluster. you
1:51:30will see how uh things uh I mean uh the
1:51:35I mean change uh because on uh Wednesday
1:51:39I will not explain on Wednesday we will
1:51:42see why we using singularity or obtainer
1:51:45rather than docker or podma on a on a sh
1:51:49resource like a archer 2 or supercomput
1:51:58Okay. Yeah. Thank you. Yeah, we are
1:52:01finishing
1:52:03soon. So, uh well, as I said, this is a
1:52:06recording. So, you want to go through
1:52:08the
1:52:10content this will be
1:52:15okay. So, uh and yeah, here what we are.
1:52:18If you have an example, you can always
1:52:20select a submit a p request. Uh this
1:52:23lesson is in the carpentry
1:52:26incubator. So uh you can suggest uh
1:52:29changes to it.
1:52:32And here there is like some theory
1:52:36that that are yeah I will go go quickly
1:52:40but well would be good like on your own
1:52:43trying to read it more carefully and if
1:52:46you have any questions so you can ask me
1:52:48on on a Wednesday uh or well or by by
1:52:53mail at any time. Again, one
1:52:59containers are quite important for uh
1:53:02reproducibility
1:53:05because uh every time that we run the an
1:53:08image uh we will use the same uh
1:53:12if we build the image uh
1:53:17specifying versions etc. we will use
1:53:20always the same uh installation of the
1:53:22same. If you open the docker file, you
1:53:25see the set of uh
1:53:28commands or instructions that you have
1:53:30followed to uh install and run the
1:53:33software. So this is documentation that
1:53:36increase the that uh that this could be
1:53:41reproduced by another person at a future
1:53:44time.
1:53:47uh sharing images you can share them on
1:53:50docker hub or another way that is
1:53:53explained here is uh you can archive uh
1:53:57and identify the container images using
1:53:59a sen the way to archive uh we can try
1:54:04that so will be podman image save and
1:54:07the name of the image our the output
1:54:10will be alpine python tar to compile it
1:54:14you need to use u I think in Windows is
1:54:17not installed but you can you can
1:54:19compile it with the uh GC or just zip
1:54:24and uh you can deposit it on seno.
1:54:29If you deposit it on seno it will create
1:54:31a DOI that this is a digital object
1:54:35identifier.
1:54:38Yeah, this make this bigger
1:54:45and you can reference on your paper or
1:54:48somewhere else on any documentation that
1:54:51you create this DOI.
1:54:58So good practice. Yeah. Again containers
1:55:01to uh because this is like everything
1:55:03documented on the docker file. You have
1:55:07a yeah you have the document what you
1:55:10have done. Why uh you have a DOI that
1:55:14you can reference. You can use make use
1:55:16of tax. Um
1:55:22yeah it's good to provide the let's say
1:55:25the build image but having also
1:55:28providing the docker file uh everyone
1:55:31will see the recipe understand and build
1:55:34it just in case that
1:55:38uh container granularity the here
1:55:44you have some positives and negatives.
1:55:47There are two options. You can the
1:55:49granularity means that you can have like
1:55:52a depending on the
1:55:55well workflow or what you want to uh you
1:55:59can have a big
1:56:01image that contains uh all of the your I
1:56:06mean if uh you have only one script fine
1:56:09you create one image and that's it. But
1:56:12if uh you have a several scripts or if
1:56:16this is part of imagine that you have a
1:56:18workflow that you have a pipeline of
1:56:22four different processes you can have
1:56:25these four different processes in the
1:56:27same image
1:56:30the advantage that yeah you have only
1:56:32one fine but disadvantage that you will
1:56:35have like a big um
1:56:38let's see that for one process you use
1:56:40Python for the second one Maybe you use
1:56:42R for the third one this runs uh using a
1:56:48no for so in one image you will have
1:56:51everything installed so in that case
1:56:54maybe more recommen it's recommended to
1:56:57use several
1:56:59here you have the advantages and
1:57:00disadvantages
1:57:03the advantages is simple to document but
1:57:06yeah could be very large in size having
1:57:09multiple
1:57:11is
1:57:13well more difficult to comment or maybe
1:57:15maintain but it will make things uh
1:57:20uh clearer. So uh yeah it's up to the
1:57:24test case that uh but uh yeah my
1:57:28recommendation or I would be tempted to
1:57:31yeah to have a multiple smaller
1:57:33container images rather than a big one.
1:57:39This is uh the what I wanted to cover uh
1:57:44today on the
1:57:46if you have any
1:57:50other questions before we
1:57:54leave. If not uh
1:58:01we can yeah we'll be uh we'll uh
1:58:06yeah so that's uh yeah that's all for
1:58:09today. Uh here yeah you have
1:58:12documentation have a look and play with
1:58:14the samples. So you have the article is
1:58:17like on the is here
1:58:22at the end. Yeah, the article is here in
1:58:25the in the lesson
1:58:29number.
1:58:32Yeah, seven. So, this is what we have
1:58:34covered. So, we in your own time have a
1:58:38quick look on and yeah, thank you for
1:58:42attending today. See you on Wednesday.
1:58:44On Wednesday, we will use uh Archer to
1:58:47uh to uh to run uh uh Singularity to uh
1:58:53we will see how to uh uh download uh
1:58:56images and run them on Archer 2. Uh have
1:59:00a nice rest of the day. Um yeah, see you
1:59:03on Wednesday. Bye. Bye for now.
1:59:05>> Thanks. Thanks a lot.

About this transcript

This page contains the full transcript of ARCHER2 - Containers for Reproducible Research: Introduction to Podman and Singularity - Session 2 by ARCHER2 HPC, generated from the public captions YouTube serves with the video. The transcript has 11,863 words across 1,889 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.