YT YouTube2Text

YouTube2Text → Transcripts → InfiniBand Principles Every HPC Expert MUST Know (Part 1)

InfiniBand Principles Every HPC Expert MUST Know (Part 1) — Transcript

by insideHPC Report · 8,964 words · 1,282 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:03okay I'll begin first with the
0:06introduction again my name is dead paws
0:09I'm from Mellanox I'm working in
0:12Mellanox for three and a half years
0:14about and doing training I'm part of a
0:18support division which means we are
0:21involved in all the issues if you like
0:23and technologies that Mellanox is
0:26dealing with some of you might have met
0:30me maybe last year on Twitter go well
0:33last year and my main intention today is
0:40actually to provide the HPC experts
0:44because you are actually HPC experts to
0:48the guys among you which are not
0:51familiar with Beth basics of InfiniBand
0:56and the most important issues in
1:00InfiniBand skeleton if you like or
1:03fabric I would like to make it clear to
1:07most of us so this is my main intention
1:11today I have additional small target and
1:16the small target is to provide you an
1:19information about Mellanox academy
1:24Mellanox academy actually is a
1:28organization that can help you get
1:32different type of information we still
1:35have problems with that yeah okay so we
1:44would like to allow you to offer you
1:47training services that could be online
1:50service services that could be a
1:54instructor-led training could be also
1:58web training or okay that's better I
2:00suppose and this is part of Mellanox
2:03Academy so a lot of subjects that some
2:09of you have holes of knowledge in them
2:13are covered as part of Mellanox Academy
2:16and as you can see as I was saying we
2:20are providing in a instructor-led
2:23training which is me and some other
2:26people we have online training and we
2:30also have certification processes in
2:33order to allow people of your
2:35organization to verify that they have
2:40the right knowledge to provide support
2:42or even just to to verify that they have
2:46the basic of the InfiniBand technology
2:49in order to get most of that so this is
2:53actually the Mellanox Academy if you
2:56will enter to our Mellanox Academy you
2:58will see that we have the white papers
3:01you have videos you have simulators and
3:04you have other tools that can help you
3:08get your vertical knowledge as well as
3:10practical knowledge if you are going
3:13into configuration or installation
3:17sessions so our sessions does not force
3:21you to take again instruct
3:25instructor-led training you can also do
3:28a lot of jobs by taking the e-learning
3:31sessions I'm not going to go into the
3:37specific solutions that they are
3:40provided actually I told you we what we
3:44are providing and as I was saying we
3:49have different types of learning methods
3:54instructor web eLearning and those are
4:00the main ones we have a list here of the
4:03offering list of the training that you
4:05can have so of course you don't have to
4:08take it now but you need to be a world
4:11that you have that option and as we were
4:16saying we have flexible solution a
4:19unique solution for all knowledge levels
4:22and professional roles
4:23in the infinite built environment and
4:25InfiniBand and HPC as well
4:29we have varied content of sessions and
4:32courses then we have also flexible
4:35payment options but between you and me
4:38money is not a problem so suppose you
4:42don't care about that of course we have
4:45certified remote lab instructors and
4:49course world that you can use for that
4:52same atom so we really urge you to use
4:58Mellanox academy not for maybe your own
5:01personal use but your environment and
5:04your partners and this is about Mellanox
5:09academy this is decided by the way of
5:11the Mellanox academy maybe try to use it
5:15later and see what are the options and
5:18that was my introduction regarding
5:21Mellanox academy and let's start with
5:26the main target that we would like to
5:30cover today so the session is called
5:34infinite vendor search essentials every
5:37HPC expert must know so you are already
5:40HPC experts so I suppose you should know
5:45their following subjects some of you are
5:49familiar with all of them some of you
5:51are familiar with nothing some of you
5:53are familiar with part of that let's try
5:56to cover together again what we are
5:59going to look at the IV principles we
6:02are going to talk about the fabric
6:04components the network components and
6:06the fabric architecture we are going to
6:10talk about the network of fabric
6:14discovery mainly the jobs of the subnet
6:19manager which is in charge of basically
6:22everything in the network we are going
6:26to talk about the main things the main
6:30things that happens and the main
6:32functions of each one of the InfiniBand
6:37Oh to cool layers and we'll talk a bit
6:41about Mellanox products this dissolve
6:44the things that I'm going to cover today
6:46of course if you will have specific
6:49questions or questions that will arise
6:52during my sessions you're free to ask I
6:56hope I would be able to provide answers
6:59to all questions and in case I won't be
7:04able I promise to provide you the right
7:07answers by mail again my name is a dead
7:12or dead pass from Ella Knox I'm not
7:15going to ask you for all your names now
7:18it's beyond 20 people so I'm not going
7:20to do that but if you ask questions I
7:24will be happy if you are able to present
7:27yourself well you are coming from what
7:30is your job in the organization which
7:32organization etc thank you very much and
7:35we will begin something to ask before I
7:38begin no okay
7:44ah so when we talk about the InfiniBand
7:49a InfiniBand environment
7:52InfiniBand fabric InfiniBand network are
7:57going to provide and provide the best
8:00solution today especially to move
8:05traffic to move data in HPC environment
8:10and the reasons are because it does
8:13provide three or four features that are
8:18very very important for the users for
8:21the customers for the clients for the
8:24applications of the HPC the
8:27high-performance computing environment
8:30if you look at the slide that we see
8:34here then a network usually a fabric
8:38will have its clients the clients will
8:42be a servers that are compute servers or
8:46storage servers that have to move
8:48traffic between them
8:50in order to provide the switching
8:53between them while providing the
8:55InfiniBand switches which are working
8:57with the InfiniBand protocol the
9:01connections between the compute nodes to
9:05the switches or the storage nodes to the
9:07switches are used with InfiniBand
9:11links those InfiniBand links today are
9:15going in pace in a speed of 56 gigabit
9:20per second which is gold FDR we're going
9:24to talk about it later
9:25and later this year the maximum rate per
9:29port will reach 100 gigabit per second
9:33in a speed rate that is gold or standard
9:37that each specific part is called the
9:40EDR enhanced data rate so currently we
9:44are in the generation that allows us
9:45spell port of silver and purple to the
9:49switch of up to 56 gigabit per second if
9:55we talk specifically about Mellanox
9:58technology Mellanox technology
10:00generation today is providing the
10:04channel adapter on the server side of
10:07the network adapter what is called the
10:09HCA host channel adapters in a
10:12generation which is called connect X ray
10:14and connect IV I'm going to talk about
10:17the differences between them and the
10:20switches generation that do provide the
10:23FDR capability are called switch X -
10:27when you talk about the InfiniBand
10:31environment and InfiniBand technology
10:33you have specific integrated circuits
10:38for that Isis or chips if you like and
10:41in our case Mellanox is providing and
10:44producing and planning those chips which
10:47are part of the general adapters and
10:49parts of the switches as we will say we
10:53have the adapter cards that in the
10:55InfiniBand I'll call the host channel
10:58adapters of different types we have
11:00different types because they provide
11:04capabilities for different users and
11:07customers according to to the need and
11:10of course we have different type of
11:12switches and the the most de or the main
11:16reason for the difference between the
11:18switches will be the capacity capacity
11:20of poles capacity of servers that each
11:24one of the switches may actually support
11:26so we are talking about the switches
11:28options or InfiniBand switches that are
11:30going between range of 12 ports up to
11:35648 ports for a specific switch and then
11:40of course in order to to allow us to
11:44enhance performance of customers to
11:49enhance performance of applications we
11:52also have a different type of software
11:55solutions begins with management
11:57solution like the ufm unified fabric
12:00manager we are talking about the FCA
12:03which is now part of the standard
12:06actually part of redhead the FCA
12:10accelerator or fabric collection
12:13accelerations we have VSA for storage
12:17virtual storage accelerator and other
12:20options that will help you actually as
12:24part of your solution that you provide
12:27to your customers to provide better
12:30latency better performance better usage
12:34of the CPU and hardware resources of
12:40course part of the part of the network
12:44usually will be the media the media
12:47between the servers to the switches the
12:49media of the cables the cables are of
12:51two main categories the copper cables
12:54and the fiber cables this is the first
12:56category beyond that category of copper
13:00cables and fiber cables we have the
13:02other category which is which talks
13:05about is it a passive type of cable or
13:08active type of cable just to make it
13:13short and simple usually when we are
13:15talking about the difference between
13:18passive cable and active cable active
13:21cable will allow you to have a longer
13:23distance between two ends or two nodes
13:26in the fabric so when we talk about the
13:32targets targets of belanov solid targets
13:35of the InfiniBand technology today we
13:38are talking about different type of
13:40applications like financial services
13:42cloud database enterprises web tool and
13:46of course the most important is HPC it
13:50would be different if I'm going to talk
13:52in the cloud environment but for this
13:54sessions the HPC this is the most
13:57environment a most important one so we
14:01are talking about actually advantage and
14:05better performance enhanced performance
14:07to the HPC environment when you are
14:10working with
14:12InfiniBand specifically and of course
14:15with some of our Mellanox solutions that
14:19will allow you to provide ten times
14:23better performance higher usage or
14:27better usage of your cpu resources
14:30better latency and of course very high
14:34bandwidth from the net for the server
14:38side to the switches side in order to
14:42provide it of course we need different
14:45components in the network to be data and
14:49this is actually the job of the
14:52technology the job of the fabric that
14:55will provide the service for you so what
15:00we are going to see today is what are
15:03the basics again of the technology what
15:06are the main components what is their
15:08job and how could we make it better as I
15:13was saying before in order to provide
15:17the main components in the network
15:20Mellanox
15:22has the technology for the channel
15:25adapter which is today in a generation
15:27which is called connect x3
15:31and in the switch's environment we are
15:34talking about Switch x2 both of those
15:38guys are able to provide today FDR 56
15:43gigabit per second purport of the
15:47channel adapter or purport of the switch
15:54about to connect x3 as you can say
15:57connect x3 will allow you to connect on
16:00the bus side on the server side to
16:03connect with PS PCI generation number 3
16:08PCI s3 will be supported by connect x3
16:13channel adapter cards any question in
16:16this stage no okay so when we talk about
16:24InfiniBand InfiniBand generation and
16:27finland planning what what is the plan
16:30ahead what is the history of the
16:32InfiniBand so InfiniBand is the network
16:35InfiniBand Eze
16:36as a pipe provider was planned in around
16:412000 and in around 2000 the port
16:46capability in the InfiniBand environment
16:48it was huge for that specific period we
16:54could have provided 10 gigabit per
16:57second per port and here was 2001 in
17:042005 we came to the second generation
17:07that was called DDR double data rate
17:11with the capability of 20 gigabit per
17:14second per port the next generation that
17:17I suppose part of you're still working
17:21with was a QD au q dr r is able to
17:26provide 40 gigabit per second per port
17:29can you tell me who in the audience has
17:32in his cluster kudiye kudiye so 70
17:39percent is actually still working with q
17:41do so
17:43can you also tell me what is the
17:46capability in data that every QDR port
17:50can provide can support how much data
17:56can you forward on every QDR port you
18:04are pointing that and this is you're
18:06saying 40 gigabit what is your name yep
18:12get really saying 40 gigabit do you
18:15agree well he sank 32 okay so uh well
18:25QDR
18:26provides 40 gigabit per second of signal
18:30you may say so the signal the bits if
18:33you will con them you will see 40
18:34gigabit per second but actually we will
18:38see that in the QDR generation we have
18:40encoding which is got 8 by 10 it
18:43actually takes 20% of the line rate if
18:47you like and provide it for for the
18:50overhead encoding overhead so this is
18:54the QDR generation both of you also have
18:58ports a general adapters and switches of
19:02the FDR generations who has FDR today so
19:07we're coming to 10%
19:10FDR 56 gigabit per second and this is
19:14today and if we are going to talk about
19:18the end of this year I don't want to
19:20provide you specific dates then we would
19:24be able to provide also 100 gigabit per
19:28second per port and this is the new
19:33generation or the current generation at
19:35the end of 2014 which will be the idea
19:38generation enhance data rate 100 gigabit
19:43per second per port of your server 100
19:46gigabit per second per port on the
19:49switch
19:51of course the other a element the other
19:54parameter which is very important
19:56specifically for high performance
19:58compute will be the latency how much
20:02time does it take for a packet for
20:05calculation to go between one server to
20:09the other server and this is actually
20:12the latency the latency is the latency
20:15when I'm talking here about the latency
20:16what is the latency that I am as a
20:19switch as InfiniBand switch I'm going to
20:22add to the packet that comes into my
20:26switch and then goes out I'm not going
20:29to talk about the latency that is
20:31actually happens if you like in the
20:33server it happens in the process between
20:37the application upper layers to the
20:39lower layers I'm talking about the
20:42latency that is added by the switch the
20:46latency that is added by the switch
20:48specifically we will talk about it it
20:50may be again later when a packet comes
20:54into InfiniBand switch of Mellanox in
20:58our example until it goes out every
21:01switch or every hope of a switch will
21:06add 117 nano second of course you can
21:12tell me all that but usually we are
21:14going when a packet enters to the fabric
21:17many times it is actually passing more
21:21than one switch more than one hop it may
21:25go to the first switch and then to the
21:28second switch and then to the third
21:30switch so how do you count that of
21:32course I'm going to add the delay so if
21:37i have delay 170 seconds in one switch
21:41and then another one and then another
21:43one
21:43then i'm going to to count for five to
21:47seven hundred nanoseconds of delay which
21:49is still very very short and good delay
21:53and this is in the color generation of
21:56FDR
21:59but what was the beginning of the
22:02InfiniBand the beginning of the
22:04InfiniBand 1999 there is organization
22:08which is called was set up organization
22:12that was called the IB ta InfiniBand
22:15trade association found in 1999 and the
22:19target of the IB ta was to provide the
22:22network and the protocol for data Sony
22:27this is not something which is new we
22:31already had networks like X 25 and is
22:35the N and Ethernet and Fibre Channel and
22:39frame relay
22:41so all those network are actually also
22:45providing data and networks so what is
22:49the difference what is the main
22:52difference or what is one of the main
22:54differences between date data network of
22:57InfiniBand to the other data network so
23:00first it was created it was thought
23:02about in 2000
23:04so in 2000 they were thinking already
23:07about customers of the 2000s
23:10applications of mm performance capacity
23:15latency application HP sees that cannot
23:21work with the other protocols because
23:24they are too slow because they are too
23:27narrow and because also because of
23:32another a very important factor one of
23:36the things that was not thought about in
23:38the other network like Ethernet
23:41switching they were not thinking about
23:45what happens to a packet of data to a
23:49message of data in the server itself
23:54when you are taking a packet of data in
23:58the tcp/ip environment that packet of
24:01data is taken and then it is put into
24:05frames in the higher layers and then it
24:09goes in the server to the Kern
24:12environment to the tcp/ip Colonel
24:15environment and what happens there that
24:19the colonel environment has two poses
24:21each one of those packets each one of
24:26the data packets has to be processed in
24:28the kernel environment and then to be
24:30sent to be copied to another buffer and
24:33then from the buffer to be sent to the
24:35channel adapter and in the channel
24:38adapter to send it out that is normally
24:42what happens in the other type of
24:44protocols one of the things that was
24:47thought about in the InfiniBand was how
24:52to reduce the time that the packet will
24:56take from the application layer until it
25:01goes out from the server to the other
25:04side that time has to be shortened that
25:07time has to be cut and this is something
25:11which is very unique to the InfiniBand
25:15protocol and InfiniBand environment how
25:18do you make it more efficient how do you
25:22make it more short how do you make the
25:26latency better within this distance
25:29between the higher layer protocols to
25:33the channel adapter before it goes out
25:35to the other side so this is also
25:37something that was a part of the
25:40thoughts of the guys that were behind
25:43the cavity of the InfiniBand behind the
25:48planning of the InfiniBand like IBM ella
25:51knox oracle HP cray intel and others
25:54that probably i did not count here so
25:59the i InfiniBand is actually switched
26:02fabric architected architecture
26:04interconnected technology connecting
26:06cpus and iOS which provides super high
26:12performance which provides high
26:14bandwidth okay starting at 10 gigabit
26:18per sector per second up to - ha 100
26:22gigabit per second pair
26:24port purport of a silver per port of the
26:28switch the low-latency that is required
26:32by your customers should be challenged
26:35by the protocol and should be responded
26:39by the provided protocol and as I was
26:41telling you they minimal latency the
26:45today's switch will head is 117 nano
26:51second and the other challenge that I
26:54was talking about how do we cut how do
26:59we enhance the performance of the
27:01application and the way that it sends
27:03the information from the higher layer
27:05protocols until it goes to the channel
27:08adapter and the main mechanism which is
27:12used for that and I suppose all of you
27:16probably familiar with is using the
27:21process using the mechanism which is
27:24called the our DMA remote direct memory
27:30access which will provide a better
27:34latency for every packet that goes from
27:36the upper layer protocol up to the our
27:40channel adapter and will also allow us
27:43to have a lower CPU a or better CPU
27:49utilization better CPU utilization
27:52because most of the traffic that was
27:55processed until the usage of all the our
27:58DMA that was processed in the kernel now
28:03should not be processed in the kernel
28:06this is one of the advantages of the our
28:10DMA so traffic communication now in the
28:16technology that we are planning the
28:18traffic communication will bypass the
28:20operating system will bypass the kernel
28:24and it means that the CPU could be used
28:30through much more processes which are
28:33other than
28:35dealing with the traffic itself
28:42InfiniBand was originally designed for
28:44loud scale grids and clusters it came to
28:49increase the application performance it
28:51can provide solutions for local area
28:54network for an for a bigger network for
28:58storage area networks and application
29:00communications it will provide high
29:04reliability cluster management what does
29:08it mean high reliability cluster
29:10management well this is a bit vague I
29:14must say but the high reliability of
29:17course means that if you have something
29:19or an entity that will manage the
29:22network and that entity fails down
29:25automatically you will have a backup to
29:28that entity that will take its place we
29:32are talking about the first Network
29:34which is an SDN Network Rail Sdn network
29:40which means software-defined network
29:43software managed Network which means
29:47that we don't need someone we don't need
29:50an administrator that will configure all
29:54parameters all the parameters in your
29:59network all the parameters up to a level
30:01of a port can be defined today by the
30:05entity that manages the network by
30:08software and that entity that manages
30:11the InfiniBand network by software is
30:14called the subnet manager we're going of
30:18course to talk about it before we
30:21continue
30:21since our DMA is one of the main
30:25processes or one of the main solutions
30:27here
30:28what is the advantage of our DMA our DMA
30:31again for those well somebody hill does
30:34not know what our DMA is and please
30:37raise your hand okay
30:42so don't laugh if you have that question
30:45it seems funny there are people in the
30:48world which are not familiar with our
30:50DMA okay so since everybody knows what
30:55our DMA is and I suppose that you know
30:57that without our DMA we have the
31:00application message that goes on the
31:03buffers on the user layer and from the
31:06buffers in the user layer they have to
31:08be copied every packet has to be copied
31:12to the buffers in the operating system
31:14level and from the operating system
31:17level after it is processed in the
31:20tcp/ip layers it has to be copied again
31:24to the channel adapter layer the channel
31:28adapter or the new claret is as it is
31:31called here or as it is called in the
31:33InfiniBand environment host channel
31:37adapter so every packet is to go user
31:40copy caramel copy a caramel process
31:43kernel copy to the channel adapter and
31:46go then goes to the other side this is
31:50what happens without our DMA without DMA
31:53we have a zero copy we have a zero copy
31:57we don't have to copy the packets which
32:00are the data packets we don't have to
32:02copy them to the kernel we don't have to
32:05copy them to the kernel so we saved it
32:07we save the time we save the copy
32:10process and we just take those packets
32:14directly from the user layer of one
32:18server we send it directly to the user
32:22layer of the other server and this is
32:25the RDMA over InfiniBand
32:27that saves time saves latency enhance
32:32the latency and of course allows us to
32:35do much more work in the operating
32:38system this is actually we can probe
32:41again again between 60 to 80 percent of
32:46improvement all efficiency if you like
32:49in the cpu
32:53because we don't need to process the
32:55traffic in the operating system in the
32:59kernel so let's look at the InfiniBand
33:03architecture InfiniBand architecture
33:06this is the basic structure like most of
33:11the other data network that you know do
33:14you have the laser pointer are working
33:16in your yeah probably
33:30okay data networks are Luke's looks
33:37actually very similar if you'll take
33:39Ethernet frame relay ISDN X 25 whatever
33:44data network it will look like that
33:47right so we have to talk about the
33:51network itself and what are the parts
33:53which are special in those network so
33:56first let's talk about our customers our
33:59customers are yours your customers as
34:02well those are the HPC computing systems
34:05or computing servers and the HPC storage
34:09servers so those are my customer here I
34:12have the application let's say that one
34:15of the most common applications in your
34:17environment will be a openmpi protocol
34:21will be GPFS nvidia or some other type
34:27of applications which are working here
34:30on your clients then we are taking that
34:33application and that application
34:35calculation result is to go to the other
34:39computation a server in order to move
34:42the traffic to the other side we have to
34:45take that packet and then we have to go
34:49out to the other server how do we go to
34:51the other server for the purposes are we
34:55need switches the switches of doing the
34:57switching so we are taking packets from
35:00server number X to server number Y later
35:04we are going to talk about addresses and
35:06addressing so we are going to take the
35:09packet and send it via the network cloud
35:12of the InfiniBand environment the
35:15network out of the InfiniBand
35:17environment is called the HCA the hosts
35:20channel adapter the hosts channel
35:23adapter has actually two sides one side
35:26is the side of the bus of the first
35:29server which is the PCI bus and the
35:32other side is the InfiniBand link the
35:35InfiniBand link could go with the
35:38relevant media to the switch what is the
35:42relevant
35:42idea the relevant media could be a Capel
35:45if you would like to have higher
35:47distance instead of copying or taking
35:50fiber right okay so you took the
35:54InfiniBand link up to the switch that
35:57switch may have different type of our
35:59ports in terms of capacity it could be a
36:02switch that takes 12 volts it could be a
36:05switch to take 648 ports in order to
36:10move the traffic between debt server
36:12server a to debt service lb I have to go
36:16to at least one switch and sometimes to
36:20several switches so therefore we are
36:22going into the network to are between
36:25one switch to another switch and then we
36:29send information to the end destination
36:33the data network or the fabric the
36:38InfiniBand fabric here is working today
36:43in a subnet or one subnet only the data
36:49network today is working as a layer 2
36:54data network what is the difference
36:58between layer 2 data network to layer 3
37:01data network
37:08now this is a question that I know that
37:10maybe some of you do not know this is
37:13why I'm asking unlike our DMA yes IP
37:20packet okay well of course you're right
37:29about in part of the answer somebody
37:33would like to expand No
37:36okay so when I'm talking about one
37:39subnet about switching of layer 2
37:42switching layer 2 means that all of my
37:46nodes so the difference is wrong ok so
37:51all of my nodes belong to the same
37:53subnet right now when I pee is a term
38:00from another environment ok because in
38:05the InfiniBand we have different types
38:07of addressing okay so one subnet means
38:11that all my nodes belong to the same
38:14subnet to the same subnet therefore if
38:17you have 20 nodes or two thousand nodes
38:21all of them are part of the same subnet
38:25and the switches currently here our
38:29switches of layer 2 so it doesn't matter
38:34of which organization here you are
38:36coming today you're working with layer 2
38:41network in the InfiniBand environment so
38:45those are our switches about your
38:48dressing we're going to talk in the next
38:50stage so in most of your environments
38:55you have the InfiniBand environment for
38:58the HPC a for calculation for GPFS or
39:02nvidia for storage for computing edge
39:04storage and most of it is InfiniBand in
39:08some of your network you have kind of
39:10mixed environment and mixed environment
39:14environment means
39:16that maybe maybe two years from today or
39:21twins are back
39:23you had another network that is maybe
39:25Ethernet network or fiber channel
39:29network and sometimes you need to move
39:33information between the part which is
39:36the InfiniBand part
39:38InfiniBand environment to the part which
39:41is the ethernet environment so you have
39:44two computers one computer one node is
39:48in the ethernet environment and one node
39:51is in the InfiniBand environment how do
39:55you move traffic between two different
39:57environments the answer is how using
40:05routing no the answer is thank you by
40:12the way for the response the answer here
40:15is a gateway it's not very far of course
40:20from the answer the answer is a gateway
40:23what the Gateway is a gateway is
40:25actually if you like a converter
40:28a converter between two protocols a
40:31converter between two languages so what
40:35I would like to do here in my example I
40:37would like to take a packet that is on a
40:40server here in that environment - a
40:43packet to a server in that environment
40:46because here they talk InfiniBand ish
40:51you know that language and here they
40:55talk Ethernet we have actually to take
40:58packets from the ib2 Ethernet and from
41:02the ethernet - to IB for that purpose we
41:05have two gateway this is the job of the
41:08Gateway to enable us to talk between
41:11ethernet InfiniBand or fiber channel to
41:14InfiniBand the component is called
41:16gateway good clear
41:22usually the information or usually the
41:25type of packets of course that will go
41:27between the ethernet environment to the
41:30InfiniBand environment will be IP
41:33packets we are using in the IB we are
41:38going to talk about it we are using a
41:41function which is called IP over IP IP
41:48over IP that will be the way we are
41:52going to take packets IP packets here
41:54and send them to IP packets on the
41:57internet environment IP over IP now
42:01let's talk about the management entity
42:04one of the things that will make the
42:07InfiniBand environment special and
42:09unique is the capability of the
42:12InfiniBand to be Sdn software-defined
42:16Network and software-defined network
42:19somebody here has managed internet
42:21environment working with switches of
42:27Ethernet whoo you're saying you're
42:32nodding like Gabriel so if you would
42:35like for example to change parameter in
42:40on a villain or trunk or speed what do
42:44you have to do you have to do it
42:45yourself so it's kind of do-it-yourself
42:47network but you have to do it yourself
42:50so you need an administrator in Ethernet
42:54environment in InfiniBand environment
42:57actually as I was saying it is actually
42:59the first network type that is Sdn
43:03software-defined network so who is that
43:07entity the entity in the InfiniBand
43:10protocol that entity is called subnet
43:14manager sm the subnet manager is the
43:18most important entity in the network
43:21because it does all the configuration up
43:28to a level port of each one of the
43:31switches of each one of the cell
43:35every parameter is managed by the subnet
43:39manager this is the configuration you
43:44don't need any administrator to do any
43:47changes here the subject manager will do
43:49it all it does work we have we have an
44:05unsatisfied customer
44:08so the way to do that today in marketing
44:13they are saying you're right Gabriel
44:14what is the problem we're going to solve
44:17it it's not that I'm dissatisfied is
44:20just it's not just switching ok
44:25first let's talk about it ok but we will
44:29do it in one of the next stages ok so
44:36this is the subnet managers that are
44:38going according to the theory in most of
44:41the time should work and the subnet
44:44manager is doing everything what happens
44:49if that subnet manager fails if the
44:56subnet manager phase let's call him the
44:58actives that a subnet manager if the
45:01active subnet manager fails of course we
45:03need another subnet manager in the
45:07network the other subnet manager that
45:10will be somewhere here will be the
45:12standby or the fallback subnet manager
45:17so we have in every network we must have
45:20a fallback and an active subnet manager
45:24by the way the protocol itself or the
45:28standard itself of the InfiniBand allows
45:31you to have more than two subnet
45:33managers we suggest to have only two
45:36subnet manager in a in the network so we
45:41were talking about currently about a
45:43layer 2 network although it will support
45:46in the future in layers
45:48the main components are the customers
45:51itself the notes channel adapter that
45:54will allows you to do the network
45:55connection
45:56the InfiniBand link itself that goes
45:59from the server to the switch the
46:01switches that currently are doing layer
46:032 switching the nodes belong all of them
46:07to the same network or to the same
46:09subnet and in order to move packets
46:13between InfiniBand environment to other
46:16type of protocol environment we have a
46:19functionality which is called gateway
46:22questions ok so we talked about the
46:29different sorry the different elements
46:34and first the host channel adapter the
46:40host channel adapter device that
46:43terminates the InfiniBand link here we
46:46have the connector this is the host
46:48channel doctor looks like I suppose
46:52other channel adapters that you have
46:54seen up today but the connector is
46:57different the connector today in the
46:59InfiniBand is called q SFP connector for
47:05those of who for those of you I suppose
47:08small number who are not familiar with
47:10that then a QFP is a different connector
47:13is not SSP it's not as a free plus it's
47:17not rj45 it is q SFP so we have a QFP
47:23connector that goes out to the towards
47:27the switch this is the channel adapter
47:28of course it's part of the server and
47:31this is decided go to the bus of the
47:34server the PCIe bus the PCI generation
47:38today that we are talking about is PCI
47:41generation 3 then we have the switches
47:45the switch is a device that moves
47:47packets from one link to another or from
47:50one node to another on the same
47:53InfiniBand subnet on the same InfiniBand
47:56subnet this is why we call it layer 2
48:01reaching okay then we have an element
48:06which is called the router and now you
48:10can ask me a dead but you told us that
48:12we are working in layer 2 network so why
48:17do you need a router the answer is that
48:24currently practically we are not working
48:27without us currently we have router in
48:31the protocol we have router in the
48:34standard we have layer three for routing
48:37of InfiniBand we have addresses for
48:41routing of InfiniBand routing will allow
48:44us to move traffic between one subnet to
48:49another submit so we have it in the
48:53stand out we everything the theory
48:56currently it's not yet implemented
49:01almost 100% and then we have the gateway
49:05or a Bridget it is it called and the
49:08Gateway job will be to Anne I enable us
49:10to move packets between InfiniBand
49:15environment to ethernet environment Auto
49:19fibre channel environment we talked
49:23about the channel adapter and we said
49:26that the journal adapter is connected
49:28with a link to the switch the link could
49:32work with different type of speeds it
49:36could be SDR Oh d-dear
49:39o QD r o FD r at the end of the year it
49:43will be also ideal this is the way if
49:47you like so if we are going back to 2001
49:51the only capability was HDL single data
49:56rate then we went to a detail double
50:01data rate and then we have a queue do a
50:05quadruple data rate and then we'll go
50:08into FDR 14 data right this is the car
50:11and generation and at the end of the
50:14or God's will we're going to have the
50:17India intense data wretches I think it's
50:20actually a what 25 words alien not 25
50:28sorry it's a mistake well please edit it
50:36in the dick yeah well ah actually it was
50:43put there intentionally to check if you
50:46are really checking what is said here
50:50right
50:51okay so that was easier let's now go to
50:54the addressing when we talk about the
50:58dresses of protocols first we have the
51:01physical address each one of us is a
51:04physical address kind of genes I suppose
51:08oh whatever the physical address in the
51:12environment of InfiniBand is called
51:15gooood gooood is global unique
51:20identifier of that specific note so all
51:29the elements in the environment are
51:31nodes each node will have his own good
51:34who is a node a channel adapter of a
51:38server is a node a switch is also a node
51:45okay a gateway is also a node so each
51:49one of them is having physical address
51:53which is a bit like a Mac if you're not
51:57familiar with good yet then you can
51:59compare it to a Mac so this is my
52:03physical risk global unique identifier
52:09the global unique identifier is an
52:12address which is comprised of 64 bits
52:16part of those bits of course will be the
52:19vendor ID ok 64 bits
52:24it is assigned by the InfiniBand vendo
52:28and it is persistent through reboot so
52:31if you're taking that channel adapter
52:32with it support if the server goes down
52:36and up the good is going to change yes
52:44the going is the good is going to change
52:46no okay that was a right so it is
52:52persistent it is a fixed one the gooood
52:55is a fixed parameter fix identification
52:58of the node now this is for channel
53:04adapter so now here I would like to
53:06emphasize that point every port in a
53:11channel adapter has its own gooood do
53:15some of you have more than one port on
53:17the same server it's like two ports on
53:21the same server yes
53:23how many goods do you have for two ports
53:28one for each port right so this is
53:31obvious Satna who is working with switch
53:35with 36 ports your name oh sorry Chris
53:47Chris how many goods do you have two
53:49deaths which with 36 ports one good
53:54right so in a switch you do not have a
53:58specific gooood to every port the switch
54:02you have only one Guido at least one
54:05main grid let's call it a node grid of
54:08the switch so the switch has the node
54:12grid so how will we identify specific
54:16port in that switch if you have felt its
54:22exports how will you identify the port
54:27put a the port number exactly so each
54:31port has a port number so if you like to
54:35identify a specific port you actually
54:37take the gooood of the switch plus the
54:41port number right
54:43there is another type of grid which is
54:46called the system grid someone is
54:49familiar with system good who has a
54:55chassis a switch or modular switch which
55:00is more than 36 ports no one here he'll
55:04you have which type of very switch big
55:10fabric one if I have a big fabric one it
55:13means that on the same chassis on the
55:15same physical chassis you have actually
55:17multiple switches so each one of those
55:21switches will have its own node good but
55:25the chassis as a whole will have another
55:28grid which is called a system grid so
55:32the system go it will be the identifier
55:34of the whole chassis and each one of
55:37those switches within the medulla will
55:40have a specific grid okay IB fabric IB
55:48fiber fabric basic building block when
55:52you have InfiniBand fabric InfiniBand
55:55network that InfiniBand network is built
55:58according to the needs what are what is
56:01the basic need usually the basic need
56:04will be the capacity how many nodes do
56:07you need in your cluster right of course
56:12this is where only one parameter so when
56:17we are talking about the fabric basic
56:21building block each one of those block
56:23is going to be a switch by the way each
56:28one of the switches is based on an ASIC
56:32what is the ASIC
56:35actually the ASIC is the heart of your
56:37switch basic is a switch basic is a chip
56:42the exit is it is a switching chip and
56:47today this generation every ASIC has 36
56:52port why do I say this generation
56:55because if you're looking at two
56:57thousand and four or five the ASIC the
57:00basic ASIC was actually structured of 24
57:04ports today basic is 36 ports when you
57:10build a network the network is built
57:13according again to the number of nodes
57:15that you have to provide that you have
57:17to support it could be a network for
57:21some of you you have I suppose 100-day
57:24nodes some of you might have 600 nodes
57:29some of our customers have 10,000 nodes
57:3510,000 and more so of course the number
57:39of switches that you will have to
57:41provide will be a different number
57:43according to the number of nodes that
57:46you will have to connect okay here let's
57:54look at that slide everybody can see the
57:56slide even from the last lines I hope
58:00okay so every switch here let's take a
58:04basic switch in our cluster and as you
58:10can see the nodes are connected the
58:13nodes are connected to switches the
58:17switches that are used for the direct
58:20connection the direct connection to the
58:23nodes those switches have a name that
58:27somebody can tell us what is the name of
58:28those switches
58:33yes exactly those switches are called
58:37the lifts
58:38okay so Dec denotes the servers or are
58:43connected to the leaf switches each one
58:46of those is a leaf now let's say that in
58:50that example as you can see every leaf
58:54is going to be connected with 18 ports
58:58to 18 nodes so 18 ports of dead leaf and
59:0318 ports of dead leaf and 18 ports of
59:07dead leaf and so on so if you are using
59:1118 ports from the leaf to their nodes
59:15what do you need the idea of 18 ports
59:18for so I have felt its exports 18 are
59:27used for the connection to the nodes
59:30themselves what do I do with those 18
59:35ports trunks okay the other 18 ports are
59:41actually used as connections to the
59:43other to the next layer the next layer
59:47in the fabric is called how how do you
59:52call the art those those switches oh yes
59:59oh it has another name
1:00:04backbone also spine spine switches so we
1:00:11have two basic type of switches we have
1:00:15the leaf switches which are connected
1:00:18directly to the server and we have the
1:00:20spine switches and the spine switches
1:00:24have one main job the spine switches are
1:00:29actually will help us to make
1:00:32interconnection between servers that do
1:00:37not belong to the same to the same what
1:00:44live to the same leaf because let's say
1:00:47that you have those servers here servers
1:00:51that belong to leaf a and here you have
1:00:54servers that belong to leaf D how will
1:00:57you connect between those servers to
1:01:00those of us the answer is you need
1:01:03another layer that layer will allow you
1:01:06to make inter connection between
1:01:11different Leafs you would be able to
1:01:15move packets between Detlef to that leaf
1:01:20via dead spine okay so this is the spine
1:01:26layer this is the job of the spine layer
1:01:29to allow connections between traffic
1:01:31that goes or belong to different lifts
1:01:35now if you paid attention i took 18
1:01:40ports to the servers and then I have 18
1:01:44poles that are going to the spine
1:01:46switches so actually 18 ports to towards
1:01:50the servers and 18 ports towards the
1:01:53spines for interconnection why did I
1:02:00take 18 ports here and 18 ports for
1:02:03interconnection
1:02:08yes at the same time right so actually
1:02:17one of the things you would like as HPC
1:02:20customers you would like the network to
1:02:23be non-blocking network and one of the
1:02:32ways to get close to that target is that
1:02:36if for example you have 18 nodes or 18
1:02:41servers in order to verify that they
1:02:46will get the bandwidth that they need
1:02:49when they want to send out information
1:02:51to other 18 servers you would like to
1:02:56have the same bandwidth of the nodes
1:03:01will be supported by equal bandwidth in
1:03:05the interconnection fabric so if you
1:03:10would like to move 100 gigabit per
1:03:13second from the nodes you need 100
1:03:17gigabit per second in the
1:03:19interconnection environment to support
1:03:21it so actually if you have a number of
1:03:2818 ports of FDR to the nodes you will
1:03:34need 18 ports of FDR in the
1:03:38interconnection if you have 72 ports of
1:03:42FDR to the nodes you will need 72 port
1:03:48of FDR in the interconnection in that
1:03:52way you will enhance the chances that
1:03:55every time you will have been with that
1:03:59is required for me node to be to go out
1:04:02to another server you will have that
1:04:06bandwidth in the interconnection
1:04:08environment this is called non-blocking
1:04:11go if you would like virtual or semi non
1:04:14blocking
1:04:15in the interconnection environment now
1:04:20of course it's not the total proof it's
1:04:25not 100% but this is one of the rules
1:04:30that we use in order to have kind of
1:04:34semi a non-blocking environment of
1:04:38course we can we could have taken
1:04:41instead of 18 4 0 and 18 ports in the
1:04:44interconnection environment we could
1:04:46have taken for example 27 here and 9
1:04:50here ok what happens if you have 27
1:04:54ports here and 9 ports that are going to
1:04:58the interconnection you have actually a
1:05:01different ratio between the
1:05:06interconnection pause - the connections
1:05:09that are going to the servers and the
1:05:12different ratio means that you have
1:05:14higher number of node poles then calls
1:05:18of interconnection you will get a
1:05:21blocking ratio okay let's go to the next
1:05:29one if you have questions please ask me
1:05:31yes please why do the fighting bull have
1:05:3636 ports of example a power of 230 wide
1:05:41the switches do not have a different
1:05:43number by this choice in this case what
1:05:47long standing car 36 ports okay the
1:05:50reason is the reason is that the
1:05:54integrated circuits which are produced
1:05:58the Asics
1:05:59which are produced the basic ASIC today
1:06:03has 36 ports although I can tell you
1:06:07that we also have a switch of 12 ports
1:06:10for example which is also possibility
1:06:14which is clear because of the hardware
1:06:16but exactly
1:06:17six when Isaac was designed by 36 was
1:06:20I'm not sure I can tell you I can tell
1:06:29you that the previous generation was 24
1:06:37you're talking about the flip-flops
1:06:40inside and so on I suppose anyway this
1:06:43is the structure this is the internal
1:06:45structure okay so in that example we
1:06:48have actually a fabric that will support
1:06:5372 volts or 72 notes
1:06:57why 72 notes because we have created we
1:07:02have created an environment that has
1:07:04four lives each one of the Leafs has 36
1:07:10ports and demand those lt6 ports we have
1:07:1418 poles to the nodes so 18 18 18 18 872
1:07:20ports towards the nodes and then we have
1:07:2372 ports which are providing us the
1:07:27support in the interconnection
1:07:28environment question
1:07:36so we have four chips which our function
1:07:39is lines or Leafs and two chips which
1:07:43will function as goals or spines okay so
1:07:51that was a main a structure of the
1:07:55fabric let's go to the next stage we
1:07:57talked about the physical address that
1:08:00we call the gooood let's go to the next
1:08:02stage when we are moving traffic between
1:08:05one node to another node we need to have
1:08:08an address that rests which is used is
1:08:12the layer to address so when I would
1:08:14like to talk with the Chris or when I
1:08:17would like to talk with Gabriel each one
1:08:21of them will have a layer to address the
1:08:24layer to address in InfiniBand
1:08:26environment is called lead local
1:08:29identifier this is the lead the local
1:08:33identifier it has 16 bits 16 bits it
1:08:38means that the range that is supported
1:08:41could be up to 2 by 16 which is about
1:08:45how many addresses about 65,000 right
1:08:51yes this is the layer to address range
1:08:54who is providing the addresses well one
1:08:59function this is the subnet manager each
1:09:02one of the nodes in this room is going
1:09:05to get lead provided by the subnet
1:09:08manager the lid again is called local
1:09:14identifier so each one of your servers
1:09:17has one or two leads it depends of the
1:09:23number of the ports that you are
1:09:24connected with it is assigned by the
1:09:27subnet manager and it is theoretically
1:09:31not persistent doing reboot
1:09:37theoretically why do I stay to radically
1:09:40if I will give you the option what would
1:09:43you like to what do you prefer would you
1:09:45like would you like it to be changed
1:09:47we sit on the server what who said no no
1:09:57yeah you wouldn't like you would like
1:10:00you will prefer your network you will
1:10:04prefer your fabric to say to stay as
1:10:06stable as possible you would like that
1:10:09if Greece would like to call Gabriel
1:10:12even if great brain went down because of
1:10:16some issue with well even in this case
1:10:24when it comes up we would like him to
1:10:27stay with the same lid so of course
1:10:29there are parameters that will allow you
1:10:31to say that even if server goes down we
1:10:35would prefer that when it comes up we
1:10:38would prefer him to come with the same
1:10:41layer to address with the same lid when
1:10:45we talk about the leader dresses we have
1:10:48two ranges of addresses because we have
1:10:50two types of communication in the
1:10:53InfiniBand environment the first type of
1:10:56communication and InfiniBand environment
1:10:58is the unicast communication when I talk
1:11:02with Gabriel only two of us of us this
1:11:05is unicast when we talk here in the room
1:11:08and everybody heals this is what when I
1:11:15send information to a group of listeners
1:11:19it is is it unicast what is it it is
1:11:25multicast of course somebody will ask
1:11:28yeah but it's like a broadcast actually
1:11:30but in InfiniBand we do not have
1:11:33broadcast InfiniBand we have only
1:11:37multicast and the simulation if you
1:11:41would like to do like a broadcast you
1:11:43will actually create a multicast group
1:11:45that will include all the participants
1:11:49so InfiniBand there is no broadcast only
1:11:54multicast so if you talk about the
1:11:57specific addresses you have a unicast
1:12:00address range which
1:12:01is between 1/2 bfff and you have a
1:12:04multicast range which is between cw-1 to
1:12:08ffff II these are the lead addresses who
1:12:14provides the leads subnet manager okay
1:12:22the next element that I wanted to show
1:12:25you is that so as I was saying we have
1:12:29only one subnet in your organization
1:12:33most of you one of the questions will be
1:12:37and it might come from you so what if we
1:12:41would like if we would like to have
1:12:44different departments or different
1:12:47customers in our organization in our
1:12:51network how can we do that in InfiniBand
1:12:55we know how to do it in ethernet how do
1:12:58we allow different departments to have
1:13:02the different kind of environment in
1:13:04InfiniBand we have only one subject can
1:13:10I ask if some of you in his environment
1:13:12is such a need or thought of it such a
1:13:17need who knows what the solution is well
1:13:22since um there is no answer I will
1:13:25answer myself I will ask an answer so
1:13:29the answer will be partitioning
1:13:34partitioning is the way that you can
1:13:37actually implement in your environment
1:13:39on your InfiniBand fabric in order to
1:13:43have to define different partitions for
1:13:47different customers you may have the red
1:13:51partition and the green partition and
1:13:53the blue partition which all belonged to
1:13:56the same subnet but their packets have
1:14:01different color actually it's called
1:14:04different identifiers and that specific
1:14:08identifier is called partition
1:14:13partition key okay is actually the
1:14:18identifier of every different partition
1:14:22so now you can say I have the storage
1:14:26guys I have the openmpi guys I have
1:14:31other type of guys IP over IB guys and I
1:14:36would like to hit two to have them on a
1:14:39separated environment although all of us
1:14:43belong to the same subnet that is called
1:14:48partitioning so those dead partitioning
1:14:55can help us to provide different
1:14:58environment to different applications or
1:15:03different environment because of
1:15:05security purposes because for example in
1:15:08your Institute you have two researchers
1:15:12one of them is for University and the
1:15:17other one is for Boeing or Airbus and
1:15:22the research for Airbus the Airbus guys
1:15:25who are paying money for that they are
1:15:27telling you we wouldn't like to have our
1:15:30part of the network to have the packets
1:15:34touch with the other common people
1:15:37packets can you separate them the answer
1:15:41is yes partitioning
1:15:46the packets of the red environment will
1:15:49have packets or partition key ID - and
1:15:53the green environment will have a
1:15:55partition kd3 this is not the only thing
1:16:01the other thing that you can have with
1:16:04partitioning is quality of service which
1:16:07could be different could be different
1:16:11it's not the only tool but is one of the
1:16:14tools that will allow you to provide
1:16:16different quality of service to open MPI
1:16:20and other quality of service - GPFS why
1:16:26because I might put them I might place
1:16:30them on different environment or
1:16:32different partition yeah okay let's stop
1:16:40there let's stop here we'll have some
1:16:43coffee break and we will continue in 15
1:16:4715 minutes

About this transcript

This page contains the full transcript of InfiniBand Principles Every HPC Expert MUST Know (Part 1) by insideHPC Report, generated from the public captions YouTube serves with the video. The transcript has 8,964 words across 1,282 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.