WEBVTT 00:00.000 --> 00:12.000 Perfect, can everybody hear me? 00:12.000 --> 00:13.000 Oh, great. 00:13.000 --> 00:16.000 Thank you all for coming for the first presentation. 00:16.000 --> 00:19.000 It's always a bit nervous to be first out. 00:19.000 --> 00:22.000 But this is a presentation about this big end, 00:22.000 --> 00:27.000 which is a project that I've been working on for the last 83 years. 00:28.000 --> 00:31.000 So quickly about me, my name is Phil Plain. 00:31.000 --> 00:34.000 I'm Swedish and Finnish, living in Berlin. 00:34.000 --> 00:36.000 I'm the creator of Spiegel. 00:36.000 --> 00:41.000 I used to also be a Quarming Caner of a project called Flux 00:41.000 --> 00:44.000 that does like continuous delivery for Kubernetes. 00:44.000 --> 00:47.000 Sadly, I had to this morning edit these slides 00:47.000 --> 00:50.000 because I got laid off from work 00:50.000 --> 00:52.000 so I don't have a employer on my slides anymore. 00:52.000 --> 00:55.000 So if anybody out there is hiring, 00:55.000 --> 00:58.000 just like a shameless plug to reach out. 00:58.000 --> 01:01.000 So yeah, let's get into it. 01:01.000 --> 01:05.000 So just a quick introduction where we are today 01:05.000 --> 01:10.000 and give you a background of what we're going to be talking about. 01:10.000 --> 01:15.000 So in typical container environments, 01:15.000 --> 01:20.000 we've come to the point where running highly available clusters 01:20.000 --> 01:24.000 is become relatively simple at least. 01:24.000 --> 01:27.000 It's not that big of a challenging more to run workloads 01:27.000 --> 01:30.000 that are distributed, running in multiple replicas. 01:30.000 --> 01:33.000 We're probably not managing VMs anymore. 01:33.000 --> 01:38.000 So we've created a new type of bottleneck, 01:38.000 --> 01:42.000 which is distributing the actual application workloads 01:42.000 --> 01:49.000 to the runtime instead of basically having the problem of 01:49.000 --> 01:52.000 like scaling the actual application workload. 01:52.000 --> 01:56.000 Now about distributing the actual image. 01:56.000 --> 01:59.000 There are some challenges in these types of situations 01:59.000 --> 02:01.000 because obviously what we're doing is we're running some sort of 02:01.000 --> 02:06.000 image registry that Kubernetes cluster is pulling from. 02:06.000 --> 02:10.000 So there's a challenge of availability of making sure 02:10.000 --> 02:12.000 that is always available. 02:12.000 --> 02:16.000 There is no point of having this cluster that can scale to 02:16.000 --> 02:19.000 10,000 nodes if you can't pull down the image 02:19.000 --> 02:20.000 and the action will run. 02:20.000 --> 02:23.000 There's also a question of speed. 02:23.000 --> 02:25.000 So we're seeing, especially now, in these 02:25.000 --> 02:29.000 damages of AI models, speed becomes a critical point 02:29.000 --> 02:31.000 when your images are growing constantly because you're putting 02:31.000 --> 02:33.000 large machine learning models in them. 02:33.000 --> 02:35.000 But also cost. 02:35.000 --> 02:40.000 So pulling images from public registries is something 02:40.000 --> 02:43.000 that actually includes a pretty large cost at 02:43.000 --> 02:46.000 the end of the day if you're cloud-built. 02:46.000 --> 02:48.000 So that, I'm the wild. 02:48.000 --> 02:50.000 We see this as like rate limits. 02:50.000 --> 02:52.000 I think most people have experienced the great limits 02:52.000 --> 02:55.000 with like Docker Hub out there probably. 02:55.000 --> 02:59.000 The Kubernetes registry does some sort of like IP 02:59.000 --> 03:02.000 rate limiting, especially if you're running workloads 03:02.000 --> 03:04.000 and perks in a cloud for somewhere reason. 03:04.000 --> 03:06.000 There's like a lot of like rate limiting going on 03:06.000 --> 03:08.000 with their IPs. 03:08.000 --> 03:12.000 Outages is very common and this isn't like 03:12.000 --> 03:15.000 this like to any rate issue provider out there at all. 03:15.000 --> 03:17.000 I think all of them have had outages at one point 03:17.000 --> 03:18.000 another. 03:18.000 --> 03:20.000 Like Docker Hub, get a container registry used to be 03:20.000 --> 03:22.000 like last year, used to be down a couple of times 03:22.000 --> 03:23.000 about a year. 03:23.000 --> 03:26.000 Quay, but even like private cloud providers like 03:26.000 --> 03:30.000 ACR, ACR also have their own outages. 03:30.000 --> 03:33.000 VanWitz are pretty limited, especially 03:33.000 --> 03:34.000 Docker Hub. 03:34.000 --> 03:37.000 I think limits to like 100 megabits a second when 03:37.000 --> 03:39.000 you're downloading images. 03:39.000 --> 03:41.000 But also we're seeing image removals. 03:41.000 --> 03:43.000 I think it was like a year or two ago. 03:43.000 --> 03:45.000 Quay was like cleaning up a lot of their images 03:45.000 --> 03:47.000 and we're actually like removing a lot of tags, 03:47.000 --> 03:50.000 which like caused like a couple of outages 03:50.000 --> 03:52.000 actually where people are explicitly 03:52.000 --> 03:55.000 depending on these images being like available 03:55.000 --> 03:58.000 and then just like over like a night or two 03:58.000 --> 04:00.000 like just like disappeared. 04:00.000 --> 04:03.000 So like the two options that we really have 04:03.000 --> 04:05.000 or that most people have is like their 04:05.000 --> 04:08.000 copying images to a private registry. 04:08.000 --> 04:10.000 So what they're doing is they're running their own 04:10.000 --> 04:12.000 registry and then they're saying, okay, we're just going to 04:12.000 --> 04:14.000 define all the images that we need. 04:14.000 --> 04:16.000 If they're a public or a private and we're going to 04:16.000 --> 04:18.000 move them over here and put them there. 04:18.000 --> 04:19.000 And that's going to be safe. 04:19.000 --> 04:21.000 We're going to have control over this. 04:21.000 --> 04:24.000 Or you're building some sort of like pull through cash. 04:24.000 --> 04:27.000 For example, harbor where you're 04:27.000 --> 04:30.000 pointing all your configuring all your nodes 04:30.000 --> 04:32.000 basically to pull through a mirror cash 04:32.000 --> 04:34.000 as you're getting the image for the first time. 04:34.000 --> 04:36.000 It catches it and then makes it available 04:36.000 --> 04:38.000 locally for you. 04:39.000 --> 04:42.000 The first one kind of depends on the fact that 04:42.000 --> 04:44.000 you're not going to have an outage with your 04:44.000 --> 04:45.000 registry. 04:45.000 --> 04:47.000 So you're kind of just like shifting the 04:47.000 --> 04:50.000 bottleneck and dependencies closer to home 04:50.000 --> 04:53.000 but they're still a risk of like you're 04:53.000 --> 04:55.000 really nice cluster not being a little 04:55.000 --> 04:58.000 scale because you're now managing your own 04:58.000 --> 04:59.000 registry. 04:59.000 --> 05:00.000 The other one is kind of like a best 05:00.000 --> 05:02.000 effort thing where it becomes a 05:02.000 --> 05:04.000 problem because you're not really like 05:04.000 --> 05:06.000 defining what images 05:06.000 --> 05:08.000 you're actually want to cash until you 05:08.000 --> 05:10.000 actually pull them down. 05:10.000 --> 05:12.000 Both of these kind of created like a 05:12.000 --> 05:14.000 management overhead where you're having to 05:14.000 --> 05:16.000 deal with defining what images do we need 05:16.000 --> 05:19.000 where they're coming from and constantly 05:19.000 --> 05:21.000 having to like clear up images that you 05:21.000 --> 05:22.000 no longer need. 05:22.000 --> 05:24.000 So like upgrading versions for example. 05:24.000 --> 05:26.000 And obviously there's like a very 05:26.000 --> 05:28.000 noticeable cost if you're running this 05:28.000 --> 05:31.000 at like a large end of a scale. 05:31.000 --> 05:34.000 So just some quick background I realize 05:34.000 --> 05:37.000 when I present this before that not everybody 05:37.000 --> 05:39.000 is like very into the OCI world. 05:39.000 --> 05:41.000 So it comes a lot helpful to just give 05:41.000 --> 05:43.000 some like some quick background like how 05:43.000 --> 05:46.000 OCI images and artifacts work. 05:46.000 --> 05:49.000 So like the image spec that is public on 05:49.000 --> 05:51.000 get up kind of defines like how a OCI 05:51.000 --> 05:53.000 artifact or like what we maybe know is a 05:53.000 --> 05:54.000 Docker image. 05:54.000 --> 05:57.000 How it is structured and defined. 05:57.000 --> 06:03.000 So basically the top level of a 06:03.000 --> 06:06.000 OCI image is what is called an index for 06:06.000 --> 06:07.000 the most part. 06:07.000 --> 06:08.000 It could be a manifest. 06:08.000 --> 06:11.000 These are all basically JSON files that 06:11.000 --> 06:15.000 reference other components of the image 06:15.000 --> 06:17.000 spec. 06:17.000 --> 06:20.000 What we have and what basically happens is that 06:20.000 --> 06:23.000 in the index you define all the 06:23.000 --> 06:27.000 manifests that are part of the image. 06:27.000 --> 06:29.000 These are reference through 06:29.000 --> 06:32.000 Digests which are hashes of the 06:32.000 --> 06:34.000 manifest file that you're actually 06:34.000 --> 06:35.000 referencing. 06:35.000 --> 06:38.000 And there can be like circular 06:38.000 --> 06:40.000 recursive referencing basically also. 06:40.000 --> 06:43.000 So if a good example of this is 06:43.000 --> 06:45.000 showing a probability to sum it. 06:45.000 --> 06:48.000 So this is like the index file for 06:48.000 --> 06:49.000 spagan itself. 06:49.000 --> 06:52.000 It has a list of manifests and then 06:52.000 --> 06:54.000 it has like digest. 06:54.000 --> 06:56.000 So we can like click at the digest and see 06:56.000 --> 07:01.000 like the manifest for for the 07:01.000 --> 07:04.000 AMD 64 build off spagan and then 07:04.000 --> 07:06.000 it in itself references a bunch of 07:06.000 --> 07:07.000 layers and it has sizes. 07:07.000 --> 07:09.000 So like this is like basically a 07:09.000 --> 07:11.000 mercury that is being created and this is like 07:11.000 --> 07:14.000 how OCI images are composed by these 07:14.000 --> 07:16.000 sets of references. 07:16.000 --> 07:18.000 Let's see here. 07:18.000 --> 07:21.000 This is not going to go full screen. 07:21.000 --> 07:24.000 Maybe loading. 07:24.000 --> 07:25.000 Maybe we shouldn't have done that. 07:25.000 --> 07:27.000 Oh, there we go. 07:27.000 --> 07:29.000 Perfect. 07:29.000 --> 07:30.000 Great. 07:30.000 --> 07:32.000 The second part we should be aware of is 07:32.000 --> 07:33.000 the distribution spec. 07:33.000 --> 07:36.000 So this is what defines like how clients are 07:36.000 --> 07:39.000 pulling images from a registry. 07:39.000 --> 07:42.000 What the distribution spec allows you to do is 07:42.000 --> 07:45.000 you can implement parts of the spec. 07:45.000 --> 07:47.000 So that's kind of what we're going to be focusing on. 07:47.000 --> 07:49.000 We're talking to like pushing in these types of things. 07:49.000 --> 07:51.000 We're only going to look at how a client pulls 07:51.000 --> 07:53.000 an image from a registry. 07:53.000 --> 07:55.000 So there are like basically three end points. 07:55.000 --> 07:57.000 You have to like implement v2, 07:57.000 --> 07:59.000 select v2 and it's basically to like tell the client 07:59.000 --> 08:01.000 that is supports it. 08:01.000 --> 08:03.000 Then there is like an end point to fetch 08:03.000 --> 08:05.000 manifest contents. 08:05.000 --> 08:09.000 The reference there is basically a tag or a digest. 08:09.000 --> 08:11.000 This only works for a manifest but what basically happens 08:11.000 --> 08:13.000 when you pull by tag it actually just resolves 08:13.000 --> 08:16.000 the tag to a digest and then returns the digest content. 08:16.000 --> 08:18.000 And then it is there is an end point to get 08:18.000 --> 08:21.000 the blocks content, which is like the layers 08:21.000 --> 08:25.000 of the actual binary data that creates the operating system 08:25.000 --> 08:27.000 that you're running. 08:27.000 --> 08:31.000 So this is basically like what you need to implement 08:31.000 --> 08:34.000 a read-only registry. 08:34.000 --> 08:39.000 It's like kind of simple actually which surprised me. 08:39.000 --> 08:46.000 So back in, I tried to look this up and I couldn't find it. 08:46.000 --> 08:49.000 But this was probably back in like 2018 08:49.000 --> 08:51.000 some time around there. 08:51.000 --> 08:55.000 It was a while ago considering that 08:55.000 --> 08:57.000 this was back when Kubernetes was still running 08:57.000 --> 08:59.000 like Docker as it's like a container run time 08:59.000 --> 09:01.000 in our community. 09:01.000 --> 09:05.000 I was adding me up listening to a 09:05.000 --> 09:07.000 pretty large company I'm not going to call them out. 09:07.000 --> 09:11.000 Talk about a 1am outage that they had. 09:11.000 --> 09:13.000 We're Docker hub went down. 09:13.000 --> 09:15.000 And at the same time they're seeing 09:15.000 --> 09:16.000 an influx in customers. 09:16.000 --> 09:18.000 I don't know why it 1am but it was probably 09:18.000 --> 09:20.000 in another time zone. 09:20.000 --> 09:23.000 But basically they couldn't scale their clusters 09:23.000 --> 09:26.000 because they had critical dependencies on images 09:26.000 --> 09:27.000 from Docker hub. 09:27.000 --> 09:30.000 So he was trying to send a solution about 09:30.000 --> 09:32.000 how he solved this. 09:32.000 --> 09:33.000 And it was basically him 09:33.000 --> 09:36.000 as a staging into a old node that had the images 09:36.000 --> 09:38.000 running Docker export. 09:38.000 --> 09:42.000 And then SEPing the exported tar 09:42.000 --> 09:45.000 to the new node that was scaling up. 09:45.000 --> 09:47.000 And then as a cluster was scaling up he was just like 09:47.000 --> 09:50.000 continuously like SEPing the images to the new 09:50.000 --> 09:53.000 nodes and then basically importing them. 09:53.000 --> 09:55.000 And it worked. 09:55.000 --> 09:59.000 Like even if it's like a very dirty solution. 09:59.000 --> 10:03.000 It kind of got them out of that kind of scenario. 10:03.000 --> 10:08.000 But it really planned to seed in my head about 10:08.000 --> 10:14.000 while having a physical person wake up at eight 10:14.000 --> 10:19.000 1am doing this work kind of as crazy. 10:19.000 --> 10:25.000 It grew from a technology standpoint that 10:25.000 --> 10:28.000 why aren't we always doing this? 10:28.000 --> 10:32.000 If we could automate this type of process we'd actually 10:32.000 --> 10:37.000 remove a critical dependency towards something external. 10:37.000 --> 10:40.000 And it kind of like start barking 10:40.000 --> 10:43.000 and trying to understand how does this work? 10:43.000 --> 10:44.000 How does Docker export work? 10:44.000 --> 10:46.000 Like where are the images coming from? 10:46.000 --> 10:49.000 At this time I was like basically varying you within 10:49.000 --> 10:52.000 the OCI specification. 10:52.000 --> 10:56.000 So I really didn't really understand everything. 10:56.000 --> 10:59.000 But like I had a couple of other questions. 10:59.000 --> 11:01.000 And so it was like okay so images. 11:01.000 --> 11:03.000 How are they stored in container D? 11:03.000 --> 11:06.000 Like where are they getting pulled from when we're 11:06.000 --> 11:07.000 exporting images? 11:07.000 --> 11:09.000 As I talked before about registries. 11:09.000 --> 11:11.000 Like is it possible to implement my own registry? 11:11.000 --> 11:15.000 Or is like Docker hub like the only alternative out there? 11:15.000 --> 11:18.000 Then it was just like I was thinking about mirroring. 11:18.000 --> 11:21.000 So how are mirror configurations and 11:21.000 --> 11:23.000 continuity configured and like what are the possibilities 11:23.000 --> 11:24.000 around there? 11:24.000 --> 11:27.000 And then obviously like discovery because like my idea is 11:27.000 --> 11:29.000 basically okay if I want to do this at scale 11:29.000 --> 11:32.000 I'd need to like have some sort of component that does 11:32.000 --> 11:38.000 discovery and like to basically find which nodes 11:38.000 --> 11:41.000 has which images. 11:41.000 --> 11:45.000 Eventually the solution I came up with is the project 11:45.000 --> 11:48.000 that I've been working out for a while now called Spiegel. 11:48.000 --> 11:52.000 It basically allows Kubernetes nodes to pull images 11:52.000 --> 11:54.000 from each other. 11:54.000 --> 11:57.000 It also has like a nice fallback solution where if the 11:57.000 --> 12:00.000 image doesn't exist anywhere in the cluster it basically 12:00.000 --> 12:03.000 just falls back to pulling from the original registry. 12:03.000 --> 12:06.000 So this is as I like to call it a 12:06.000 --> 12:09.000 stateless OCI registry. 12:09.000 --> 12:11.000 And that kind of is like a juxtaposition of like 12:11.000 --> 12:13.000 people like this isn't really right. 12:13.000 --> 12:18.000 But like they'll kind of explain a bit more about that. 12:18.000 --> 12:21.000 But through the benchmarks that I've built what we're 12:21.000 --> 12:25.000 basically seeing is an up to 80% improvement in image 12:25.000 --> 12:26.000 full times. 12:26.000 --> 12:30.000 There are obviously scenarios that are sub-optional 12:30.000 --> 12:34.000 for Spiegel but like I've built a it's also open source 12:34.000 --> 12:38.000 a benchmarking tool so you can compare your different 12:38.000 --> 12:42.000 pull performances. 12:42.000 --> 12:46.000 So the major breakthrough that really allow me to build 12:46.000 --> 12:52.000 Spiegel is I realized that container D when it pulls 12:52.000 --> 12:58.000 an image actually stores the uncompressed layers on disk. 12:58.000 --> 13:03.000 So this is disk space that is always used in any type of 13:03.000 --> 13:04.000 container environment. 13:04.000 --> 13:07.000 So the images have always existed in the cluster. 13:07.000 --> 13:10.000 So in all of these outages we're basically going, 13:10.000 --> 13:12.000 oh we can't pull from this registry. 13:12.000 --> 13:15.000 Basically there's somewhere in the cluster most likely 13:15.000 --> 13:18.000 your images have already existed. 13:18.000 --> 13:22.000 These are stored in yeah in valid container D. 13:22.000 --> 13:26.000 Just as like the file names are the hashes or the digest. 13:26.000 --> 13:30.000 Basically there is like some other trickery that you have to do 13:30.000 --> 13:33.000 to like talk to container D to get like the names of the images 13:33.000 --> 13:37.000 and like the tags but that's like pretty easy to solve. 13:37.000 --> 13:40.000 But this basically means that Spiegel can like piggyback 13:40.000 --> 13:42.000 on top of container D. It doesn't have to. 13:42.000 --> 13:45.000 It's not doing any type of storage. 13:45.000 --> 13:48.000 It's not doing any type of like cleanup. 13:48.000 --> 13:51.000 In Kubernetes garbage collection is triggered automatically 13:51.000 --> 13:53.000 on disk pressure. 13:53.000 --> 13:57.000 So you're basically really just acting as a proxy 13:57.000 --> 14:00.000 to serve content that is already there. 14:00.000 --> 14:05.000 And that's kind of like the cool thing of like there is no state at all. 14:05.000 --> 14:09.000 So the architecture is actually very simple. 14:09.000 --> 14:14.000 On each node that you run in the cluster you're running Spiegel. 14:14.000 --> 14:23.000 It basically runs a OCI registry that is available in localhost. 14:23.000 --> 14:29.000 Container D is configured basically as it has a mirror configured. 14:29.000 --> 14:34.000 So when you try to pull an image the first registry that it tries is not the upstream 14:34.000 --> 14:37.000 registry is actually localhost. 14:37.000 --> 14:42.000 Spiegel will then receive this request and try to discover another node 14:42.000 --> 14:47.000 in the cluster that has this layer that's being requested. 14:47.000 --> 14:49.000 And just proxy the request. 14:49.000 --> 14:52.000 That's like all it does. 14:53.000 --> 14:58.000 And in simplicity Spiegel is really just like three different components. 14:58.000 --> 15:00.000 So it's the registry. 15:00.000 --> 15:03.000 It's the routing or discovery mechanism. 15:03.000 --> 15:07.000 And it is the advertising mechanism. 15:07.000 --> 15:09.000 So and it's core sense. 15:09.000 --> 15:12.000 It isn't really more than that. 15:12.000 --> 15:18.000 Content discovery was probably the hardest part in figuring out how to implement these types of things. 15:19.000 --> 15:23.000 I this is not my background in like these types of distributed systems. 15:23.000 --> 15:28.000 But as some might like recognize and like the the diagrams and everything. 15:28.000 --> 15:30.000 This looks very much like the torrent. 15:30.000 --> 15:32.000 And that's basically what it is. 15:32.000 --> 15:39.000 It's a bunch of clients advertising content that they have to other clients that want to discover it. 15:39.000 --> 15:44.000 After some reading you kind of figure out that nearly. 15:45.000 --> 15:53.000 Most of these solutions today use an algorithm called cademlia, which is like a implementation of a distributed hash table. 15:53.000 --> 15:57.000 There is a really nice product called libp to p. 15:57.000 --> 16:01.000 That is used in like IPFS and some other things. 16:01.000 --> 16:02.000 Shout out to these guys. 16:02.000 --> 16:05.000 They have a really good implementation of cademlia and go. 16:05.000 --> 16:10.000 So it comes it was really like this is doing most of the heavy lifting. 16:10.000 --> 16:12.000 I am not smart enough to implement this myself. 16:12.000 --> 16:15.000 So very happy that somebody else did it. 16:15.000 --> 16:23.000 But basically what happens is it's basically looking at all the content locally in this and advertising it on to like the distributed hash table. 16:23.000 --> 16:32.000 And then as a request for coming in basically there's a look up done on the distributed hash table to like find the content. 16:32.000 --> 16:41.000 There's some compatibility issues mostly it's based it's dependent on like how the container de default configuration is. 16:42.000 --> 16:48.000 It works in a case it works in a case you just need like do some minor configuration changes not that bad. 16:48.000 --> 16:53.000 Currently it's embedded into both the k3s and rk2. 16:53.000 --> 17:01.000 So these are like embedded features they don't run as containers actually use import as big as a library into k3s. 17:01.000 --> 17:10.000 And it basically is it's from my perspective honestly a better solution than running it on top as a container. 17:10.000 --> 17:11.000 Sorry. 17:11.000 --> 17:14.000 So use cases. 17:14.000 --> 17:22.000 So when I built this I was working on another company running Kubernetes clusters for like large customers. 17:22.000 --> 17:25.000 So we used it mostly as a best there for cash. 17:25.000 --> 17:39.000 One thing that I've realized and that's like the major user of this project are home libraries kind of makes me happy but like the majority of users out there are using this to avoid a doctor hub's rate limiting. 17:39.000 --> 17:47.000 Which is like as if anybody's run like a small cluster at home is kind of like the major pain point where you just like I don't. 17:47.000 --> 17:58.000 I'm just doing this for fun I maybe don't want to pay for a subscription to pull an image but because you're restarting your nodes or trying something at home you're just like constantly getting rate limited. 17:58.000 --> 18:04.000 Yeah, so there's a lot of people they're using this as like a just like a small hobby project really or like just for their hobby projects. 18:04.000 --> 18:09.000 But also I'm seeing this talk people using this in air gap environments. 18:09.000 --> 18:16.000 I've received information that people are trying to run this on trains which is interesting. 18:16.000 --> 18:22.000 But also like there's there's people using these types of projects or this project to distribute AM models. 18:22.000 --> 18:33.000 So especially in situations where you're running large AM models and you want to like not pull these models from like a external registry you can basically because it's like private local. 18:33.000 --> 18:43.000 Like private local networking you can basically distribute these models a lot quicker internally. 18:43.000 --> 18:46.000 Okay, I will. 18:46.000 --> 18:52.000 Quickly talk about challenges and then maybe we can do questions. 18:52.000 --> 18:59.000 So the challenges that I'm currently working on is mostly about like topology or writing. 18:59.000 --> 19:06.000 There's like a lot of like problems and they're not problems for challenges and can only have because can only is only aware of like the larger network. 19:06.000 --> 19:11.000 It doesn't understand like networking topology and regions and these types of things. 19:11.000 --> 19:17.000 So there is some research work that that's out there that is trying to like. 19:17.000 --> 19:25.000 Nudge networks into like choosing the physically closer nodes compared to more distant nodes. 19:25.000 --> 19:30.000 But it's actually it seems like a very complicated thing and but it's like there's some work that being on there. 19:30.000 --> 19:32.000 Fair load balancing is nothing. 19:32.000 --> 19:36.000 It turns out it's very difficult to do. 19:36.000 --> 19:44.000 Implement load balancing algorithms on distributed systems where you don't even know the peers that you're going to route to before you get the request. 19:44.000 --> 19:55.000 So it becomes like a very interesting problem of calculating bandwidth and not basically overloading a single peer in your cluster. 19:55.000 --> 20:04.000 And then there's like some challenges in how goes HB coffee performance works in proxies basically everything is done in like user space. 20:04.000 --> 20:08.000 You're basically copying memory back and forth between like a response and a request. 20:08.000 --> 20:12.000 So there's like a lot of challenges there and like how. 20:12.000 --> 20:20.000 Like there is some performance degradation when running through a proxy compared to like hitting a registry immediately. 20:20.000 --> 20:26.000 I think there have some more things here but we can skip these. 20:26.000 --> 20:28.000 Thank you very much. 20:28.000 --> 20:39.000 If you have any questions we can ask them here. 20:39.000 --> 20:41.000 You can reach me by email also. 20:41.000 --> 20:45.000 My username at most platforms is Filippaba. 20:45.000 --> 20:49.000 Here's a link to the website and also to get a repo. 20:49.000 --> 20:51.000 There's a QR code there if you want to scan it. 20:51.000 --> 20:53.000 But we can start with questions maybe. 20:53.000 --> 20:54.000 Yes. 20:54.000 --> 20:55.000 I think for the talk. 20:55.000 --> 20:57.000 It's very interesting in sounds very useful. 20:57.000 --> 20:59.000 I was wondering if there is. 20:59.000 --> 21:07.000 I was wondering if there is an authentication mechanism or how does it work in general with the authentication. 21:07.000 --> 21:17.000 So currently it is completely unauthenticated because it's within your own private network. 21:17.000 --> 21:21.000 So the aspect of like. 21:21.000 --> 21:25.000 There are obviously security concerns. 21:25.000 --> 21:33.000 What you're kind of relying on is the fact that you can't modify the existing like mercury. 21:33.000 --> 21:37.000 So like if you're pulling images by digest reference. 21:37.000 --> 21:43.000 What you're getting back because the client will always validate the digest to the content it's getting. 21:43.000 --> 21:45.000 Can't be modified. 21:45.000 --> 21:47.000 Obviously if you're pulling by tag. 21:47.000 --> 21:49.000 You're trusting the upstream registry. 21:49.000 --> 21:52.000 K3S implement TLS. 21:52.000 --> 21:55.000 So it has like some it has it. 21:55.000 --> 21:58.000 We've worked together to add some additional authentication features. 21:58.000 --> 22:01.000 Out of the box bigum doesn't it's something I've looked at. 22:01.000 --> 22:09.000 It's very complicated to like generate certificates and a distributed workflow that is stateless. 22:09.000 --> 22:11.000 It's kind of if somebody wants to help with that. 22:11.000 --> 22:13.000 I'd love to have help. 22:13.000 --> 22:14.000 Yeah. 22:14.000 --> 22:15.000 Hello. 22:15.000 --> 22:16.000 Thank you. 22:16.000 --> 22:17.000 Oh. 22:17.000 --> 22:18.000 I'm. 22:18.000 --> 22:19.000 Speak out of the manager. 22:19.000 --> 22:21.000 The end user. 22:21.000 --> 22:23.000 I could not imagine. 22:23.000 --> 22:26.000 When we discover and the ability. 22:26.000 --> 22:27.000 For example. 22:27.000 --> 22:28.000 Sorry. 22:28.000 --> 22:30.000 How does it manage the end user? 22:30.000 --> 22:32.000 If we need to. 22:32.000 --> 22:34.000 Did it this image is in the plus. 22:34.000 --> 22:36.000 Or even the cabinet's crystal. 22:36.000 --> 22:38.000 Oh, it's bigger. 22:38.000 --> 22:40.000 Yeah. So so basically. 22:40.000 --> 22:43.000 Image layer image advertising is done with a TL. 22:43.000 --> 22:45.000 So there's like a time to live. 22:45.000 --> 22:49.000 What happens is if a so it's bigger doesn't. 22:49.000 --> 22:50.000 Remove any images. 22:50.000 --> 22:53.000 So that would be up to the garbage collector to do that. 22:53.000 --> 22:55.000 When an image is removed. 22:55.000 --> 22:57.000 It's going to be stopped to be advertised eventually. 22:57.000 --> 23:01.000 The time to live is going to invalidate the advertised key. 23:01.000 --> 23:06.000 What happens is if you could end up in a situation where. 23:06.000 --> 23:08.000 The images removed from disk. 23:08.000 --> 23:10.000 But the time to live has an expired. 23:10.000 --> 23:11.000 If what happens. 23:11.000 --> 23:13.000 It's just like it gets a four or four back. 23:13.000 --> 23:15.000 And they just tries to expire. 23:15.000 --> 23:17.000 So that's basically what happens. 23:17.000 --> 23:19.000 Yeah. 23:19.000 --> 23:20.000 I'm happy. 23:20.000 --> 23:22.000 Have you considered running it. 23:22.000 --> 23:24.000 The the other way around. 23:24.000 --> 23:29.000 You said that you have implemented it to circumvent. 23:29.000 --> 23:32.000 Limit rate limiting and stuff like that. 23:32.000 --> 23:36.000 But still you mentioned that there's one like overbearing one single 23:36.000 --> 23:38.000 node that has to serve too many images. 23:38.000 --> 23:43.000 Which is one of the things that I've had a question about when when you started. 23:43.000 --> 23:46.000 And now I was thinking is there possibility to switch around. 23:46.000 --> 23:49.000 I mean, if you don't have a problem with rate limiting. 23:49.000 --> 23:52.000 Could this be something to have more of a resilience increase. 23:52.000 --> 23:55.000 If you have a risk placement for the cluster where you. 23:55.000 --> 24:00.000 Debut default pull the image and only if like this connection does not work. 24:00.000 --> 24:04.000 It's like jumps to to speak a land then gets from another node. 24:04.000 --> 24:07.000 So you run doesn't or run into this issue. 24:07.000 --> 24:10.000 You can you can hear yes do this and container. 24:10.000 --> 24:13.000 Configure like mirror configuration. 24:13.000 --> 24:14.000 You could actually add. 24:14.000 --> 24:18.000 The mirror the original registry as the first mirror. 24:18.000 --> 24:21.000 And then have container default back if that mirror fails. 24:21.000 --> 24:23.000 That is that is 100% of possibility. 24:23.000 --> 24:26.000 I still think that like you probably want to serve it internally. 24:26.000 --> 24:30.000 Like it's it's mostly like a load balancing question of. 24:30.000 --> 24:34.000 Like the the rate limiting internal thing has mostly to do with. 24:34.000 --> 24:38.000 situations where one node has an image and then you spin up. 24:38.000 --> 24:41.000 200 more replicas and they all just go. 24:41.000 --> 24:45.000 Give me the image and there there has to be some sort of like because. 24:46.000 --> 24:53.000 From a from a sort of perspective it's like it would be more beneficial for. 24:53.000 --> 24:58.000 The request to queue because as each image is they are each node is getting pulled. 24:58.000 --> 25:03.000 It's actually helping in in the cluster of distributing that image. 25:03.000 --> 25:08.000 So like but we don't want is like the whole thundering herd problem where 200 clients all come. 25:08.000 --> 25:10.000 We we want some sort of like organization. 25:10.000 --> 25:12.000 I think that it could be solved. 25:12.000 --> 25:16.000 So I like I prefer to solve that rather than just go pull from the external registry. 25:16.000 --> 25:17.000 I just don't. 25:17.000 --> 25:21.000 I don't know the right words for this type of problem. 25:21.000 --> 25:22.000 It's kind of my yeah. 25:22.000 --> 25:23.000 Yeah thank you very much. 25:29.000 --> 25:30.000 Question up there. 25:30.000 --> 25:31.000 I've got a quick question. 25:31.000 --> 25:35.000 You said that you're using a harbour for image as a back end. 25:35.000 --> 25:39.000 Did you reconsider different things like a result project. 25:39.000 --> 25:40.000 Sorry what? 25:40.000 --> 25:44.000 Did you try a result project for that instead of harbour? 25:44.000 --> 25:49.000 Yeah so like all running all these like thoughts, harbour or like the distribution, 25:49.000 --> 25:56.000 registry whatever they're like multiple of these like types of registries that kind of solve these types of problems. 25:56.000 --> 25:58.000 But they they're all stateful. 25:58.000 --> 26:03.000 That you all you like in these types of solutions you're basically having to like run something that has 26:03.000 --> 26:06.000 state that is using disk and that's kind of like the challenge. 26:10.000 --> 26:12.000 Hi. 26:12.000 --> 26:15.000 First of all thanks for the things ready to talk. 26:15.000 --> 26:18.000 I was wondering about what about latest stack. 26:18.000 --> 26:19.000 How are you dealing with that? 26:19.000 --> 26:21.000 What's latest stack? 26:21.000 --> 26:23.000 It's on request. 26:23.000 --> 26:25.000 Yes. 26:25.000 --> 26:26.000 That is a problem. 26:26.000 --> 26:27.000 There is in the FAQ. 26:27.000 --> 26:29.000 I have document how to solve this. 26:29.000 --> 26:37.000 If you're only pulling by latest yes you will basically lock in a single version of that latest. 26:37.000 --> 26:39.000 You can't update it. 26:39.000 --> 26:46.000 There are projects like Kate's digester that like runs the muting web hook and resolves the latest tag for you and then 26:46.000 --> 26:48.000 append the digest. 26:48.000 --> 26:51.000 That's kind of the solution you have to go to. 26:51.000 --> 26:56.000 I will not preach about my opinions about using reusing tags. 26:56.000 --> 27:03.000 Do you have experience with which non-image blocks can be stored locally? 27:03.000 --> 27:04.000 I'm here by the way. 27:05.000 --> 27:14.000 If you have all of us or something where some at the station is attached in the OCI image structure would contain the cache. 27:14.000 --> 27:17.000 Anything of that or would be done to pull that? 27:17.000 --> 27:20.000 Not like non-doctor image. 27:20.000 --> 27:24.000 This is like I skipped one of those slides but it was basically. 27:24.000 --> 27:27.000 Container D is coming with like OCI volumes. 27:27.000 --> 27:30.000 Soon hopefully there's like a there's a PR for it. 27:30.000 --> 27:37.000 That's going to raise like the ability to run like to like mount OCI artifacts as volumes into a container. 27:37.000 --> 27:40.000 When that comes it will work. 27:46.000 --> 27:48.000 Maybe I'm running out of time. 27:48.000 --> 27:52.000 I don't want to mess up somebody else. 27:52.000 --> 27:55.000 Just checking this in one more question maybe. 27:55.000 --> 27:56.000 Okay. 27:56.000 --> 27:57.000 Who was this? 27:57.000 --> 27:58.000 I think he was it. 27:58.000 --> 28:01.000 We can we can talk in the corridor otherwise. 28:01.000 --> 28:02.000 I think he was. 28:02.000 --> 28:04.000 Then the rest you can ask afterwards. 28:04.000 --> 28:11.000 Hey, uh, I was just wondering is it there anyway to use it outside of Kubernetes? 28:11.000 --> 28:17.000 Let's say with part which pulls OCI images for Mac VMs which are quite big for instance. 28:17.000 --> 28:18.000 Wait, where? 28:18.000 --> 28:22.000 There's a protocol taught which runs maximum Mac VMs. 28:22.000 --> 28:23.000 Oh yeah. 28:23.000 --> 28:27.000 I know that there are people running this in Nomad. 28:27.000 --> 28:31.000 Like if as long as you're running container D it should work. 28:31.000 --> 28:39.000 There is like a slight requirement in like how you do the bootstrapping of like the distributed hash table. 28:39.000 --> 28:43.000 If you want to have support for something that isn't no matter Kubernetes, 28:43.000 --> 28:48.000 I'd happily work with you and like try to figure stuff out. 28:48.000 --> 28:49.000 Great. 28:49.000 --> 28:50.000 Cool. 28:50.000 --> 28:51.000 Thank you. 28:51.000 --> 28:52.000 Thank you very much. 28:52.000 --> 29:02.000 Here we go. 29:02.000 --> 29:07.000 All right, so I'm making up. 29:07.000 --> 29:17.000 Thank you. 29:37.000 --> 29:47.000 Thank you. 29:47.000 --> 29:56.000 Thank you. 29:56.000 --> 29:58.000 Thank you. 29:58.000 --> 30:01.000 Thank you. 30:07.000 --> 30:10.000 Thank you.