WEBVTT 00:00.000 --> 00:05.000 Thank you. 00:05.000 --> 00:07.000 So welcome to my session, 00:07.000 --> 00:10.000 State of Checkpoint Restore and Kubernetes. 00:10.000 --> 00:11.000 My name is Art van Rébo. 00:11.000 --> 00:14.000 I'm really happy to be at first them again this year. 00:14.000 --> 00:18.000 So I work at Red Hat since about 10 years now. 00:18.000 --> 00:21.000 I'm involved in process migration checkpoint restore, 00:21.000 --> 00:25.000 which is the basis for what I'm talking about today for at least 15 years now. 00:25.000 --> 00:30.000 The project we are using today to do checkpoint restore processes 00:30.000 --> 00:34.000 and containers is called CREU. 00:34.000 --> 00:37.000 We'll talk a bit about that also. 00:37.000 --> 00:39.000 There's a problem. 00:39.000 --> 00:42.000 There's also a tool I'm talking about, a container engine, 00:42.000 --> 00:44.000 which is called CRIO with the O at the end. 00:44.000 --> 00:47.000 So don't get too confused. 00:47.000 --> 00:50.000 I'm involved in CREU maybe since 2012. 00:50.000 --> 00:57.000 Checkpoint restore user space CREU. 00:57.000 --> 00:59.000 That's the tool we're using. 00:59.000 --> 01:03.000 So why this name, you might ask, may ask. 01:03.000 --> 01:05.000 The reason is there were, 01:05.000 --> 01:09.000 checkpoint restoring things in processes is the thing 01:09.000 --> 01:14.000 which has been done in operating systems for a long time. 01:14.000 --> 01:18.000 And also people are looking at this in Linux for a long time. 01:18.000 --> 01:22.000 It kind of started in high performance computing. 01:22.000 --> 01:25.000 For more or less the same reasons as today, 01:25.000 --> 01:31.000 you need four tolerance or you want to load balancing something. 01:31.000 --> 01:37.000 So you need to find a way to migrate a process from one system to another somehow. 01:37.000 --> 01:40.000 And there were some implementations, 01:40.000 --> 01:43.000 maybe 15, 20 years ago, they were using some external code module, 01:43.000 --> 01:45.000 which was never upstream. 01:45.000 --> 01:49.000 Then there was some library interception intercepting system calls, 01:49.000 --> 01:52.000 so you can know what the process is doing. 01:52.000 --> 01:56.000 And all those implementations were not perfect because 01:56.000 --> 01:59.000 they were not what you would call a transparent checkpoint restore 01:59.000 --> 02:01.000 because you had to prepare your process. 02:01.000 --> 02:04.000 You had to know at the point of time when you start the process, 02:04.000 --> 02:06.000 at some time you want to check it. 02:06.000 --> 02:08.000 For high performance computing, you would say, 02:08.000 --> 02:10.000 okay, I do this with all my processes, 02:10.000 --> 02:12.000 but it still limits you a bit. 02:12.000 --> 02:14.000 And so maybe, I don't know, 02:14.000 --> 02:17.000 15, 18 years ago there was an approach 02:17.000 --> 02:20.000 to bring checkpoint restore into the Linux kernel, 02:20.000 --> 02:22.000 and it didn't have a name, 02:22.000 --> 02:25.000 but it would probably be called checkpoint restore in kernel space. 02:25.000 --> 02:28.000 It was not accepted by the Linux community, 02:28.000 --> 02:31.000 because it was a big and complex at a time. 02:31.000 --> 02:33.000 And so a new approach was started, 02:33.000 --> 02:35.000 checkpoint restore uses space, 02:35.000 --> 02:37.000 kind of the contrast to the previous one. 02:37.000 --> 02:38.000 And the thing which, 02:38.000 --> 02:41.000 Cree you would try to do, or try to do from the beginning, 02:41.000 --> 02:46.000 is use existing interfaces of the Linux kernel as much as possible. 02:46.000 --> 02:52.000 And with this, it was possible to have some basic checkpoint restore 02:52.000 --> 02:54.000 working maybe, I don't know, 02:54.000 --> 02:58.000 12 or even longer when it the project started a year ago. 02:58.000 --> 03:02.000 And over the years, Cree you introduced additional interfaces 03:02.000 --> 03:03.000 into the Linux kernel. 03:03.000 --> 03:06.000 And the interesting thing about those interfaces is 03:06.000 --> 03:10.000 they are never really only for checkpointing restoring. 03:10.000 --> 03:12.000 They are also used by other tools. 03:12.000 --> 03:16.000 They basically they give you additional information about the process, 03:16.000 --> 03:19.000 which you can read out while the process runs. 03:19.000 --> 03:22.000 And then there's also usually some interface 03:22.000 --> 03:26.000 to restore this value to set something in the process, 03:26.000 --> 03:29.000 which makes it restoreable things like this. 03:29.000 --> 03:33.000 So let's come to the Kubernetes topic of this talk. 03:33.000 --> 03:37.000 So in 2015 actually there was a ticket opened 03:37.000 --> 03:41.000 to get something into Kubernetes, 03:41.000 --> 03:47.000 which you might call port or container migration. 03:47.000 --> 03:50.000 If you read a ticket, it's still open. 03:50.000 --> 03:52.000 It's a very, very high level discussion, 03:52.000 --> 03:54.000 so it's not about technology. 03:54.000 --> 04:00.000 It's more about the ideas how to have container migration 04:00.000 --> 04:02.000 possible in Kubernetes. 04:02.000 --> 04:07.000 And one of those things which is often difficult for a container world 04:07.000 --> 04:09.000 is that container are supposed to be stateless. 04:09.000 --> 04:11.000 So why would you want to need to migrate them? 04:11.000 --> 04:12.000 Because it's stateless. 04:12.000 --> 04:15.000 I'm just stopping it and starving it somewhere else. 04:15.000 --> 04:19.000 The reality has shown that containers in fact are not stateless. 04:19.000 --> 04:20.000 They are stateful. 04:20.000 --> 04:25.000 And people are today using 10 years later container migration. 04:25.000 --> 04:27.000 In production it's maybe nothing Kubernetes, 04:27.000 --> 04:30.000 but there are a lot of different container run times today 04:30.000 --> 04:32.000 and engines which support checkpoint, 04:32.000 --> 04:35.000 restore all the level down. 04:35.000 --> 04:37.000 You have run C, C run. 04:37.000 --> 04:39.000 You have Docker, podman, container D, 04:39.000 --> 04:41.000 LexD, Inclus. 04:41.000 --> 04:44.000 They all in some kind of way work together with 04:44.000 --> 04:48.000 crew to enable you to checkpoint and restore containers. 04:48.000 --> 04:55.000 So that's why we started to look closer at how to do 04:55.000 --> 04:59.000 checkpoint restore and maybe at some point container migration 04:59.000 --> 05:04.000 in Kubernetes around, I think 2020 was our first ideas 05:04.000 --> 05:08.000 when we talked about this to actually implement it. 05:08.000 --> 05:11.000 So let's talk a bit about use cases. 05:11.000 --> 05:14.000 Why would you want to checkpoint restore containers? 05:14.000 --> 05:16.000 Why would you want to migrate them? 05:16.000 --> 05:20.000 And most of the use cases are actually what people use today 05:20.000 --> 05:25.000 to do exactly that. 05:25.000 --> 05:28.000 And one really popular use case is the quick startup. 05:28.000 --> 05:32.000 So you have and in most cases it seems to be a Java container. 05:32.000 --> 05:35.000 It takes some time to start maybe a really long time. 05:35.000 --> 05:38.000 So what you can do with checkpoint restore is, 05:38.000 --> 05:41.000 so this is your container and this is your host running the container. 05:41.000 --> 05:44.000 So you can take a copy of the container right into this. 05:44.000 --> 05:47.000 Once it has everything loaded in initialized and then we write 05:48.000 --> 05:52.000 a stateful copy of the container to disk with all the information. 05:52.000 --> 05:56.000 And once I don't know, maybe a user or customer once to use the service, 05:56.000 --> 06:01.000 we can quickly deploy additional containers from this pre-initialized 06:01.000 --> 06:03.000 pre-initialized state. 06:03.000 --> 06:06.000 And the container starts up much, much quicker. 06:06.000 --> 06:12.000 And this is because preview just reads all the memory pages from 06:12.000 --> 06:14.000 storage back into memory. 06:14.000 --> 06:19.000 And there's no complicated library loading or data loading or whatever 06:19.000 --> 06:20.000 necessary. 06:20.000 --> 06:24.000 It's just we just loaded in one go basically. 06:24.000 --> 06:29.000 And I have looked at container migration times, 06:29.000 --> 06:31.000 especially the downtime. 06:31.000 --> 06:36.000 How long will my application be down if I try to do a live migration? 06:36.000 --> 06:41.000 And the most time you spend for the migration is always the network 06:42.000 --> 06:44.000 writing the checkpoint to disk. 06:44.000 --> 06:47.000 And you can, I don't know, you do it to RAM disk. 06:47.000 --> 06:52.000 And it's even faster or restoring it from disk is never the bottleneck. 06:52.000 --> 06:54.000 It's always the network. 06:54.000 --> 07:00.000 And so what I want to say is reading and writing the checkpoint is a fast operation 07:00.000 --> 07:01.000 usually. 07:01.000 --> 07:05.000 So another use case which is kind of, it's a similar use case. 07:05.000 --> 07:08.000 It's like the quick startup a bit. 07:08.000 --> 07:09.000 It's a reviewed and safe state. 07:09.000 --> 07:12.000 So again, you have a container which takes a long time initialized. 07:12.000 --> 07:14.000 You don't want to have a long downtime. 07:14.000 --> 07:17.000 So you, but you need to install some updates on your system. 07:17.000 --> 07:18.000 So what you can do. 07:18.000 --> 07:19.000 You install the updates. 07:19.000 --> 07:21.000 You take a checkpoint of your container. 07:21.000 --> 07:22.000 Write it to disk. 07:22.000 --> 07:24.000 Then the container is there alone. 07:24.000 --> 07:25.000 The system is gone. 07:25.000 --> 07:27.000 It reboots into your new system. 07:27.000 --> 07:28.000 A new kernel. 07:28.000 --> 07:29.000 A green one. 07:29.000 --> 07:32.000 And the container can be restarted from the checkpoint. 07:32.000 --> 07:34.000 It's a bit like the quick startup use case. 07:34.000 --> 07:38.000 But it's a bit, also a bit different. 07:38.000 --> 07:42.000 Then container life migration is kind of the combination of those use cases. 07:42.000 --> 07:45.000 So you have a container running on one system. 07:45.000 --> 07:48.000 And maybe you have additional containers running there. 07:48.000 --> 07:52.000 And resources are getting low or your machine has some troubles. 07:52.000 --> 07:56.000 And you want to migrate your workload from one system to another. 07:56.000 --> 08:00.000 And then the interesting discussion here I often have is life migration on 08:00.000 --> 08:01.000 life migration. 08:01.000 --> 08:05.000 And I think the definition of life migration is a really difficult one. 08:05.000 --> 08:08.000 Because even if you think about life migrating VMs. 08:08.000 --> 08:13.000 There will be some time when the VM is not running. 08:13.000 --> 08:18.000 So I guess you could also call this container life migration. 08:18.000 --> 08:22.000 Because we have to shut it down and move it somewhere else. 08:22.000 --> 08:27.000 We can employ and create the same technologies as we do into EMU. 08:27.000 --> 08:30.000 We have pre-copy migration, we have post-copy migration. 08:30.000 --> 08:37.000 So we can minimize the downtime in pre-using the same techniques you have in virtual machines. 08:37.000 --> 08:41.000 So from my diagram is you have the container running on one system. 08:41.000 --> 08:46.000 You take a checkpoint, you write it to some storage, share storage, whatever. 08:46.000 --> 08:49.000 And then you can start it multiple times on the destination system. 08:49.000 --> 08:55.000 And if you stop a container after the checkpoint or if you keep it running, 08:55.000 --> 08:57.000 it doesn't make a difference for the tool. 08:57.000 --> 09:02.000 Because this is what Cree you does, Cree you basically, 09:02.000 --> 09:07.000 and in Cree you speak, it infects the process with a parasite code. 09:07.000 --> 09:11.000 And then the process is running under the control of Cree you. 09:11.000 --> 09:17.000 And once we have written all the information from within the address space of the process, 09:17.000 --> 09:21.000 Cree you is then cheering the process and removes the parasite again. 09:21.000 --> 09:24.000 So the process will never know that it was checkpointed. 09:24.000 --> 09:30.000 It will just stop for short time and this is basically the time it takes to write the memory to disk. 09:30.000 --> 09:36.000 It's the most information written and depending on your process, this can be a lot of data. 09:36.000 --> 09:41.000 Another use case which we initially didn't really talk about, 09:41.000 --> 09:45.000 but people reached out to us are spot instances. 09:45.000 --> 09:49.000 So spot instances from what I've been told are instances with a really cheap, 09:49.000 --> 09:55.000 but they can go away really fast in 30 seconds or 2 minutes depending on your cloud providers. 09:55.000 --> 10:02.000 And so what people do today is they use Cree you to wait for a signal that the spot instances spot instance might go down. 10:02.000 --> 10:09.000 Then the evacuated node quickly using Cree you writing the checkpoint maybe to some S3 or whatever. 10:09.000 --> 10:14.000 And then they can continue to work on some other spot instance or wherever they want, 10:14.000 --> 10:18.000 without losing any of the work they've done so far. 10:18.000 --> 10:27.000 And last year there was a lot of interest from people how to use Cree you in combination with GPUs. 10:27.000 --> 10:31.000 And if I have had this talk last year then I would have said, 10:31.000 --> 10:47.000 it worked with ANG GPUs but not with NVIDIA GPUs but we are lucky that NVIDIA released a tool to actually actually do the checkpointing of the GPU state into the main memory of the host. 10:47.000 --> 10:57.000 And then Cree you can check point this state and so we can today with the help of an external tool called CUDA checkpoint. 10:57.000 --> 11:06.000 We can also migrate workloads using NVIDIA GPUs and AMD also contributed like two or four, 11:06.000 --> 11:11.000 maybe four years ago code into Cree you to do actually that. 11:11.000 --> 11:34.000 I want to know more about GPU checkpoint there's tomorrow talk 9 a.m. in the HTC Deaf room and they talk about exactly this how they use Cree you in combination with Kubernetes and especially GPU checkpointing to optimize the utilization of the GPUs node without having them sit with nothing to do for a long time. 11:35.000 --> 11:47.000 From the youth cases back to Kubernetes so we tried we tried to think how can we introduce this into Kubernetes. 11:47.000 --> 12:00.000 And I mentioned before the container community often things as containers is stateless and we come along and say well we want to migrate them and then they say, 12:00.000 --> 12:13.000 we don't know it's necessary. So we try to find a way how can we convince the Kubernetes community to get checkpointing into Kubernetes with the minimal change as possible. 12:13.000 --> 12:26.000 So we want to be as least invasive as possible when we introduce this into Kubernetes and the idea the story we put around this and it's important to know this is just a story around it. 12:26.000 --> 12:53.000 The actual functionality today in Kubernetes is not about forensic container checkpointing it's container checkpointing but we used this story to bring this feature into Kubernetes and one of the nice things about forensic container checkpointing was it's only about checkpointing the idea of of this idea is the idea of this idea is intent of this idea is that. 12:53.000 --> 13:10.000 So there's a container running on your system and there is maybe you have you think I might be in a tech going on or not and what what what's done today is if you have an intrusion detection system something like this then maybe it detect something it kills the container and yourself. 13:10.000 --> 13:28.000 And the idea is with forensic container checkpointing we now take a checkpoint of the container and this means we have all the memory pages from that point in time we have all and it's important we have all the memory pages means we have all passwords or random numbers or secrets whatever there is. 13:28.000 --> 13:41.000 But we also have the complete stage so whatever the attacker might be doing in the container we can now reproduce it in a sandbox environment we can restore the container maybe we can just look at the memory pages. 13:41.000 --> 13:59.000 We also offer to convert the checkpoint into core files so you can run it through gdb and see what was going on so there's a lot of possibilities you can now do with the checkpoint of your container and this is also what especially. 13:59.000 --> 14:11.000 As far as I've been told banks are interested in they actually want to have a copy of the thing they stopped for later analysis to make sure if there wasn't a tech going on or not. 14:11.000 --> 14:26.000 So this work started in 2020 we the code was was really simple to do this but we discussed this for at least two years how to how to do this correctly. 14:26.000 --> 14:40.000 And in 2022 with Kubernetes 125 this was released as an alpha feature alpha feature in Kubernetes means it's default off but anyone who wants to use it can turn it on and we. 14:40.000 --> 15:09.000 I have implemented the feature so so you need to implement it in Kubernetes and in the container engine currently I think there are mainly two container engines for Kubernetes it's container D and cryo and we implemented the feature in cryo 125 cryo has the same release numbers as Kubernetes I think so with the release of Kubernetes 125 we also had a container engine supporting this so with Kubernetes 125 it was possible to create checkpoints of your container. 15:10.000 --> 15:37.000 And write them to disk and look at the image so and Kubernetes 130 in 2024 we moved the feature in Kubernetes from alpha to beta and beta means defaults to on so now we have a feature which is always on and Kubernetes and anybody can use it without any additional configuration so you can still turn it off if you think this is unsafe for my system. 15:37.000 --> 16:00.000 I don't want it but currently it defaults on and with the release of or shortly after the turning it to better we also got the changes into container D so today it's possible to create a checkpoint with container D and cryo using the latest version of container D so you need at least two dot oh. 16:01.000 --> 16:10.000 So we have check pointing in beta so let's talk a bit about container restore I already mentioned that the whole story about forensic container check pointing was. 16:10.000 --> 16:25.000 just a story and we we put on on this to ensure it's easier to accept kind of so but we also were thinking about. 16:25.000 --> 16:43.000 and restoring containers and we we left it out of the enhancement proposals for Kubernetes but we find a way to kind of trick Kubernetes to restore container and we were able to include this code also in cryo 125 so what we do is if you if Kubernetes stops a container it usually first does a. 16:43.000 --> 16:49.000 container create and then does a container stop and we now hook in cryo into container create and container stop. 16:49.000 --> 16:58.000 We see if the image path is a checkpoint image if there is a checkpoint image then we actually restore the container and do not create it from scratch. 16:58.000 --> 17:10.000 And then we say to Kubernetes there's a container so Kubernetes restore the container without never knowing it's a restore it thinks I just created a container I'm greater that and so. 17:10.000 --> 17:31.000 And this is also available since cryo 125 the pull request for container D is open for half a year now and they have been some feedback from the container D maintainers so it seems like they are interested in it as far as I would say it just takes some time to go over. 17:31.000 --> 17:52.000 These changes in in container D to to get emerged but I'm positive we will also have the possibility to restore containers in container D is soon and you can always try out my pull request so let's do a quick demo here so now so this is. 17:52.000 --> 18:02.000 System on my so this is the m on my home system couple of hundred kilometers away and I will start a really simple container here. 18:02.000 --> 18:18.000 So it's it's I guess it's the most simple Yamazaya for a Kubernetes container it's a state full container of course because if I want to migrate it to the m locally it doesn't make me make much sense if it's stateless so let's start this container. 18:19.000 --> 18:32.000 So it's starting you see great so I mentioned great install so this is what Kubernetes does for each container and that's where we hook into a restore so let's talk to my container. 18:32.000 --> 18:43.000 So it's really state full and now I create a checkpoint the checkpoint is a tube let only API at this point I will talk a bit about this later more. 18:43.000 --> 19:01.000 We have created a tool called checkpoint control to inspect the checkpoint image and it gives you some information here it's the image it's based on when it was checkpoint the engine the IP address the sizes and the process tree and it also gives you much more information if you want that. 19:01.000 --> 19:11.000 So this is everything the tool can extract and this was written by some Google summer of code students were really thankful for the work so we see. 19:11.000 --> 19:33.000 Kubernetes metadata annotations more annotations annotations annotations then we see the mount points from the annotation point of view then we see our process tree again so you see the complete command lines and the PID from within the PID name space you see all the environment variables which we've said. 19:33.000 --> 19:42.000 And files another process in the container more variables you see also there's a TCP socket I'm talking to you see that one. 19:42.000 --> 20:00.000 And then again mountains now from the point of view of the tool how we see it in the in the meta data and not in the Kubernetes meta data we all need all this information to later restore the mount points so what we do during restore we restore all the mount points mount points as they were before so that. 20:00.000 --> 20:11.000 Tools can find all the files and it had before so now I will convert convert this to an OCI image using builder so I say. 20:12.000 --> 20:33.000 So I'm copying my tar file to the container and then I add a annotation to so that cryo knows it's a checkpoint archive and then I do a commit and then I removed intermediate image and then I. 20:34.000 --> 20:35.000 And then I push it to a registry. 20:39.000 --> 20:40.000 There it is. 20:44.000 --> 20:56.000 So the checkpoint image is really small it's like eight nine megabytes because we have only the differences only the memory in my test cases really small so let's go to my local VM here so let's see describe. 20:56.000 --> 21:09.000 And now I have again a really simple gamma file and it's now. 21:09.000 --> 21:12.000 Where is it 73. 21:12.000 --> 21:19.000 You see I've done this a couple of times already. 21:19.000 --> 21:25.000 And now I tell Kubernetes please run this container and I find out how to describe. 21:25.000 --> 21:34.000 You see it said it pulled the image and it created the container in started a container and now if I talk to my container I should get a. 21:34.000 --> 21:40.000 A three thank you and not a zero and there it is so you've seen. 21:40.000 --> 21:42.000 Thank you. Thank you. 21:46.000 --> 21:50.000 So this is possible since 2022 so. 21:51.000 --> 21:54.000 Okay, this was the demo I have five minutes of perfect. 21:54.000 --> 22:15.000 So next step so I opened Kubernetes enhancement proposal pull request to declare the feature as stable the requirements in our so we defined the requirements early in 2020 what other requirements that it requirements to container engines support it. 22:15.000 --> 22:32.000 No serious bugs I think and and test yeah so okay test so everything is available right now so today if you do a pull request against the Kubernetes main repository container the will run checkpoint with a checkpoint tests using. 22:32.000 --> 22:39.000 Our our code from container D and our test so it's all is there so there shouldn't be much. 22:39.000 --> 22:44.000 Again, moving it to stable or GA so let's see if this works. 22:44.000 --> 23:05.000 The next thing I mentioned is it's a cube let only API at this point so I opened Kubernetes enhancement proposal to move it to cube ctl because cube let only API currently means mainly you have to go to the node where the cube let runs and talk to the cube let directly if you have it in cube ctl then you can do it from wherever cube ctl is running. 23:05.000 --> 23:13.000 If you have the right roles defined to your user account so this is also something I started. 23:13.000 --> 23:20.000 The the process to get this into into Kubernetes this will probably take take a bit longer so. 23:20.000 --> 23:34.000 The summary is the idea of container pot migration was already first mentioned in Kubernetes in 2015 we went off of the feature 2022 beta and 2024 we hope to be stable in 2025 and we are. 23:34.000 --> 23:40.000 Working on cube ctl integration and with this I'm the end thank you for your time. 23:40.000 --> 24:00.000 Thank you for your contribution I've been following the container D you have to. 24:00.000 --> 24:05.000 Sorry I've been following the container D for request for a while thank you for the contribution. 24:05.000 --> 24:17.000 I have two questions actually like first question because I've been doing this in production using everything you built that basically first question is what is the replacement for page server. 24:17.000 --> 24:25.000 So you're getting more and more traction and spot instances without arm is very difficult to handle. 24:25.000 --> 24:35.000 The second question is very difficult nowadays to separate the process itself from the state that's persisting in the machine. 24:35.000 --> 24:47.000 So you move the process but you have a file open for example the famous storage do we have any traction to actually move files with the process itself in the same way. 24:47.000 --> 24:48.000 Thank you very much. 24:48.000 --> 25:04.000 So for the first question about page server we don't we don't yet work on any of the optimizations on the Kubernetes level we only have work with the optimizations for the migration on the lower levels. 25:04.000 --> 25:21.000 For Kubernetes we want to make sure the feature works and then we look into optimizations for the files is we include all changed files in the checkpoint image so if I have a lock file or an input file and output file they are all part of the checkpoint image and they are restored on the destination system. 25:21.000 --> 25:34.000 If there are inside of the container if you have a bind mount from the outside then the bind mount is restored and you have to make sure that the content you bind mount into is the same. 25:34.000 --> 25:45.000 If you use a volume thing in Kubernetes then we do not handle it because the volumes are on a pot level and we do only container single container migration. 25:45.000 --> 25:52.000 Pot migration is also something we loosely discuss but we focus on one thing right now. 25:52.000 --> 25:57.000 Thank you I have one question. 25:57.000 --> 25:59.000 Oh yes one. Sorry. 25:59.000 --> 26:02.000 When do you think we're going to get this? 26:02.000 --> 26:04.000 I don't hear your sorry. 26:04.000 --> 26:14.000 I don't know what to do. 26:14.000 --> 26:33.000 So the question is when can we expect that to make it to the public cloud type Kubernetes offering EKS? 26:33.000 --> 26:40.000 So I can I can talk about for example I'm a redhead so I can talk about what happens with open shift. 26:40.000 --> 26:48.000 So what happened with open shift is because it it defaults to on in the upstream repository it also defaults to on in open shift. 26:48.000 --> 26:58.000 So if Google or Amazon are not actively disabling it an upstream feature then as soon as you have one thirty I guess. 26:58.000 --> 27:03.000 But you must be able to access the Cupid API. 27:03.000 --> 27:06.000 I guess that's the hard part. 27:06.000 --> 27:12.000 All right thanks a lot. Any more questions you can probably find Adrian somewhere. Thank you.