WEBVTT 00:00.000 --> 00:29.840 How many of you run the NS server, how many of you vote the NS server, how many of you 00:29.840 --> 00:36.840 would not never do that again? 00:36.840 --> 00:47.840 All right, we can start at end, so a little bit of a story by nine, it's started actually 00:47.840 --> 00:52.840 in the last millennia. 00:52.840 --> 00:59.840 It is a spiritual successor of bind 4 in bind 9, bind 8, sorry, and when people say, 00:59.840 --> 01:06.840 oh, there are so many security issues with bind, they usually speak about bind 4 in bind 8. 01:06.840 --> 01:10.840 And because bind 9 was written from a scratch, it's a different story. 01:10.840 --> 01:16.840 Sure, we have a security issues, but a different kind of them, because there's something called 01:16.840 --> 01:23.840 contract designed by contract, which basically makes the server crash if there's anything unusual, 01:23.840 --> 01:30.840 which is much better than revealing a memory or writing a memory stuff like this. 01:30.840 --> 01:33.840 It was originally single-fredded and multifredded. 01:33.840 --> 01:40.840 You can choose that as a compound option by nine, and because of that, it's own event-based 01:40.840 --> 01:42.840 copper of scheduling. 01:42.840 --> 01:49.840 It uses locking for, you know, the multifredded, it has own RW lock that allows downgrades 01:49.840 --> 01:55.840 and upgrades of the lock, which the standard POSIX RW lock doesn't allow. 01:55.840 --> 01:59.840 So, 1998, how all this that? 01:59.840 --> 02:07.840 So, for the same money roughly, you can get it like 4 CPU threads, 500 megahertz 02:07.840 --> 02:11.840 clock, and you know, 4 gigab hours, and it was really, really cool machine. 02:11.840 --> 02:18.840 Nowadays, it's like, you know, you can have a really, really be finished in for the same amount of money. 02:18.840 --> 02:28.840 So, the technology is from 1998, a full phone, right? 02:28.840 --> 02:31.840 So, more recent changes we did in bind 9. 02:31.840 --> 02:37.840 So, in bind 9, the 14, those are already end of life. 02:37.840 --> 02:43.840 We started requiring the atomics. 02:43.840 --> 02:50.840 Here, it was not only standard atomics, but it was also, well, bind would be able to build 02:50.840 --> 02:55.840 with the GCC atomics that's built in. 02:55.840 --> 03:02.840 So, if you look at the source code, I wrote a, like, a shim on top of the Windows interlock 03:02.840 --> 03:07.840 API to emulate standard atomics on top of the Windows API. 03:07.840 --> 03:14.840 It was a horrible, horrible, like, if you ever wanted to see a horrible C macros that's the place. 03:14.840 --> 03:19.840 So, I'm kind of a shim and proud at the same time. 03:19.840 --> 03:27.840 By 916 already, also end of life, there's a new networking layer for, but only for the DNS server 03:27.840 --> 03:28.840 part. 03:28.840 --> 03:32.840 So, you know, the machine is half steam, half rocket. 03:32.840 --> 03:38.840 And the bind 918, which will be end of life like next year. 03:38.840 --> 03:44.840 So, it all uses the new networking layer for even the client part and the server part, 03:44.840 --> 03:47.840 but some of the parts are still the steam machine. 03:47.840 --> 03:54.840 It also, like, we ripped out the home cook allocator, which was really good at the time. 03:54.840 --> 04:00.840 It was written, but nowadays, we would, like, as a DNS server, we should focus on DNS and 04:00.840 --> 04:01.840 not brightening allocators. 04:01.840 --> 04:08.840 So, we are relying on G, uh, malloc, uh, for, uh, like handling the memory fragmentation and 04:08.840 --> 04:10.840 stuff, and it's really good. 04:11.840 --> 04:17.840 You can copy it even without it, because there's, like, shim just to work with the standard, uh, 04:17.840 --> 04:24.840 lib CL skater about, uh, the JMLoc is really, really much better option for using, uh, with bind. 04:24.840 --> 04:29.840 So, uh, now, do you do, like, that more complicated part. 04:29.840 --> 04:35.840 So, what makes part of programming hard is actually a name of the book by McKenney. 04:36.840 --> 04:42.840 Uh, and this is really good book on, like, understanding the architecture of the more 04:42.840 --> 04:43.840 computers. 04:43.840 --> 04:44.840 It's open source. 04:44.840 --> 04:50.840 Uh, so, term of foretling, memory barriers, atomic operations, pipeline flashes, 04:50.840 --> 04:56.840 cash misses, IO completion, memory, uh, sort of memory references. 04:56.840 --> 05:02.840 It all makes, like, the CPU runs slower, because it needs to throw away the cash, 05:02.840 --> 05:09.840 and, you know, the cash is really close to the CPU, but when it's not, then it's much, 05:09.840 --> 05:14.840 you know, the operations are much, become much slower and slower and slower. 05:14.840 --> 05:21.840 So, every time you need to stop the CPU needs to stop, like, synchronizing memory 05:21.840 --> 05:27.840 between each other, uh, and every time it needs to, like, fetch a memory that that's really, 05:27.840 --> 05:33.840 or forwarded cash, everything makes the program slower, even though it's not visible on, 05:33.840 --> 05:37.840 like, in a C or even a higher language is. 05:37.840 --> 05:43.840 Uh, so that's, uh, for, for people, not, not programming close to hardware. 05:43.840 --> 05:46.840 This is really, like, complicated stuff. 05:46.840 --> 05:52.840 Uh, like, convinced me that it is not the case, but usually people, like, writing in Java 05:52.840 --> 05:56.840 or, you know, the frameworks like Flutter, they don't really understand, 05:56.840 --> 06:01.840 don't know if it, because they just, like, this is all hidden by the frameworks. 06:01.840 --> 06:09.840 And, uh, I think it's, uh, it's a big shame that, like, many, sort of engineers don't understand 06:09.840 --> 06:14.840 the hardware, they're everything, this program's they've, uh, on, because it, it's cost us, 06:14.840 --> 06:20.840 it cost money, electricity, you know, performance, everything. 06:20.840 --> 06:32.840 Uh, yeah, memory reclamation is also, like, the, the huge part we can talk about, 06:32.840 --> 06:35.840 many days about memory reclamation techniques. 06:35.840 --> 06:42.840 So, uh, by nine secretion, uh, you know, uh, still post six locks for the music. 06:42.840 --> 06:48.840 Uh, there's a new RW lock, uh, it's not one to one implementation, 06:48.840 --> 06:52.840 because that requires a lot of memory for the new mom, like, machines. 06:52.840 --> 06:58.840 So, it's, let's little is simplified, but it's, you know, it gives an improve, 06:58.840 --> 07:00.840 improvement in the, in the performance. 07:00.840 --> 07:03.840 I'll show you, uh, flight and it uses thermodynamics. 07:03.840 --> 07:06.840 So, that, that's what's required right now. 07:06.840 --> 07:09.840 Um, so, of course, people using, you know, right, 07:09.840 --> 07:12.840 had seven hours, uh, sent us seven. 07:12.840 --> 07:15.840 That's already been, and of life are now complaining. 07:16.840 --> 07:19.840 So, the effect of the custom RW lock. 07:19.840 --> 07:24.840 Uh, so this is the main branch, and this is the, you know, 07:24.840 --> 07:28.840 I created a branch just to switch to the default, uh, preferred RW lock. 07:28.840 --> 07:32.840 So, this is, uh, so I should explain this graph. 07:32.840 --> 07:35.840 This is logarithmic, this is reverse, logarithmic graph, 07:35.840 --> 07:38.840 actually invented by the powerliness people here. 07:38.840 --> 07:43.840 So, this basically means that, uh, if you look at this line, 07:43.840 --> 07:46.840 this is, uh, response time in, uh, milliseconds, 07:46.840 --> 07:48.840 and this is, uh, slowest percentile. 07:48.840 --> 07:53.840 So, this basically means that something like 95 percent, uh, 07:53.840 --> 07:57.840 uh, of queries were, uh, answered under one millisecond. 07:57.840 --> 08:00.840 So, this looks huge, but it's actually, 08:00.840 --> 08:02.840 the smallest part of all of it. 08:02.840 --> 08:07.840 So, uh, so you won't, basically, this is, to be, 08:07.840 --> 08:10.840 as small as possible, but as you can see, 08:10.840 --> 08:13.840 this is, like, slowest percentile is, like, five, 08:13.840 --> 08:16.840 fifth, slowest percentile goes, like here, 08:16.840 --> 08:20.840 and something like one and a half percentile of the queries 08:20.840 --> 08:24.840 are not answered ever, uh, which could be for various reasons. 08:24.840 --> 08:28.840 So, this is just the effect of the RW lock. 08:28.840 --> 08:31.840 Uh, this is, uh, this is the, this is the call cache, 08:31.840 --> 08:34.840 this is the hold cache, the, the second picture. 08:34.840 --> 08:38.840 Uh, so not much, much, much is the effect for the, for the hold cache, 08:38.840 --> 08:41.840 but for the case, it's quite significant. 08:41.840 --> 08:44.840 Um, so, and I'd like, like, complicated, 08:44.840 --> 08:46.840 who knows what the read copy update is. 08:46.840 --> 08:48.840 Okay, not bad. 08:48.840 --> 08:52.840 So, uh, there's a really cool series, uh, from the altars, 08:52.840 --> 08:56.840 uh, and there's a talk tomorrow in the kernel, uh, 08:56.840 --> 08:58.840 uh, from one of the altars. 08:58.840 --> 09:02.840 Uh, so basically, if you look at the RW lock, 09:02.840 --> 09:06.840 uh, it, it stops everything when you are, you know, 09:06.840 --> 09:10.840 for, so readers can, can work together, 09:10.840 --> 09:13.840 but if there's a writer, it stops all the threats, 09:13.840 --> 09:15.840 and so even the readers need to wait, 09:15.840 --> 09:18.840 and the writer will do the work, and then they will continue. 09:18.840 --> 09:20.840 So, it basically stops everything. 09:20.840 --> 09:22.840 So, for the RCU, uh, 09:22.840 --> 09:26.840 the mechanism is that the readers can, you know, 09:26.840 --> 09:29.840 continue while this changes, 09:29.840 --> 09:31.840 and so you start a change, 09:31.840 --> 09:34.840 basically here, like, the pictures removal. 09:35.840 --> 09:39.840 So, you start a change, and then you wait until, uh, 09:39.840 --> 09:41.840 the last reader that started before, 09:41.840 --> 09:44.840 you started a change, has finished, 09:44.840 --> 09:47.840 and then, uh, you can, uh, free the memory. 09:47.840 --> 09:50.840 So, this, this is a memory reclamation mechanism. 09:50.840 --> 09:53.840 Uh, and again, you can see it, this is from the, 09:53.840 --> 09:56.840 the, uh, the, uh, all WN article, uh, 09:56.840 --> 09:59.840 that it takes, uh, like, 09:59.840 --> 10:02.840 uh, it's much faster that way because it doesn't, 10:03.840 --> 10:05.840 doesn't make the other, for it to stop. 10:05.840 --> 10:08.840 So, this is one of the mechanisms that we are using. 10:08.840 --> 10:11.840 We are, right now we are using just the RCU API. 10:11.840 --> 10:13.840 This is right, the readcopy updates, 10:13.840 --> 10:16.840 and they, they provide data structures, 10:16.840 --> 10:19.840 our hash tables that are already, like, 10:19.840 --> 10:21.840 lock free or, or wait free. 10:21.840 --> 10:26.840 Um, so, um, 10:26.840 --> 10:27.840 by nine. 10:27.840 --> 10:31.840 Hi. 10:31.840 --> 10:35.840 Uh, thank you. 10:35.840 --> 10:39.840 Uh, so, uh, by nine, twenty, uh, 10:39.840 --> 10:41.840 it uses USB-RCU. 10:41.840 --> 10:44.840 We replace the locks that are WX in couple places, 10:44.840 --> 10:46.840 like the logging subsystem configuration. 10:46.840 --> 10:48.840 So, if you change something in the logging, 10:48.840 --> 10:50.840 before there was a locking, 10:50.840 --> 10:51.840 now it's just like, 10:51.840 --> 10:52.840 swipe the pointer, 10:52.840 --> 10:55.840 and when the grace period is over, 10:55.840 --> 10:57.840 it just throws away the old configuration, 10:57.840 --> 10:59.840 and there's no locking whatsoever. 11:00.840 --> 11:02.840 It's not a big change, but it's like, 11:02.840 --> 11:04.840 we are learning it. 11:04.840 --> 11:07.840 Also, it's not so easy to, uh, 11:07.840 --> 11:09.840 to switch the, like, the mindset, 11:09.840 --> 11:11.840 how to do this, you know, 11:11.840 --> 11:13.840 do the things in a new way. 11:13.840 --> 11:16.840 Also, we had a, like, weak references. 11:16.840 --> 11:20.840 So, the, the stronger references that, uh, 11:20.840 --> 11:27.840 do, do, do, do, do, 11:27.840 --> 11:29.840 do, just close the door, please. 11:29.840 --> 11:31.840 It would come in or out. 11:31.840 --> 11:36.840 So, the stronger references that you know, 11:36.840 --> 11:37.840 it's there. 11:37.840 --> 11:39.840 So, if you, if you, if you, 11:39.840 --> 11:41.840 do reference the pointer, you know, 11:41.840 --> 11:43.840 that you have a value or not. 11:43.840 --> 11:44.840 For the weak one, 11:44.840 --> 11:45.840 it there could be, uh, 11:45.840 --> 11:47.840 it could be now, uh, 11:47.840 --> 11:48.840 or there could be objects, 11:48.840 --> 11:50.840 so we, you know, use, 11:50.840 --> 11:52.840 RCU for the weak references now. 11:52.840 --> 11:54.840 And, uh, 11:54.840 --> 11:55.840 there's some, 11:56.840 --> 11:59.840 smaller places where you reuse, uh, 11:59.840 --> 12:01.840 that the lock free rate free data structures, 12:01.840 --> 12:02.840 like, 12:02.840 --> 12:04.840 something like called, 12:04.840 --> 12:06.840 callbackstable that's used for catalogue zones. 12:06.840 --> 12:07.840 And, and RPC, 12:07.840 --> 12:08.840 it needs, 12:08.840 --> 12:10.840 well, when you basically load, 12:10.840 --> 12:12.840 a new version of this catalogue zone, 12:12.840 --> 12:14.840 it needs to call back something and call, 12:14.840 --> 12:15.840 do something to, 12:15.840 --> 12:16.840 we use, 12:16.840 --> 12:17.840 in the small, 12:17.840 --> 12:18.840 small places, 12:18.840 --> 12:20.840 at the DNS dispatch, 12:20.840 --> 12:22.840 you use the lock free, 12:22.840 --> 12:23.840 uh, 12:23.840 --> 12:24.840 hash table, 12:24.840 --> 12:26.840 and the bedcash, 12:26.840 --> 12:27.840 uh, 12:27.840 --> 12:29.840 so this is the place where you be stored, 12:29.840 --> 12:31.840 that the server is unreachable, 12:31.840 --> 12:32.840 for the name, 12:32.840 --> 12:33.840 and stuff like this. 12:33.840 --> 12:36.840 So, for the bedcash API, 12:36.840 --> 12:37.840 formally, 12:37.840 --> 12:38.840 it was a, like, 12:38.840 --> 12:39.840 bucketed lock list, 12:39.840 --> 12:40.840 uh, 12:40.840 --> 12:42.840 where it was linear list, 12:42.840 --> 12:45.840 with the DNS hash names, 12:45.840 --> 12:46.840 as in index, 12:46.840 --> 12:48.840 and the list that we use are, 12:48.840 --> 12:49.840 are RU, 12:49.840 --> 12:52.840 the list recently used list, 12:52.840 --> 12:53.840 for cleaning, 12:53.840 --> 12:54.840 uh, 12:54.840 --> 12:55.840 right now, 12:55.840 --> 12:56.840 it's completely lock free, 12:56.840 --> 12:57.840 uh, 12:57.840 --> 12:58.840 there's a single hash table, 12:58.840 --> 12:59.840 and, 12:59.840 --> 13:00.840 paraphrase, 13:00.840 --> 13:02.840 there's a parap, 13:02.840 --> 13:03.840 paraphrase, 13:03.840 --> 13:04.840 are you list, 13:04.840 --> 13:05.840 um, 13:05.840 --> 13:07.840 so you put it in the right place, 13:07.840 --> 13:08.840 so you don't have to, 13:08.840 --> 13:10.840 there's no locking whatsoever, 13:10.840 --> 13:11.840 um, 13:11.840 --> 13:12.840 originally, 13:12.840 --> 13:13.840 I just, 13:13.840 --> 13:14.840 like, 13:14.840 --> 13:15.840 fixed this in the recent release, 13:15.840 --> 13:16.840 originally, 13:16.840 --> 13:17.840 I did it in a wrong way, 13:17.840 --> 13:18.840 that's, 13:18.840 --> 13:19.840 I'm saying, 13:19.840 --> 13:20.840 it might be somebody, 13:20.840 --> 13:21.840 something harder to do that. 13:21.840 --> 13:22.840 So there was, like, 13:22.840 --> 13:24.840 opportunistic cleaning of the table, 13:24.840 --> 13:25.840 which, 13:25.840 --> 13:27.840 basically meant the memory just blew up. 13:27.840 --> 13:29.840 So this is now fixed, 13:29.840 --> 13:30.840 uh, 13:30.840 --> 13:31.840 by using the paraphrase, 13:31.840 --> 13:32.840 are you list, 13:32.840 --> 13:33.840 uh, 13:33.840 --> 13:34.840 and, uh, 13:34.840 --> 13:35.840 a proper cleaning of the, 13:35.840 --> 13:36.840 uh, 13:36.840 --> 13:37.840 of the table. 13:37.840 --> 13:38.840 Um, 13:38.840 --> 13:40.840 so for 920, 13:40.840 --> 13:41.840 uh, 13:41.840 --> 13:42.840 so again, 13:42.840 --> 13:43.840 the, 13:43.840 --> 13:44.840 918, 13:44.840 --> 13:45.840 uh, 13:45.840 --> 13:46.840 and before, 13:46.840 --> 13:47.840 it has the, 13:47.840 --> 13:48.840 uh, 13:48.840 --> 13:50.840 the all cooperative scheduling, 13:50.840 --> 13:51.840 so, 13:51.840 --> 13:52.840 and, 13:52.840 --> 13:53.840 sometime with, 13:53.840 --> 13:54.840 sometime with, 13:54.840 --> 13:57.840 between 916 and 918 and 920, 13:57.840 --> 13:58.840 it was, like, 13:58.840 --> 13:59.840 a kind of hybrids. 13:59.840 --> 14:00.840 This is one on lip UV, 14:00.840 --> 14:02.840 that we use for networking. 14:02.840 --> 14:04.840 So lip UV has also haven't loop, 14:04.840 --> 14:07.840 so we just replace the old one, 14:07.840 --> 14:09.840 to run on top of this, 14:09.840 --> 14:12.840 and 920 just ripped all of the old code, 14:12.840 --> 14:13.840 uh, 14:13.840 --> 14:14.840 uh, 14:14.840 --> 14:15.840 out, and, uh, 14:15.840 --> 14:18.840 so there are just lip UV haven't loops for each fret, 14:18.840 --> 14:20.840 and stuff runs on top of it. 14:20.840 --> 14:21.840 It's a simple, 14:21.840 --> 14:23.840 but more powerful, 14:23.840 --> 14:24.840 uh, 14:24.840 --> 14:25.840 simpler, 14:25.840 --> 14:27.840 but sometimes more complicated, 14:27.840 --> 14:28.840 but it's, 14:28.840 --> 14:29.840 uh, 14:29.840 --> 14:30.840 because it, 14:30.840 --> 14:31.840 it allows more stuff. 14:31.840 --> 14:32.840 So, 14:32.840 --> 14:33.840 um, 14:33.840 --> 14:36.840 the easiest API is I see job. 14:36.840 --> 14:37.840 There's no, 14:37.840 --> 14:38.840 no looking. 14:38.840 --> 14:39.840 It's on, 14:39.840 --> 14:40.840 on loop, 14:40.840 --> 14:41.840 very fast. 14:41.840 --> 14:42.840 It does no allocations, 14:42.840 --> 14:43.840 because, 14:43.840 --> 14:44.840 uh, 14:44.840 --> 14:45.840 like the, 14:45.840 --> 14:47.840 the part of the memory 14:47.840 --> 14:49.840 that's needed to run the job, 14:49.840 --> 14:50.840 has to be on the, 14:50.840 --> 14:51.840 in the structure, 14:51.840 --> 14:52.840 uh, already. 14:52.840 --> 14:53.840 Uh, 14:53.840 --> 14:54.840 and it uses the, 14:54.840 --> 14:55.840 there's a, 14:55.840 --> 14:57.840 like main fret pool of the, 14:57.840 --> 14:58.840 event pools, 14:58.840 --> 14:59.840 uh, 14:59.840 --> 15:00.840 and it uses the, 15:00.840 --> 15:01.840 this one, 15:01.840 --> 15:02.840 the shared one. 15:02.840 --> 15:03.840 So then there's, 15:03.840 --> 15:04.840 I see a thing, 15:04.840 --> 15:05.840 uh, 15:05.840 --> 15:06.840 this similar, 15:06.840 --> 15:07.840 it uses the weight free, 15:07.840 --> 15:08.840 uh, 15:08.840 --> 15:09.840 uh, 15:09.840 --> 15:10.840 but it's, 15:10.840 --> 15:11.840 uh, 15:11.840 --> 15:13.840 but it's, 15:13.840 --> 15:14.840 uh, 15:14.840 --> 15:16.840 but it's, 15:16.840 --> 15:17.840 you see the, 15:17.840 --> 15:18.840 the very free queue, 15:18.840 --> 15:19.840 uh, 15:19.840 --> 15:21.840 um, 15:21.840 --> 15:23.840 so it's for, 15:23.840 --> 15:25.840 also for fast operations, 15:25.840 --> 15:26.840 um, 15:26.840 --> 15:30.840 and it uses the same fret pool as the I see jobs. 15:30.840 --> 15:31.840 Then we had to, 15:31.840 --> 15:32.840 there was a, 15:32.840 --> 15:34.840 uh, 15:34.840 --> 15:36.840 vulnerability in the NSEC, 15:36.840 --> 15:37.840 uh, 15:37.840 --> 15:39.840 as in the protocol, 15:39.840 --> 15:40.840 uh, 15:40.840 --> 15:41.840 where if you just, 15:41.840 --> 15:43.840 like hit the server with, 15:43.840 --> 15:44.840 like huge, 15:44.840 --> 15:46.840 the NSEC operations that would just, 15:46.840 --> 15:48.840 like slow down everything. 15:48.840 --> 15:49.840 So we had to introduce a, 15:49.840 --> 15:51.840 yet another fret pool. 15:51.840 --> 15:52.840 So, 15:52.840 --> 15:54.840 and this is only used for, 15:54.840 --> 15:55.840 uh, 15:55.840 --> 15:56.840 slow, 15:56.840 --> 15:57.840 crypto operation. 15:57.840 --> 15:58.840 So basically, 15:58.840 --> 15:59.840 when there's a, 15:59.840 --> 16:00.840 the NSEC validation, 16:00.840 --> 16:02.840 we receive it on, 16:02.840 --> 16:03.840 on this, 16:03.840 --> 16:04.840 on this shared, 16:04.840 --> 16:05.840 for pool, 16:05.840 --> 16:07.840 and then passed the NSEC, 16:07.840 --> 16:09.840 operation to this different one. 16:09.840 --> 16:10.840 So, 16:10.840 --> 16:11.840 this one can still, 16:11.840 --> 16:12.840 uh, 16:12.840 --> 16:14.840 dispatch everything else that's, 16:14.840 --> 16:16.840 already being in the cache, 16:16.840 --> 16:17.840 or, 16:17.840 --> 16:18.840 is alternative data. 16:18.840 --> 16:20.840 So the NSEC operations, 16:20.840 --> 16:22.840 doesn't slow down the, 16:22.840 --> 16:24.840 the rest of the operations. 16:24.840 --> 16:26.840 So that's why there's so many of them. 16:26.840 --> 16:27.840 And there's I see work, 16:27.840 --> 16:28.840 this is using the, 16:28.840 --> 16:29.840 lip UV, 16:29.840 --> 16:30.840 uh, 16:30.840 --> 16:31.840 uh, 16:31.840 --> 16:32.840 uh, 16:32.840 --> 16:33.840 for pool, 16:33.840 --> 16:34.840 and this is for really, 16:34.840 --> 16:35.840 very slow stuff, 16:35.840 --> 16:37.840 like writing the zone on a disk. 16:37.840 --> 16:38.840 So we just pass it there, 16:38.840 --> 16:40.840 and just passes back when it's done, 16:40.840 --> 16:41.840 and we don't care. 16:41.840 --> 16:42.840 So, 16:42.840 --> 16:43.840 um, 16:45.840 --> 16:46.840 this uses, 16:46.840 --> 16:47.840 the whole thing, 16:47.840 --> 16:48.840 the whole new thing, 16:48.840 --> 16:50.840 uses the fact that, 16:50.840 --> 16:51.840 uh, 16:51.840 --> 16:54.840 kernel is much better at scheduling than we are. 16:54.840 --> 16:55.840 So, 16:55.840 --> 16:57.840 this cooperative scheduling, 16:57.840 --> 16:58.840 uh, 16:58.840 --> 17:00.840 was probably okay in, 17:00.840 --> 17:01.840 you know, 17:01.840 --> 17:03.840 the year that the bind started, 17:03.840 --> 17:04.840 uh, 17:04.840 --> 17:05.840 but right now, 17:05.840 --> 17:07.840 we basically removed everything 17:07.840 --> 17:09.840 that had to do with the scheduling, 17:09.840 --> 17:11.840 like assigning threats to, 17:11.840 --> 17:12.840 uh, 17:12.840 --> 17:13.840 to CPU cores. 17:13.840 --> 17:14.840 Uh, 17:14.840 --> 17:16.840 and so we are not trying to be smarter than, 17:16.840 --> 17:17.840 than the kernel, 17:17.840 --> 17:19.840 because the kernel is smarter than us. 17:19.840 --> 17:20.840 Uh, 17:20.840 --> 17:21.840 um, 17:21.840 --> 17:25.840 the other big change we did in the, 17:25.840 --> 17:26.840 in the 1920, 17:26.840 --> 17:28.840 and there's this working progress. 17:28.840 --> 17:31.840 We have a new QP3 database. 17:31.840 --> 17:32.840 Uh, 17:32.840 --> 17:35.840 this replaced the RBT database, 17:35.840 --> 17:36.840 which is the, 17:36.840 --> 17:38.840 the RBT is a bit of misleading, 17:38.840 --> 17:40.840 because it's more like red black forest, 17:40.840 --> 17:41.840 because it's, 17:41.840 --> 17:42.840 uh, 17:42.840 --> 17:47.840 RBT3 of the RBT3 is because of how the DNS structure works. 17:47.840 --> 17:48.840 Um, 17:48.840 --> 17:49.840 and, 17:49.840 --> 17:51.840 is it key value storage suitable for DNS, 17:51.840 --> 17:53.840 it uses RCU for reads, 17:53.840 --> 17:56.840 it blocks forcing donations of rights. 17:56.840 --> 17:57.840 Uh, 17:57.840 --> 17:59.840 but that's had, 17:59.840 --> 18:00.840 it still has, 18:00.840 --> 18:01.840 uh, 18:01.840 --> 18:02.840 uh, 18:02.840 --> 18:04.840 so the underlying structure is much better, 18:04.840 --> 18:07.840 but we just like bolted it to the old, 18:07.840 --> 18:08.840 old one, 18:08.840 --> 18:11.840 so it still uses the RWOX for storing DNS data, 18:11.840 --> 18:13.840 and this is bucketed and stuff. 18:13.840 --> 18:16.840 Um, 18:16.840 --> 18:18.840 is very technical detail, 18:18.840 --> 18:19.840 so, 18:19.840 --> 18:20.840 uh, 18:20.840 --> 18:21.840 it's okay. 18:21.840 --> 18:22.840 But I want you to show you, 18:22.840 --> 18:23.840 uh, 18:23.840 --> 18:24.840 the difference. 18:24.840 --> 18:27.840 Uh, 18:27.840 --> 18:29.840 so this is the UDP response latency. 18:29.840 --> 18:30.840 Uh, 18:30.840 --> 18:32.840 the blue one is the 1916, 18:32.840 --> 18:34.840 and I can't run the 1916 anymore, 18:34.840 --> 18:36.840 even in our, 18:36.840 --> 18:37.840 the 1918, 18:37.840 --> 18:38.840 and the, 18:38.840 --> 18:39.840 uh, 18:39.840 --> 18:40.840 uh, 18:40.840 --> 18:41.840 green one is 1920. 18:41.840 --> 18:42.840 So I would say that's, 18:42.840 --> 18:43.840 uh, 18:43.840 --> 18:45.840 nice improvement in the response latency. 18:45.840 --> 18:46.840 For the memory usage, 18:46.840 --> 18:47.840 uh, 18:47.840 --> 18:48.840 the same colors. 18:48.840 --> 18:49.840 Um, 18:49.840 --> 18:50.840 but, 18:50.840 --> 18:51.840 uh, 18:51.840 --> 18:52.840 this is, 18:52.840 --> 18:53.840 older picture, 18:53.840 --> 18:54.840 I just, like, 18:54.840 --> 18:55.840 this week fixed, 18:55.840 --> 18:56.840 uh, 18:56.840 --> 18:57.840 uh, 18:57.840 --> 18:58.840 or made up optimization that the, 18:58.840 --> 19:00.840 the 1918 memory is much lower now. 19:00.840 --> 19:01.840 Um, 19:01.840 --> 19:02.840 so, 19:02.840 --> 19:04.840 how do we measure the hotspots? 19:04.840 --> 19:06.840 There's a Detroit system tape, 19:06.840 --> 19:07.840 a system tap. 19:07.840 --> 19:08.840 So, 19:08.840 --> 19:09.840 basically you can measure, 19:09.840 --> 19:10.840 uh, 19:10.840 --> 19:12.840 when you start locking, 19:12.840 --> 19:13.840 when you, 19:13.840 --> 19:14.840 like, 19:14.840 --> 19:15.840 actual acquired lock, 19:15.840 --> 19:16.840 and when you release lock. 19:16.840 --> 19:17.840 So, 19:17.840 --> 19:18.840 uh, 19:18.840 --> 19:19.840 we have kind of, 19:19.840 --> 19:20.840 our own detrace, 19:20.840 --> 19:21.840 uh, 19:21.840 --> 19:22.840 uh, 19:22.840 --> 19:23.840 uh, 19:23.840 --> 19:24.840 the script looks like, 19:24.840 --> 19:25.840 something like this, 19:25.840 --> 19:26.840 that you, 19:26.840 --> 19:27.840 define, 19:27.840 --> 19:28.840 uh, 19:28.840 --> 19:29.840 basically, 19:29.840 --> 19:30.840 these are tables, 19:30.840 --> 19:31.840 uh, 19:31.840 --> 19:32.840 inside the kernel, 19:32.840 --> 19:33.840 and then you measure, 19:33.840 --> 19:34.840 and you have a mutex, 19:34.840 --> 19:35.840 and free mutex, 19:35.840 --> 19:36.840 acquired, 19:36.840 --> 19:37.840 and it takes release, 19:37.840 --> 19:39.840 and the output could look like something like this, 19:39.840 --> 19:41.840 and you have a, 19:41.840 --> 19:42.840 how, uh, 19:42.840 --> 19:44.840 how often it was, 19:44.840 --> 19:45.840 uh, 19:45.840 --> 19:46.840 acquired, 19:46.840 --> 19:47.840 and how much time it you spend. 19:47.840 --> 19:48.840 This is, like, 19:48.840 --> 19:49.840 sorted in, 19:49.840 --> 19:50.840 uh, 19:50.840 --> 19:51.840 uh, in the table. 19:51.840 --> 19:52.840 So, these are the, 19:52.840 --> 19:53.840 out spots where, 19:53.840 --> 19:54.840 this is, 19:54.840 --> 19:55.840 again, 19:55.840 --> 19:56.840 a little bit older data, 19:56.840 --> 19:57.840 but, 19:57.840 --> 19:58.840 uh, 19:58.840 --> 19:59.840 so these are the, 19:59.840 --> 20:00.840 the spot there are more, 20:00.840 --> 20:02.840 like most heavy content in, 20:03.840 --> 20:04.840 in our code. 20:04.840 --> 20:07.840 So, for 921 and beyond, 20:07.840 --> 20:08.840 uh, 20:08.840 --> 20:09.840 we, again, 20:09.840 --> 20:10.840 we want to, 20:10.840 --> 20:11.840 so it was the best thing, 20:11.840 --> 20:12.840 memory synchronization, 20:12.840 --> 20:14.840 because I'm not synchronized at all. 20:14.840 --> 20:16.840 So, we want to keep all the data, 20:16.840 --> 20:17.840 local to threats, 20:17.840 --> 20:18.840 unless we have to. 20:18.840 --> 20:20.840 So, we will share only what, 20:20.840 --> 20:21.840 absolutely needs to be shared, 20:21.840 --> 20:22.840 like a cache that's, 20:22.840 --> 20:24.840 they make sense to share it. 20:24.840 --> 20:25.840 Uh, 20:25.840 --> 20:26.840 So, 20:26.840 --> 20:28.840 caller is in typical cases to RCU, 20:28.840 --> 20:29.840 uh, 20:29.840 --> 20:30.840 API. 20:30.840 --> 20:31.840 Uh, 20:31.840 --> 20:32.840 not everything can be converted. 20:32.840 --> 20:33.840 So, if you have a, 20:33.840 --> 20:35.840 some long lift objects that, 20:35.840 --> 20:36.840 uh, 20:36.840 --> 20:37.840 like, 20:37.840 --> 20:38.840 on one tick of the oven loop, 20:38.840 --> 20:39.840 you acquire it, 20:39.840 --> 20:40.840 and then it does something, 20:40.840 --> 20:41.840 and then you restore it, 20:41.840 --> 20:42.840 and then, 20:42.840 --> 20:43.840 basically, 20:43.840 --> 20:44.840 you can't use RCU, 20:44.840 --> 20:45.840 you need to use reference, 20:45.840 --> 20:46.840 counting, 20:46.840 --> 20:47.840 or something. 20:47.840 --> 20:48.840 Uh, 20:48.840 --> 20:49.840 and, 20:49.840 --> 20:50.840 uh, 20:50.840 --> 20:51.840 there's a, 20:51.840 --> 20:53.840 I'm going work to replace the configuration, 20:53.840 --> 20:54.840 uh, 20:54.840 --> 20:55.840 so it will also use, 20:55.840 --> 20:56.840 uh, 20:56.840 --> 20:57.840 key value storage, 20:57.840 --> 20:58.840 and some callbacks, 20:58.840 --> 21:00.840 and remove the global, 21:01.840 --> 21:03.840 some more stuff, 21:03.840 --> 21:04.840 but I, 21:04.840 --> 21:05.840 you have, 21:05.840 --> 21:06.840 some weird pictures. 21:06.840 --> 21:07.840 Uh, 21:07.840 --> 21:08.840 so this is, 21:08.840 --> 21:09.840 this is recent, 21:09.840 --> 21:10.840 so this is like, 21:10.840 --> 21:11.840 a very heavy, 21:11.840 --> 21:12.840 uh, 21:12.840 --> 21:13.840 QPS. 21:13.840 --> 21:14.840 So, 21:14.840 --> 21:15.840 by nine, 21:15.840 --> 21:16.840 18 versus by nine, 21:16.840 --> 21:17.840 20. 21:17.840 --> 21:18.840 Uh, 21:18.840 --> 21:19.840 so again, 21:19.840 --> 21:20.840 this is very, 21:20.840 --> 21:21.840 very heavy, 21:21.840 --> 21:22.840 uh, 21:22.840 --> 21:23.840 hit server. 21:23.840 --> 21:24.840 So, 21:24.840 --> 21:25.840 um, 21:25.840 --> 21:26.840 since like, 21:26.840 --> 21:27.840 300K for cold cache. 21:27.840 --> 21:28.840 Uh, 21:28.840 --> 21:29.840 but, 21:29.840 --> 21:30.840 I have, 21:30.840 --> 21:31.840 really, 21:31.840 --> 21:32.840 I have, 21:32.840 --> 21:33.840 like, 21:33.840 --> 21:34.840 this is my, 21:34.840 --> 21:35.840 what the fuck moment. 21:35.840 --> 21:36.840 So, 21:36.840 --> 21:38.840 these top lines are, 21:38.840 --> 21:39.840 like, 21:39.840 --> 21:40.840 one and two threats. 21:40.840 --> 21:41.840 And, 21:41.840 --> 21:42.840 the other one, 21:42.840 --> 21:44.840 our cleaning is much better, 21:44.840 --> 21:46.840 if you run it with more threats. 21:46.840 --> 21:47.840 I, 21:47.840 --> 21:48.840 don't know why. 21:48.840 --> 21:49.840 Uh, 21:49.840 --> 21:52.840 and I'm probably not going to debug this into much detail, 21:52.840 --> 21:53.840 because, you know, 21:53.840 --> 21:54.840 uh, 21:54.840 --> 21:55.840 this is the nine, 21:55.840 --> 21:56.840 twenty, 21:56.840 --> 21:57.840 and it looks much better. 21:57.840 --> 21:58.840 So, 21:58.840 --> 21:59.840 for the CPU, 21:59.840 --> 22:00.840 again, 22:00.840 --> 22:01.840 by nine, 22:01.840 --> 22:02.840 18, 22:02.840 --> 22:03.840 so it's like, 22:03.840 --> 22:05.840 one threat to threats for, 22:05.840 --> 22:06.840 eight, 22:06.840 --> 22:07.840 and 16. 22:07.840 --> 22:08.840 And, 22:08.840 --> 22:09.840 you know, 22:09.840 --> 22:10.840 if you compare this to nine, 22:10.840 --> 22:11.840 twenty, 22:11.840 --> 22:12.840 nine, 22:12.840 --> 22:13.840 twenty looks like, 22:13.840 --> 22:14.840 we, 22:14.840 --> 22:15.840 well, 22:15.840 --> 22:17.840 this could be just measurement artifact, 22:17.840 --> 22:18.840 who knows. 22:18.840 --> 22:19.840 But, 22:19.840 --> 22:20.840 if you look at the, 22:20.840 --> 22:21.840 so load for, 22:21.840 --> 22:23.840 eight threats is almost, 22:23.840 --> 22:24.840 almost, 22:24.840 --> 22:25.840 eight hundred percent, 22:25.840 --> 22:28.840 basically means that it used all the CPUs it can. 22:28.840 --> 22:29.840 Uh, 22:29.840 --> 22:34.840 So, now we need to focus how to reduce the work we actually do inside. 22:34.840 --> 22:35.840 That's, 22:35.840 --> 22:37.840 there's a only way for what. 22:37.840 --> 22:38.840 Thank you. 22:38.840 --> 22:39.840 Thank you. 22:39.840 --> 22:41.840 Thank you. 22:41.840 --> 22:42.840 Thank you.