WEBVTT 00:00.000 --> 00:26.360 Hello, welcome everybody. Thanks for coming to my talk about attacking ballfields 00:26.360 --> 00:33.280 with this workshop title that I'll explain in a second. My name is Vinci here. I'm an interesting 00:33.280 --> 00:37.280 developer. I've been on the business for more than 10 years and I think I've had fun 00:37.280 --> 00:45.880 stuff. I was a cool. I was a customer at the time and nothing else was there. I was so 00:45.880 --> 00:50.120 feel of easier if you were a developer. It sounds complicated. It's like, oh, we can do it 00:50.120 --> 00:58.280 here and so. It's about a second. So I'm not basically three topics. I want to explain 00:58.280 --> 01:01.160 what I'm going to explain in a second case. We all know I think most of you probably know 01:01.160 --> 01:07.800 what it is. I'll talk about this tool. I developed a cool tracker and it's history. It's 01:07.800 --> 01:15.800 the more than 10 years history now. I started in 2014 and how I exist from the cloud in a sense 01:15.800 --> 01:25.160 because in the beginning it was like cloud native serverless at the time. Now I'm muted. 01:25.160 --> 01:32.400 Nowadays it's a standalone application that runs on netbistiefoundation server. So package 01:32.400 --> 01:40.960 source is the netbistief package collection. It's not just for netbistief. So it runs, I put 01:40.960 --> 01:49.440 here this number. I was slightly surprised. It runs on 23 OSS, including netbistief. All the 01:49.440 --> 01:59.440 other BSDs, Linux, Mac OS, Illumos, etc. And all of these have multiple architectures. So you 01:59.440 --> 02:04.160 might be using it on netbistief and B64 and you think you're in the majority use case which 02:04.240 --> 02:11.200 is probably true. But maybe you have a pine go pro running a 64-bit arm chip and you can also 02:11.200 --> 02:18.320 use package stuff. We don't have binary packages for all these architectures. There's some 02:18.320 --> 02:27.680 sort of like core platforms that we provide packages for. But also we release quarterly. So the 02:27.760 --> 02:33.920 last release was called 2024 Q4 was released at the end of the year and it is the 85th quarterly 02:33.920 --> 02:40.720 release. There's some discussion that we might not have quarterly release in future but we'll 02:40.720 --> 02:49.600 see about that. I think we will. And the release consists of like branching off the CVS tree, yes, 02:49.600 --> 02:55.840 it's still CVS unfortunately. So you have a stable source of source I guess and you can build 02:55.840 --> 03:02.480 your packages from there. There's security updates during the for the latest branch which is nice. 03:03.680 --> 03:10.400 Now, package source does not have a CI-CD pipeline the way you would think about it. Like, 03:10.400 --> 03:14.320 I don't know, I don't know how free BSD probably has something like, I don't know, they seem 03:14.320 --> 03:21.440 way more professional in these matters than we are. Like, intuitively you would expect, you know, 03:21.520 --> 03:25.440 somebody doesn't commit updates of package, let's say, and then there's some sort of 03:26.400 --> 03:32.400 demon that builds the package and maybe it's dependency, see if everything still works. 03:33.280 --> 03:40.640 We don't have that. However, we have sort of, in a way, outsource or distributed the problem, 03:40.640 --> 03:47.280 which is that anyone including you can do bulk builds of either all of the 30,000 packages 03:47.360 --> 03:55.920 or just a part that interests you. I'm using this software called P bulk and publish your results. 03:56.720 --> 04:01.360 And then we can look at other people's build results. Like, I don't know, this guy builds 150 04:01.360 --> 04:07.600 packages on this exotic platform you can see what is broken and whatnot. I actually looked into 04:07.680 --> 04:18.480 porting Pudria from Fabius D, but it seems hard. Yeah. So P bulk is also specific to package 04:18.480 --> 04:30.080 stores. It consists of some C and some shell. Basically, it first does a scan to find out what 04:30.080 --> 04:36.560 depends on what. So you can do it in dependency order, obviously. Then it builds the packages or 04:36.720 --> 04:41.840 some subset of packages. And it sends the report, usually via email to this mailing 04:41.840 --> 04:49.760 this package source bulk. And also the report contains a link to a file called the machine readable 04:49.760 --> 04:56.320 report that you usually also upload somewhere. And the machine read the report is a lot longer 04:56.320 --> 05:01.440 and it has all the individual results. It's not very easy to read as a human, but it's useful for 05:01.520 --> 05:07.360 software and we'll get to that. So that's what the email looks like. You see you have 05:08.400 --> 05:15.520 machine readable version here. This is an official net-based build. You see out of the 29,354 05:15.520 --> 05:21.520 packages, 143 failed to build. And then there's this list here which is sorted by the number of 05:22.160 --> 05:28.640 packages broken by this breakage. So like, if you're, I don't know, say your Python build is broken 05:28.640 --> 05:32.320 and there's a couple thousand packages that can't build, which show up at the top here. So that's 05:32.320 --> 05:39.120 an important signal. And this is really, really important, especially when you're preparing 05:39.120 --> 05:43.200 this quarterly release because there's a freeze. You want to treat to stabilize. You want to see 05:43.200 --> 05:50.880 that at least the platforms you care about have no breakage or nothing important is broken in 05:50.880 --> 05:56.480 the sense. But because these are hard to read, I started building this web app, so it's called 05:56.480 --> 06:04.320 pull tracker. And what it does is it actually subscribes to the mailing list. Any mail that contains 06:04.320 --> 06:09.520 the report is parsed. I put it to a database, so you already have a sort of metadata, you know, 06:09.520 --> 06:14.880 the counts and whatnot. And you have the URL to the full file, it downloads the full file, 06:14.880 --> 06:21.840 parsed as well, and puts individual records for each package. And then you can query that database. 06:22.480 --> 06:28.160 So, so this is the current address where you can see and the relange side, let me see 06:28.160 --> 06:34.800 release engineering, this is the source, my GitHub. This is what it looks like. This is the start, 06:34.800 --> 06:42.640 so you can either start by selecting a platform and looking at the latest build or you can start 06:42.640 --> 06:47.520 by, you know, going by a category to a package that interests you and see the latest results for that 06:47.600 --> 06:53.440 package. Maybe I can actually do this little live demo. Let's see if I can do that. 06:58.160 --> 07:09.440 One second. So here's a page of a build. This one is arm 64 build on a stable branch. 07:10.160 --> 07:18.800 It has a little graph, like 26,000. Okay. And then I should explain the status as maybe. 07:18.800 --> 07:23.840 Prefailed means it didn't even attempt to build the package because it knows it's not going to work. 07:23.840 --> 07:30.800 For example, something that is not for this architecture or marked this broken. Then there's 07:30.800 --> 07:39.040 indirect prefailed, which is when one of the dependencies is prefailed. And there's failed, 07:39.040 --> 07:44.400 which means it's tried to build the package, but it failed. And there's indirect fail, which means 07:44.400 --> 07:49.680 one of the dependencies failed to build. So you can't even attempt to build. And again, you have the 07:49.680 --> 07:54.880 category list. And you have this thing here, which is relatively new. We have some what we call 07:54.880 --> 08:01.920 Sentinel meta packages. So meta package is a package that has nothing in it, just dependencies. 08:01.920 --> 08:08.640 And you can use that to make lists basically. And we have added a bunch of these, like bulk test. 08:09.440 --> 08:15.200 Let's do bulk test go for example. It's one of the most familiar with. So whenever you update 08:15.200 --> 08:21.120 the go compiler, you want to build the good bulk test go package and, you know, it's dependencies 08:21.200 --> 08:28.880 to see that stuff still works. And so the status of these Sentinel packages is a useful signal. 08:28.880 --> 08:34.080 You can see some of these are marked as indirect failed. And you can go directly click through to 08:34.080 --> 08:40.880 the things that failed at the time. So like, I don't know, you'll go to MongoDB. And then you see 08:41.840 --> 08:47.280 it MongoDB broke two others, both of them are the bulk test packages, because they're marked 08:47.280 --> 08:53.360 as depending on it. One annoying thing is you have these links to the actual reports, but you 08:53.360 --> 08:58.160 don't know which is the failing state. So I suppose it's probably failed in the build. So you can 08:58.160 --> 09:04.800 click through and you can scroll through the bottom very quickly. That's a lot of text. It was a bad 09:04.800 --> 09:10.960 example. I don't know. Let's not go there. Let's not go there. Anyway, so you can actually read the 09:10.960 --> 09:17.760 build reports. The logs, the logs, the failed pegs are also uploaded. And you can try to debug 09:17.760 --> 09:26.320 from there and maybe you know what the problem is right away. So that was the, there was a little 09:26.320 --> 09:32.560 tour of the UI. There's a bit more to it, but yeah, those were the most important things. 09:32.560 --> 09:44.560 Also, oh yeah, also on the build page, if we go back to the build, below this is the same 09:46.080 --> 09:52.000 list that we saw from the mail, like the package is breaking most other packages. 09:52.000 --> 09:57.840 You see PHP-56 failed to build and 151 other packages didn't work. And some of them also pre-failed. 09:58.800 --> 10:06.720 In this particular example, it's a bit weird because these are net-based, he has X11 from 10:06.720 --> 10:11.920 base or X11 from tech source. And if you set it from base, it will not try to build these, 10:11.920 --> 10:15.920 so they show up as pre-failed. But that's an artifact you can ignore those. 10:16.640 --> 10:33.600 All right, going back to my slides, going back to my slides. So this thing was started in 10:34.720 --> 10:43.840 2014, and this is where I'll go sort of maybe, I'll say some critical words about the cloud. 10:44.800 --> 10:50.880 So I started this as a serverless application using a platform called Google App Engine that was 10:50.880 --> 10:59.200 the style at the time, I guess. All the back end is written in Go, and Google App Engine had 10:59.200 --> 11:04.560 good support for Go and had plenty of APIs that are useful for this kind of thing. You can receive 11:04.560 --> 11:11.520 emails, you can download files, you have a database, not a SQL database though, but a document database, 11:12.480 --> 11:18.160 which were, okay, it's called cloud data store. You have memcache, you have this thing called delay 11:18.160 --> 11:26.800 function, which lets you schedule expensive processing out of the request. I used all of those things, 11:28.400 --> 11:35.680 and was nice, and I paid for it out of pocket, it steadily became more expensive because 11:35.840 --> 11:41.440 one major downside of the cloud data store is, it will not let you do any queries that are 11:41.440 --> 11:48.960 not backed by an index. So you have more and more indices, and an index means write amplification. 11:48.960 --> 11:56.160 So you write, say, you know, 30,000 records for a build, but in reality with all the indices, 11:56.160 --> 12:02.960 you end up writing maybe 100,000. That's very annoying. So you have, say, 300 kicks of data, 12:02.960 --> 12:08.800 but only maybe 50 kicks in there are actually like build data that is useful, the rest is overhead. 12:10.400 --> 12:16.720 But also, in the beginning, I was thinking, you know, who my user is, and I thought net 12:16.720 --> 12:23.600 BST folks maybe don't have a full featured browser, so I'm not going to use any JavaScript. Everything 12:23.600 --> 12:31.680 was rendered on the server. And in the end, I came back from that, you know, my own understanding of 12:31.680 --> 12:38.800 like web UI is probably evolved. It turns out, there are advantages to like having a JSON API 12:38.800 --> 12:45.200 and doing XHR requests, because one that your query doesn't block the rendering of the page, 12:45.200 --> 12:52.320 it's very nice, you can render the summary maybe, and for the full results, you can wait one or 12:52.320 --> 12:57.120 two seconds until the full results start to appear, and two is also very, very easy to do. 12:57.360 --> 13:04.880 Especially, we've got, okay, so this thing I talked about in 2015 at a package source 13:04.880 --> 13:11.680 con that probably not many of you were at, but so there's also an API, you could do, 13:11.680 --> 13:16.880 you could use it as a JSON API, although it's not documented as such, but it's there. 13:18.160 --> 13:22.720 And now the thing, the thing about the cloud, so Google App Engine, as these things like to do, 13:22.800 --> 13:29.200 Google App Engine continues to evolve. So over time, it became less attractive to host your 13:29.200 --> 13:35.680 application on Google App Engine, and I don't know if they even, if it even shows up in documentation 13:35.680 --> 13:41.360 today, because there's new hotnesses like cloud functions, cloud run whatnot. I would not 13:42.320 --> 13:48.720 recommend anyone use this. Let's put like this. And also, the data store is kind of annoying, 13:48.880 --> 13:53.360 not only because of the right-emplification, but also it's slow. However, they fix that, 13:53.360 --> 13:59.040 they actually use a completely different server software called the firebase data store that 13:59.040 --> 14:04.080 was, they came to Google through the firebase acquisition, and added a shim on top that made 14:04.080 --> 14:09.520 it behave like the old cloud data store. But that's very integrated, everyone's data on to it, 14:09.520 --> 14:17.760 and it's much faster. So, okay. And also, as I said, the cost was starting to rise, because as 14:18.640 --> 14:25.280 I was writing this, I added ingestion, but it did not add division, or at least not automated, 14:25.280 --> 14:31.520 so like the amount of data continues growing, and the cost continues increasing, and in the end, 14:31.520 --> 14:38.480 it was about $60, which I paid out of pocket. But then it continued to evolve further. And I got 14:38.480 --> 14:43.040 an email saying, you are on Google App Engine Classic, and that should be read slightly immediately. 14:43.040 --> 14:47.680 If your infrastructure provider tells you you're on the classic tier, that means they're about to 14:47.680 --> 14:54.560 delete what you're using. So, you should migrate to the latest and greatest App Engine 14:56.160 --> 15:01.840 flexible, or I don't know, like the new tier at the time. Oh, by the way, all those APIs that you're using, 15:03.120 --> 15:08.720 they're gone. Like, there was no API for incoming mail, there was no API for fetching files, 15:08.800 --> 15:14.000 there was no API for memcache, memcache, they say, we don't have anything above memcache, like, 15:15.280 --> 15:23.760 I don't know. So, and there's a direct sort of associated to it. Like, if you don't migrate to 15:23.760 --> 15:27.680 the new thing, you will not be able to deploy new versions after the next February or something, 15:27.680 --> 15:34.240 so I started rewriting, and then it's hit me. Why am I doing this? If I rewrite stuff anyway, 15:34.320 --> 15:42.640 I might as well not target a proprietary platform, but just don't stand alone. And that also, 15:44.080 --> 15:49.680 that also meant I could adopt a SQL database, which it turns out is much better suited for 15:49.680 --> 15:57.280 this kind of cross correlation query workloads. And because I was too lazy to configure 15:57.920 --> 16:05.840 PostgresQL, my first thought was Postgres is great. I started with SQLite, and it turns out 16:05.840 --> 16:13.360 SQLite is very good. So, it is not a toy. We still have, like, I don't know, a couple hundred 16:13.360 --> 16:18.240 gigabytes of data in there. It does not break a sweat. It's totally fine. The one thing you kind of 16:18.240 --> 16:24.320 give up is that you can't really serve, you can't really have replication, or only with extra 16:24.320 --> 16:30.400 work. So, basically, you're kind of limited to one instance running at a time, which is totally 16:30.400 --> 16:37.040 fine for us because there's not that many visitors. So, SQLite is actually pretty great for this, 16:40.160 --> 16:47.360 and it's also much faster than what we had before. And then, and then I thought, now that it 16:47.440 --> 16:53.040 can run on any OS, on any, you know, just like you just started on the command line, it runs 16:53.040 --> 16:58.480 open support. Can I get the net-based foundation to host it? On of hand net-based users. So, 16:58.480 --> 17:04.800 started that process. They asked, like, okay, but how much RAM does it need? How much space does 17:04.800 --> 17:10.400 it need? We made up some numbers, and they were like, yeah, sure, it can fit in a corner of that 17:10.480 --> 17:20.800 server over there. And so, that happened. And today, the old address, a book tracker.appspot.com, 17:20.800 --> 17:24.800 hosts a page that says, this instance was shut down, and here's the replacement link. So, that's 17:24.800 --> 17:32.160 that's the last deploy I did. I posted a static page that says, we're gone. I deleted the database 17:32.240 --> 17:41.600 and now we're native. Quick overview over the APIs that we used. My absolute favorite one is 17:41.600 --> 17:54.000 the incoming mail API. It's not forward. So, the app engine incoming in mail API, it'll transform 17:54.000 --> 18:01.360 an incoming mail into a post request to this slash a h slash mail endpoint. And it turns out you 18:01.360 --> 18:07.120 can do that in a single line and it dot forward with curl as your mda as your mail delivery agent. 18:07.920 --> 18:17.920 So, that's very nice. For writing the HTTP server, there's the built-in one and go. So, 18:17.920 --> 18:23.360 that's fine for SQL actually using ORM. If anyone knows what an ORM is an object, 18:24.320 --> 18:30.320 it's a very simple thing. You write your SQL queries in a file and it generates functions for it 18:30.320 --> 18:38.400 and types for the results and the inputs. So, that's nice. It's called SQL C. It's a little 18:38.400 --> 18:44.400 coaching. Instead of a mem cache, I just cache data in process. You know, I have a map. And it has 18:44.400 --> 18:55.840 the key to value and an expiry and the values are regularly removed. For chart, instead of the Google 18:55.840 --> 19:00.080 chart API, I use something called chart.js, which is a single JavaScript file that's self-contained. 19:00.080 --> 19:08.160 So, that's also nice. For the App Engine Log API, I use this thing called Log Rust, which 19:08.240 --> 19:16.960 is one of those fancy-go logging packages. It doesn't have those. It's fine. Any other 19:16.960 --> 19:22.800 putting would also do. So, what this logs is mainly debug logs. When there was an error connecting 19:22.800 --> 19:30.000 to the database or things like that, in practice, nobody really reads the logs. The request logs 19:30.080 --> 19:38.240 themselves are logged by the foundation in the Engine X reverse proxy, which sits in front of 19:38.240 --> 19:47.600 this. And one also bigger reflection I had to do is that they hosted at a sub-directory. So, 19:47.600 --> 19:53.520 slash pull trackers, I had to fix all the URLs to include the sub-directory because previously 19:53.600 --> 20:01.120 was running on slash. And for the files fetch API also, how do you fetch the files just 20:01.120 --> 20:08.960 making outgoing HTTP requests you've done? I actually develop both versions inside by site and 20:08.960 --> 20:14.800 different branches of the same GitHub repo. And eventually, I just declared the branch 20:14.800 --> 20:20.080 called SQL the default. And this is where development continues. And they all main branch is just 20:20.160 --> 20:28.320 dead. And we also didn't do any data migration, because the data comes in by itself. 20:28.320 --> 20:31.840 Like you start with an empty database, you let it run for two, three days, you have a couple builds 20:31.840 --> 20:41.200 already, and you'll find. Nobody is that interest in all data, most people want the newest data. 20:41.200 --> 20:48.160 Sometimes it's useful to go back to see like, at what version did this package stop building, 20:48.160 --> 20:53.920 maybe? Like, was it caused by something else? Or was it caused by the update that had to 20:53.920 --> 21:01.520 two months ago, if you have two months of data, then you can see it. That's nice. And yeah, 21:01.520 --> 21:11.360 and it's there now on the net-based server for release engineering. And I want to talk a little bit 21:11.360 --> 21:16.160 about the community involvement in this also. So, I'm a net-based developer, where this is 21:16.160 --> 21:21.600 developed outside of the tree. Obviously, it only runs like with package source stuff. So, 21:21.600 --> 21:29.680 there's no users that are not package source. And I'm also kind of still the only developer. 21:29.680 --> 21:36.400 However, what has changed, especially recently, like this, this becoming hosted on a net-based 21:36.400 --> 21:41.360 the order domain has actually given it a big push in like visibility and legitimacy. And all the 21:42.240 --> 21:48.640 release engineers and all the people care about stuff not being broken, they use this. And they 21:48.640 --> 21:53.360 tell me that it's useful for them. And they come back with requests, for example, the Sentinel package 21:53.360 --> 22:00.240 table that was a recent request about half a year ago. So, yes, that's kind of a one-man show, 22:00.240 --> 22:07.200 but it is kind of motivating because you get these testimonials back saying, hey, thanks for doing this. 22:07.280 --> 22:14.160 It's very useful. You know, if in your project, you have somebody who does something like that 22:14.160 --> 22:19.600 on their own, you should tell them that sometime. It's the probably one of the biggest motivators 22:19.600 --> 22:29.440 there is. One other thing also, I started doing releases. I hadn't realized how satisfying 22:29.440 --> 22:34.160 it is to not just deploy, you know, you can deploy anytime you want, but like to say, I'm going 22:34.160 --> 22:40.480 to deploy this thing now and I'm going to call it the 2025.02 release. And then write a little change 22:40.480 --> 22:45.600 lock saying, in this release, I fixed this bug and I fixed this bug and this new enhancement. 22:45.600 --> 22:51.600 And then you send it to like some mailing list and then you get things back like, you get some 22:51.600 --> 22:56.560 reactions back and so on. So, it's nice. It gives sort of a closure to having on some work. 22:56.960 --> 23:04.400 And in the future, there's still a lot of stuff for improvement. I noticed that some of the 23:04.400 --> 23:09.840 actually the oldest bugs that were there from 2014 are still open. That's because they're 23:10.800 --> 23:20.640 basically requests for redesigning the thing, so it's hard. There's some requirements that people 23:20.720 --> 23:28.080 wanted to see such as, we should follow the commits and highlight when a new package has not 23:28.080 --> 23:32.800 been built yet, for example, or be able to diff, like, what's has changed between the net 23:32.800 --> 23:40.480 beasty build, a week ago, and the net beasty build today. This would all be, I think following 23:40.480 --> 23:43.920 package source is kind of hard, but like the diff is kind of straightforward to implement just 23:43.920 --> 23:51.040 takes time, somebody has to do it, or search by maintainer. There's a related website that some 23:51.040 --> 23:56.400 of you might have used, it's called Repology. It's, it also, it follows different repositories, 23:56.400 --> 23:59.760 the packages, just one of them, it also follows free beasty ports and deviant and one not. 24:00.800 --> 24:08.800 And it'll tell you which repo has which version of which package, and Repology also allows 24:08.800 --> 24:13.200 searching by package maintainers. So, I can go into Repology type my name and see if the 24:13.280 --> 24:18.160 packages I maintain are up to date, or maybe there's an open security patch that they haven't 24:18.160 --> 24:27.520 applied. So, something like that would be useful. Also, if any of you, I don't know, open beasty 24:27.520 --> 24:33.520 folks, maybe, or, or, previously, I don't know, or some substrol, want to use something like this 24:33.520 --> 24:39.680 for your builds, you should come and talk to me, we can, we can do things like that for one 24:40.160 --> 24:45.040 to. And I'm just going to, I guess, be working on it. It's kind of low intensity, it's maybe, 24:45.040 --> 24:50.720 I don't know, few, a few hours of work every quarter, it's, it's fine. 24:52.560 --> 24:57.360 And that was kind of all I had. Thanks very much. The slides are online on my blog. 25:04.080 --> 25:06.640 And again, the URL is here. Are there any questions? 25:10.160 --> 25:15.680 Okay. Do you want to say a latest question on the latest, why can it's for the 25:17.040 --> 25:18.400 thing? So, can you repeat? 25:18.400 --> 25:22.480 Do you want to get about the latest release on the latest data? 25:22.480 --> 25:24.480 Yeah. What makes best of format release? 25:24.480 --> 25:32.720 Um, well, the question was, why do we care about all your data at all? Why not just 25:33.360 --> 25:40.960 call that release? You mean, you mean package source or the, uh, the, uh, the app? 25:40.960 --> 25:47.120 Uh, the app? Well, you don't, you don't actually need releases, per se. It's just a way of, um, 25:50.560 --> 25:56.000 it's just a way of announcing what's new every once in a while, uh, showing people that, you know, 25:56.000 --> 26:01.280 it's, it gets updated. And basically, I, I just tag a release every time I deploy to the server. 26:02.720 --> 26:06.320 There's also a little deploy pipeline, uh, that gets triggered by the release. 26:13.680 --> 26:14.720 Are there any other questions? 26:17.840 --> 26:25.920 How would you measure the, uh, the comparison? Complaints versus thank you for the, to the team. 26:27.040 --> 26:30.480 So, the question was about complaints versus thank you. I think 26:33.040 --> 26:42.000 I, uh, I think most people do kind of both in the same email. There's there. 26:44.000 --> 26:48.160 You know, they're saying things like, I really enjoy using this thing, but it would be great if it also did 26:48.960 --> 26:49.760 X. 26:53.760 --> 26:55.760 Yeah, most of the, most of the people are very nice. 27:01.280 --> 27:07.600 It's not a question, but I just want to, uh, make it, make it, what, when many says that 27:07.600 --> 27:13.200 the package source is more to platform means that for example, for your mark. 27:13.280 --> 27:14.560 Yes. 27:35.280 --> 27:42.640 Yes. Yes. So, so one of the, one person works full-time on package source, uh, is, uh, employed a 27:42.640 --> 27:48.240 giant and they, they produce this OS called smart OS, but they also provide, as you said, 27:48.240 --> 27:52.320 Mac OS, uh, binary packages, they're very good. So, if you have a market set of going with home 27:52.400 --> 27:55.920 rule, um, why not, you know, why not try package source very good. 28:03.760 --> 28:05.920 All right. Thanks again.