WEBVTT 00:00.000 --> 00:08.700 You're becoming, you know, it's been a long day and I know tomorrow is going to be another 00:08.700 --> 00:12.600 longer day, who's looking forward for the after parties today? 00:12.600 --> 00:13.600 Yes? 00:13.600 --> 00:14.600 Yes? 00:14.600 --> 00:15.600 Who's enjoying Brussels? 00:15.600 --> 00:16.600 Good? 00:16.600 --> 00:17.600 All right, I need to break the eyes because I don't know you. 00:17.600 --> 00:22.200 This is the first time I've paused them, so I want to get to know you. 00:22.200 --> 00:27.000 All right, so first of all, this is who I am, and I'm going to apologize for those 00:27.000 --> 00:29.000 in the recording, especially because I move a lot. 00:29.000 --> 00:33.800 And I like moving a lot, so it helps me deliver better, and I think so. 00:33.800 --> 00:37.600 I speak too fast, so let me know if I'm not pronouncing correctly. 00:37.600 --> 00:43.080 I'm Mexican, my primary language Spanish, so I have a hard accent. 00:43.080 --> 00:47.000 Let me know if I'm not pronouncing anything, or you don't understand what I'm saying. 00:47.000 --> 00:51.600 So I worked with the Linux Foundation, hence the Penguin. 00:51.600 --> 00:56.920 I am the general manager for Drunkot, Drunkot is a project within the Linux Foundation. 00:56.920 --> 01:00.120 We specialize in drone stuff. 01:00.120 --> 01:08.320 I am initially a contributor, so I'm not a business developer or a sales or marketing person. 01:08.320 --> 01:13.320 Although I had to be training that because the position required it, but I was initially 01:13.320 --> 01:17.320 an individual contributor, and I'm still I'm, and that's one of the reasons why I'm here 01:17.320 --> 01:21.720 because I actually need help, and that's kind of like the reason why I'm here to recruit 01:21.720 --> 01:23.720 some of you. 01:23.720 --> 01:27.720 I've been working for more than 10 years in robotics, some of my colleagues are here, 01:27.720 --> 01:32.720 so if I say hi, I call it the Roth Aerial Robotics Workgroup. 01:32.720 --> 01:38.720 Has anyone here been working with robotics at all before? 01:38.720 --> 01:41.720 Okay, like a quarter of you, nice. 01:41.720 --> 01:43.720 Anyone know about Ross? 01:43.720 --> 01:44.720 Good. 01:44.720 --> 01:50.400 All right, so there's an Aerial Robotics Ross community workgroup that I call it with Kimberly 01:50.400 --> 01:52.720 Maguire sitting right there. 01:52.720 --> 01:57.280 And then I also call it the Spaceware Linux Initiative that started last year in which we're 01:57.280 --> 02:02.560 trying to build a Linux distribution that is certifiable for space, and we're working with a few 02:02.560 --> 02:05.720 companies, NASA included, and that's a really cool effort. 02:05.720 --> 02:11.760 Hopefully next year at 1226, I get to talk about that, but today we're going to be talking 02:11.760 --> 02:13.760 about drones. 02:13.760 --> 02:16.160 All right, so what is drone code? 02:16.160 --> 02:26.200 So we're an open source UAV ecosystem, but honestly, when you hear about a foundation, 02:26.200 --> 02:29.600 like an open source foundation, what is the first thing that comes to mind? 02:29.600 --> 02:30.600 Like anyone? 02:30.600 --> 02:31.600 No, my. 02:31.600 --> 02:32.600 No, my. 02:32.600 --> 02:33.600 No, my. 02:33.600 --> 02:35.240 Yes, that's a very good one. 02:35.240 --> 02:39.840 It's like the funding to the project, right? 02:39.840 --> 02:42.600 But it's a bit more than that. 02:42.600 --> 02:47.400 And sometimes some of the foundations miss the opportunity to tell that story, and I'm 02:47.400 --> 02:51.840 not right to do a good job here, but this is like the second time I do it this time. 02:51.840 --> 02:55.800 So we're trying to be a neutral home for the open source projects. 02:55.800 --> 02:56.800 What does that mean? 02:56.800 --> 03:02.600 It means that any company within the ecosystem can contribute, even if they're competitors. 03:02.600 --> 03:05.240 And that causes a lot of challenges. 03:05.240 --> 03:09.800 We're here to promote open collaboration, open standards, open source, and accelerate the 03:09.800 --> 03:11.000 innovation in the industry. 03:11.000 --> 03:16.440 Robotics industry has been around for, I don't know, for longer than I existed, probably. 03:16.440 --> 03:21.600 But it's been very slowly moving towards commercialization. 03:21.600 --> 03:27.160 To the point where right now, you see a human, I robots, and like their famous with AI. 03:27.160 --> 03:30.320 But honestly, there's still a kind of gimmicky. 03:30.320 --> 03:32.240 There's not real applications yet. 03:32.240 --> 03:37.160 Like I haven't seen one that works perfectly yet, or that has been deployed out there. 03:37.160 --> 03:38.160 Maybe I'm wrong. 03:38.160 --> 03:39.160 Where's your hands if I'm wrong? 03:39.200 --> 03:40.680 You know, an example? 03:40.680 --> 03:41.680 No? 03:41.680 --> 03:42.680 All right. 03:42.680 --> 03:46.120 It's good that I'm still right about this. 03:46.120 --> 03:49.600 And we want to support commercial adoption of open source, which is the biggest thing here. 03:49.600 --> 03:54.920 Because if there's no commercial adoption, there's no industry, there's no industry. 03:54.920 --> 03:56.160 How are we going to fund this, right? 03:56.160 --> 03:58.320 Because initially, this is what we're trying to do here. 03:58.320 --> 04:01.920 We want to make sure that the project continues to be alive. 04:01.920 --> 04:05.160 So what we form here is an ecosystem. 04:05.160 --> 04:08.280 So drone code sits at the center. 04:08.280 --> 04:11.280 But we have the open source projects. 04:11.280 --> 04:15.120 These are some examples I'm going to talk about the rest and develop it. 04:15.120 --> 04:22.040 We also have industry standards, which is like an open hardware project, a map link, which 04:22.040 --> 04:28.840 is the standard for communication, a protocol, which we can define the behavior of certain 04:28.840 --> 04:31.040 components of drones. 04:31.040 --> 04:35.080 We have industry members with some of the companies that actually support us. 04:35.080 --> 04:37.280 And actually contribute back. 04:37.280 --> 04:41.320 And industry associations, like the Linux Foundation, the open source robotics alliance, 04:41.320 --> 04:45.040 which hosts Ross, and we work with them, or the OpenCV Alliance as well. 04:45.040 --> 04:50.680 So we try to put everything together and make sure that there's a concise industry. 04:50.680 --> 04:54.600 And then there's a network, a concise, sorry. 04:54.600 --> 04:59.400 There's a network that we're trying to push things forward together. 04:59.400 --> 05:01.320 Why am I telling you all of this? 05:01.320 --> 05:05.720 Because like you just saw not a lot of you know about drones and I just wanted to give 05:05.720 --> 05:08.200 you a quick context and what's going on right here. 05:08.200 --> 05:13.320 We've been around since 2014, but the projects have been around since 2009. 05:13.320 --> 05:14.320 It's not something new. 05:14.320 --> 05:15.560 It's been around for a while. 05:15.560 --> 05:20.160 It actually was born out of ETH in Zurich. 05:20.160 --> 05:22.960 But it has grown tremendously out of that. 05:22.960 --> 05:29.800 To the point where I just pulled the analytics and we have already 13,000 unique contributors 05:29.800 --> 05:30.800 over time. 05:30.800 --> 05:37.400 So even though I say we're humble as more community, we're growing over time a lot. 05:37.400 --> 05:42.600 But we're struggling with some of the things that will allow us to grow more, like 05:42.600 --> 05:46.400 CI. 05:46.400 --> 05:49.920 Just some more stats, so you know exactly what's happening here. 05:49.920 --> 05:54.480 Almost 60 million lines of code distributed across all the repositories. 05:54.480 --> 05:57.160 Which are almost like 100 repositories. 05:58.120 --> 06:02.040 We have 20 companies that are supporting us and we have the top five projects which are 06:02.040 --> 06:03.040 there. 06:03.040 --> 06:07.440 PX4, which is the firmware for drones, picks up, which is open hardware, the flight controllers 06:07.440 --> 06:12.680 that actually control the drones, map link, the message protocol that I was explaining, map 06:12.680 --> 06:17.760 SDK, which is an SDK that allows you to talk map link because map link is basically 06:17.760 --> 06:19.480 an XML protocol. 06:19.480 --> 06:28.240 So map SDK is a wrapper on top of that that helps you build APIs to control it. 06:28.240 --> 06:32.840 And Q1 control, which is a very interesting project based on QT, and it's a mission planning 06:32.840 --> 06:37.280 software or a ground station, how we call it, which allows you to control the freaking drones, 06:37.280 --> 06:38.800 which is pretty cool. 06:38.800 --> 06:42.560 All right, so let's hope the internet works. 06:42.560 --> 06:46.800 2009, this is where we started. 06:47.040 --> 06:52.880 And it's important that we see where we started. 06:52.880 --> 06:55.880 Has anyone heard about PX4 or PX4? 06:55.880 --> 07:00.720 Yeah, just a few. 07:00.720 --> 07:05.560 So this is like, look at the machine, like, that is actually an Intel NUC. 07:05.560 --> 07:07.320 Like, very old Intel NUC. 07:07.320 --> 07:11.920 Those are parts put together from off the shelf in the electronic store. 07:11.920 --> 07:13.320 There's nothing secret here. 07:13.360 --> 07:17.000 This was like, a lot of pieces that were put together from electronics that were already 07:17.000 --> 07:19.360 available at the time. 07:19.360 --> 07:23.960 And we started from there. 07:23.960 --> 07:29.440 We moved a little bit further in a few years because it was a basically a research platform. 07:29.440 --> 07:32.080 And as you can see, you have more stable flight now. 07:32.080 --> 07:37.960 It's still a NUC machine, but no GPS flights now. 07:37.960 --> 07:41.640 It's autonomous flights. 07:41.640 --> 07:44.160 We have plane detection, a non-board camera. 07:44.160 --> 07:49.720 So we started getting tonsils, putting more features on top of this. 07:49.720 --> 07:50.920 I'm not going to play the whole video. 07:50.920 --> 07:53.640 I'm going to share this slide later, so you can see it at that home. 07:53.640 --> 07:56.320 But this is one of my favorite videos that we have. 08:01.120 --> 08:06.360 This is actually came out in, or developer, about into 1222. 08:06.360 --> 08:08.320 Look at the configuration of this drone. 08:08.320 --> 08:14.160 Look at how it's going to be moving in different directions. 08:14.160 --> 08:15.760 This is an opening copter. 08:15.760 --> 08:21.000 So PX4 allows you to fly the vehicles that you saw already, and then PX4's like this. 08:21.000 --> 08:28.800 I have a non-traditional geometry, and allow you to do this type of movement. 08:28.800 --> 08:32.240 This type of autonomy. 08:32.240 --> 08:37.360 So as you can imagine, this is going to bring a lot in a lot of complications, especially 08:37.360 --> 08:41.480 when you find the test things. 08:41.480 --> 08:44.280 These are some of the other applications. 08:44.280 --> 08:52.640 Walmart was using or firmware, and or hardware to do the delivery in the US. 08:52.640 --> 08:56.080 There's companies that do mission inspection like Wintra. 08:56.080 --> 08:58.560 This is a vertical and takeoff. 08:58.560 --> 09:04.560 It takes off with the propellers, and then it transitions as a fixed wing mid-air. 09:04.560 --> 09:08.280 This is a vehicle that has a very funny characteristic. 09:08.280 --> 09:12.080 The payload, the camera, it's like a $100,000 camera. 09:12.080 --> 09:13.720 It's more expensive than the drone. 09:13.720 --> 09:17.880 So you've got to make sure that that camera is going to be safe at the end of the day, right? 09:17.880 --> 09:20.280 So how do we actually ensure that? 09:20.280 --> 09:21.280 It's hard. 09:21.280 --> 09:23.120 It's actually hard. 09:23.120 --> 09:24.880 We have another vertical takeoff and landing. 09:24.880 --> 09:27.120 This is actually a very cool vehicle. 09:27.120 --> 09:29.480 They can configure it differently. 09:29.480 --> 09:39.720 This company is called quantum systems out of Germany, and we also support other water vehicles. 09:39.720 --> 09:48.720 So how do we make it so that all of these freaking applications fit in the same firmware? 09:48.720 --> 09:52.080 How do we make it so that we can test those things? 09:52.080 --> 09:54.680 And how do we prove that we're safe? 09:54.680 --> 09:59.760 So the honest truth here is that we can not guarantee that. 09:59.760 --> 10:04.120 We are only a humble open source project that when we depend on their contributions. 10:04.120 --> 10:06.840 I'm not a new authority to be able to test this. 10:06.840 --> 10:14.200 Like the FAA in the US actually runs tests for aviation companies to be able to guarantee the safety. 10:14.200 --> 10:17.000 You cannot expect an foundation to be able to do this. 10:17.000 --> 10:24.600 But what you can expect is us doing our best to the best of our ability to actually guarantee the safety to the best of 10:24.600 --> 10:30.040 our possibilities within our resources because it's expensive. 10:30.040 --> 10:34.200 So the answer to how do we do this is the PX4 out of the pile of the firmware. 10:34.200 --> 10:38.280 And this is going to be a little bit deeper into the technical side, but I'm going to 10:38.280 --> 10:43.640 try to explain the functionality of the firmware so that I can finally go into the CI and testing 10:43.640 --> 10:46.200 and how do we do it. 10:46.200 --> 10:52.240 So we're running on top of the narthos, everyone knows what the narthos is, real-time operating 10:52.240 --> 10:53.240 system. 10:53.480 --> 10:59.080 Basically it's like an operating system, the guarantees the execution of the commands whenever 10:59.080 --> 11:03.560 you say you wanted to do it and it's a gross characterization, but it's like a think of fancy 11:03.560 --> 11:07.960 scheduler that guarantees that whatever you said is going to happen happens. 11:07.960 --> 11:15.880 We're a C++ based, we're modular, meaning that within the operating system, different components 11:15.880 --> 11:20.200 of the firmware actually run out of different modules, like think of different apps. 11:20.360 --> 11:26.120 Like the app that runs logging is different from the app that runs the estimation or the control. 11:26.120 --> 11:31.560 And that's important because you don't want the logger to actually fail and then the drone 11:31.560 --> 11:33.720 is going to crash because the log is not working. 11:34.520 --> 11:38.440 That's not something that we want so we want to make sure that it's separate. 11:39.000 --> 11:45.480 But how do we actually connect all everything like all the modules through a middleware, middleware 11:45.480 --> 11:52.520 is basically a published and subscribe component of our system that actually just basically says. 11:53.240 --> 11:58.440 Okay, this is the driver for the GPS and this is the location that we're publishing. 11:59.160 --> 12:04.040 And then there's a different modules that want to maybe subscribe to the location so they do that. 12:05.080 --> 12:10.200 And we do that through a middleware we call Europe, which is the DS compatible for those of you 12:10.200 --> 12:14.600 that know Ross, it's actually compatible with Ross and I'm going to talk a little bit about that. 12:16.200 --> 12:20.520 Everything is better realized in Trebsafe. We have great hardware support, I'm going to talk a little 12:20.520 --> 12:25.560 more in the next slide. There's support for custom builds, which is one of the biggest things you're 12:25.560 --> 12:31.480 in something I wanted to talk to you about because we have hardware support, like I don't know, 12:31.480 --> 12:35.320 think of this as one of the hardware. Okay, we support this hardware, but we have 12:35.320 --> 12:39.800 variants of that hardware too, maybe you want this version of the firmware that doesn't have 12:39.800 --> 12:44.840 this feature, we support that. And sometimes we even support upstream, which means we have to test for this. 12:46.760 --> 12:52.120 We have more than a million of vehicles using PX4 right now. This is data that we surveyed from 12:52.120 --> 12:56.680 the companies that are within our memberships, but actually we don't really know. It's impossible 12:56.680 --> 13:01.400 to know for the open source project because we are not charging for our license, we're not actually 13:01.560 --> 13:07.480 demanding from the companies that are implemented, that they share this info, but the members 13:07.480 --> 13:11.880 that are the companies in Green Drone Code, some of them are manufacturers of the components. 13:14.440 --> 13:17.800 I'm 10 minutes over. Okay, good. So I'm going to go quickly. 13:18.440 --> 13:24.680 And we have 30,000 developers like SLB4. We got flight modes, which are provided instead of 13:24.680 --> 13:31.160 helpers, the developers to actually be able to sort of autonomy. We have flight tests that help the 13:31.160 --> 13:37.640 flight modes to actually be able to control the behaviors. We have throners where you can say, 13:37.640 --> 13:41.240 if you're going to take off, take off a certain altitude always. Maybe you don't want to 13:41.240 --> 13:45.320 every time you take off, set that. We have a device interface. If you're a developer, you want to 13:45.320 --> 13:50.040 make sure you can subscribe to things. And then we have control allocation. We infranslate the trust 13:50.040 --> 13:56.520 and the pork. Like I was showing you, we support different geometries. So it will be hard for 13:56.520 --> 14:02.760 the developers to do the controllers to actually define and hard constrain the controllers 14:03.800 --> 14:08.520 to the geometry because we want to support multiple geometries. So we have a translation layer that 14:08.520 --> 14:15.080 helps you write the controllers so you don't have to think of the geometry. You think about it later. 14:15.080 --> 14:20.760 And we also support cross 2. This is an example of a hardware. This is a pixel. This is the 14:20.760 --> 14:25.320 one of the projects that I was telling you about. We support more than 80 boards. We have more 14:25.320 --> 14:30.200 than 100 sensors that we support within all of the hardware collections. Mostly, the architecture 14:30.200 --> 14:38.200 has the M32 based, IMAX and RIS5. But we also support payloads, gimbals, batteries, the bloggers. 14:38.200 --> 14:43.080 There's a whole ecosystem that is just doing this. And like I said, the mission planning software, 14:43.400 --> 14:48.600 it's also part of the ecosystem, something that we also have to test. We have the rawst to support 14:48.600 --> 14:52.680 with things to the middleware. I'm going to gloss through this because we're running out of time. 14:52.680 --> 14:58.920 We also have a simulation engine based on gazebo, also based on open robotics projects, 14:59.640 --> 15:04.440 which allows the developers to actually do very advanced things and test them. 15:05.240 --> 15:10.280 But how do we actually test it? And sorry that I took half of the thought for this. In my mind, 15:10.280 --> 15:16.280 there's going to be five minutes. So the developer testing, I separated this two things. 15:16.280 --> 15:21.480 Unitests. Integration tests with simulation tests. Hardware in the loop tests. Okay, that's what 15:21.480 --> 15:28.920 we can do in the cloud. We run in CI. But what about this? Flashing and installing on the hardware. 15:29.560 --> 15:33.320 Setting up this hardware. Setting up the parameters. How do you want to tune this? 15:33.320 --> 15:39.560 Actually, flying with an RC, or maybe flying an admission, autonomous mission. And how do you 15:39.560 --> 15:44.760 validate those missions and those controlling those flights actually work? So we do two different things here. 15:44.760 --> 15:49.320 Testing the cloud or mostly in the cloud because we also do hardware in the loop. 15:50.120 --> 15:57.240 Let's run with the unitesting. So, or fortunately, our unitests coverage is less than 60%. 15:57.960 --> 16:03.480 Less than 60%. Why? Because it's been really hard for us to enforce this. 16:04.440 --> 16:09.560 We've been slowly building and growing this number, but it's really, really, really slow. 16:10.360 --> 16:16.040 It's very hard to incentivize the contributors right now to get this. But we do have them, 16:16.040 --> 16:21.640 and we do run them. The test run in about seven minutes per pull request. And we have about, 16:21.640 --> 16:28.440 I don't know, like 10 to 20 pull requests per day. So sometimes, in a given negative in Monday, 16:28.440 --> 16:34.120 I would wake up, go to my computer, and we would have like 50 pull requests that came in on Monday. 16:34.120 --> 16:38.760 Because I don't know, developers worked over the weekend and just took me to the pull request on Monday. 16:39.560 --> 16:46.040 And I'll surprise. Everything in there was running on greekobactions and the runners from greekob, 16:46.520 --> 16:52.840 which meant that a pull request was sitting in queue for hours, sometimes days. 16:54.040 --> 16:59.480 Because the collection of every test we did just took hours. So, this is an example right now. 16:59.480 --> 17:05.480 This is the current set up and this is just only taking seven minutes. We do integration tests. 17:06.200 --> 17:12.840 This is a bit more complex. This is actually three step. We have a test runner, 17:12.840 --> 17:17.320 which is a Python script, and we have different tests. This are just an examples, but we have way more. 17:18.360 --> 17:24.360 And each one of those actually launches px4 from 0, from scratch. And then px4 launches gasivo, 17:24.360 --> 17:29.560 from scratch for each one of those, which meant that at least you have five minutes wasted in there, 17:29.560 --> 17:36.520 in setting up the environment. And then you run the tests and it took like 20 minutes each. 17:36.600 --> 17:44.280 We fixed those. That set up. I actually fixed that last October. 17:45.080 --> 17:52.280 And then we realized that when we fixed the execution time, we found out that there was actually tests 17:52.280 --> 17:56.040 in there that were failing because there were real errors. But we couldn't see them because there were 17:56.040 --> 18:01.640 timeouts in the execution because sometimes when the test runner launched, it tried to launch px4. 18:01.720 --> 18:07.000 And for whatever reason, the resources in the system were lost or maybe it didn't launch 18:07.000 --> 18:11.800 actually launch px4. So there was a timeout, but you didn't know about the timeout until like it was too late, 18:11.800 --> 18:18.200 like 60 minutes late. So we had so many failures and we didn't actually know because we couldn't 18:18.200 --> 18:25.400 have, we didn't have the observability into those issues. So I took it to the task. I tried to fix 18:25.400 --> 18:30.760 those issues and then I found way more. These are just the only tests that here that sometimes fail 18:30.760 --> 18:37.320 in terms intermittently. And we haven't figured out why yet. Thank you. I'm almost done. 18:39.560 --> 18:43.480 This is just an example of what's happening. Actually, this is from the screenshot from the 18:43.480 --> 18:49.160 last 9-15 minutes execution time overall for all of the tests. It was a huge effort. 18:49.960 --> 18:55.000 But mostly it was me scratching my head against Gid Cup runners because I actually didn't know how 18:55.080 --> 19:02.280 they worked. And the whole setup was to me honestly something that I never dealt with. 19:03.800 --> 19:08.840 And then finally we do hardware in the loop and we do this through Jenkins because one of the 19:08.840 --> 19:16.120 maintainers in his house has a test rack with the USB hub with a bunch of FTDI to USB cables connected 19:16.120 --> 19:21.640 to all of the hardware. And every pull request guess what? All the hardware got flushed. 19:22.360 --> 19:27.800 All the firmware got flushed and then you configure everything. And then you check the status 19:27.800 --> 19:33.240 and then you reset devices and then you run tests for everything. This used to take hours to 19:33.640 --> 19:38.920 per pull request with crazy. It's crazy. Like why did we think this was a good solution? 19:38.920 --> 19:43.160 Because we didn't have anything. We wanted to have something and we had to start somewhere. 19:44.040 --> 19:48.760 The only reality is that we shot this down last month because again, 19:49.480 --> 19:55.240 the hardware, first of all, we didn't have every hardware that we support. We only had the ones 19:55.240 --> 20:00.360 that the manufacturers shipped us or the ones that we could afford. And second, we had a lot of 20:00.360 --> 20:05.160 time loss too when we tried to flash. Sometimes the FTDI drivers didn't work. I don't know why. 20:05.880 --> 20:12.360 It was a Windows machine and then we switched to Ubuntu machine. We had driver issues. We had to 20:12.360 --> 20:16.120 reconfigure Jenkins. It just wasn't nightmare. We couldn't support it as a maintainer. 20:16.760 --> 20:20.840 We had to spend more time reviewing the pull request and actually fixing this infrastructure. So we decided 20:20.840 --> 20:27.480 to turn it off. So if anyone here wants to pitch in, thank you. I would please talk to me. 20:28.360 --> 20:32.440 All right. So lastly, the most complex thing of all, how do we actually do build, 20:32.440 --> 20:37.880 how do we publish artifacts? So our primary development environment is Ubuntu. 20:38.280 --> 20:43.880 Whatever the LTS is, right now it's 24 or four. We build using GCC. But since we do 20:46.040 --> 20:52.920 embed it, we have to use the ARM cross build GCC version and we're locked down to version nine right 20:52.920 --> 20:59.080 now because of historical reasons. I'm actually working in trying to put that to I think 13. 20:59.080 --> 21:05.160 I don't remember which one this. But we have a script that installs all the dependencies. 21:05.160 --> 21:10.040 Actually, we have that running in CI and we know that if you want to run the PX4, 21:10.040 --> 21:14.520 we guarantee that if you run Ubuntu.ca station, the fresh machine is going to work because we 21:14.520 --> 21:19.160 have it in CI. Finally, we didn't have that last the beginning of last year. 21:20.280 --> 21:25.080 So this is how you run it. 9 PX4 and then the variant of the hardware. 21:25.640 --> 21:29.560 So you always need to specify which hardware. So how do we make it so that we actually 21:29.560 --> 21:35.320 burnt all of the 80 hardware boards that we support? Well, we used to have a hard-coated file 21:35.880 --> 21:39.960 in which if you add a new support, you also have to make sure that you add that line 21:39.960 --> 21:45.400 then make sure that the string name is right. What happened was that sometimes the manufacturers 21:45.400 --> 21:50.040 forgot to add it there and then someone reported an issue and guess what, it wasn't in CI, so we 21:50.040 --> 21:55.800 never catch that. All right, that's an issue. Second issue, it took so much time to run everything 21:55.880 --> 22:01.160 because we were running a matrix and github actions which meant that it was every put request 22:01.160 --> 22:08.760 was launching 80 runners. So if you have 10 pull requesting queue, that means 10 plus 80, 22:08.760 --> 22:16.680 how much is it? 800. And each one of those took 15 to 20 minutes. That was impossible for us. So 22:18.520 --> 22:24.840 we created the Python script and we actually grouped together by manufacturer and we started 22:24.840 --> 22:31.320 caching those builds and that made it so that if you're using the same manufacturers, sometimes 22:32.200 --> 22:39.560 the dependencies when you're making the build are the same. So you could save like up to 80 22:39.560 --> 22:48.360 percent of your time on builds. So we wrote this script and we put it on github and we 22:48.440 --> 22:56.120 trim it down from 80 to 23 jobs. So we only now have 23 runners. We also did one something else 22:56.120 --> 23:04.040 which is we moved into self-coasted runners in the AWS using a platform called runs on. 23:04.040 --> 23:09.480 You have never heard of runs on. They make it so easy for you to actually run your own infrastructure 23:09.480 --> 23:15.240 and really happy any three-for-open source. So if you really want to try it out, it's really easy. 23:16.040 --> 23:22.600 So this is one group, not xpx4, and it's like all of the different versions. This is v6x, 23:23.240 --> 23:26.920 all of these are the same hardware, but these different variants. This is the part that was actually 23:27.560 --> 23:32.120 killing us. But thanks to the caching, thanks to the new runners, like look, 23:32.680 --> 23:37.560 58 seconds in running one build. It took 10 minutes, at least before. Now it's taking a minute. 23:38.280 --> 23:45.560 So it was a result. What took hours now takes 16 minutes. But still, it's a lot. 23:49.480 --> 23:55.960 Here's an example from pull request from last night. Lastly, 57 checks. I didn't talk about 23:55.960 --> 24:03.800 all of the things we do in CI. We have about 57 different checks. And an example, the container 24:03.800 --> 24:10.440 build failed, because I wanted to be smart, and I wanted to push every container building to the 24:10.440 --> 24:19.480 Geek of Registry. So I was doing a deal. I talked to my API token, and I was basically running 24:19.480 --> 24:23.960 through the threshold and like the first hours of the day, which meant that from there on, 24:23.960 --> 24:28.120 throughout the day, every PR is going to fail. So this is still out there. So if you want to 24:28.120 --> 24:33.240 contribute something, does it really easy fix? And I'm not going to suddenly not going to be able 24:33.240 --> 24:39.000 to do the human in the loop test, but we actually have a flight test team, and they do actually 24:39.000 --> 24:45.800 do the testing. And they submit logs through the website that we have, which is called PX4LOG 24:45.800 --> 24:51.720 review. You do upload the logs. They give you nice charts. You know what happens. And then the 24:51.720 --> 24:55.800 developers actually tell you, oh, your motor is not aligned. This is why you have so much of 24:55.800 --> 25:01.240 a ratio. Maybe this is why you're not able to move the drone direction you want. As I 25:01.240 --> 25:06.840 conclusion, my friends, testing in robotics is tremendously hard. My robotics colleagues 25:06.840 --> 25:11.800 here is not going to let me lie. We've been suffering, and we need your help. So if you're ever 25:11.800 --> 25:18.040 thinking of collaborating without robotics project, I know the few. Tomorrow, we have a 25:18.040 --> 25:26.920 debt room, and which building team is at UV? UV21 something. It starts to meet there, and there's 25:26.920 --> 25:37.560 really cool content. So if you have time, please go there. And thank you so much.