WEBVTT 00:00.000 --> 00:11.120 Okay, just test. Yeah, that seems to be working. So, hello, welcome everyone. So, today we're 00:11.120 --> 00:16.000 doing a small presentation about a tool that we came up at work with that's called 00:16.000 --> 00:23.400 Film Reaction. It's as the title says, helps building you all kinds of filmwares and we 00:23.400 --> 00:29.960 don't go into more detail here yet because filmreaction can be used for a multitude of things. 00:29.960 --> 00:35.360 So first of all, who are we? I'm Marvin, a filmre developer working at nine elements 00:35.360 --> 00:43.000 as my shirt says here and this is my lovely co-worker. Hello, I'm Woike, can I do DevOps 00:43.000 --> 00:57.680 mostly? I did, so it's good. So, what we want to go over today is building filmre is usually 00:57.680 --> 01:02.080 hard to do. For some reason, there are so many build systems and it's always difficult to get 01:02.080 --> 01:07.400 a hold of all of them, understand all of them especially if you're working in more than one project. 01:07.400 --> 01:14.160 Then we go over what is actually filmreaction and how does it help us? So, short introduction 01:14.160 --> 01:19.840 to Jocker and a helper tool called Dagger and then we do an example and maybe if the 01:19.840 --> 01:29.600 demo gods are on our side also working demonstration. So, first of all, building filmwares. 01:29.600 --> 01:34.840 If you come across building host filmwares, BMC filmwares embedded filmre not so creative 01:34.840 --> 01:40.520 name, it's an action, it's an action that builds filmre. So, what is it? It's kind of 01:40.520 --> 01:46.480 similar to what you know from make file or task file. It just calls into different things 01:46.480 --> 01:50.320 and abstract things the way you even further and yet now you might be saying oh yeah, 01:50.320 --> 01:55.120 yet another build system. How does that improve the whole situation? Did you want to get 01:55.120 --> 02:01.440 rid of some build systems? And it's not really a build system per say. It's simply just 02:01.440 --> 02:08.880 a executor of commands under known circumstances and doesn't really build stuff itself. 02:08.880 --> 02:15.760 So, it gives us a unified environment using container technology. It doesn't need to be 02:15.840 --> 02:24.240 darker just anything that speaks OCI images and builds in CI and builds locally are always 02:24.240 --> 02:28.640 the same given that we now have containers on our side and it doesn't matter if you're building 02:28.640 --> 02:32.800 on windows, if you're building on Mac, if you're building on Linux, you get the same results 02:32.800 --> 02:41.040 throughout anyone using the system. It's written and goes so it can cross compile to almost any 02:41.040 --> 02:46.640 architecture that goes supports. We still have some minor issues with paths but that's up to 02:46.640 --> 02:52.240 be fixed. It has get up action integration already but it is very low effort to integrate into 02:52.240 --> 02:59.120 other CI-CD systems that are not GitHub. It just started out there and we also packaged version 02:59.120 --> 03:07.920 for the AUR. So, what you usually see when you split some CI systems and your local development, 03:08.000 --> 03:17.120 you have some build Yamels or other scripts, whatever GitHub uses right now and locally you might 03:17.120 --> 03:22.640 have some additional shell script to just call into all your tools so that you don't need to type 03:22.640 --> 03:30.400 all of the Mac commands or whatever tool is being used in that and this all together then fetches 03:30.400 --> 03:35.920 obviously you code that you have locally or in your remote repository and builds the thing. 03:35.920 --> 03:43.600 And now with our unification we have the same results in CI and local by just using the tool 03:43.600 --> 03:49.680 plus its configuration which is written in JSON and obviously the code and configurations of 03:49.680 --> 03:56.960 your tool that you previously had. So with that I'll switch over to Voytech to I'll 03:57.280 --> 04:05.840 explain a bit more about Docker and how Dagger fits into the builds. Hello, hello. 04:06.800 --> 04:11.040 So, does any of you actually know Docker have you heard about that? 04:12.240 --> 04:20.880 Oh, nice. So, essentially Docker is a somewhat a virtualization environment for 04:21.680 --> 04:27.120 often used for development. It allows you to package together all kinds of dependencies 04:27.840 --> 04:32.880 for all kinds of systems, doesn't matter if you are running or a Fedora, you can still 04:32.880 --> 04:40.800 Dockerize debion packages for example. It's very convenient, you can use different versions, 04:40.800 --> 04:49.200 you can use all the versions. So, it is super great thing for development and hence we wanted to 04:49.440 --> 04:55.600 use it for this but we couldn't use it standalone that's where the Dagger comes in. 04:56.400 --> 05:04.400 Dagger is a Docker orchestrator. It sits on top of it and essentially it allows you to take a 05:04.400 --> 05:12.080 Docker container and execute commands inside it. Define environment variables also things like that. 05:12.240 --> 05:20.720 So, it's programmable CI-CD engine essentially. The idea is that you could create your CI-CD 05:21.600 --> 05:30.000 once in Dagger and then run it everywhere. In GitHub, locally, GitLab, source hard, whatever comes to your 05:30.000 --> 05:40.960 mind. So, we decided to use this to basically leverage what we wanted to do. And that's the 05:40.960 --> 05:50.640 core technology. So, the example that we have made, this is from the example repository, 05:50.640 --> 05:57.920 this is the JSON configuration. So, for example, over there the fourth line you have SDK URL 05:57.920 --> 06:02.960 that's the URL to the container you want to use to build in this case, core boot. 06:03.680 --> 06:09.200 We pre-build all of the containers so you don't have to build them yourself. We have multiple 06:09.280 --> 06:16.400 versions, dating back, I think the oldest is for 19 something like that. We also have 06:16.400 --> 06:25.280 EDK2, Linux, whatever you name it. So, you define your container there and then you define 06:25.280 --> 06:29.200 additional paths like where is the core boot repository that you're trying to build. 06:30.160 --> 06:36.640 There is the configuration file for your core boot and so on. Where do you want to output the result 06:36.640 --> 06:45.600 of this endeavor? You can also define blobs like Intel FSB, management engine, things like that. 06:45.600 --> 06:53.360 It will automatically figure out what to do with them and put them in. So, that's a very nice thing 06:53.360 --> 07:02.480 if I do say so myself. Also, at the bottom you can define inputs. So, the whole idea behind this 07:02.640 --> 07:10.480 is that you can define a core boot module. You can also define EDK2 module, U-Root module, Linux 07:10.480 --> 07:17.280 module and then you can pipe output of one into the next one. So, you can build the whole stack in 07:17.280 --> 07:29.520 one single loop for which we have a recursive build. In GitHub, this is the Yamalo that you could use. 07:29.600 --> 07:34.720 You have some kind of checkout of your repository where you might have the core boot or other 07:35.440 --> 07:42.160 projects as a GitHub modules or whatever you fancy as alternative. Then you just call 07:42.160 --> 07:47.920 the interaction with the configuration file which module you want to build and if you want to build 07:47.920 --> 07:54.320 it recursively means it will build all of the dependencies like Linux, U-Root, whatever you might 07:54.400 --> 08:02.320 define there. In the end, you just upload the artifacts to the GitHub. It's as easy as that. 08:02.320 --> 08:10.160 If you want to run the same thing locally, you just install the interaction or compile it from 08:10.160 --> 08:16.960 the sources called build and again config target and if you want to do recursive build or not. 08:18.640 --> 08:23.680 If you want to abstract this way a little bit more and make it a little bit more convenient, 08:23.680 --> 08:29.920 you could also use something like task file where you define that you want standalone 08:29.920 --> 08:36.400 core boot or you want core boot with Linux, big thing. You can define some variables and so on. 08:38.160 --> 08:47.040 So, essentially that's the whole premise. Then we have a demo if that will work. So, let's try that. 08:47.680 --> 08:59.040 Okay. So, while you're preparing the demo, another good thing is about that all of these containers 08:59.040 --> 09:06.720 one thing about dagger is it will use API calls into any container engine. So, the default is to 09:06.720 --> 09:12.880 use Docker but if you have popmen or just raw container D or whatever, what it will do is whatever 09:12.880 --> 09:20.160 it finds on your system. It will run API calls. Those are defined directly in the go code and 09:20.160 --> 09:26.640 based on what you've written in your in an outputs will turn this into a familiarly created containers, 09:26.640 --> 09:31.840 mutate them on the fly and outputs whatever you define there. So, essentially you tell it what 09:31.840 --> 09:38.720 goes in what goes out and how to operate with that and then it will just in those containers which 09:38.800 --> 09:43.280 have a specifically known good environment for each of these firmware stacks that we already 09:43.280 --> 09:49.440 predefined call their build systems. It will not touch any of their build systems. So, it will 09:50.080 --> 09:55.760 operate as if you're just doing that locally except now much more conveniently and with much more 09:56.800 --> 10:01.600 confidence that your environment does not affect the build if it's running on your machine or if 10:01.680 --> 10:11.040 it's running on a co-work machine. Yes, so here's the example of the JSON configuration file 10:11.040 --> 10:19.280 right now we support only JSON but we might extend it to more system slater. So, in this example 10:19.280 --> 10:29.920 we have a co-boot which is supposed to have a Linux kernel baked in. So, here you define that it 10:30.000 --> 10:38.400 depends on the Linux kernel. Here you define the Linux kernel. You also define that Linux will use 10:38.400 --> 10:45.760 its own container made for that and you can also say this Linux container will use a 10:45.760 --> 10:55.760 urode inside and here below you define your urode and with these dependencies then you can 10:55.760 --> 11:12.720 have the whole tree defined like so. All right, so I already tried to execute this because 11:14.640 --> 11:22.000 it's a demo but yeah demos are always very funky so let me clear this. 11:22.960 --> 11:31.360 Perfect. So, I have defined the task file which does only finger action dash build dash dash 11:31.360 --> 11:39.600 config dash dash target. In this case it's built the co-boot with Linux and because I already 11:39.600 --> 11:44.720 built it before it has already everything cached it detects there are no changes in the sources 11:44.720 --> 11:50.160 so it will not try to rebuild them. It just shows you everything is up to date. So, the urode is up to 11:50.560 --> 11:56.640 that's the one module that has no dependency at all Linux which depends on the urode. It's also 11:56.640 --> 12:02.640 up to date and then the core boot which depends on the Linux also up to date. So, basically you're done. 12:03.520 --> 12:13.120 I could try to maybe how much time we have? 12:17.120 --> 12:27.360 Oh, nice. So, if we, how many seconds? 12:33.520 --> 12:43.120 The core idea which is part of the project? So, the core idea which is part of the project is that 12:43.120 --> 12:49.440 oftentimes these build systems especially if they consist of multiple parts take away from 12:49.440 --> 12:54.800 developers' time you need to fill around with them too much and especially if you're changing 12:54.800 --> 13:00.800 something in Linux when you have a core boot Linux boot urode stack and then you need to figure 13:00.880 --> 13:05.040 out which you need to rebuild now. Do I need to rebuild only the kernel or do I need to change 13:05.040 --> 13:11.440 something here now as well? Now, you can focus solely on just coding and rerun the build system 13:11.440 --> 13:16.480 and it will figure out for you where things changed where it knows how to rerun some stuff. 13:18.000 --> 13:24.800 Plum things together and you can just focus on coding, run building and you get a binary out of it. 13:25.440 --> 13:32.400 Just so that more time can be spent doing things that are actually 13:33.920 --> 13:38.880 I'd say important to do and not to care about because building something should be easy and it 13:38.880 --> 13:45.040 should always be the same so this is how we created it. I think it's where mostly at a point where 13:45.040 --> 13:51.680 we can already move into a round of questions. Okay, so this is an example where he removed some of the 13:51.840 --> 13:58.320 caches. I didn't really remove the caches. I just touched the read me in Linux kernel so 13:58.960 --> 14:04.320 now it will not rebuild your root because that's already up to date but it will try to rebuild 14:04.320 --> 14:10.720 the Linux kernel because I just changed one file so it is rebuilding that and then it will also 14:10.720 --> 14:15.760 rebuild the core boot because that depends on this Linux kernel so that's the whole idea of this 14:15.840 --> 14:25.680 dependency tree. So I guess now questions. Yes. 14:33.760 --> 14:40.080 So it does require some Windows 2 links and it actually just expects the Windows user space 14:40.160 --> 14:47.120 for the FSP. We did some hex there to pull in some of the Microsoft 2 links into Linux 14:47.120 --> 14:53.120 environment and make them work and it really makes it environment. It works. It's still kind of 14:53.120 --> 14:58.880 Linux container but the setup is so finicky that abstracting it away in a simple container just 14:58.880 --> 15:05.040 tells us to make this work more properly. It's basically requires what EDK2 requires with some 15:05.120 --> 15:15.200 extra sprinkled on top. Yes, we lose the changes to the vendors or customers directly so 15:16.080 --> 15:23.600 I think as a hobbyist there is no legal way to obtain them but otherwise you can obtain them from 15:23.680 --> 15:41.040 Intel with set vendor license. Yes. Sure, we could do that. It's basically a matter of 15:41.040 --> 15:47.680 figuring out the licensing because there are so many projects involved that we need to make sure 15:47.680 --> 15:52.400 that whatever we distribute is actually redistributable especially if blocks are involved 15:52.400 --> 15:57.840 because firmware action does allow to supply you a arbitrary amount of blocks and those are often 15:57.840 --> 16:04.000 tricky when it comes to licensing but other than that I see no issue in doing that. For now we just 16:04.000 --> 16:09.840 cared about the tool itself and not what it actually outputs itself. That's mostly for internal use 16:09.840 --> 16:15.600 right now but for all the public stuff I see no problem doing that. What you think in the 16:15.760 --> 16:21.840 way, firmware files that aren't really going to be used to provide damages without them and then 16:21.840 --> 16:28.240 how to script things like that. Yeah, sure. There's also a way that's just called a stitching stage 16:28.240 --> 16:33.280 where you could do exactly that already in firmware action. Just provide like everything as a 16:33.280 --> 16:40.320 block and then tell it where everything goes in the binary and then it will do just that. But you 16:40.320 --> 16:44.480 can also use any other tool. It's just convenient for us to just have one tool that does it all. 16:45.600 --> 17:00.560 Any more questions? I think that's not the case and with that we can end few seconds early.