WEBVTT 00:00.000 --> 00:12.500 Okay, so, hi, today we will talk about the ultimatum cloud that we are filming 00:12.500 --> 00:27.420 by this and we are about framework, I am Maché, Mr. Mitchell, I am from, okay, so, I am 00:27.420 --> 00:33.660 Maché, from 3M depth and generic measure currently, what we are doing, we are doing 00:33.660 --> 00:40.380 various stuff for the scope of this presentation, what might be important is that we are developing 00:40.380 --> 00:52.380 boot firmer mainly for X86, so we can simplify by saying that we are developing the UFI BIOS, 00:52.380 --> 00:57.300 but open source one, so it's the BIOS you know from your computer, but it might be more open 00:57.300 --> 01:04.560 than the traditional one, and we need to validate that to make sure it works correctly 01:04.560 --> 01:14.200 on the certain machines, so the agenda, we talk about the history of our framework, about 01:14.200 --> 01:19.880 what kind of hardware is supported there, how to interface, how we are interfacing with 01:19.880 --> 01:28.440 the hardware, and I will show you some examples scenarios of test we are running. 01:28.440 --> 01:38.120 So the history, we have been using a similar approach since around 2018, when we have 01:38.120 --> 01:44.120 been evaluating PC engines, core boot releases on a monthly basis, so we needed some kind 01:44.120 --> 01:48.640 of regression test suite that we would run, at least each month before release to make 01:48.640 --> 01:56.120 sure that everything is working as intended, using that approach is executed over 50,000 01:56.120 --> 02:05.320 tests, and publicly released over 150 binaries of open source firmer for I believe six or seven 02:05.320 --> 02:11.640 different platforms, and it was mostly an internal project, that the firmer was open 02:11.640 --> 02:24.880 so the test was not, we have published it like two years ago, small part of it earlier, 02:24.880 --> 02:29.500 what are the use cases of this dashrar open source firmer validation framework, of course mainly 02:29.500 --> 02:37.240 validation of firmer in general, not only open source, you can imagine or we even had one 02:37.320 --> 02:45.960 use case where we did tests on IMI firmer as well, but mainly we use it for testing dashrar 02:45.960 --> 02:52.600 firmer releases that we released for different hardware platforms, for test driven backfixing 02:52.600 --> 02:58.160 for regression testing, so after introducing new features we can check if we haven't 02:58.160 --> 03:05.000 break, other features before the major release we can run all of the tests we have to 03:05.000 --> 03:12.440 make sure it works as intended, it mainly works currently with dashrar with UFI payload, 03:12.440 --> 03:20.560 but we do have other firmers and cases as well, it's mostly written in robot framework 03:20.560 --> 03:27.600 as the main test environment, we do have some Python as well for level of keywalls and some 03:27.680 --> 03:36.200 shell scripts mostly as some rappers or hyper, are you familiar with robot framework, have 03:36.200 --> 03:42.360 you been using it, okay, so it's like a generic open source automation framework, it's built 03:42.360 --> 03:51.000 on top of Python, so it is a Python underneath, you can use Python to implement libraries 03:51.080 --> 03:58.040 for robot framework as well, it's from what I know, it's used widely with web application 03:58.040 --> 04:07.320 testing and many more, but it's not so widely used in the lower level, maybe validation, 04:07.320 --> 04:14.040 I know one more project, which is kind of level, which is open BMC where robot framework 04:14.040 --> 04:19.560 is also used and they are basically my kind link, but it's links distribution for the 04:19.640 --> 04:28.520 BMC on servers, I can say that robot framework has excellent documentation and great 04:28.520 --> 04:36.200 community, it's a really fun project, so what platforms are we talking about here, we have 04:36.200 --> 04:42.840 a wide variety of hardware platforms, we develop a firmer floor, so a lot of stuff as this 04:42.840 --> 04:52.040 one, we have some network appliances, we have some desk stops, so the MSI one, we bought 04:52.040 --> 04:57.400 a two-core boot, like two or three years ago, which is quite modern, so we have different 04:57.400 --> 05:05.480 kind of hardware and we developing a BIOS for them, so we need some means of testing those 05:05.800 --> 05:14.200 releases on those different hardware units, which poses different challenges, so we need to 05:14.200 --> 05:21.160 have some small lab that we put those hardware boxes in and we need some interfaces to remotely 05:21.160 --> 05:28.520 run some tests to let the developers work on them, we need at least some power control, 05:28.520 --> 05:37.080 we need some testing interface, we need some GPA control sometimes, we will walk through those 05:37.080 --> 05:47.400 in details right now, so power control is perhaps the most important one, most of the time we 05:47.400 --> 05:55.960 use one of these two, we have like a custom board, with this relay, we can switch power on and off 05:55.960 --> 06:03.880 and for some other boards, so it's not possible, we just have some Wi-Fi plug, equivalently, 06:03.880 --> 06:11.560 which we have a lot of firmer, which works pretty nice, and we can use API to the power supply unit, 06:13.400 --> 06:17.640 and at the end of the board, we've developed a couple years ago, with open source hardware 06:17.640 --> 06:24.280 settified, you can find a lot of the schematics if you are interested in, and we develop some 06:24.280 --> 06:29.960 server application on top of that, which allows also to use HTTP API to control the power, 06:30.520 --> 06:39.720 GPAios, and so on on the device on the test. Next, if you already can control the power supply 06:39.720 --> 06:45.560 unit, we also need to control the test flow itself, we need to send some commands to the device 06:45.560 --> 06:51.960 and to receive some responses, so for that we mostly prefer to use serial ports whenever possible, 06:52.920 --> 07:00.280 so we, in fact, you'll turn it, so we connect the serial port from the device to the other board, 07:00.280 --> 07:07.640 our RTE, and then we set to net service, it's like a service, which basically exposes this 07:07.640 --> 07:13.640 serial interface in local network, and we can use it from different machines via turnet, 07:14.600 --> 07:22.200 this is the primary one, there are also others like SS site, but SS site is not really enough 07:22.200 --> 07:26.840 for firmware validation, but we can use it in a lesson, in some cases, when we are doing some 07:26.840 --> 07:34.840 OS level testing, we have also, in terms of input, we have also used the keyboard simulation, 07:34.840 --> 07:40.360 in some cases we can only read from serial, but we need to write from USB keyboard, because that is 07:40.360 --> 07:50.520 like one line only for you up, in some words, we need to control the PIOs, most important 07:50.520 --> 07:57.480 ones are power, and reset the buttons, because obviously we need to be able to have that 07:57.480 --> 08:04.360 level of control over the board mounted in lab, in some cases we need some more, like power 08:04.360 --> 08:11.560 let status, so we can read if the board is in fact already booted or not, or at least 08:11.560 --> 08:18.120 powered on, there are also many custom GPS on different boards needed as well, sometimes we need 08:18.120 --> 08:23.080 to, I don't know, reset signals after we flash firmware, and we need to build different kinds of 08:23.080 --> 08:30.840 circuits to support that, and all of those are connected to our RTE control board, and then we can 08:30.840 --> 08:41.560 use REST API to control the device, maybe the most important is the firmware flashing, we need to 08:41.560 --> 08:48.760 provide external interface for developers, whenever they are developing firmware, more often than 08:48.760 --> 08:56.760 not they will break the machine, and they need to externally flash that motherboard, so depending 08:56.760 --> 09:02.600 on the board we have either some flashing headers, or we need to come up with some different 09:02.600 --> 09:10.440 creative ideas, such as clips to clip onto the SPI flash chip to flash the bias externally, 09:12.440 --> 09:17.000 that's also connected to this RTE control board, and we can flash all of that remotely, 09:17.000 --> 09:28.520 to support all of those hardware interfaces, we have the CLI tool, which allows developers to 09:28.520 --> 09:37.800 easily run the typical scenarios, for assets management, we are using sniped currently, so 09:37.800 --> 09:43.560 we can check out the device, so to indicate that I'm working on this one, so no other people will 09:43.560 --> 09:50.200 be using it at the time, then I can reach some backup firmware right my own firmware, power on the 09:50.200 --> 10:00.360 supply unit, power button press, increase the pressing and so on, and grab serial output from 10:00.360 --> 10:07.080 the device on the test, so I can execute the hold of a lot of the flow on the machine remotely in 10:07.160 --> 10:16.680 the lab. The same set of libraries we have for the CLI tool, we are using as a robot framework 10:16.680 --> 10:25.320 libraries in test environment, previously it was a problem that we had like different set of code paths 10:25.320 --> 10:33.000 and some scenarios worked well with test environment, some not in CLI tool, and the other way 10:33.000 --> 10:42.600 around, so that's now integrated to use the same set of libraries, and now I would like to 10:42.600 --> 10:50.840 describe what kind of tests we need to perform on our firmware, so typically it consists of 10:50.840 --> 10:59.720 like, and time to set up, switch some option, put in some OS and check if this option works as it 10:59.720 --> 11:08.440 should work right, so that's end-to-end functional testing on the actual hardware. So in this 11:08.440 --> 11:13.640 example scenario we put the platform, so we need to send some commands to our control devices 11:13.640 --> 11:20.440 to power on the board, then we start reading serial until we see the very first strings, 11:21.320 --> 11:28.600 data as which key needs to be pressed to enter this setup, so we enter setup, then we get this 11:29.880 --> 11:37.080 menu of our serial, we need to parse it and compute calculate how many key strokes we need to 11:39.480 --> 11:45.880 for instance how many press down we need to execute to reach the position we need, 11:47.320 --> 11:56.600 so we enter for the in this case the shower system features, then we are in the next menu, 11:57.480 --> 12:04.680 here we just need to press enter to enter the shower security options, and here in this 12:04.680 --> 12:09.480 example we want to enable an SMM BIOS write protection, so we again need to 12:09.480 --> 12:17.560 read out this setup, this menu layout over serial, parse it and calculate that we need to 12:17.560 --> 12:25.560 one out of them and enter to select this option, and then we need to also to press F9 to size, 12:25.560 --> 12:32.280 then agree to save restart, and that's it in terms of setting this option state, 12:34.440 --> 12:40.360 here's how this example look in robot framework code, so we have all of these actions, 12:40.920 --> 12:45.240 I just described but in robot framework keyboard, so we have the keyboard implemented that 12:45.960 --> 12:54.440 do that kind of actions like going to certain menu, then sub menu, then enable 12:54.520 --> 12:59.240 some option, so I think it's taken through in this case, then we have keyboard to save changes 12:59.240 --> 13:06.440 and reset, then to boot from a selected operating system, and finally we execute some command in 13:06.440 --> 13:13.160 this case flash from command in the OS to see if we have the desired outcome, 13:14.440 --> 13:20.760 but some that we can parse or file the certain case case, and here we have some random amount, 13:21.640 --> 13:32.440 example how to run such test, we can run this in QMU as well, so we have some 13:32.440 --> 13:39.800 wrapper script that can allow us to spin up the QMU with the server firmware, so on the left side 13:39.800 --> 13:44.200 you can see the execution from robot, and on the right side you can see the execution from QMU, 13:44.200 --> 13:50.360 so we can in lifetime we can expect how the robot framework test phase pass or file, 13:50.360 --> 13:57.000 on the right side you can inspect how the machine is being powered on, part of the menus are being 13:57.000 --> 14:03.880 changed and so on, some links to the community if you'd like to reach out and discuss 14:05.480 --> 14:09.000 other contact information and that's it for the main part.