WEBVTT 00:00.000 --> 00:13.000 Good afternoon, Faustem, Bongeur, come in, hello. 00:13.000 --> 00:14.000 Hello. 00:14.000 --> 00:16.000 There we go. 00:16.000 --> 00:18.000 It's absolutely wonderful to be here. 00:18.000 --> 00:21.000 Thank you so much for coming to my presentation. 00:21.000 --> 00:22.000 I really do appreciate it. 00:22.000 --> 00:26.000 I know there are loads of great talks going on at the same time. 00:27.000 --> 00:29.000 I am here to discuss this. 00:29.000 --> 00:34.000 The lessons that I learned, open-sourcing the UK's COVID contact tracing app. 00:34.000 --> 00:39.000 And how you can apply those lessons when you are open-sourcing your own very important work. 00:39.000 --> 00:46.000 But first, I can't really talk about COVID the app without talking about COVID the disease. 00:46.000 --> 00:53.000 And I need to take a moment to acknowledge the huge toll that this horrific disease is taken on all of us. 00:53.000 --> 00:58.000 I'm sure everyone here has lost friends, family, co-workers and loved ones. 00:58.000 --> 01:05.000 And I do invite you to take just a moment to reflect on all the people who really should be here at Faustem with us. 01:05.000 --> 01:17.000 I am going to try and make this talk entertaining and interesting, but I do bear in mind that a lot of us will have been traumatized by this event. 01:18.000 --> 01:25.000 By way of a content warning, obviously this talk contains discussion of disease and death and cyber bullying. 01:25.000 --> 01:31.000 And it does feature some clips of British politicians, but I will warn you when those are coming up. 01:31.000 --> 01:34.000 So let's talk apps. 01:34.000 --> 01:38.000 Every country had their own app. The UK's app looked like this. 01:38.000 --> 01:41.000 I'm sure yours looked reasonably similar. 01:41.000 --> 01:46.000 And I'm sure everyone here at received a ping at some point saying, oh, you were near someone. 01:46.000 --> 01:49.000 I know I did. 01:49.000 --> 01:53.000 So in fact, this is the ping that I received on my phone. 01:53.000 --> 01:57.000 That's a screenshot of my phone when someone near me had sneezed. 01:57.000 --> 02:08.000 So I'm telling you this because I want to emphasize these are my personal recollections about a very strange time in my life and indeed in the world. 02:08.000 --> 02:21.000 I was working for NHSX, which was a brand new startup type government department, which was trying to modernize technology in the UK's national health service. 02:21.000 --> 02:24.000 And that's me just there. 02:24.000 --> 02:27.000 There are the launch party for NHSX. 02:27.000 --> 02:31.000 There were hundreds and hundreds of people working there. 02:31.000 --> 02:37.000 I'm working on this project, all doing a fantastic job in incredibly difficult circumstances. 02:37.000 --> 02:40.000 But I'm here to describe my memories, my experiences. 02:40.000 --> 02:44.000 I don't want you thinking that this was the, the Terence Eden show. 02:44.000 --> 02:50.000 It wasn't I was a small cog in a big machine, but hopefully a very useful cog. 02:50.000 --> 02:56.000 This, this isn't a story about, I can't have to re emphasize it. 02:56.000 --> 03:01.000 This isn't a story about one person alone in their bedroom coding away and doing something. 03:01.000 --> 03:03.000 This is a story about everyone. 03:03.000 --> 03:07.000 So the story starts about a year before the pandemic. 03:07.000 --> 03:12.000 I had spent my time on knowing he laying the groundwork for what I was about to do. 03:12.000 --> 03:18.000 By going around to conferences and speaking at events like this saying, hey open source, it's pretty good isn't it? 03:18.000 --> 03:24.000 And, oh, if we're going to buy something for the NHS, we'd really like it to comply with open standards. 03:24.000 --> 03:28.000 Open open open, that's all I said in every single meeting. 03:28.000 --> 03:35.000 I literally made it a point whenever I was in a meeting to go, oh, could we open source that? 03:35.000 --> 03:39.000 Or, oh, do we think we can release open data from this? 03:39.000 --> 03:41.000 Does this comply with open standards? 03:41.000 --> 03:48.000 People were probably sick to the back teeth of me being in meetings and saying open open source, open source. 03:48.000 --> 03:55.000 But the, the whole point of this exercise was I was trying to normalize the idea of what open source was. 03:55.000 --> 04:00.000 Lots of people haven't heard of it and when they do hear of it, it sounds weird, because it is weird. 04:00.000 --> 04:04.000 But by saying again and again, hey, this is what we do. 04:04.000 --> 04:05.000 Hey, we want to open source it. 04:05.000 --> 04:15.000 It just builds up people's acceptance of the fact that as an organisation wanted to be open, which was fantastic. 04:15.000 --> 04:22.000 And it led to lots of positive feedback people at conferences with blog and post on social media saying, oh, hey, look, they're going to be open. 04:22.000 --> 04:30.000 That's fantastic. And I also presented internally as well, telling everyone inside what I'm telling you now that, hey, open source. 04:30.000 --> 04:32.000 It's a pretty good idea. 04:32.000 --> 04:39.000 And the end result was this, a culture change that made being closed impossible. 04:39.000 --> 04:42.000 That's what you have to do. You have to do the work. 04:42.000 --> 04:44.000 You've got to signal intent. 04:44.000 --> 04:46.000 You've got to actually release stuff. 04:46.000 --> 04:51.000 You've got to build consensus with your peers and treat them as equals. 04:51.000 --> 05:01.000 That is how you build a culture of open source in an organisation, which I'm sure is exactly what Jack Dorsey would have said if it turned up. 05:01.000 --> 05:06.000 Look at that. You're getting two talks for the price of one here. 05:06.000 --> 05:15.000 So, by the time the pandemic struck, we were discussing, hey, look, do we need a contact tracing app? 05:15.000 --> 05:20.000 There was no discussion about whether it should be open source or not. 05:20.000 --> 05:24.000 Because, obviously, it's going to be open source. That guy, Sharon, is always talking about open source. 05:24.000 --> 05:28.000 That's what we do, isn't it? That's the culture of this organisation we release in the open. 05:28.000 --> 05:34.000 Yeah, yeah. We discussed where to host it and what license and technologies and all sorts of things. 05:34.000 --> 05:39.000 But at no point, did anyone say, hmm, are you sure about open sourcing this? 05:39.000 --> 05:44.000 The culture had changed. If you take nothing else away from this talk, it's this. 05:44.000 --> 05:52.000 You have got to talk internally and externally and make people aware that open source is the way we do things in this organisation. 05:52.000 --> 06:02.000 Don't make open source sound, esoteric and scary, even though open source is esoteric and scary, it just has to be normalized within your organisations. 06:02.000 --> 06:13.000 So, the pandemic struck all the work that I'd been doing on a changing technology stopped and I was given a new mission, 06:13.000 --> 06:24.000 which was this. It was Albert Hubbard who said that all you need to succeed is a definite plan and not enough time. 06:24.000 --> 06:33.000 Well, we had a definite plan and the plan was, on the day that people get the live app into their hands, the source code needs to be released. 06:33.000 --> 06:39.000 There we go. That's our plan. That's our deadline. Do we have enough time? No. 06:39.000 --> 06:52.000 But having a deadline was the most important factor in getting this released. If we had said, ah, you know open source is great. No, we do want to open source it, but we'll think about that when it's done. 06:52.000 --> 07:02.000 Can we just wait a bit and then, you know, we'll do it later. If we had said that, it would never have been open source. 07:02.000 --> 07:11.000 In a large organisation, a decision deferred is a decision that never gets resolved because you can always push it back later. 07:11.000 --> 07:17.000 But right from the start, we said internally, it is going to be open source. It has to be open source on this date. 07:17.000 --> 07:27.000 We also said that externally, we signalled intent. We made announcement, we blocked about it, we posted on social media. 07:27.000 --> 07:37.000 We even managed to get politicians to save the words open source to the press, which made it was imprinted in the press, which gives a huge amount of pressure internally. 07:37.000 --> 07:47.000 Because once the boss has said, yes, of course it's going to be open source. You can't backtrack on that. You need to make binding commitments. 07:47.000 --> 07:54.000 And obviously, that does increase the pressure on the team. And the pressure was absolutely enormous. 07:54.000 --> 08:06.000 And it wasn't helped by a massive shift in our plan. So this is me when this happens smiling on the outside, hollow on the inside. 08:06.000 --> 08:16.000 You've all been there. You start working on a project. It's going great. You're hitting all your KPIs, or your story points that Daniel, Gerritic, it's a falling wonderful. 08:16.000 --> 08:23.000 And then the bug happens. Something goes wrong and all of your plans go out the window. 08:23.000 --> 08:30.000 In our case, we had started developing our app long before Google and Apple had said, hey, we're going to do something around COVID. 08:30.000 --> 08:42.000 We had something out and working, but there was a bug in it. I'm going to pass you on to the press conference, given by the UK Secretary of State for Health and Social Care. 08:42.000 --> 08:51.000 And I'm going to put my mic to my laptop and let's hope this works. 08:51.000 --> 08:59.000 Because of this testing, we discovered a technical barrier that every other country building their own app is also now hitting. 08:59.000 --> 09:14.000 We found that our app works well on Android devices, but Apple software prevents iPhones being used effectively for contact tracing, unless you're using Apple's own technology. 09:14.000 --> 09:26.000 I hope you all heard that. But that's weird, isn't it? That is a prime time press conference from the Secretary of State of Health, talking about software bugs on the news. 09:26.000 --> 09:33.000 I mean, that doesn't happen in normal times. But so there was this bug. 09:33.000 --> 09:43.000 If we had carried on with our own solution, it would have been less effective than if we had switched to Google Apple, expose your notification framework. 09:43.000 --> 09:53.000 So we made this switch, which brings up this really interesting question. What does it mean when your country's health policy is controlled by Apple and Google? 09:53.000 --> 10:02.000 Now, you can argue about whether politicians have our best interests at heart, but ultimately they are democratically accountable. You can vote them out. 10:02.000 --> 10:15.000 Apple and Google are not democratically accountable. You can use freedom of information laws to get source code out of governments. You cannot get anything out of Apple and Google that they do not want you to have. 10:15.000 --> 10:25.000 The original technology that we had worked. It worked really well. We did lots of testing with it, and it was good, but it wasn't good enough. 10:25.000 --> 10:34.000 So we made a decision. We could have kept our app completely open source, completely under our control, or we could have said it needs to use this one proprietary thing. 10:34.000 --> 10:38.000 The rest of it will be open source, but it needs to use this proprietary thing. 10:38.000 --> 10:49.000 And sometimes you don't get to be ideologically pure. Ideological purity doesn't help. It never has. What you need is to get something out of the door and something that works. 10:49.000 --> 10:58.000 We had stayed 100% open source, self-sovereign. Maybe more people would have died. That is the choice we made. 10:58.000 --> 11:04.000 I don't know if it's the right choice. So what we're going to do is we're going to take a legally binding vote. 11:04.000 --> 11:15.000 So I'm going to ask you to put your hands up if we were right to use Google Apple and use their proprietary stuff, or if we should have kept things self-sovereign and 100% open. 11:15.000 --> 11:25.000 If we were right to use Google, that's quite a lot of hands, thank you, thank you. And hands up if we should say to 100% open not using anything. 11:25.000 --> 11:37.000 Yeah, I see maybe like 5% of people. As I say, I don't know if we made the right choice, but every single open source project has to make a similar choice. 11:37.000 --> 11:49.000 Do you use this proprietary library? Do you call that API, which is restricted? Do you run it on AWS or some cloud server, which isn't under your control? 11:49.000 --> 11:55.000 These are the choices you make which bind you. And these are the choices which piss people off. 11:55.000 --> 12:05.000 So I am, of course, talking about trolls. It is very easy to bully people online. It's quite fun too, apparently. 12:05.000 --> 12:17.000 In a world of lockdown where everyone was feeling tense and frustrated, lots of people turned their obsessions to us and vented all their fears at us. 12:18.000 --> 12:28.000 If you've read about the exe backdoor issue that happened last year, you will know it is terrifyingly easy to bully people in open source into doing the wrong thing. 12:28.000 --> 12:39.000 Our number one job as a community has to be to find a way to keep ourselves safe and to keep our friends and colleagues safe. 12:39.000 --> 12:44.000 Working in the open makes you a target. 12:45.000 --> 12:56.000 Working in the open for the government makes you fair game in some people's eyes. And that is not a very comfortable position. 12:56.000 --> 13:01.000 Here is a good example. So I'm going to show you a real tweet a couple of tweets. 13:01.000 --> 13:05.000 These are real. I've likely redacted the names and filled around some details. 13:05.000 --> 13:13.000 So at the bottom, someone asks a very reasonable question. Why is the UK building their own up? Perfectly reasonable question. 13:13.000 --> 13:20.000 And this guy at the top says it's the old boys network. They're all detonions. It's sarcastic. 13:20.000 --> 13:26.000 It's Dominic Cummings and Vote Leave. And it's all, that's all Bullocks. It's not true. It's rubbish. 13:26.000 --> 13:32.000 But this is the worst thing about working in the open, especially on a large project. 13:32.000 --> 13:39.000 People will just lie about you. They will make up stuff. They will push their own agenda out there. 13:39.000 --> 13:45.000 And just for the record, I didn't go to the Eaton Public School. I went to a normal comprehensive. 13:45.000 --> 13:50.000 I don't think anyone from Circa was on my team. I didn't see anyone from Vote Leave. 13:50.000 --> 13:58.000 And yeah, they wouldn't even let me email Dominic Cummings. But apart from that, 100% accurate. 13:58.000 --> 14:05.000 But paradoxically, this is the superpower of open source. Because it makes it so easy to see what's going on. 14:05.000 --> 14:11.000 When someone says, oh, the government are going to track you with GPS everywhere you go. 14:11.000 --> 14:16.000 You can say, well, look, there's a source code, mate. Show me where it does that. Oh, it doesn't. 14:16.000 --> 14:24.000 Great. Now, that is nakedly cynical. But it is effective. My fully believe the open source is a moral good. 14:24.000 --> 14:30.000 I'm public, money, public code. If the taxpayer has paid for it, the taxpayer should see the source code. 14:30.000 --> 14:38.000 But from a really cynical point of view, open source means that you can cut off a huge number of complaints by saying, look, it's there. 14:38.000 --> 14:41.000 Show me where the problem is. 14:41.000 --> 14:49.000 Now, of course, we say it's going to be open source. And not everyone was satisfied with what we did. 14:49.000 --> 14:56.000 So here's another genuine message from someone saying that they have 40 years over 40 years of software development. 14:56.000 --> 15:07.000 Fifteen years of top-level Apache project management. And they can tell the difference between open source and a PR stunt. Thanks. 15:07.000 --> 15:14.000 Here's the thing. If you work for the public sector, you can't reply in public to this. You just can't even. 15:14.000 --> 15:19.000 If you work for a large company, you, your PR team do not want you replying to this message. 15:19.000 --> 15:26.000 I know in your bio, it says, my views don't represent those of my employer, doesn't make a difference. 15:26.000 --> 15:30.000 When I blocked people like this, they started complaining to my boss. 15:31.000 --> 15:48.000 When I muted them, they instigated pylons. I get that in the early days of the pandemic, people were scared and frustrated, but it is really difficult to maintain enthusiasm for open source work when people are abusing you. 15:48.000 --> 15:55.000 I mean, I had people writing blog posts, mentioning me by name saying how evil I was for working on this project. 15:55.000 --> 16:04.000 I had people sending me really nasty direct messages saying how terrible all this was. It was absolutely awful time in my life. 16:04.000 --> 16:12.000 And I'm so incredibly grateful to the support of my wife Liz for helping me through it because it was just, it was awful. 16:12.000 --> 16:23.000 The lesson here is you have to protect your team members. You need to make sure that they know you are not listening to the rubbish being talked about them. 16:23.000 --> 16:31.000 And you need to find a way to protect yourselves. And if you see your friends getting some abuse, check in on them. 16:31.000 --> 16:40.000 Because it is heartbreaking to be pouring your heart out and building something that you hope is going to be wonderful and just to receive abuse like this. 16:40.000 --> 16:43.000 I am being really down on social media. 16:43.000 --> 16:51.000 Honestly, every message like this that we got, we had dozens which were supportive. People say, oh wow, it's open source, that's wonderful. 16:51.000 --> 17:00.000 We took screenshots of every single one of those messages, put them in a big document and email them to our boss and to the politicians and said, look, people love that we're open source. 17:00.000 --> 17:07.000 Ignore these people here, look how one look at the PR bonus that we are getting from being open source. 17:07.000 --> 17:17.000 You've got to reflect that love back up the chains so that your managers will see and understand it. 17:17.000 --> 17:25.000 Maybe, just maybe this guy has a point though, maybe. What is an open source? 17:25.000 --> 17:31.000 Well, the easiest way to tell is to look at the license. 17:31.000 --> 17:39.000 But no one will ever be happy with the license at YouTube. I would be delighted after this talk to discuss licenses with you. 17:39.000 --> 17:45.000 The only stipulation I have is you need to buy me one beer per license you wish to discuss. 17:46.000 --> 17:57.000 Internally, we had lots of discussions about what we were going to choose. And in the end, we went with the MIT license. 17:57.000 --> 18:02.000 I love the MIT. Now, why did we choose the MIT? What the first? 18:02.000 --> 18:08.000 And this is probably, I don't know if it's the most important, but other government departments use the MIT license. 18:08.000 --> 18:14.000 So we didn't have to evaluate it. We could say, look, they've chosen it. They say it's okay. Great. Move on. 18:14.000 --> 18:22.000 You do not want to be in long discussions about licenses. If you can point to someone else in your community who is already chosen that, that is half the battle. 18:22.000 --> 18:34.000 It's good enough for them. It's good enough for us. The second point about the MIT license is, like, the government wasn't trying to make money off this app. 18:34.000 --> 18:43.000 We didn't need that sort of very restrictive, very viral GPL of it. We wanted people to use the source code. 18:43.000 --> 18:47.000 So we didn't need to lock it down anyway. We just wanted to get it out. 18:47.000 --> 18:55.000 But the third, and I think definitely the most important reason why we chose MIT, is that the MIT license is short. 18:55.000 --> 19:03.000 It is really, really short. And when you are dealing with government lawyers who charge by the word, brevity is important. 19:03.000 --> 19:10.000 It means that you can show it to anyone in your community and they can read it in a minute and go, yeah, no, I get that. 19:10.000 --> 19:19.000 You cannot read a GPL in a minute. And if you take 10 minutes reading it, you still won't get it. 19:19.000 --> 19:27.000 Is the MIT license the most ideologically sound, most free software Europe approved license in the world? I don't know. 19:27.000 --> 19:32.000 But it was quick and easy for us to use and it meant that we can open source it. 19:32.000 --> 19:38.000 So I'd like to take another entirely legally binding vote. I'm going to ask you to stick your hands up if we were right to choose MIT, 19:38.000 --> 19:46.000 and then hands up if we should have chosen literally any other license. So hands up MIT was the best license for us to choose. 19:46.000 --> 19:51.000 Okay, pretty good. Hands up if we should have chosen anything else. 19:51.000 --> 20:01.000 Or I'd say about 75, 25 split there. If you would like to explain your choice of license, I'm very happy to remember one beer, one license. 20:01.000 --> 20:09.000 Thank you. So we announced, hey, it's going to be MIT, and the internet had opinions. 20:10.000 --> 20:19.000 If it was a GPL, we'd get the packet. MIT is a license of choice for people who do not understand open source and public goods. 20:19.000 --> 20:26.000 So there, all of you people who put your hands up for MIT, you're wrong. This man on the internet says so. 20:26.000 --> 20:30.000 I mean, maybe I don't understand these things only, spent half my life working on them. 20:30.000 --> 20:38.000 But it highlights a really interesting lesson. We didn't talk about why we chose MIT. We did discuss it. 20:38.000 --> 20:43.000 We had some good reasons for doing it, but we never publicly said why. 20:43.000 --> 20:53.000 And when you don't explain your decisions, people will ascribe malice and incompetence when there is at best in difference and laziness. 20:54.000 --> 21:00.000 You know, the license you choose, please, into the public's perception of what something is for. 21:00.000 --> 21:07.000 So please, if you are going to choose a license, and I recommend you do, it's for STEM, say why you've chosen it. 21:07.000 --> 21:12.000 People can disagree with you, but at least they can see your thought process. 21:12.000 --> 21:21.000 And I'll take questions later. And incidentally, it doesn't give me any pleasure to say this whatsoever. 21:21.000 --> 21:29.000 But this person is wrong. They were completely and utterly wrong, because we did release the back end and the front end and the test and the architecture diagrams and all the documentation. 21:29.000 --> 21:36.000 And yes, the MIT does let you do that. So I highly recommend that as a software license. 21:36.000 --> 21:45.000 But how do you explain what you're doing? How do you explain this, this concept of open source to normal people? 21:45.000 --> 21:51.000 I had the number 10 Downing Street Press team asked me a brilliant question. 21:51.000 --> 21:58.000 We were prepping to open source, and they said, why are you publishing two versions of the source code? 21:58.000 --> 22:07.000 What you mean? Well, you've got one version of the source code for Android, and one version of the source code for iPhone. Why are they different? 22:07.000 --> 22:14.000 That's a pretty good question, isn't it? I know. I'm very happy to turn this into a lecture on the architectural differences between those two platforms. 22:14.000 --> 22:22.000 But you need to find something that can be condensed into a paragraph, which can then be regurgitated as a single sentence in the newspapers. 22:22.000 --> 22:30.000 You need to find ways to bring people with you on this journey, so they understand what you're doing, why you're doing it. 22:30.000 --> 22:42.000 And we had to justify all sorts of things. So we had to justify why we were hosting the source code on GitHub, which is owned by Microsoft. 22:42.000 --> 22:51.000 We should have paid quite a lot of money to run and maintain our own GitHub instance, which would have been a waste of taxpayers money. 22:51.000 --> 22:59.000 These choices you will never get right, but at least if you explain them openly, you might send a chance to justify them. 22:59.000 --> 23:12.000 Look, this is incredibly vain, but I say it wasn't down to people in their bedroom. This is me in my bedroom. This is the moment that we open sourced the app on GitHub. 23:12.000 --> 23:19.000 We'd been working in a private repository, and on release date I received contradictory messages. I had my boss saying, yeah, go go release. 23:19.000 --> 23:27.000 And then they're boss saying, no, no, no, wait. And then my press team saying, where is it? We sent the press release. And then the number 10 press release team saying, no, no, you've got to wait. 23:27.000 --> 23:31.000 It was, you know, it was a bit of a mess to be honest. 23:31.000 --> 23:43.000 My lesson for you here is have a robust procedure that you have rehearsed beforehand. If you are going to release something, no who gets to say yes, we do it. 23:43.000 --> 23:55.000 So who gets to copy and paste things from GitHub and paste them into to open sourced it. It was a bit of pandemic pandemonium, but I think, you know, it was our first time doing it. So I think we can be forgiven for that. 23:55.000 --> 24:04.000 Anyway, people downloaded the app, and they could download the source code at the same time, and, you know, we released it. 24:04.000 --> 24:15.000 And now it's a wonderful, people still weren't happy. So we announced, hey, everyone, you can get the source code, get hub.com, slash whatever, slash whatever. 24:15.000 --> 24:27.000 And oh, boy, men on the internet had opinions. Again, this is, this is an absolute genuine message that we received and then had to talk to the press team about. 24:27.000 --> 24:36.000 GitHub is the shareware developers free for all programmers site. EG, the app was written by hobbyists. 24:36.000 --> 24:42.000 There you go. All of you using GitHub for professional purposes. No, you're not. This man says so. 24:42.000 --> 24:52.000 All right, it is kind of easy to laugh for people like this. But how do you explain to everyone what you are doing, why you were doing it, why you have chosen, 24:53.000 --> 25:00.000 GitHub, why open source in the first place? Internally, it wasn't a battle, but we did find ourselves explaining it again and again. 25:00.000 --> 25:10.000 Like, all the techies that we worked with get it, but we were dealing with lawyers and doctors and politicians and people who are really intelligent, excellent at their jobs, 25:10.000 --> 25:25.000 but had never heard the words open source before. So, a lesson for you is, try and find the five minute presentation you can give to everyone to bring them up to speed on what we're doing, why you're doing it. 25:25.000 --> 25:35.000 Let's take a moment to talk about history, get history specifically. So, I mentioned before, we initially developed in a closed repo and then we opened it. 25:35.000 --> 25:46.000 We obviously had hundreds of people working on this, all doing individual commits, you know, Alice committed this line of code and Bob approved it great. 25:46.000 --> 26:06.000 But we made the very difficult decision to squash our get history. Every time there was a new release of the app, we did a new full release of the source code. So, you could dip between them, but you couldn't see individually, Bob did this, Alice did that, so did that. Why did we do this? 26:06.000 --> 26:21.000 First, and I feel very uncomfortable saying this, security by obscurity. It's not great, but we were terrified that someone would slip up, they would commit a key or a piece of debug code or something, and we just wouldn't have time to fix it. 26:21.000 --> 26:31.000 We said, okay, every release, squash history, or sanitize it, make sure there's nothing bad in there, and push it out. Okay, you can disagree with me on that in a bit. 26:31.000 --> 26:46.000 The second lead was the privacy of the people working on the app. So, we, none of us want to be the main character on social media, because we left out a semi colon on line 137, or anything like that. 26:46.000 --> 26:55.000 The pressure on the team, as I said, was enormous. Do you want people saying, why isn't this person committing code at three o'clock in the morning? 26:55.000 --> 27:08.000 Proper developers are fueled by Red Bull at three o'clock in the morning. No, no, we didn't want that. We wanted to keep our people safe. We also wanted to protect their identities from state sponsored attackers or targeted fishing attacks. 27:08.000 --> 27:15.000 Things like that. So, again, we said, we will anonymise it. It will just be one big death. 27:15.000 --> 27:22.000 Again, I don't know if that's the right thing to do. I'm happy to justify it, so I'm going to ask the audience one more time. 27:22.000 --> 27:32.000 It's going to be hands up. We were right to squash the history, or hands up. We were wrong to squash and we should have been everything. So, hands up. We were right. That's very gratifying. Thank you. 27:32.000 --> 27:38.000 Hands up. We were wrong. We should have done individual deaths quite a few. I would like sort of 10-25% there. 27:38.000 --> 27:44.000 I fully respect your view on that. We can argue about it in the pub later. 27:44.000 --> 27:58.000 So, comments are a natural part of source code. I really hope you will comment your code and make it nice and easy to read. 27:58.000 --> 28:06.000 We all comment our code. Yes? Good. So, obviously, we want to leave comments in so that people can understand the app code. 28:06.000 --> 28:12.000 But the problem we had is that programmers think they're funny. 28:12.000 --> 28:23.000 We had to let them know that you mustn't put any comment in the source code that you want on the front page of a newspaper tomorrow. 28:23.000 --> 28:27.000 I mean, I would love it if the Daily Mail ran with the headline. 28:27.000 --> 28:33.000 Government insists on tabs over spaces and it's your children who are paying for it. 28:33.000 --> 28:42.000 Just keep everything calm and you have people putting in sort of joky commits and comments and saying, 28:42.000 --> 28:48.000 this bloody thing doesn't work or personal information. I need to tell Fred to fix that. 28:48.000 --> 28:52.000 We just had to go through the tedious process of snipping some of them out. 28:52.000 --> 28:58.000 But of course, the problem is that comments, even accurate comments, can be a bit misleading. 28:58.000 --> 29:04.000 So, we had a process which had a load of sub-processes and sometimes they timed out and then you had to read. 29:04.000 --> 29:10.000 This is incredibly boring. This is the comment that went with that. 29:10.000 --> 29:15.000 When the signal is received, terminate all non-responsive children. 29:15.000 --> 29:21.000 Do not log any data and spawn a YouTube run. 29:21.000 --> 29:25.000 Now, I know what they're talking about and you know what they're talking about. 29:25.000 --> 29:36.000 It doesn't, if you are seat in 5G COVID conspiracies, this doesn't look good, does it? 29:37.000 --> 29:47.000 So, there we go. There we go. How do you make your code and your comments understandable to non-coders? 29:47.000 --> 29:55.000 I mean, there is a limit to what you can do, but people are going to be reading through your code and they need to have an understanding of it. 29:55.000 --> 30:03.000 You also need to think about, if someone were to maliciously misunderstand your comments, how could you prevent that? 30:04.000 --> 30:12.000 One thing that we did was we had lots of Q&A, so people would write to their MPs, the members of parliament asking questions which got forwarded to us. 30:12.000 --> 30:22.000 We did blog posts and we had comments and we also had to get up, we were wrong get up, so we had get up issues, people could raise issues and they could also raise technical questions. 30:22.000 --> 30:30.000 Now, has anyone who has ever managed a forum knows, technical questions quickly become political questions. 30:30.000 --> 30:36.000 So, we had a code of conduct and I, unfortunately, was its ruthless enforcer. 30:36.000 --> 30:47.000 So, this is me saying, no, sorry, we're going to stop this conversation here, locked and then very occasionally saying, sorry, your band, we're not having any more of this out you go. 30:47.000 --> 30:50.000 This isn't what we want from this community. 30:51.000 --> 30:57.000 This is an impossible balance to get right, I don't just mean pruning out the spam, you know, the obviously flaming comments. 30:57.000 --> 31:05.000 Sometimes you have to go look, this is really interesting, but it is so off topic and distracting from here, we're going to stop it, take it elsewhere. 31:05.000 --> 31:12.000 This moderation is a professional skill, it is a skill I do not possess. 31:12.000 --> 31:18.000 And after a while of doing this, I went to my bosses and said, I can't do this, I am not good enough at this job. 31:18.000 --> 31:26.000 So, we bought in professional moderators who professional social media, people who could answer questions and lock comments and things. 31:26.000 --> 31:33.000 If you have a big or important repo, I thoroughly recommend getting professional moderators in. 31:33.000 --> 31:41.000 If you can, do not get the the bluntest, most forthright code that you have to answer everything and get it. 31:41.000 --> 31:47.000 For some things, it's great, but you do not necessarily want them waiting in on sensitive discussions. 31:47.000 --> 31:51.000 Again, I am being down on social media. 31:51.000 --> 32:02.000 The thing is, get hop was brilliant, it was fantastic, we got so many positive contributions, we got pull requests, we got bug reports, we had people saying, I've written to you on hack a one, I've disclosed something responsibly. 32:02.000 --> 32:13.000 We were taking screenshots of all of these and sending them to our bosses and saying, look how wonderful it is, look how brilliant open sources and then just like that it was over. 32:13.000 --> 32:21.000 One day, my phone pinged with that message and it was like a punch to the gut. 32:21.000 --> 32:25.000 They were shutting down my project. 32:25.000 --> 32:37.000 This was the biggest thing I had ever worked on, it is probably going to be the most important thing, I will ever work on and they were killing it. 32:37.000 --> 32:42.000 I think whenever you work on a project, if it's big or small, you just want it to go on forever, right? 32:42.000 --> 32:49.000 But if this up was still running today, it would have meant that we had failed. 32:50.000 --> 33:06.000 You need to lay the groundwork for stopping your project, here's another lesson, who holds the API keys, what's the contract details, who can actually switch this up, what order do we need to turn things off, how do we do a graceful shutdown? 33:06.000 --> 33:12.000 That is, I think, as important as how you start up your project. 33:12.000 --> 33:21.000 Actually, I wasn't around to see that a few months after I clicked the button to make things open source, I returned to my old job in a different team. 33:21.000 --> 33:26.000 But all the source code was open, so mission accomplished. 33:30.000 --> 33:35.000 We're never going to be truly post pandemic, there is always going to be another disease. 33:35.000 --> 33:43.000 We've got through a combination of vaccines and changed behaviour and lots of hand sanitizer, we've turned a corner. 33:43.000 --> 33:49.000 And the app was part of that, the legacy of open sourcing this app was a mixture of things. 33:49.000 --> 33:57.000 First and foremost, the app worked, it saved lives, it reduced the strain on the NHS and it continued to do so, all the time it was working. 33:57.000 --> 34:03.000 And open sourcing was part of that. Now, it wasn't the be all an end all, even I recognized that. 34:03.000 --> 34:11.000 But if it hadn't been open source, I think most people in this room would have told their friends and family, don't install that, you can't trust it. 34:11.000 --> 34:15.000 I would have told friends and family, don't install it, you can't trust it. 34:15.000 --> 34:27.000 And open source increased trust in the government and the health care service at a time when trust was low. 34:27.000 --> 34:32.000 We now know much about how we're going to deal with the next pandemic. 34:32.000 --> 34:36.000 We know what technology works, we know what measures don't work. 34:36.000 --> 34:39.000 All of us, I'm sure, have learned lessons from this horrific experience. 34:39.000 --> 34:48.000 And I hope one of the lessons that governments around the world have learned is that open source is a necessary part of any technological response. 34:48.000 --> 34:54.000 I say around the world because I've given the UK perspective, but of course, loads of countries were doing this. 34:54.000 --> 35:06.000 I spent hours on calls with people from all around the world who were going through exactly the same problems and we exchanged ideas and tips and strategies and yes we exchanged code as well. 35:06.000 --> 35:16.000 Now, as far as I can tell, these are all the national COVID tracing apps which were based as open source about about a dozen, two dozen countries there. 35:16.000 --> 35:20.000 And as you can see, they use a wide variety of technologies. 35:20.000 --> 35:31.000 So, G-A-E-N is the Google Apple, notification explanation notification framework, Bluetooth, some people went with D-P-3-T and G-P-L range. 35:31.000 --> 35:37.000 So, not everyone did the same choices we did earlier of saying yes, we were right to use Google Apple. 35:37.000 --> 35:43.000 You can see not all of them used MIT, there's lots of, you know, there's some Apache, E-U-P-L. 35:43.000 --> 35:47.000 Is there anyone from Latvia in the room? 35:47.000 --> 35:51.000 Because I would love to know why Latvia chose Creative Commons attribution share alike. 35:51.000 --> 35:57.000 For they can no shame, I will buy you a pint if you can explain that one to me. 35:57.000 --> 36:02.000 I'm going to leave you with a couple of things. 36:02.000 --> 36:12.000 So, this is a speech that the former Prime Minister, Rishi Sunak, gave just before the general election. 36:12.000 --> 36:20.000 And this is, Rishi Sunak, the then Prime Minister saying, we are pro open source, open source drives innovation. 36:20.000 --> 36:26.000 There must be a very high bar for any restrictions on open, I can't explain how weird this is 36:26.000 --> 36:32.000 to see your Prime Minister speaking about open source positively in a bid for reelection. 36:32.000 --> 36:42.000 I mean, all of this is like, we want AI to be open pro innovation pro, this is the Prime Minister saying we are pro open source and the app. 36:42.000 --> 36:45.000 I'm sure, played a large part of that. 36:45.000 --> 36:51.000 Now, in fairness, Rishi Sunak did lose that election and he did lose quite badly. 36:51.000 --> 36:58.000 But I don't think it was a Tory party stance on Linux versus Windows, which caused that defeat. 36:58.000 --> 37:05.000 And in the spirit of political neutrality, here's the current Prime Minister also talking about open source. 37:06.000 --> 37:18.000 And there's a reason, Mr. AI, on the champions of open source, have just announced a UK office too. 37:18.000 --> 37:24.000 That is the legacy of us managing to open source for COVID tracing app. 37:24.000 --> 37:31.000 The words open source, spoken at the highest, are you waving at me or someone else? 37:31.000 --> 37:37.000 Okay. That is the legacy of open source in this app. 37:37.000 --> 37:44.000 The words open source being spoken at the highest level of government and being spoken about positively. 37:44.000 --> 37:53.000 We are not relegated to page 500 of a trade treaty or we'll do a bit of open source to stay off competition concerns. 37:53.000 --> 37:59.000 This is open source being seen as a force for good as something you want to mention in an election speech. 37:59.000 --> 38:03.000 Something you want to say in a major speech about your new industrial strategy. 38:03.000 --> 38:07.000 Open source is important and it's important to government. 38:07.000 --> 38:12.000 And that legacy isn't mine, that legacy is yours. 38:12.000 --> 38:18.000 It's every single person who has been in a meeting with a politician or written to their MP and said, 38:18.000 --> 38:19.000 Why isn't this open source? 38:19.000 --> 38:23.000 It's everyone who's responded positively on social media and said, 38:23.000 --> 38:26.000 Oh, I'm so happy that this is open source, let me send a poll request. 38:26.000 --> 38:34.000 It is the legacy of everyone who has started a project on GitHub or just ranted about it to their friends and family. 38:34.000 --> 38:43.000 This is the lesson that I learned from open sourcing the UK's contact tracing app is that open source is not about code. 38:43.000 --> 38:49.000 Open source is about community and it is community that saved the day. 38:49.000 --> 38:51.000 So thank you very much indeed. 38:51.000 --> 38:52.000 Thank you very much. 38:53.000 --> 39:05.000 I thank you so much. I really appreciate that. 39:05.000 --> 39:07.000 I think we do have some time for questions. 39:07.000 --> 39:11.000 Remember, licensed questions do come with a mandatory one beer minimum. 39:11.000 --> 39:16.000 There are two beers for that person as well. 39:16.000 --> 39:20.000 Anything else I'm very happy to take questions? 39:20.000 --> 39:27.000 I don't know if we've got anyone any questions. 39:27.000 --> 39:30.000 I don't know if someone can run with a microphone. 39:30.000 --> 39:36.000 There we go. 39:36.000 --> 39:41.000 In the meantime, if anyone closer to the front has a question. 39:41.000 --> 39:44.000 I see we've got one over there. 39:44.000 --> 39:47.000 Do you want to shout out? 39:52.000 --> 40:03.000 Within public sector institutions, what can we do to support other technical professionals in open source seeing their work? 40:03.000 --> 40:05.000 That is a crucial point. 40:05.000 --> 40:08.000 Lots of people don't know how to open source their work. 40:08.000 --> 40:11.000 How to do is explain to the people working on this. 40:11.000 --> 40:17.000 Not only why we were doing it, how they should behave in public as well. 40:17.000 --> 40:21.000 One of the support we used to give, when I was in the public sector, I'm not there anymore, 40:21.000 --> 40:26.000 was we told people that if they wanted to, they could use their real name on commits, 40:26.000 --> 40:31.000 but if they didn't want to, they could make up a new GitHub profile and use that. 40:31.000 --> 40:36.000 That's a balance to get right because some people want their name against big projects 40:36.000 --> 40:37.000 to go and interview elsewhere. 40:37.000 --> 40:39.000 They can say, look, that's me. 40:39.000 --> 40:42.000 I don't want to receive any abuse for this. 40:42.000 --> 40:44.000 I want to be anonymous. 40:44.000 --> 40:48.000 A lot of the support is technical saying, okay, this is how we do it, but a lot. 40:48.000 --> 40:51.000 Any cool amount is pastoral saying, this is how we will protect you. 40:51.000 --> 40:54.000 This is how to keep yourself safe online as well. 40:54.000 --> 40:55.000 Thank you. 40:55.000 --> 40:57.000 There's a really good question. 40:57.000 --> 40:59.000 I don't know where the microphone is. 40:59.000 --> 41:00.000 It's up here. 41:01.000 --> 41:07.000 Right, I'm also prepared to discuss for the price of a bear, if any bell to variety, 41:07.000 --> 41:13.000 the MIT stuff, because I was there when that was actually decided in the broader 41:13.000 --> 41:14.000 government. 41:14.000 --> 41:20.000 No, the question I want to ask is, so I understand the reasons why we wanted to hide 41:20.000 --> 41:21.000 the exact commits. 41:21.000 --> 41:25.000 Do you think, however, not with saying that, it would have been more effective in some 41:25.000 --> 41:28.000 of these things if you'd be able to open source it from earlier in the project. 41:29.000 --> 41:31.000 We should have open source earlier in the project. 41:31.000 --> 41:34.000 In an ideal world, yes. 41:34.000 --> 41:37.000 But we weren't operating under ideal conditions. 41:37.000 --> 41:39.000 We were rushing to get something ready. 41:39.000 --> 41:43.000 We were testing it at a British military base with military personnel, 41:43.000 --> 41:47.000 and so there was some hard-coded stuff in the original version. 41:47.000 --> 41:50.000 I would have loved to have open source it literally from the beginning, 41:50.000 --> 41:53.000 but there were very practical reasons why we couldn't. 41:53.000 --> 41:56.000 The thing that I always sort of like in it, too, 41:56.000 --> 42:00.000 is if you knew that a meteor was coming to hit London, 42:00.000 --> 42:03.000 and you had to develop a meteor avoidance app, 42:03.000 --> 42:07.000 you would probably want to keep it secret until you had announced it to the public. 42:07.000 --> 42:11.000 Otherwise, it would be a bit upsetting to suddenly look going to get up and 42:11.000 --> 42:14.000 discover, oh, look, the government knows that a meteor is coming down. 42:14.000 --> 42:18.000 So in an ideal world, yes, it should have been open source from the beginning, 42:18.000 --> 42:22.000 but ideology is trumped by practicality every time. 42:23.000 --> 42:26.000 I think we would have struggled to get agreement to do it, 42:26.000 --> 42:28.000 so the compromise was it will be done, 42:28.000 --> 42:30.000 and the public announcements for it, 42:30.000 --> 42:33.000 in exchange for it being private to start with. 42:33.000 --> 42:35.000 But all that code is still available, 42:35.000 --> 42:38.000 even from the earliest one you can go on and you can look through the history 42:38.000 --> 42:40.000 back to the earliest commits. 42:40.000 --> 42:43.000 So a good compromise, I think. 42:48.000 --> 42:50.000 I don't know where the microphone is. There we go. 42:50.000 --> 42:53.000 Yeah, yeah, a really great presentation. 42:53.000 --> 42:57.000 I'm curious how you handed contributions from outside 42:57.000 --> 43:01.000 from the community and what do your experiences there? 43:01.000 --> 43:03.000 How do we get contributions from the outside? 43:03.000 --> 43:08.000 So yes, we had people raising bug reports on GitHub, 43:08.000 --> 43:10.000 which was great. That's a valid contribution. 43:10.000 --> 43:15.000 We had people doing UI changes and things in pull requests. 43:15.000 --> 43:19.000 We did accept some pull requests, 43:19.000 --> 43:22.000 but because of, you know, 43:22.000 --> 43:25.000 if you're working on something which is sensitive for the government, 43:25.000 --> 43:28.000 sometimes you do want to do my background checks on your developers 43:28.000 --> 43:32.000 and things, so we didn't accept all of them. 43:32.000 --> 43:37.000 What we did have was I think it was Hacker 1 for responsible disclosure, 43:37.000 --> 43:41.000 and so we accepted a fair number of responsible disclosure, 43:41.000 --> 43:46.000 which also came with patches and fixes there. 43:46.000 --> 43:49.000 Someone wrote a wonderful blog post. 43:49.000 --> 43:53.000 They submitted a pretty good security bug to us. 43:53.000 --> 43:55.000 And we said, thank you very much and they went, 43:55.000 --> 43:57.000 can I publish it? 43:57.000 --> 43:59.000 Yeah, just let us push out the new version first. 43:59.000 --> 44:02.000 We pushed out the new version and a few days later, 44:02.000 --> 44:03.000 they wrote a blog post saying, 44:03.000 --> 44:07.000 and it was really easy to contribute to a UK government open source thing. 44:07.000 --> 44:11.000 Yes, so that got screen shot and sent up as well. 44:11.000 --> 44:13.000 Thank you. 44:13.000 --> 44:14.000 Hi. 44:14.000 --> 44:15.000 Yeah. 44:15.000 --> 44:18.000 And there's a few saying about the government kind of like talking 44:18.000 --> 44:21.000 positive about open source was crazy cool. 44:21.000 --> 44:24.000 And I was kind of wondering if you worry that that could fade with time 44:24.000 --> 44:27.000 as like governments, interesting things, sometimes can. 44:27.000 --> 44:32.000 And do you have any kind of stuff in mind for keeping it in their conscience 44:32.000 --> 44:35.000 and making sure that this stays like goings down? 44:35.000 --> 44:37.000 Fantastic question. 44:37.000 --> 44:40.000 I mean, the UK government, at least, is staff, 44:40.000 --> 44:42.000 with lots of people who get open source, 44:42.000 --> 44:46.000 there's no longer work there, but they are pushing heavily. 44:46.000 --> 44:49.000 What I would say to all of you is, 44:49.000 --> 44:53.000 you've got to be the change that you want to see. 44:53.000 --> 44:56.000 So, on one hand, star government repose on GitHub 44:56.000 --> 44:59.000 and simple requests, right to your members of Parliament 44:59.000 --> 45:01.000 or other elected officials and say, how happy you are 45:01.000 --> 45:04.000 or say you really want this thing to be open source. 45:04.000 --> 45:05.000 You've got to put the pressure on. 45:05.000 --> 45:07.000 But I would also say, apply for jobs. 45:07.000 --> 45:09.000 Go in there, do it yourself. 45:09.000 --> 45:11.000 You know, the best way to make a change. 45:11.000 --> 45:14.000 I mean, there's that meme of the person who gets a job, 45:14.000 --> 45:16.000 does a one line fix and then leaves, 45:16.000 --> 45:17.000 because how finally it's fixed. 45:17.000 --> 45:20.000 I wouldn't go that far, but go in, 45:20.000 --> 45:23.000 and you take for open source externally and internally. 45:23.000 --> 45:26.000 That's probably the best way to keep this momentum up. 45:26.000 --> 45:28.000 I've only got five minutes left, 45:28.000 --> 45:30.000 so we'll take a couple more, 45:30.000 --> 45:33.000 and then get set up for the next speaker. 45:33.000 --> 45:34.000 Hi. 45:34.000 --> 45:38.000 Thank you for your talk. 45:38.000 --> 45:41.000 I had a question regarding the development 45:41.000 --> 45:43.000 of applications in times of need, 45:43.000 --> 45:46.000 so in times of really, 45:46.000 --> 45:49.000 where you immediately need to release an application 45:49.000 --> 45:50.000 like you did for a COVID. 45:50.000 --> 45:53.000 I'm guessing this is a situation that's going to be more frequent 45:53.000 --> 45:57.000 in the future, not necessarily due to COVID or pandemics, 45:57.000 --> 45:59.000 but also due to other reasons, 45:59.000 --> 46:01.000 like, for example, the kind of change, 46:01.000 --> 46:04.000 and you had the sore ear in, 46:04.000 --> 46:06.000 and great bit not too long ago. 46:06.000 --> 46:08.000 You had a tornado in the coronal and so on. 46:08.000 --> 46:10.000 So I can't quite hear you. 46:10.000 --> 46:14.000 So are you talking about how rapidly we updated the code? 46:14.000 --> 46:15.000 Sorry, I meant, 46:15.000 --> 46:20.000 do you have suggestions for application developers 46:20.000 --> 46:26.000 or even for the public sector to have the habits of deploying 46:26.000 --> 46:30.000 very fast applications in times of need? 46:30.000 --> 46:31.000 Yeah. 46:31.000 --> 46:34.000 The best thing for people working in the public sector 46:34.000 --> 46:37.000 to get things out in times of need is to, 46:37.000 --> 46:38.000 well, there's two things. 46:38.000 --> 46:40.000 One is have good frameworks, 46:40.000 --> 46:42.000 so the UK government and the UK health service 46:42.000 --> 46:45.000 have a design framework and design language, 46:45.000 --> 46:47.000 which means that you couldn't sort of drag and drop 46:47.000 --> 46:51.000 an app prototype and get it looking right. 46:51.000 --> 46:54.000 And then the second thing is to practice. 46:54.000 --> 46:57.000 At some point, you will almost certainly want to run 46:57.000 --> 46:58.000 an exercise of, 46:58.000 --> 47:01.000 let's imagine that the prime minister has announced 47:01.000 --> 47:05.000 that we all need to have an app to track how much 47:05.000 --> 47:07.000 swans are in our garden. 47:07.000 --> 47:09.000 Great, how quickly could we do that? 47:09.000 --> 47:12.000 And just do a sprint of how quickly can we develop something? 47:12.000 --> 47:14.000 So I think it's that. 47:14.000 --> 47:16.000 Make sure you've got the tools available. 47:16.000 --> 47:18.000 Make sure you know how to use them. 47:18.000 --> 47:20.000 I think that's time for one more question. 47:20.000 --> 47:22.000 If there is one. 47:24.000 --> 47:26.000 If you were doing it all again, 47:26.000 --> 47:27.000 what would you have done differently? 47:27.000 --> 47:29.000 If I could do it all again, 47:29.000 --> 47:31.000 I wouldn't have taken the job in the first place. 47:33.000 --> 47:36.000 It was stressful. 47:36.000 --> 47:38.000 It was annoying. 47:38.000 --> 47:42.000 It was far too much pressure for the entire team. 47:42.000 --> 47:43.000 And it was, you know, 47:43.000 --> 47:46.000 it is the best thing that I've ever done. 47:46.000 --> 47:48.000 I think I'm. 47:50.000 --> 47:52.000 What would I have done differently? 47:52.000 --> 47:53.000 Nothing. 47:53.000 --> 47:56.000 I mean, we should have telegraphed and we should have discussed 47:57.000 --> 48:00.000 our reasoning, especially behind the licenses a bit more. 48:00.000 --> 48:01.000 But you know what? 48:01.000 --> 48:05.000 We got various high level government ministers to say the words open source 48:05.000 --> 48:08.000 in press conferences and have it reported in the news. 48:08.000 --> 48:11.000 We got the app out by the time everyone had it in their hands. 48:11.000 --> 48:13.000 We kept open sourcing it. 48:13.000 --> 48:15.000 Every time there was a new release. 48:15.000 --> 48:17.000 We dealt with all the trolls. 48:17.000 --> 48:20.000 We dealt with people contributing. 48:20.000 --> 48:23.000 You know, if I wouldn't live through it all again, 48:23.000 --> 48:26.000 but I don't think I would have changed anything. 48:26.000 --> 48:28.000 Anyway, thank you so much for coming to my talk. 48:28.000 --> 48:29.000 Thank you so much. 48:29.000 --> 48:30.000 Cheers.