WEBVTT 00:00.000 --> 00:10.560 Hi, I'm Caroline, I'll introduce myself in a second, this is Antonia, and this is our talk 00:10.560 --> 00:13.360 design in five minutes, okay, 20 minutes. 00:13.360 --> 00:18.160 We're going to be covering ecosystem mapping and user research, or really say to be here. 00:18.160 --> 00:22.480 We did design this talk for folks that are new to design, but if you have a long experience 00:22.480 --> 00:29.120 with design, I think we have some really interesting case studies for y'all, so yeah. 00:29.120 --> 00:32.560 I'm Caroline Senders, I'm the founder and principal designer of 00:32.560 --> 00:39.040 communication, research and design, and I'm Antonia Valencia, the design researcher in 00:39.040 --> 00:42.880 communication, research and design. 00:42.880 --> 00:48.880 So today we're here to talk about open source projects, and many times when we think of open 00:48.880 --> 00:54.720 source projects, this can be or seem a little bit like a multiplayer game, so there's 00:54.800 --> 01:01.520 many contributors, everyone is participating, and it's sometimes beautiful, sometimes very chaotic. 01:03.760 --> 01:10.160 You fix one bug, another one appears, somebody does a request at 3 a.m., but it's kind of the magic, right? 01:11.920 --> 01:17.120 But for your project to really thrive in this ecosystem, you really need to understand 01:17.120 --> 01:19.760 the broader picture, so where does it actually fit? 01:20.480 --> 01:26.720 So again, an open source project really exists in a vacuum. 01:26.720 --> 01:33.200 Everything is connected, you're building tools, libraries, platforms, you're contributing 01:33.200 --> 01:39.120 to different web or other projects, technologies, and collaborating with people. 01:40.160 --> 01:48.880 So design really has this capability of building bridges, so knowing where your project fits, 01:48.960 --> 01:54.880 within this nice ecosystem, it's not something, it's not like a nice to have, but it's something 01:54.880 --> 02:02.320 essential. Understanding where it fits, how you can collaborate, and the perspective of design 02:02.320 --> 02:10.160 is very important for this building of the ecosystem mapping. So it's going to become let's hope 02:10.160 --> 02:20.080 one of a key element in your design process. So we like to think of ecosystem mapping as a 02:20.080 --> 02:26.720 calling of an API between your project and the environment that it actually exists inside. 02:27.760 --> 02:33.200 So what are the dependencies? How does it connect to who are the users, the contributors, the 02:34.080 --> 02:41.840 monitors, this is something key. So how do humans actually interact within this whole system? 02:42.880 --> 02:49.280 So building this map of connections and identifying the relationships will actually sit 02:49.280 --> 02:56.240 to a year project within this ecosystem that surrounds it, and it's also going to help you find potential 02:56.240 --> 03:03.760 collaborators, competitors, even, and hopefully understand the interactions of use of your 03:03.760 --> 03:10.160 users and of everyone around it. So now let's dive into the things that you can do right now, 03:10.160 --> 03:16.960 that is going to be mapped out your ecosystem, and also try to debug this human experience and 03:16.960 --> 03:28.800 interactions. And then now we're thinking about what actually does it take to do ecosystem 03:28.800 --> 03:34.400 mapping. So this is where I think it's important to define this opening objectives of your project. 03:34.400 --> 03:38.880 A lot of you probably do that, but it's always good to write it out explicitly for the entire team 03:38.880 --> 03:43.760 you're working with. I think the next step is important to then figure out who are the key stakeholders 03:43.840 --> 03:47.760 and communities that are either going to be involved in the project or will utilize it. 03:49.440 --> 03:54.480 Then I think it's important to analyze the interaction, and by that is how people engage with it, 03:54.480 --> 03:59.360 but also what are potential interactions with similar projects with dissimilar projects? 03:59.360 --> 04:04.480 Where do you sort of sit in a space of let's say the entire community? Where are their places to 04:04.480 --> 04:08.480 leverage collaborations, and where are there also potentials to sort of say like actually even though 04:08.480 --> 04:13.680 this seems really similar to us, we're actually doing this very very different kind of thing, 04:13.680 --> 04:19.200 even if it's on let's say like it's a similar programming language or similar again type of project, 04:19.840 --> 04:24.880 and then lastly documenting findings and insights for your team. So sort of taking all this together 04:24.880 --> 04:29.680 and presenting it back to think about how do we position ourselves not only inside of an ecosystem, 04:29.680 --> 04:34.880 but also how do we talk about with our community where we think the project also fits? 04:35.840 --> 04:43.280 And we're going to get a little more into this. So these next steps are going to focus on 04:44.080 --> 04:48.000 it's almost like a deeper dive into steps two and three. Mapping just isn't about 04:48.000 --> 04:54.400 coder dependencies, it's also very much about people. And thinking about people is this experience of 04:54.400 --> 04:59.280 thinking about these invisible bugs. Why aren't new contributors contributing or sticking around? 04:59.280 --> 05:03.520 What are the barriers they're facing to engaging with a project? How do you balance sort of 05:03.520 --> 05:09.120 onboarding let's say new community members who have a variety of different backgrounds or skills? 05:09.120 --> 05:13.920 Does your project have space for less technical and more technical folks? How do you want to 05:13.920 --> 05:19.200 engage with them? How would you like them to engage with the project? What are barriers for people 05:19.200 --> 05:24.880 getting understanding the project? Is it just language or is it how to actually find let's say 05:24.880 --> 05:30.320 materials about the project? I think all these things are really really important and this is also 05:30.320 --> 05:36.160 where we're going to get into user research to understand your community more or understand 05:36.160 --> 05:44.080 who's actually using the project. So user research tips and tricks. So what is user research? 05:44.080 --> 05:48.560 Something we could call qualitative research, are like to think of it as talking to people or 05:48.560 --> 05:54.240 engaging with your community. I'm involved things like interviews, focus groups, surveys, 05:54.240 --> 05:58.640 and some cases you might do something called diary studies where folks will be sort of writing 05:58.640 --> 06:04.080 down how they're engaging with something on a day-to-day basis, but also uses a variety of other 06:04.080 --> 06:13.360 tools. And so here's what I kind of or what we sort of think is the the bearist of bones 06:13.360 --> 06:20.000 of user research. And it also I think helps to define like why you would do it. So it helps us 06:20.000 --> 06:27.680 identify a problem or focus area to outline what it is you're doing. It helps us understand why 06:27.760 --> 06:32.320 I like to often say is how do you know how do you know that this is a problem. An answer could be what 06:32.320 --> 06:38.480 we saw people complaining on GitHub or to our volunteers or we just saw things online across a variety 06:38.480 --> 06:44.480 of spaces. And so then we spoke to a few people that complained or had thoughts. This is a great 06:44.480 --> 06:49.040 way to sort of validate how do you know. But it's also the one I think one the most important 06:49.040 --> 06:54.240 things. There's a problem how do you know it's a problem where did it come from? And then the next 06:54.240 --> 07:01.600 step is responding to that problem to create a goal. And what I mean by goal is like what what 07:01.600 --> 07:06.720 are the next steps? What would you like to solve? How do you start to distill that into let's say 07:06.720 --> 07:14.240 an area to focus like a question or just really sort of narrowing narrowing that down. So a goal can 07:14.240 --> 07:18.880 be like we know that users keep repeating the same mistake. Maybe it's maybe when they're engaging 07:18.880 --> 07:25.440 with certain API using too large of a dataset. Why are they doing that? And you know then it could 07:25.440 --> 07:31.200 be and we know this because we got X complaints. So the goal might be we need to learn or understand 07:31.200 --> 07:36.240 better how this mistake keeps sort of refracting inside the community over and over again. Our goal 07:36.240 --> 07:42.000 is to understand more and try to come up with a solution. Then the last thing is a plan. 07:42.960 --> 07:48.560 Or not the last things. Sorry, but then you can create a plan towards actually engaging with 07:48.560 --> 07:53.520 users and interviewing them. That could be we decided to run a short survey or we decided to speak 07:53.520 --> 07:59.760 to like 20 users, some who are much older users who are maybe higher impact contributors 07:59.760 --> 08:04.480 and newer users. What are their experiences and how do they differ? How does someone have a lot 08:04.480 --> 08:11.440 more knowledge about a project than than others, right? Then it's from that plan synthesizing results. 08:11.520 --> 08:16.000 What did you find? What are the patterns? What are the patterns tell us? And so then 08:16.000 --> 08:22.160 instead of taking all of these things for next steps to think about what a solution is. So validate 08:22.160 --> 08:27.680 findings, right? Say this is our thought of next steps of our solution. What does the community think? 08:27.680 --> 08:31.600 Ask for feedback. And then so on and so forth. 08:31.600 --> 08:44.720 All right, so now that we got a first look at user research, we go back into ecosystem mapping. 08:44.720 --> 08:50.720 So this really comes into play when you start understanding who is around your project, 08:50.720 --> 08:55.440 who are the users that are going to be interacting with it. So here are the tips and tricks 08:55.440 --> 09:03.200 for ecosystem mapping. To make this ecosystem thinking part of your process, you first need to start 09:03.200 --> 09:11.600 mapping early. So don't wait until your project has so much complexity that is going to be hard 09:11.600 --> 09:18.640 to jot it down on a piece of paper or a map or whatever means you end up using. Then you can keep 09:18.720 --> 09:25.040 it simple. So you can even do a hand drawn map to try to put down these interactions. 09:25.840 --> 09:31.920 Then share and update these maps. It's a live document. It's never going to be a static piece. 09:32.560 --> 09:38.000 And finally involve your community. So mapping is always better done with different perspectives, 09:38.000 --> 09:46.080 different opinions, and start doing all of this collaboratively. So why ecosystem mapping? So 09:47.040 --> 09:52.960 this tool really helps you understand what else is out there, what is similar, what is 09:52.960 --> 09:59.040 this similar, what makes your project stand out between the rest. It also helps you understand 09:59.040 --> 10:04.720 challenges and that other projects have already undergone and how they really took a particular 10:04.720 --> 10:12.320 view or a focus. It also helps you visualize, so visualize the gaps in this ecosystem and find 10:12.320 --> 10:18.720 the opportunities where your project or your community could try to help fill and it allows you to 10:18.720 --> 10:25.120 understand and explain the why and why this project in particular, whether the characteristics 10:25.120 --> 10:32.320 that make it different from the rest. And finally it really allows you to engage in a companionship 10:32.320 --> 10:38.560 in working together to build this tool and building your project in a more wide view. 10:39.520 --> 10:46.160 So to further understand this tool, we're going to present two case studies. The first one of them 10:46.160 --> 10:52.640 is the Red Decoratia Hital, which is a project done with school teachers in Puerto Williams, 10:52.640 --> 10:59.360 Chile, together with the Favla Baustral, which is the Favla in this island in Chile. 11:00.640 --> 11:06.160 And this project was situated in a very particular context, first of all, it was in a remote 11:06.160 --> 11:13.600 island, at the end of Chile, in the southernmost Favla been the world. So it has very particular 11:13.600 --> 11:19.440 characteristics. This island is very small but has like very different stakeholders inside of the 11:19.440 --> 11:27.120 island. And we use this ecosystem mapping tool to try to comprehend the context around technological 11:27.120 --> 11:33.440 education. So what are the gaps and opportunities to create this more profound implementation 11:33.440 --> 11:42.720 of these skills? So the subject of technological education is immensely large and it is a domain 11:42.720 --> 11:48.160 that we wanted to get our hands in, but this tool really helped us that the ecosystem tool really 11:48.160 --> 11:54.640 helped us to guide us in which direction to actually take. So the first insight that we found 11:54.640 --> 12:02.400 is that we were going to work directly with the educators. So by doing this, it would enhance 12:02.480 --> 12:08.640 their understanding of the technological language, the tools, and thus they would take these skills 12:08.640 --> 12:14.880 to then design the curriculum to take to the classroom. This also allowed us to onboard different 12:14.880 --> 12:22.240 educators from the island that in this particular case, this only one school and one pre school 12:22.240 --> 12:27.680 establishment. So getting together with a couple of teachers would make the rest of them start 12:27.680 --> 12:32.080 getting involved and onboarding themselves to the project, inviting others to collaborate. So 12:32.080 --> 12:40.160 also creating a community between them. And after this, we also thought that we wanted to identify 12:42.240 --> 12:47.280 different hubs and technology clubs around Chile, being a very long country and them being 12:47.280 --> 12:52.960 more in isolation. We started collaborating with different of these actors around the country 12:52.960 --> 13:00.960 and also introducing this technological hub that we are trying to create in this broader network. 13:00.960 --> 13:05.600 So at the end of the day, a ecosystem mapping really allowed us to define the scope, 13:05.600 --> 13:10.480 identify the users, the collaborators, and also to visualize these gaps that we had found 13:10.480 --> 13:16.480 to create the outputs and actual implementation of the project. Great. Thank you. 13:17.520 --> 13:21.600 I'm going to talk about another project that I worked on. So I used to be a design researcher at 13:21.600 --> 13:27.600 the Wikimedia Foundation on the Anti-Arassment Team, which is a sub unit of the trust and safety 13:27.600 --> 13:35.200 team. So one of the things I worked on there was called the Interaction Timeline. So this project 13:35.200 --> 13:39.680 I'm going to talk about covers both ecosystem mapping and user research, because Wikimedia itself 13:39.680 --> 13:44.560 isn't incredibly large ecosystem. We're talking hundreds of different language wikis, 13:44.560 --> 13:50.880 Wikimedia Commons, things called village pump, etc. These all serve exist inside of this one, one space. 13:51.280 --> 13:59.040 And one of the things that we actually utilize inside of the Wikimedia community, something 13:59.040 --> 14:04.400 at the time we called the Wikimedia Community Wishlist. I mean, have y'all heard of this. 14:05.440 --> 14:09.920 So the community wishlist was a way that across the year different community members could list 14:09.920 --> 14:15.360 things that they would like to see. The team I worked on there specifically focused on. I was 14:15.360 --> 14:22.000 on a team that was called like community and building. So that would be like if you saw a bug on 14:22.000 --> 14:25.200 something you really wanted to fix, if there was a new feature you'd like to see, or if there's 14:25.200 --> 14:30.800 something that we should focus on. And then we would look at how people had sort of voted or ranked 14:30.800 --> 14:35.360 all the things that ended up on the wishlist. We would go internally to our team and sort of talk 14:35.360 --> 14:41.920 about, okay, well, how difficult will it be to implement this one thing if it's really easy, 14:42.000 --> 14:47.680 even if it's lower on the wishlist, we are going to sort of like maybe prioritize building that 14:47.680 --> 14:52.000 really early on so take a short amount of time. Other things might be very, very important but 14:52.000 --> 14:55.920 might take many years. And so what we would do is we would then present back to the community, 14:55.920 --> 15:00.400 the wishlist and say, here's the order of operations we're going to go in and then here's our 15:00.400 --> 15:04.640 rationale. So this one thing you really want, we're going to take it back to like the board of Wikimedia 15:04.640 --> 15:08.640 and talk about how we actually need a fundraise for this thing, but it's maybe not something we 15:08.640 --> 15:14.080 can build this year. So the reason that that matters in the context of this thing called the 15:14.080 --> 15:19.360 interaction timeline is we get a lot of requests about updating tools and solving problems. 15:19.360 --> 15:23.600 And one of the things at the time that we were dealing with and Wikimedia is still dealing with is a 15:23.600 --> 15:28.160 well-known toxicity and harassment problem. Now if you're not an editor, you might not know about this, 15:28.160 --> 15:33.440 but this is something a lot of editors deal with on a day-to-day basis is sort of growing toxicity 15:33.440 --> 15:38.320 and say the community. So this is something we saw pop up again and again on the wishlist. 15:38.320 --> 15:43.200 And so we were wondering, is there some kind of tool or intervention that we could engage with, 15:43.200 --> 15:47.440 for example, of social networks like Reddit or Twitter, have a variety of what we call trust 15:47.440 --> 15:52.080 and safety tools that help them analyze and spot patterns inside of things like toxicity. 15:52.720 --> 15:57.680 And so we were thinking, is that something we can do? One of the things Wikimedia 15:58.000 --> 16:04.240 has, so we don't have moderators, we have editors. We have special roles that editors can hold 16:04.240 --> 16:08.240 called administrators. And those are actually the folks that when there is, let's say, 16:08.240 --> 16:12.560 some kind of disagreement inside of Wikimedia, the administrators will step in. 16:12.560 --> 16:16.400 Then there's a special committee called the arbitration committee when different kinds of 16:16.400 --> 16:21.680 harassment cases sort of keep on going and ongoing. You have to bring in this very special group. 16:21.680 --> 16:27.760 We like to sort of describe them as the Supreme Court of the Wikimedia of English language 16:27.760 --> 16:35.840 Wikipedia. And the reason that that matters is so when I, when I worked at the Wikimedia 16:35.840 --> 16:41.840 Foundation in 2018, we knew that, again, the harassment was this really big issue. 16:43.120 --> 16:47.120 And we had seen sort of this request again for something to be, to be done about it. 16:47.520 --> 16:54.480 So we sort of utilized requests from the community wish list as well as just conversation 16:54.480 --> 16:59.680 that we had picked up from different Wikimedia conferences to sort of go back to the community 16:59.680 --> 17:03.440 and say, this is something we're going to focus on. Is this something you think we should focus on? 17:03.440 --> 17:09.840 And we got a lot of, like, yes. But then what do we focus on? This is where user research comes in. 17:10.480 --> 17:18.400 So we did a variety of surveys and quality of interviews with all different kinds of community 17:18.400 --> 17:24.000 members. So folks that ran specific meetups related to English English Wikipedia like women and 17:24.000 --> 17:30.560 red, which is an initiative to get more female scientists inside of the Encyclopedia. 17:31.680 --> 17:37.280 Also with folks that had faced high amounts of harassment, as well as different members of 17:37.280 --> 17:42.240 administrators in the arbitration committee or our comp as we call it. 17:43.040 --> 17:52.400 We ran these two surveys and also, again, held in-person focus groups and in, like, 101 interviews 17:52.400 --> 17:58.960 at events. And so we had to sort of then present all this back to the community. All in all, 17:58.960 --> 18:04.800 I think we worked with or spoke to over 400 contributors. This also includes our surveys. We weren't, 18:04.800 --> 18:09.760 you know, one to one interviewing, 400 people. But again, this really matters. And one of the things 18:09.760 --> 18:19.040 we kept hearing are one of the things we started to realize is that there might be a space for 18:19.040 --> 18:27.520 making a tool to alleviate certain problems we were having. And tool changes and prototyping might 18:27.520 --> 18:32.640 go a lot faster than us trying to change policy while we work towards changing or thinking about policy. 18:32.640 --> 18:40.480 One of the things we kept hearing from these surveys and speaking to the administrators is this 18:40.480 --> 18:46.480 thing called sock puppeting. So sock puppeting is a major harm. It's where you sort of pretend 18:46.480 --> 18:54.320 to be a user, which you're not allowed to do. And so that was one thing that popped up that was 18:54.320 --> 19:02.000 a big concern. This is something where let's say someone who works for a film production company 19:02.000 --> 19:06.960 will write a page, like a wiki page about a movie. You're not allowed to do that. You're not allowed 19:06.960 --> 19:14.640 to be paid to write wiki pages. So we'd also see people who had been banned from one of the, 19:14.640 --> 19:20.480 like, like, from let's say English English Wikipedia coming back and creating a new account 19:20.480 --> 19:25.520 and then going and engaging further forms of toxicity. So we knew that this was a major problem. 19:25.520 --> 19:30.000 We also knew that something called wiki-hounding, which is digital stocking, was also a problem. 19:32.480 --> 19:39.280 And so from our user research, we suddenly could isolate right these two major issues that we 19:39.280 --> 19:46.320 had to do something about. This helped us provide, again, examples. Like, we could start creating 19:46.320 --> 19:50.400 these anonymous examples to better understand harassment, also presented back to the community. 19:51.040 --> 19:55.200 But from sock puppeting and wiki-hounding, the reason I bring those up is we started to realize 19:55.200 --> 20:00.800 that there might actually be something we could use that could help our community administrators, 20:01.520 --> 20:07.760 which is trying to visualize the activity of a sock puppet account and potentially visualize 20:07.760 --> 20:13.360 the activity of someone engaging in wiki-hounding or digital stocking. Because a lot of what we learned 20:13.360 --> 20:19.520 to the metrics of those accounts would be doing is they would be editing or changing edits, 20:19.520 --> 20:24.560 or, again, writing and doing something to engage with another user. And all this actually is 20:24.560 --> 20:31.120 trackable inside of the wiki-mediate projects. So one of the things the administrators mentioned 20:31.120 --> 20:37.600 over and over again was not being sure of what a sock puppet account was doing of like how much 20:37.600 --> 20:42.880 damage let's say they had done inside the encyclopedia, like how could they understand to build a 20:42.880 --> 20:49.280 case to then say like maybe we should ban this account. Same with wiki-hounding. How much like how long 20:49.280 --> 20:54.880 has this been going on? How do we know that it's happening? This sort of came up again and again 20:54.880 --> 21:00.720 and again in our findings. And so we create this tool that I'll quickly show you now the 21:00.720 --> 21:06.080 back to this slide. So we create this tool called the interaction timeline. And the way that we 21:06.080 --> 21:12.400 also decide to create this tool is by engaging in ecosystem mapping to see what kinds of other tools 21:12.720 --> 21:19.360 existed. And it turns out that an editor seven years prior to my team in 2018 starting this 21:19.360 --> 21:26.000 head built a tool called the interaction timeline. But it didn't have a visual interface to it. 21:26.000 --> 21:31.200 So all you could see was like line line line line line line line. And it hadn't been maintained. 21:31.920 --> 21:36.240 So one of the things we thought was maybe this is something that could actually be helpful. 21:36.240 --> 21:41.120 We created a series of wire frames. We tested with different members who are administrators 21:41.120 --> 21:45.680 and are upon to see what they would think. And we also tested with high impact editors who 21:45.680 --> 21:51.120 did not hold those roles. We actually got a lot of really great feedback. One thing I will say is we 21:51.120 --> 21:58.800 design this for the wiki media community. So a big thing was can you have there be less space? 21:59.840 --> 22:06.480 Can you have there be like less color, meaning like they like things very condensed and sort of 22:06.480 --> 22:12.240 close together? We're like, yeah, we can do that. Why not? So then it's sort of going back to this. 22:12.800 --> 22:19.200 This is the success of the tool was very action specific. It's one small tool inside of a 22:19.200 --> 22:24.800 large ecosystem of problems. The goal is not like we cannot fix harassment with one tool and with the 22:24.800 --> 22:30.240 action of like one piece of software. But it was something we knew that would alleviate the pain 22:30.240 --> 22:34.400 administrators were facing. Because to report her harassment side of wiki media, there's not 22:34.480 --> 22:39.840 a reporting tool. You go to a public page called admins no sports slash incidents and you have to 22:39.840 --> 22:47.280 write what is happening. And admins don't have to go and track down what is happening between 22:47.280 --> 22:52.960 two users. And this tool ended up has been very integral and those administrators to see what 22:52.960 --> 22:58.880 happens. Because the administrators are the ones that weigh in and make a decision on any kind of action. 22:58.880 --> 23:05.840 So if they ban someone from interacting with another user, they have to back up why? Because it's 23:05.840 --> 23:13.040 wicked media. You have to cite your sources. So anyway, we're at time when we're going to quickly 23:13.040 --> 23:20.240 say this really quickly. How did this sort of use both user research and ecosystem mapping? 23:20.240 --> 23:24.960 The user research helped us realize we had a problem toxicity and helped us to refine 23:24.960 --> 23:30.480 actually very specific nuances to that problem. A lot of this sort of here and see what 23:30.480 --> 23:34.480 what data data administrators were facing and understanding some of the major issues they had, 23:34.480 --> 23:39.360 which was admins no sports slash incidents. That was very toxic place. It was like the place that 23:39.360 --> 23:44.480 they had to engage in. And what they needed was help in understanding the cases. And this is where 23:44.480 --> 23:49.920 this tool came in. We also looked at other tools inside of the trust and safety world to see if 23:49.920 --> 23:56.080 there was some other kind of tool we could use. So we also looked outside of outside of the 23:56.080 --> 24:00.560 with media ecosystem. But we realized that a tool that an editor made was the best one. 24:00.560 --> 24:04.480 And also for us, the reason that we also wanted to work on that tool maintained that tools 24:04.480 --> 24:09.280 because it came from the community. It was designed for our community. And so we went and asked 24:09.280 --> 24:13.920 that editor, do you mind if we update this? And he said yes of course. So then now it's something 24:13.920 --> 24:19.280 that's updated and maintained by the foundation, by a specific team. So we know that it's continually 24:19.280 --> 24:26.480 maintained and updated. And one of the things I'll say really quickly is, regardless of your project, 24:26.480 --> 24:31.600 or product or community, it's always worth thinking about consistent ways for users to get feedback 24:31.600 --> 24:35.360 and interact with you. And that's a big thing we had at Wikimedia. We had a very special page 24:35.360 --> 24:40.400 for our team. We had a community liaison who sat on our team. So you could always leave comments 24:40.400 --> 24:46.240 or email us. And we shared everything in real time, also on wiki. So people could provide 24:46.320 --> 24:51.200 real feedback, also with a suggestion of please email us. We'd also set up one-on-one interviews. 24:52.880 --> 25:00.080 So thank you. Look out for patients.