1
00:00:03.580 --> 00:00:05.839
Liam Wyatt: All right. Hello, everyone.
2
00:00:05.890 --> 00:00:09.040
Liam Wyatt: My name is Liam Wyatt. I will be
3
00:00:09.100 --> 00:00:15.159
Liam Wyatt: hosting this session. This is the Wikimedia Foundation. Future audiences. Discussion
4
00:00:15.440 --> 00:00:17.430
Liam Wyatt: approximately held monthly
5
00:00:17.660 --> 00:00:24.930
Liam Wyatt: and we'll be uploaded to Wikimedia Commons very soon after the call. Today is the fifteenth of February, 2,024,
6
00:00:25.160 --> 00:00:29.069
Liam Wyatt: and we have an agenda on the on the screen here which I'll read out.
7
00:00:30.370 --> 00:00:36.659
Liam Wyatt: Welcome what we're doing here, what is future audiences and what is happening past and future.
8
00:00:36.770 --> 00:00:39.669
Liam Wyatt: Then 2 major software development
9
00:00:39.680 --> 00:00:43.220
Liam Wyatt: reports, one about Chat Gpt Plugin.
10
00:00:43.440 --> 00:00:49.310
Liam Wyatt: It's a conclusion. And what happens next? Hint? It's called citation needed.
11
00:00:49.580 --> 00:00:53.840
Liam Wyatt: and we'll be hearing from that from Mike. So Mariana and Mike
12
00:00:54.100 --> 00:01:08.839
Liam Wyatt: followed by open QA. We. If there are any questions between each of these sections, please ask them then, because that is more relevant at the time. And we can share the video and the associated questions directly
13
00:01:08.850 --> 00:01:13.270
Liam Wyatt: to Meta, to Wikimedia community social media channels, etc.
14
00:01:13.600 --> 00:01:22.329
Liam Wyatt: So with no further ado Mariana, would you like to kick us off with what is future audiences, past, present, in the future.
15
00:01:22.380 --> 00:01:34.209
Maryana Pinchuk: Yes, I would love to thank you, Liam. Hello! For those of you who don't know me. Marianna Pinchuk, product manager at the Wikimedia Foundation. I lead the future audiences. Initiative. And I will be taking you through
16
00:01:34.240 --> 00:01:38.330
Maryana Pinchuk: what might be a reminder for most of you, but might be new for some of you.
17
00:01:38.370 --> 00:01:57.509
Maryana Pinchuk: What is the future? Audiences? Initiative? Why are we here? So just as a reminder. This is a new initiative that just kicked off in this fiscal year the Wikimedia Foundation fiscal year starts in July. So we've been operating as a team since July.
18
00:01:57.520 --> 00:02:20.669
Maryana Pinchuk: Not that much time. Actually, but the mandate of this team is as follows, we are trying to understand and identify strategies to pursue to arrive in a changing Internet and test our assumptions about those strategies with quick experiments.
19
00:02:20.670 --> 00:02:37.660
Maryana Pinchuk: So that is the set of objectives that we have but what does that actually mean? So what is a strategy? Well, a strategy that our movement has basically benefited from for the last 20 years or so
20
00:02:37.660 --> 00:02:50.959
Maryana Pinchuk: is a a interesting symbiosis with the way that technology has worked for the last 20 years. And the way that many people have used technology to find information
21
00:02:50.970 --> 00:03:17.420
Maryana Pinchuk: and what I mean by that is for the last 20 years there have been a large group of people globally, who, when interested in finding out a fact or learning more about a topic, would open up a web browser and go to a search engine, usually Google and would type in some question and would get a set of results and often the top result, or in the top results would be Wikipedia
22
00:03:17.550 --> 00:03:29.080
Maryana Pinchuk: and that is how Wikipedia has thrived in in readership and in building communities in various languages for the last 2 decades.
23
00:03:29.080 --> 00:03:50.269
Maryana Pinchuk: But that is a a very specific set of behaviors. That is changing. We know that. You know the world looks different now than it did in 2,001. There are a lot more places online to go and get knowledge. There are a lot more ways online to go and get knowledge. You can listen to a podcast you can watch a short video.
24
00:03:50.270 --> 00:04:04.829
Maryana Pinchuk: You can do all kinds of things. Now. To get information. Go to all kinds of places. And we know with generative AI. The whole concept of of getting information. Knowledge is in in flux.
25
00:04:04.830 --> 00:04:31.590
Maryana Pinchuk: So what we're trying to do as a team really is to look ahead and see what other strategies we could pursue as a movement to continue to be the incredible knowledge service to the world that we've been the last 20 years and to engage new audiences who we actually haven't been engaging over the last 20 years. Not everyone in the world knows about Wikipedia or uses Wikipedia.
26
00:04:31.670 --> 00:04:37.539
Maryana Pinchuk: So how might we see this change in technology as an opportunity. So
27
00:04:37.780 --> 00:05:05.809
Maryana Pinchuk: at a high level, II think 2 ways you could look at what a different strategy might look like. And these are kind of the areas we're poking around with experiments in to better understand both the opportunities and the the risks, the costs. Of each. These 2 areas are growing as a knowledge destination. Which just means that instead of relying on
28
00:05:05.840 --> 00:05:31.160
Maryana Pinchuk: a search engine or some other platform which, or technology to to bring people to our platform, to our projects. We could instead envision a world in which anytime anyone thought of anything they wanted to know more about, they would go directly to Wikipedia. That might bring, when many, many more readers to our projects. And potentially more folks who can contribute directly
29
00:05:31.350 --> 00:05:48.570
Maryana Pinchuk: that comes with a whole set of assumptions and risks and costs that we're that we're trying to understand experimentally. Another approach might be to instead look outward at all of the places. Now that people are getting information.
30
00:05:48.570 --> 00:06:12.569
Maryana Pinchuk: Whether that's video platforms like Youtube or Tiktok, whether that's third party chat bots like Chat Gbt, anywhere in the world that people are getting information and looking to see, could there be an opportunity for us to present our information more clearly, with attribution, with pathways to contribution. In all of those places
31
00:06:12.800 --> 00:06:18.359
Maryana Pinchuk: and I see a hand already. So I'm gonna stop and field any questions that people have.
32
00:06:21.150 --> 00:06:22.200
Liam Wyatt: Anton.
33
00:06:23.590 --> 00:06:24.570
Hello.
34
00:06:25.320 --> 00:06:28.419
Антон Обожин: May speak.
35
00:06:29.140 --> 00:06:31.310
Антон Обожин: Yes, we can hear you just go
36
00:06:33.720 --> 00:06:42.250
Антон Обожин: about that issue to become a source of knowledge
37
00:06:43.580 --> 00:06:47.109
Антон Обожин: itself instead of being surge driven, I can give an example
38
00:06:47.140 --> 00:07:13.019
Антон Обожин: which is relevant to our local community. And we have raised this question is an example that if you are using Ukrainian set to operational system windows. If, your browser is set to language, and your preferences in Google are set to bring in languages. If you Google, something new cranium languages, if
39
00:07:13.140 --> 00:07:21.839
Антон Обожин: gives you a results in Russian Wikipedia, first, then in English, and only then in Ukrainian. So
40
00:07:22.980 --> 00:07:24.110
Maryana Pinchuk: hmm
41
00:07:24.380 --> 00:07:50.059
Антон Обожин: have ways to manipulate what they are showing manipulate which content from Vicky, which is not enter, in which priority so deprive them. Of this possibility can be shown on the Wikipedia. And what can be?
42
00:07:50.080 --> 00:08:08.970
Антон Обожин: There may be an popular article with 100,000 views per day which would be easily given in a search engine like Google. And there may be an article with one view per day.
43
00:08:08.990 --> 00:08:24.040
Антон Обожин: and it will be completely ignored by search engine. And it may be a wonderful article. But no one ever sees it because Google never shows it in its search results. That is a problem.
44
00:08:25.160 --> 00:08:34.919
Maryana Pinchuk: Yeah, absolutely. Yeah. I think one of the one of the things that we're doing is trying to not just understand what are the good things about
45
00:08:34.919 --> 00:08:55.839
Maryana Pinchuk: each of these different directions, but what are potentially the risks and the costs. And you're certainly right. That, relying on other platforms to syndicate our content, which we have for the last 20 years again. Really does come with that risk that the different platforms, choices of what? Content to show
46
00:08:55.860 --> 00:09:16.320
Maryana Pinchuk: potentially their political affiliation, whatever might influence, what kind of information is is available to to readers. And that's a big danger. It's been a big danger with Google, and it will continue to be a big danger as long as we allow other platforms to be our main mode of syndication. Absolutely
47
00:09:20.340 --> 00:09:21.930
Maryana Pinchuk: any other questions
48
00:09:22.270 --> 00:09:37.049
Maryana Pinchuk: or thoughts about. And yes, Sandy, thank you for linking to the the retro board. So in December we had a a chat about sort of the different opportunities and risks we see in all of the different
49
00:09:37.150 --> 00:10:03.180
Maryana Pinchuk: potential strategies that we might undertake. And I think there were some good pros and cons. Very, very significant pros and cons for for undertaking either of these strategies the destination strategy, while it does have the benefit of allowing us total control over what we showed to to readers as a as a movement. Comes with also a number of really big unknowns. Cost
50
00:10:03.180 --> 00:10:11.550
Maryana Pinchuk: whether people will change their behavior to look for knowledge in a specific place.
51
00:10:11.580 --> 00:10:12.400
Maryana Pinchuk: yeah.
52
00:10:17.510 --> 00:10:21.569
Liam Wyatt: Anton, I see your hand up again. Is that the same hand? Or is that a different question.
53
00:10:26.890 --> 00:10:28.669
Maryana Pinchuk: I think that might have been a legit.
54
00:10:29.390 --> 00:10:34.460
Liam Wyatt: Pepsi. Okay. Mariana, please. Okay. Well.
55
00:10:34.530 --> 00:11:03.319
Maryana Pinchuk: yes, I will continue. So so yes. So our team. As I said, we are a testing team and a learning team. We aren't building fully finished products. We aren't putting a big bet on either of these directions yet, because we don't know whether these are even possible to pursue. So so I really just wanna stress that everything that you're seeing and everything that we'll be talking about. I'm gonna hand it over to Mike in a second
56
00:11:03.320 --> 00:11:07.959
Maryana Pinchuk: is experimental work. It's not meant to be like.
57
00:11:07.960 --> 00:11:32.019
Maryana Pinchuk: This is the new big 1 million dollar bet from Wikimedia about how we're going to. Use AI it, we're really just trying to learn and understand the opportunities and the risks of of these different strategies and make recommendations for other teams at the Wikimedia Foundation to pursue bigger investment if we feel like it is an important and
58
00:11:32.370 --> 00:11:34.310
Maryana Pinchuk: potentially fruitful area.
59
00:11:35.120 --> 00:12:00.379
Maryana Pinchuk: Okay, so I'm gonna hand it off to Mike now to talk about our very first completed experiment which we started the year with which was building a Plugin for Chat Gp. That would pull knowledge specifically from Wikipedia and summarize it, and you'll see the screenshot here shows essentially how it looked. So these are our key research questions that we had going in
60
00:12:00.380 --> 00:12:16.300
Maryana Pinchuk: this experiment. In the future. Audiences kind of framework was trying to help us understand whether Chat Gp is becoming the new Google and if so, what are those implications for? For how our knowledge is showing up there to readers?
61
00:12:16.300 --> 00:12:31.270
Maryana Pinchuk: And how might we deal with that? And there is a lot of learning and insight. That I'm excited for you to hear about. So, Mike, please take it away. I'm gonna stop sharing my screen so you can share your slides. Yeah.
62
00:12:31.810 --> 00:12:39.089
miishen: sounds good. Thanks. Great Hi, everyone. Please bear with me oops
63
00:12:39.420 --> 00:12:44.629
miishen: I'm I had Covid recently, so I'm like I might have a brain for it once in a while.
64
00:12:45.120 --> 00:12:48.140
miishen: okay, can everyone see this?
65
00:12:48.480 --> 00:12:50.159
miishen: I'm just gonna use this mode.
66
00:12:50.310 --> 00:13:06.369
miishen: Alright. So I'm not gonna go through every single slide in detail here, just out of the interest of time. But I think I'm gonna try to do a quick overview of the things we learned the link to this. This is on our Meta Wiki page. So anyone can go look at these slides as well.
67
00:13:06.810 --> 00:13:19.980
miishen: so I won't recap too long over what the Wikipedia Plugin for Chatty Bt. Was just because I think most people on this call are somewhat familiar with it. Basically, it was a plugin for Chat Gbt, that would
68
00:13:20.120 --> 00:13:36.590
miishen: be activated if Chat Gt. Thought that the query was related to general knowledge, and something that Wikipedia could answer, and then it would go search Wikipedia, and then summarize the content that we return back to it and provide links and references and some disclaimers.
69
00:13:36.660 --> 00:13:42.350
miishen: and basically just marking that the information was coming from Wikipedia in a very conspicuous way.
70
00:13:43.180 --> 00:14:07.089
miishen: So I think overall, this is kind of the executive summary of this experiment. We wanted to know if Lms and chat assistance like Chat Gbt, were the new paradigm for information seekers. If you can imagine. A year ago, when everyone was talking about chat Gvt, this is there was this kind of impending, almost doom feeling of like, oh, everything's gonna go away as we know it, and it's going to be replaced by Chat Gbt.
71
00:14:07.140 --> 00:14:14.040
miishen: so there's a lot of urgency behind trying to learn more about whether this is going to happen, or how or what that world looks like.
72
00:14:14.050 --> 00:14:18.879
miishen: And that was kind of a huge motivator. For why we wanted to do this experiment quickly.
73
00:14:18.930 --> 00:14:36.589
miishen: and we can say that it hasn't happened. It may still happen. We don't really know. But that same level of urgency is kind of diminished a little bit. We haven't seen like Google disappear. For example, we haven't seen, you know, as we'll see in other slides like web traffic to Wikipedia go down, or anything.
74
00:14:36.720 --> 00:14:44.320
miishen: But I don't think we can dismiss it completely. The the industry is changing very quickly. People are making improvements. Things could still change in the time.
75
00:14:44.690 --> 00:15:11.909
miishen: And similarly, like, I said, our chat assistance displacing traffic to Wikipedia. The answer is, no, there's no evidence that any of the Chat. Gbt. Anything has caused traffic to Wikipedia to change in any noticeable way, mostly because there's just so much traffic to our website that, like it's, you know, every chat Gbt user was like not using Wikipedia. I don't even know that would make it dead honestly. So that's kind of a good thing in a way.
76
00:15:12.370 --> 00:15:17.460
miishen: and then we were also interested in these Lm chat assistance. We're reaching new audiences.
77
00:15:17.480 --> 00:15:29.219
miishen: And we don't have complete data. But in general, from what we've seen is that it's not really reaching new people. It seems to be reaching a lot of the same people that currently use Wikipedia.
78
00:15:29.890 --> 00:15:39.090
miishen: But we would also say, a big part of this is Chatty, Pbt. To get to the Plugin and install it. You have to pay for it, anyway, so that Paywall is going to introduce a lot of barriers to
79
00:15:39.310 --> 00:15:53.070
miishen: new audiences. Who, you know, may live in person or over. They don't have enough money, basically And then, on the technological side, we're interested in how accurate the Lms were in retrieving and summarizing answers.
80
00:15:53.150 --> 00:15:55.040
miishen: And actually, they were pretty good.
81
00:15:55.060 --> 00:16:20.630
miishen: They're not perfect. They make mistakes, they do weird things. And these mistakes are constantly changing as they update the models. But I think overall from our internal auditing we saw that the results are pretty good most of the time, and they're probably only going to get better. We assume and then, I think, equally importantly, is beside the actual like performance of it. What is perception of that credibility like? What did people think about the accuracy and the relevancy.
82
00:16:20.700 --> 00:16:34.619
miishen: And in general we saw that people generally trust the information more when they know it's from Wikipedia, and so there's some strength there in the brand of Wikipedia as a source of reliable information. And that's something that we're kind of interested in exploring further as well.
83
00:16:34.750 --> 00:16:45.060
miishen: But okay, so just real quick. We will go through some data. Once again feel free to ask questions. I won't spend too much time on any individual thing.
84
00:16:45.260 --> 00:16:58.760
miishen: The I just labeled the boxes or the data in different colored boxes based on where it's coming from. Yellow is from logging data. We did a survey with about 71 participants and self reported, so that's green boxes.
85
00:16:59.810 --> 00:17:23.979
miishen: Okay. So we look at the users and the messages. We ran it for about 2 quarters. So about, let's say, 6 months, and we saw that after about a month into August we got some publicity. And so we got this kind of spike in users. Everybody suddenly knew about it. People were downloading it and solving it, and we see this kind of rise up in unique users and message count kind of more or less follows that trend
86
00:17:24.060 --> 00:17:30.589
miishen: And then, sometime around October, we saw the usage start to go down again.
87
00:17:30.750 --> 00:17:38.049
miishen: and that doesn't line up exactly with Chat Gbt. Moving away from plugins to Gpts. But I think it's pretty close
88
00:17:38.230 --> 00:17:40.350
miishen: and
89
00:17:40.450 --> 00:17:51.219
miishen: and then we ended the experiment in January. So just last month so we kind of see this rise, and we kind of see the a fall, and it seems kind of like top-out kind of around October.
90
00:17:53.590 --> 00:18:03.970
miishen: We looked at logging data and self reported data. It's kind of close, not exactly the same, but we would say about 2 thirds of the users were located in Europe and North America.
91
00:18:04.400 --> 00:18:08.319
And then, if we look at the top languages by number of messages.
92
00:18:08.380 --> 00:18:17.970
miishen: then English is always on top, and then the rest of, like Chinese, German, French, and Japanese would kind of like trade places for like week by week, which one is the most
93
00:18:17.980 --> 00:18:21.189
miishen: the most number of messages that week.
94
00:18:21.260 --> 00:18:45.639
miishen: But English is generally always on top. So this is part of what we were seeing. Where, like in terms of new audiences, you know. Are we only reaching North America and Europe in general, or are we reaching Africa Asian audiences quite as to the degree that we'd want to if we were hoping this is like a new means for other people that access Wikipedia information. And it looks like for the most part, mostly
95
00:18:45.650 --> 00:18:57.759
miishen: it's still North American Europe. There's some potential here and there. But it's very much mitigated by what open AI decides to do right like whether they have a pay wall where chatty pts, even accessible, etcetera.
96
00:18:59.620 --> 00:19:06.010
miishen: When we looked at self-reported gender and age, we saw that most people are from 26 to 45.
97
00:19:06.050 --> 00:19:10.610
miishen: Let's call that middle is aged people.
98
00:19:11.150 --> 00:19:22.749
miishen: so we don't see. II think we had one person reporting that was under 18. But II can't remember why we didn't use that, or maybe we don't report as much on under 18. But this was like
99
00:19:23.160 --> 00:19:31.710
miishen: it. It wasn't like we we threw out like 20 of self reports because they're all under 18. We just basically didn't see a lot of young people using the the plugin
100
00:19:32.020 --> 00:19:37.309
miishen: and then of the self reports we had predominantly, mostly men
101
00:19:37.560 --> 00:19:42.040
miishen: reporting usage of the Chachi team. Chat. Gt. Plugin.
102
00:19:43.680 --> 00:19:56.950
miishen: This one is just kind of interesting. I don't know what major conclusions I have from this. But we asked people what their primary purpose for using Wikipedia was, and most people use it for personal interest topics, some academic research.
103
00:19:57.200 --> 00:20:11.709
miishen: And you know there's a fair amount that said just trying out chat. Gbt, I suspect that's also why we see kind of them a decline after the peak, because, like people who are just like trying this cool new piece of technology and seeing how things work and trying out the plugin, seeing how that works.
104
00:20:12.090 --> 00:20:23.219
miishen: you know, if they did that for a few months, and then they're kind of like, okay, that was a cool little toy, like, you know, I'm gonna go back to the website, or something that we might expect to the drop off that we saw in unique users.
105
00:20:23.500 --> 00:20:29.110
miishen: And then, almost as expected as well, breaking news. Not a lot of people are looking at.
106
00:20:29.470 --> 00:20:33.749
miishen: I think, knowing that Chat Gp has a training window
107
00:20:33.800 --> 00:20:37.910
and then, knowing that Wikipedia is maybe always updated.
108
00:20:37.990 --> 00:20:50.930
miishen: you know, with the most breaking news to the very minute or something, and I think people were in patterns of like checking Twitter or checking like social media. I wouldn't expect breaking news to be like the highest use case.
109
00:20:52.090 --> 00:20:58.069
miishen: I see that there's some chats coming in, but I don't see the chats themselves. So if anybody wants to.
110
00:20:59.750 --> 00:21:05.900
miishen: that's okay, Mike, we'll tell you if there's something to respond to. That's what I was asking for.
111
00:21:06.460 --> 00:21:33.190
miishen: Okay, this is also interesting. So we asked people if they used to plug in more than the Wikipedia website. And by and large, we see that most people are still going to the website, probably at the same rate or more. Some people used to plug in. They said much more, but I think that's kind of minority. So this is part of the story of like, it's not like chatty Bt. Users, or like chat gpts like taking away like our lunch. Right? It's not like people are like abandoning Wikipedia to go find information on Chat Gbt.
112
00:21:33.440 --> 00:21:35.730
miishen: they're still good using the website.
113
00:21:37.190 --> 00:21:41.040
miishen: This is the click through rate.
114
00:21:41.350 --> 00:21:52.729
miishen: So, as we saw that each time chat our plugin would answer with it would provide a link if possible. Sometimes it's a little bad about it, but it would provide a link back to Wikipedia for people to click through and get more contacts.
115
00:21:52.870 --> 00:22:02.329
miishen: And we can see that it kind of starts up the average. Sorry. The click-through rate was similar to what we see on from Google, searches.
116
00:22:02.340 --> 00:22:04.920
miishen: And and then
117
00:22:05.430 --> 00:22:14.039
miishen: and then we have this queries per day, which is starts up kind of high and decline. So let's say, about 4 queries per day. So this is, this is kind of the
118
00:22:14.160 --> 00:22:35.290
miishen: the one that's a little bit less clear just by looking at what's on the slide. The story that I think that's coming out from this that we are seeing is that people are really using the plugin like search. They're using it as like a spe. A better natural language search to get information out of Wikipedia. And I think the 2 things that are kind of interesting here click through rate. It's similar to
119
00:22:35.500 --> 00:22:40.860
miishen: Google Search, which, of course, is kind of a not Apple's apples. But it's saying, like, you know.
120
00:22:40.910 --> 00:22:43.610
miishen: the numbers are the same. But, more importantly.
121
00:22:44.040 --> 00:23:13.710
miishen: they're not using people weren't really using it conversationally is what I'm kind of seeing here. Because if you can imagine, like the whole appeal of Chat Gbts, you have this conversational tool. So I ask it a question. I get an answer. If I want more context, or I wanna dig in. The the proposal is that I can always just ask more questions and follow up and have a conversation with this agent and get more information. If that's the case, I would expect click, click through rate to be quite low. Right? Because
122
00:23:14.220 --> 00:23:37.159
miishen: why would I click through and read the article? If I can just keep querying the system to get more information out of it. But if the quick click through rate is similar to Google than what it's telling me is that people are using the search, they get some information. And when they want more context, they're going to the website just as they would be on search rather than continuously query the system. And that's also why I'm including click our queries per day as well. Because
123
00:23:37.990 --> 00:23:43.360
miishen: you can imagine if you constantly are re querying the system, get more information you might see higher
124
00:23:44.920 --> 00:23:52.609
miishen: once again to get that from clicking. I can always send another message to get more information.
125
00:23:52.640 --> 00:24:08.520
miishen: So this isn't like, you know, the home run smoking gun. But I would say, when you kind of like, look at all these pieces, it does kind of look like people are using in the same paradigm which is search, and not this kind of new paradigm of like. Let me keep asking this robot questions over and over to get more information.
126
00:24:08.770 --> 00:24:13.120
Liam Wyatt: Mike, we have a question and a comment in the chat.
127
00:24:13.350 --> 00:24:16.080
Liam Wyatt: The the question is effectively.
128
00:24:16.240 --> 00:24:19.349
Liam Wyatt: why can't I still use it? Wouldn't it be
129
00:24:19.580 --> 00:24:24.490
Liam Wyatt: still interested to have it available. Anyway, I might be interested in
130
00:24:24.550 --> 00:24:28.759
Liam Wyatt: using it in the future so effectively. What's the harm of keeping it around
131
00:24:29.070 --> 00:24:36.999
Liam Wyatt: and the comment is that, excuse me. if we mediate on the who and how many
132
00:24:37.150 --> 00:24:47.919
Liam Wyatt: people? It's actually quite a small number of people compared to say Google with users and thinking about those people that they are mostly social media or creators.
133
00:24:47.960 --> 00:24:50.030
Liam Wyatt: anyway, which is even a smaller
134
00:24:50.080 --> 00:25:00.550
Liam Wyatt: percentage. So the implication of this comment is that it's quite a fraction of the users and their paid users, and so forth. Quite a minority of of
135
00:25:00.990 --> 00:25:04.759
Liam Wyatt: potential audience, either at the best of times. If I understand that
136
00:25:05.280 --> 00:25:07.050
Liam Wyatt: the scope of the comment.
137
00:25:08.610 --> 00:25:14.209
miishen: thanks. I'll try to address maybe both of them in a similar way, which is as Mariana said.
138
00:25:14.350 --> 00:25:27.420
miishen: the goal of this team is to run quick experiments and to learn as much as we can in a quick turnaround and that's going to mean that the data that we collect and can generalize from is going to be limited.
139
00:25:27.530 --> 00:25:29.369
miishen: And it's not complete.
140
00:25:29.460 --> 00:25:46.909
miishen: And you know, I have a lot of tension with our data scientists all the time, because they like to be very meticulous, since they like. Do we have enough data to support this. And my answer is typically going to be like, No, we don't. And II recognize that. But we're doing a little bit of reading the tea leaves here, because.
141
00:25:46.910 --> 00:26:07.040
miishen: you know, we ran an experiment in 2 quarters, which is actually quite good, and we have a lot of good information and things that are kind of pointing us in certain directions. And there's always going to be follow ups. And we can. We have better ideas of where we might want to follow up now, or whether it's worth it. But we're never gonna be able to conclusively say that this is like perfect information. So
142
00:26:07.050 --> 00:26:08.109
miishen: you know, like
143
00:26:08.590 --> 00:26:13.519
miishen: this is the best as we can know in the time that we're gonna run it. And
144
00:26:13.600 --> 00:26:18.059
miishen: this the speed at which we want to run these experiments is kind of the main priority here to get
145
00:26:18.120 --> 00:26:35.229
miishen: some indication. You know, we can get that 70% confidence level that, like, you know, we shouldn't invest like 50 million dollars into Chat Gbt, for example. Then that's like a pretty important thing for us to know as an organization rather than spending like 3 years getting like 85% confidence. That's like
146
00:26:35.460 --> 00:26:43.760
miishen: that's not going to help us in a world where things are moving very quickly. Technology and social kind of implications of that technology are moving very quickly.
147
00:26:43.860 --> 00:26:56.910
miishen: So I think that addresses kind of the second question of like, you know, limited sample size, etc. The first question of why we don't keep this running is is similar in a way, because we want to run these quick experiments, spend a lot of time
148
00:26:56.930 --> 00:27:00.370
miishen: diving in learning something, and then kind of
149
00:27:00.450 --> 00:27:02.409
moving on to the next thing, and
150
00:27:02.530 --> 00:27:21.270
miishen: there is always going to be a cost of keeping these things running right like our, we have to maintain the plugin. We have to. I mean even just to keep it running to improve it, even if we don't want to improve it. Chat Gbt changes their system. Now, everything is a Gbt, so that's some work to change the Plugin infrastructure to be a Gpt infrastructure and constantly having to follow that
151
00:27:21.630 --> 00:27:25.600
our data team has to be able to keep collecting data on it potentially
152
00:27:25.770 --> 00:27:35.780
miishen: because if we're not collecting data, then there's almost like no point in running. Continuing the experiment. The whole point of the experiment is, we're collecting data so that we can analyze and learn something from it.
153
00:27:35.830 --> 00:27:45.419
miishen: And I think this is kind of a key distinction from what Mariana was saying about like experiments versus products, right products is like, we're really trying to
154
00:27:45.460 --> 00:28:04.350
miishen: create something for users, improve it over time and have this product that is useful for people. And in an experiment. It's like we need people to use these things so that we can learn something from it. But the the ultimate point is to get these learnings. And it's not to put a product in the world that it's actually in its final form, useful.
155
00:28:04.440 --> 00:28:13.490
miishen: Which is not to say that this won't come back right like, if this was like a knockout we saw like amazing numbers, and that's a chance for us to go to leadership and say
156
00:28:13.570 --> 00:28:29.270
miishen: our recommendation is like people really love this Plugin. We should go build this. We should go find a team that actually can maintain this. We should actually go find the budget. For this. We should actually use the software that we want to use internally, that aligns with our values. That's open source. All of these things that we would normally consider for internal
157
00:28:30.160 --> 00:28:42.230
miishen: product development is that would be like starting down that path. And we can make a recommendation for that, but for us the experiment is the priority which is like, what can we learn from this? And at the point where, like.
158
00:28:42.500 --> 00:28:52.399
miishen: you know, the juice is not worth the squeeze, so to speak. Then we kind of know that this version, the the experiment it is like it's time for it to end. So I hope that it addresses the questions
159
00:28:54.220 --> 00:28:55.030
next bucket.
160
00:28:56.140 --> 00:29:03.979
Liam Wyatt: We have. We're past the half now. So we should get to the what's next. But I wanna make sure you have
161
00:29:04.700 --> 00:29:06.860
Liam Wyatt: this concluded
162
00:29:06.910 --> 00:29:12.809
Liam Wyatt: adequately. And people have an adequate time to respond to any immediate questions about Chat Gp, specifically.
163
00:29:13.400 --> 00:29:17.459
miishen: okay, cool. I'll try to fly through these slides a little bit.
164
00:29:17.690 --> 00:29:20.740
miishen: and then leave some time for questions, and we'll talk about citation needed.
165
00:29:20.780 --> 00:29:37.120
miishen: This is just Google trends, showing that the red line is searches for Chat. Gbt. It can. It's spiked, as everyone you know can see. But Wikipedia and Wiki searches. Just kind of stay level. So we didn't really see any once again, not really taking our pie taking our lunch.
166
00:29:37.180 --> 00:29:53.450
miishen: This is kind of similar. This is just the page view issue. Or this is, what is this page views you have per month, and effectively, if it looks like chaos, and it doesn't look like there's a pattern that's kind of the point, which is that chatty Pt. Launched, and it didn't really
167
00:29:53.580 --> 00:29:56.770
miishen: seem to move us in any significant way in terms of page views.
168
00:29:57.860 --> 00:30:06.889
miishen: So this is just returning to main questions. Lms are not like this new paradigm that we can see yet they're not displacing traffic to Wikipedia and
169
00:30:07.030 --> 00:30:09.679
miishen: in general, we're not really reaching new audiences.
170
00:30:09.990 --> 00:30:20.420
miishen: This is model quality. We did some internal annotation. The the punchline here is relevancy. We did pretty good. This is just whether or not the answer was relevant, whether we thought it was accurate or not.
171
00:30:20.590 --> 00:30:36.889
miishen: And it's it's pretty relevant most of the time. Accuracy is how accurate the information was to the Wikipedia source, also pretty accurate most of the time. The caveat in both of these is that there's some variation by language. But once again, there's not a lot of data. Sometimes there's only like a dozen
172
00:30:37.410 --> 00:30:42.329
miishen: you know, examples that we were able to annotate. And take a look at so
173
00:30:42.370 --> 00:30:46.279
some of these may look worse. It's kind of hard to tell in some cases.
174
00:30:46.290 --> 00:30:50.100
miishen: but overall. When we look at it overall, it looks like pretty okay.
175
00:30:51.050 --> 00:30:59.170
miishen: So in general, the the models are pretty accurate. They're pretty relevant. They could be better, but they're probably improving over time, and it's something we should keep an eye on.
176
00:30:59.330 --> 00:31:04.789
miishen: And once again, people generally trust the information more when they know it's coming from Wikipedia, which is pretty important.
177
00:31:06.410 --> 00:31:10.340
miishen: Okay, so I can kind of skip this part. This is looking forward.
178
00:31:10.430 --> 00:31:19.199
miishen: I will kind of stop it here for now and see if anybody has questions about the there is one more.
179
00:31:19.280 --> 00:31:36.999
Liam Wyatt: Okay. How confident are we about extrapolating from the Wikipedia chat Tpt user base as opposed to the general chat. Tpp user base, which is orders of magnitude larger especially considering that's a self selected sample upon a self selected sample.
180
00:31:38.120 --> 00:31:50.919
miishen: II think it depends on what we're trying to extrapolate right? Like here. We were really, really trying to learn who is using Chat Gbt, and you know what their purposes are and what their intentions are.
181
00:31:50.990 --> 00:31:57.899
miishen: So of course, there's going to be limitations about how much we can know about the people who don't use chat. Gbt.
182
00:31:58.050 --> 00:32:01.850
miishen: But in general I
183
00:32:02.080 --> 00:32:05.580
miishen: I don't see too much reason to not
184
00:32:06.840 --> 00:32:25.539
miishen: to be too like overly conservative and say, this doesn't apply at all to the general population. I would say that this should carry over to some comparable degree and this would be kind of like a good chance to do. Follow up research if we wanted to, you know, get a better idea. But I don't see any immediate reason why, you know this.
185
00:32:25.750 --> 00:32:28.430
miishen: II think we can kind of like make
186
00:32:28.540 --> 00:32:43.729
miishen: corrections in some cases, like, Okay, well, the self reported people that are using self or Chat Gbt are like mostly men, for example. So I can imagine the wider audience I would correct and say, there's probably more women using the website than like the chatty chat Gbt users.
187
00:32:43.760 --> 00:32:53.109
miishen: So I think in some cases you can treat this. What we're seeing here is either like the high end of the low end depending on you know how you think the data might skew towards self selection.
188
00:32:54.570 --> 00:32:56.569
and we had a further question,
189
00:32:57.320 --> 00:33:03.089
Liam Wyatt: that users are more interested or engaged in conversational format than reading long form articles
190
00:33:03.360 --> 00:33:10.959
Liam Wyatt: providing helpful and relevant information depends a lot on the question they're asking or you asking the right question?
191
00:33:11.180 --> 00:33:19.789
Liam Wyatt: Has there been thought, in case the previous queries that relate to the current one, and immediately suggesting for the information based on the most probable next question.
192
00:33:22.570 --> 00:33:45.260
miishen: The short answer is no, and I think that we started digging into the queries themselves. We didn't do too much with them, but we don't. I didn't show it here. But part of what the story that we're saying, that people are using it like search is that I would say at least half the the queries that we're seeing. I actually aren't even natural language questions. People are using it like search. They would just be like Istanbul Turkey.
193
00:33:45.710 --> 00:33:58.110
miishen: you know, like, that's something you might type into a Google search bar and then hope to like sort through the results. And you know the appeal of Chat Gt, is that you can actually ask a more specific question. You know, like, you know, what is
194
00:33:58.130 --> 00:34:05.150
miishen: that? That feral cat population of stable or something right? And and get the answer that you would want hopefully.
195
00:34:05.160 --> 00:34:10.890
miishen: And so we saw a mix of like all of these across the board, and I kind of bring it up, because
196
00:34:11.020 --> 00:34:31.469
miishen: that huge variation, I think, makes it a little bit more difficult to do. Some follow up behavior based on what the queries are going to be, because, especially for a search query, where you're just getting like a name. It's kind of really hard to know what to do with that afterwards, because the intention is a little bit hidden from the system itself. It's like in the users head
197
00:34:31.469 --> 00:34:49.429
miishen: that they're gonna follow up, and they know what they're looking for. But they're not really giving that information to us. So I think, as the trend of how people use these technologies changes, and maybe people are better trained to ask questions and follow ups. And it might be possible to kind of walk down that road. But it's not something that we explored in this stage.
198
00:34:50.310 --> 00:34:53.520
Liam Wyatt: Thanks, Mike. that's all the questions that have been submitted.
199
00:34:53.590 --> 00:34:55.810
People can continue obviously
200
00:34:55.840 --> 00:34:59.309
Liam Wyatt: asking questions in the chat about this section. But if you could
201
00:34:59.680 --> 00:35:02.460
Liam Wyatt: now revealed the what next?
202
00:35:02.590 --> 00:35:08.249
Liam Wyatt: Because there is one comment, that is, I think, leading basically to what you're talking about no.
203
00:35:08.560 --> 00:35:19.660
miishen: Okay, cool. I'll try to be pretty brief about this, too. I'll just show it first, and we can kind of talk a little bit about it. I think it's just easier to see it. So citation needed.
204
00:35:20.340 --> 00:35:24.630
miishen: which we are jokingly say, don't call it a fact checker. It's not a fact, checker.
205
00:35:24.920 --> 00:35:37.769
miishen: But II think there's something in this spirit that, like we kind of all recognize, which is that there's a lot of misinformation online. And there's a lot of claims online, dubious, not and
206
00:35:38.010 --> 00:36:02.579
miishen: a lot of it with AI as being, you know, AI is amplifying how much of this there could be. And so we're positioning this as an opportunity for allowing people to use Wikipedia as a way of verifying information. If it exists on Wikipedia, and seeing whether or not this these claims are supported or not, supported by Wikipedia, and, you know, kind of facilitating their journey towards like verifying information.
207
00:36:02.620 --> 00:36:06.860
miishen: you know, even teaching them, maybe problem, like information hygiene online.
208
00:36:06.980 --> 00:36:16.880
miishen: and so it looks very different behind the scenes. It's actually very similar to what we built before. We're still going through Chat Gbt just through through the Api
209
00:36:16.990 --> 00:36:20.019
miishen: and
210
00:36:20.100 --> 00:36:32.799
miishen: This is kind of like part of my belief that Chat Gbts like this great sandbox to see what these Lms can do and play with them. But ultimately we're gonna see a lot of this functionality exists behind the scenes. It's not like
211
00:36:32.800 --> 00:36:51.760
miishen: the chat interface or the chat assistant is really going to be the way we all you see technologies and interact with it. So this is kind of a more focused way of saying, like, you know, how can we actually implement these technologies in a way that's used for people that can bring information to where they are that can, like set us up as a you know, a source of
212
00:36:52.370 --> 00:37:04.690
miishen: credible, reliable information. So as you can see here. The way this is going to take shape is the experiment is going to be a browser extension for chrome, just because that's the easiest way for us to
213
00:37:05.030 --> 00:37:27.539
miishen: get in front of everybody's browsing experience when they're off the website. Because crucially, Mariana, touch on this like, how do we bring Wikipedia, content to people who are off the website and don't depend on you know you have to come to Wikipedia to get any content, you know, like, how can we bring it outwards, people? So as in the Browser extension, you know, it sits in the browser. You're browsing the Internet. You see this article on Taylor Swift.
214
00:37:27.540 --> 00:37:40.370
miishen: and you will select this claim about like this earthquake essentially, and then you will be able to search Wikipedia for information. It will return the relevant quote from the relevant article, if you can find one
215
00:37:40.530 --> 00:37:52.980
miishen: and then you can see that it returns some what's it called quality signals here. So, like the last edit date the number of references number of people who worked on this article, and then you can click through and continue reading more about this. If you'd like
216
00:37:53.130 --> 00:38:01.260
miishen: there is going to be another version we'll play with here. So this is just kind of like, search, right? This is like, we kind of give you a search. We give you a snippet that looks
217
00:38:01.700 --> 00:38:11.420
miishen: contextually relevant. And then we want to experiment with using a little bit more. AI, you know, firepower, so to speak, and having inference that says like.
218
00:38:11.660 --> 00:38:16.159
miishen: it won't be like as obvious as a green check mark or something, but it will be some kind of
219
00:38:16.210 --> 00:38:23.660
miishen: indicator, like a prime like, Hey, watch out like, maybe this is like not great information. You should like, maybe go read more about this, for example.
220
00:38:23.840 --> 00:38:31.470
miishen: or maybe we couldn't find anything on Wikipedia about this. You might want to like, go look someone else just to to fully verify this good thing.
221
00:38:31.610 --> 00:38:49.100
miishen: so II think the concept is pretty simple. There's a lot more we can kind of say about it, but I think people have a lot of questions. I can see the number in the chat, so I'll kind of open it up to the QA. Here. Well, first question in the chat here is, how does this work in different languages?
222
00:38:49.730 --> 00:39:00.600
miishen: The short answer is, it doesn't. Once again due to the scope of the experiment. We are restricting it just to English, for right now and I think that is just
223
00:39:00.750 --> 00:39:06.029
miishen: so that we can handle it. We don't have very many people working on this. We know that these models work best on English.
224
00:39:06.060 --> 00:39:14.360
miishen: and so we are restricting it to English for now and in the future. Once again, if this is like a huge smashing success, then we can
225
00:39:14.420 --> 00:39:21.839
miishen: do follow up studies or recommended to product development to do a more comprehensive kind of multilingual approach.
226
00:39:25.630 --> 00:39:37.579
Liam Wyatt: That's the only question. Explicit question in the chat. So far with regards to the to the citation needed tool. That might be because it's blown everyone's mind so much that they can.
227
00:39:37.920 --> 00:39:53.819
Maryana Pinchuk: Well, whether or not you have questions I'm just curious to get some like verbal reactions, either in chat or like. What do you think? If you think this is a terrible idea, please please speak up. We want to hear this, too. We are here for your transparent feedback.
228
00:40:02.420 --> 00:40:07.030
Maryana Pinchuk: Thank you, Andrew. All caps. I like it.
229
00:40:07.140 --> 00:40:13.219
Liam Wyatt: Could you say more about why you like it, Andrew, please, please talk.
230
00:40:13.610 --> 00:40:17.150
Liam Wyatt: You should be able to unmute yourself if if you can't.
231
00:40:18.010 --> 00:40:20.229
Andrew Lih: yeah. Can you hear me? Okay, yes, you can.
232
00:40:21.540 --> 00:40:34.689
Andrew Lih: Hey? There! Wow! I'm tilted II think any way of bridging the day to day whether you wanna call pedestrian surfing and the the flow of someone's day
233
00:40:34.720 --> 00:40:39.409
Andrew Lih: with Wikipedia in a virtuous way, is a big plus. So
234
00:40:39.440 --> 00:40:43.320
Andrew Lih: that's what II like seeing about stuff like this.
235
00:40:43.370 --> 00:40:45.259
Andrew Lih: and not only just for our core
236
00:40:45.600 --> 00:40:58.369
Andrew Lih: set of editors who tend to have this already in their mindset of like. you know, as I'm browsing the web like, oh, I wonder if that's Wikipedia? Oh, I wonder what I can improve in Wikipedia? Oh, that's already part of our DNA in many ways. But anyway, to
237
00:40:58.480 --> 00:41:00.740
Andrew Lih: you know. tap.
238
00:41:00.970 --> 00:41:14.459
Andrew Lih: Folks who have this kind of a vibe, but have never have been always puzzled by the interface and the difficulty of adding it. Any tools like this, that bridge that, and make it much lower hanging fruit I'm all for, so it's great to see
239
00:41:16.370 --> 00:41:23.989
Liam Wyatt: something. I, Anthony, have a hand hand up from Anton in the chat. Would you like to speak, or would you like to type? And I will be to that for you?
240
00:41:25.500 --> 00:41:26.260
Антон Обожин: Hmm!
241
00:41:26.400 --> 00:41:51.460
Антон Обожин: I have a small remarks that it is wonderful that you're doing that in English. But many things in Wikimedia are too much based on English language, too much biased towards English language, like Commons, is completely in English, completely in English, and the foundation gives most of it's
242
00:41:51.600 --> 00:41:56.969
Антон Обожин: attention to English Wikipedia. Exactly. And
243
00:41:57.090 --> 00:42:03.039
Антон Обожин: other language versions. Other languages like
244
00:42:03.090 --> 00:42:05.630
Антон Обожин: attention from develop
245
00:42:05.680 --> 00:42:18.070
Антон Обожин: first from foundation and the struggle a little bit. So it's important to scale this project to as many languages as possible.
246
00:42:18.100 --> 00:42:21.350
Антон Обожин: and not only focus completely on English.
247
00:42:25.460 --> 00:42:29.780
Liam Wyatt: Mike, that's 2 votes. So far for more than English, please.
248
00:42:29.780 --> 00:42:37.609
Maryana Pinchuk: Just a just a quick comment on that. Yes, I you're absolutely right. And to to go back to the very start of this presentation
249
00:42:37.610 --> 00:43:01.760
Maryana Pinchuk: the aim of any future audiences. Initiative is not to build a finished product what we're doing is trying to understand whether there is interest in demand for this experience. And if so, we will stop the experiment. And we will hand this off to a team of engineers and designers and developers and interns a internationalization localization specialists that we that we have on staff.
250
00:43:01.830 --> 00:43:10.859
Maryana Pinchuk: who would make this available in all languages? So we are. We are not the team that would do that. But we are the team that would make that recommendation.
251
00:43:14.530 --> 00:43:16.840
Liam Wyatt: It's also important to get the
252
00:43:17.370 --> 00:43:19.120
Liam Wyatt: largest visibility
253
00:43:19.240 --> 00:43:24.649
Liam Wyatt: testing platform, because if it doesn't work in English. It's not going to work in other languages.
254
00:43:26.410 --> 00:43:29.810
Liam Wyatt: the is it? Is it correct to say, Mike, that
255
00:43:29.890 --> 00:43:35.549
Liam Wyatt: this is also the back end of this system is also based on the Wikimedia enterprise. Api.
256
00:43:36.640 --> 00:43:39.099
miishen: I'm I'm confusing something else.
257
00:43:39.580 --> 00:43:41.719
I'm not fully
258
00:43:42.030 --> 00:43:52.770
miishen: cognizant of the current state of the you know architecture. II know that we use the search Api, like we did with the Wikimedia or the Chachi Bt. Plugin.
259
00:43:52.810 --> 00:44:08.609
miishen: And I know that we've been working closely with enterprise. There's some discussion right now whether it's going to be hosted on tool forage or use the enterprise, you know, back in. So that's why, that's like as of this week. So that's why I don't have like full details on exactly which format it will take.
260
00:44:08.780 --> 00:44:24.230
miishen: I mean, currently our our sole engineer working on this part time. It's kind of sitting on this machine locally, basically. And so that's why, you know, we don't really know where it's going to be hosted yet. We're kind of working out these decisions as he's building it.
261
00:44:24.380 --> 00:44:28.919
miishen: but I believe it is going to be on tool forge.
262
00:44:30.140 --> 00:44:37.990
Liam Wyatt: and could you speak unless anyone else had their hands up? Please please come up, but I will add comments at Mike to elaborate further things.
263
00:44:38.320 --> 00:44:49.680
Liam Wyatt: Could you speak a bit about the name? Choice specifically what is not named. and the significance of that in terms of
264
00:44:51.170 --> 00:44:53.730
Liam Wyatt: the expectation management
265
00:44:53.870 --> 00:44:58.360
Liam Wyatt: of both Wikimedians and the wider Internet.
266
00:44:59.420 --> 00:45:08.059
miishen: Yeah, of course, thanks for bringing this up. I think we had a lot of deliberation on the name and most notably it's not called a fact checker like I said
267
00:45:08.260 --> 00:45:16.900
miishen: and it easily could be called that, because that's seemingly what the function is. But I think there's a lot of kind of
268
00:45:17.360 --> 00:45:34.560
miishen: technical to philosophical debates about. You know what fact and truth is, and what our position in asserting it or not asserting it is, and what I really wanted to avoid is a world in which we're saying, this is the facts of the world. You know, this is the truth in the world. We are the kind of decision makers about
269
00:45:34.580 --> 00:45:37.810
miishen: what truthfulness truthiness is, or something.
270
00:45:38.250 --> 00:45:50.179
miishen: And I and I really wanted to. If you stick, take a step back and look at the chat. Utt Plugin. One of the things we've learned is people trust our brand right? And so I'm kind of trying to thread this needle of saying, like
271
00:45:50.500 --> 00:46:09.310
miishen: Wikipedia, is for most people pretty reliable good resource. And so, you know, you should look at Wikipedia as a way of like verifying or checking information. It's not going to tell you the truth. People know. Sometimes there's mistakes. People know. Sometimes things are missing, and it doesn't mean it's truthful, or it's not truthful, but
272
00:46:09.640 --> 00:46:17.249
miishen: it does lend a good resource, and it does provide a good at least first step of like credibility.
273
00:46:17.300 --> 00:46:41.229
miishen: And so part of this experiment is also kind of seeing like what the power of that, you know Brand is. You know, like how powerful is Wikipedia as a resource for people, as a trustworthy kind of source of information. And then we provide like, here's something that looks relevant to what you were searching for. And then the kind of like needle to thread it's kind of like that final mile is like if people really do trust us as a source of information. Then
274
00:46:41.580 --> 00:46:51.400
miishen: we should see that the that last gap is kind of like, okay, well, it's on Wikipedia. Then, like, maybe my trust in this information goes up, and I can do more research to kind of like.
275
00:46:52.510 --> 00:46:54.839
miishen: Continue verifying this, for example.
276
00:46:54.990 --> 00:46:59.130
miishen: but we can add a call to action of
277
00:46:59.160 --> 00:47:00.639
Liam Wyatt: come over and edit.
278
00:47:01.000 --> 00:47:05.929
Liam Wyatt: especially the content which produces a a null result, perhaps.
279
00:47:06.190 --> 00:47:11.180
miishen: Exactly. Yeah. There. There is a question. Can you please go back to slide. 9, by the way.
280
00:47:12.160 --> 00:47:16.749
Liam Wyatt: So yes, what happens if there is a null result? Or if the fact that is returned
281
00:47:16.900 --> 00:47:20.400
Liam Wyatt: is a refutation, or
282
00:47:21.120 --> 00:47:25.189
Liam Wyatt: perhaps even vandalism. What's what's the negative result?
283
00:47:26.230 --> 00:47:37.020
miishen: So I think the most neutral result is to say that this is not support. Right. If we can't find anything, then it doesn't mean it's not true. We'll just say like, Hey, we couldn't find anything on Wikipedia to support this, you know, and
284
00:47:37.120 --> 00:47:40.420
miishen: that's once again not saying it's true or not true. So that's kind of
285
00:47:40.650 --> 00:47:49.050
miishen: perhaps an easier case. Right? There's a lot of information that is not on Wikipedia that should not be on Wikipedia, and it doesn't mean that it's not real
286
00:47:49.310 --> 00:47:59.570
miishen: with the AI in some cases will be able to tell whether there's conflicting information. So one of our mockups.
287
00:48:00.040 --> 00:48:24.089
miishen: there's, you know, a claim about like, I think, like aside varies being like super food or something, and like some dubious claims, health claims and then the Wikipedia article that we have in that example says, like, you know, maybe head your bets a little bit like maybe some of these claims aren't true. And so in that case, there's clear conflict of information, and we can label it as such and say, like.
288
00:48:24.170 --> 00:48:39.119
miishen: you know, this looks wrong, basically. Or these things don't match up right. Whatever this claim is, whatever is on Wikipedia, they don't match up, and we will indicate that in some way. I think at first we had very conspicuous things like check green checks, Red X's.
289
00:48:39.350 --> 00:48:46.199
miishen: and we might wanna we're considering, maybe backing off on that like, there's some signal that's important, right? Like, there's something important about like.
290
00:48:46.270 --> 00:49:06.960
miishen: I'm being prime to read the information in a in a useful way that, like something read means like, Oh, hey! I should be careful. I should like, look into this more. Something is off. So we want to keep that feeling. But without doing it in such a way that, like X, this is wrong. We've made it like a very strong judgment that this is like false etc.
291
00:49:06.960 --> 00:49:23.640
miishen: And so we don't know what final form is going to take out or take. Yet we're working with our designer to kind of keep that, like, you know, alert mode of that prime mode, but without being too strong, where somebody just sees the x, and then like never follows up, and just like thinks that you know the claim is wrong or something.
292
00:49:24.480 --> 00:49:27.359
Liam Wyatt: Can we see the video again as a request in the chat?
293
00:49:28.200 --> 00:49:40.260
miishen: Ii didn't show the video. It's 2 min long. This is a demo. Video. This is a screenshot from it. I'm I'm happy to show it. I don't know how good the quality will be, but I'm also wanting to be aware of the time.
294
00:49:40.690 --> 00:49:46.419
Liam Wyatt: We have 5 min, and there is not a queue of questions so I think it might be
295
00:49:47.000 --> 00:49:53.050
miishen: useful. Right? You could make it full screen. Yeah, give me a second.
296
00:49:55.440 --> 00:50:15.540
miishen: Hello, folks, this is Amin from the design team. And today I'm gonna give you a workthrough extension that we are working on with future audiences team. It's called citation needed. And it's a Wikipedia and Gpt power extension that you can use to quickly verify content online as you runs the web.
297
00:50:15.790 --> 00:50:18.730
miishen: So here, imagine you're reading an article
298
00:50:18.900 --> 00:50:24.290
miishen: on an entertainment website which says that Taylor student concert caused earthquake.
299
00:50:24.520 --> 00:50:34.499
miishen: So you have to browser extension install, and you can see here your browser chrome. And if you trigger the extension tells you that you have to highlight one sentence to start.
300
00:50:34.640 --> 00:50:47.809
miishen: So we're gonna do that. And now that we have highlighted a sentence, we can trigger the session again, and we see that the sentence is here extension, and you have a big verified button. So we're gonna trigger that.
301
00:50:48.240 --> 00:50:51.300
miishen: And now the new verification process is going to start
302
00:50:51.430 --> 00:51:12.320
miishen: here. We're going to be transparent about what's happening behind the scenes. And so we're gonna say, extension users that we are reading the sentence. Now we are disrupting relevant keywords from the sentence. In order, then, to trigger a search on Wikipedia and through the search, we're gonna then find and generate ourselves
303
00:51:12.410 --> 00:51:14.840
miishen: for papers. There's to be reviewing.
304
00:51:14.970 --> 00:51:25.270
miishen: And so once the result is ready, we're gonna load the article that we found we're gonna provide rather than quotes from the article. This case is about
305
00:51:25.280 --> 00:51:35.459
miishen: relevant information on these system activity. And we're gonna also give additional information that will be full signals of quality
306
00:51:35.470 --> 00:52:05.150
miishen: regarding the latest, adding a number of references and a number of people who work article. And beside the citation result, we're all gonna give quick links or quick direct links to visiting the article page. And we also have here. Gonna click! Here it's gonna bring us to the Wikipedia article with the citation highlighted. And people can continue reading. And yeah
307
00:52:05.200 --> 00:52:08.649
miishen: came to their own conclusions after years of accession.
308
00:52:08.810 --> 00:52:12.910
miishen: And that should be all. Thank you very much, and thank you soon. Bye, bye.
309
00:52:19.570 --> 00:52:25.439
Liam Wyatt: video. There's a question about whether this could or should be a booklet, a book market.
310
00:52:25.490 --> 00:52:39.749
Liam Wyatt: or an extension, or rather than necessarily a plug, into a browser. Sorry! An extension to present. What is the other ways of doing it? I'm assuming the answer is because this is the quick and easy way to start. And in the future, if it's very successful.
311
00:52:39.760 --> 00:52:42.930
Liam Wyatt: who knows how it might become more solid?
312
00:52:43.500 --> 00:52:47.230
miishen: Yeah, I didn't really paint a lot of the
313
00:52:47.590 --> 00:52:55.950
miishen: super idealistic feature state of this once again. That that is the answer. This is the quickest, easiest way for us to just get something in front of people to get results.
314
00:52:56.250 --> 00:52:59.450
miishen: if you ask me more like
315
00:52:59.930 --> 00:53:05.880
miishen: my end goal here, if this becomes a successful product is not to build like the best browser extension on earth or something.
316
00:53:05.890 --> 00:53:12.770
miishen: What I would really like to do is to prove the value of Wikipedia content as information verification
317
00:53:12.860 --> 00:53:38.160
miishen: especially in places where they're needed, such as like social media platforms and incentivize them to build it natively. So that like on Facebook, you could like, like comment or just verify something. And they just build it themselves. And they see the value. And then we provide maybe an Api or something. Instead of having to maintain a suite of like dozens and dozens of different plugins into different ecosystems, they would want to come to us and and have some
318
00:53:38.200 --> 00:53:44.729
miishen: and have something native that's going to work better and just be able to connect it and our information in a way that makes sense.
319
00:53:45.280 --> 00:54:02.350
miishen: And then also, I make a comparison with with with ad blockers. Right where, 15 years ago, on the Internet, people installed all sorts of extensions to manage their privacy and etc. And nowadays it's kind of just a built in way of using the Internet where most browsers have some rudimentary fund functions to
320
00:54:02.420 --> 00:54:20.710
miishen: block all the garbage online. And we're just seeing that there's different types of garbage online these days. And you know, is there a world in which you know Firefox or Chrome is gonna just build this in natively, and then once again we work with them directly. But we're not on the hook for building out and maintaining like dozens and dozens of like these different
321
00:54:21.020 --> 00:54:24.400
miishen: a plugins or extensions, or what have you?
322
00:54:26.750 --> 00:54:36.470
Liam Wyatt: Alright? Thanks, Mike. We are at time, and I know that Susana would like to have a moment to talk about an event that's coming up in Helsinki talent.
323
00:54:36.950 --> 00:54:54.389
Susanna Ånäs: I can just briefly say that II pasted 2 links in the chat because I could, of course, run through the, you know, mood board slides, but I think I might save time by just pasting the links. I can do that if you wish to stay on the call. I don't know.
324
00:54:56.450 --> 00:54:58.859
Susanna Ånäs: Yeah. Can I share.
325
00:54:59.320 --> 00:55:00.869
Liam Wyatt: I think, can you?
326
00:55:01.810 --> 00:55:02.940
Susanna Ånäs: Maybe
327
00:55:03.890 --> 00:55:09.549
Susanna Ånäs: it seems so. Yes, it seems many people are here already. So
328
00:55:09.710 --> 00:55:20.750
Susanna Ånäs: It's happening in Helsinki after the Tallinn Hackathon, and this is a 2 day event starting on Monday, the sixth, and lasting until the seventh.
329
00:55:20.790 --> 00:55:47.760
Susanna Ånäs: So who are joining this is sound like finished. AI finished glams. Research projects Wikimedians from the from the hackathon, and hopefully also very much from the foundation and the the chapters and affiliates staff, and then companies as well as then other open knowledge advocates.
330
00:55:47.970 --> 00:56:00.659
Susanna Ånäs: And here's the first day we'll have inspire talks in the morning. In the afternoon. We'll be having a like a ideation and group forming exercises for the whole afternoon, and then have sound
331
00:56:00.840 --> 00:56:15.920
Susanna Ånäs: and then start, you know, like working towards the next day, which is a practical collection of different practical activities for different groups of people, some some of them learning how
332
00:56:16.520 --> 00:56:28.579
Susanna Ånäs: learning like nuts and bolts about AI, just dipping their toe in the in the topic that could be also serious hacking with serious tools.
333
00:56:28.580 --> 00:56:50.870
Susanna Ånäs: And then it could be a strategic discussion between different advocates of open knowledge like policy issues that we we discuss with different types of methods and and practices. So it's a like a big playground for activities of this type. So so this is the timeline, like from
334
00:56:50.870 --> 00:57:08.950
Susanna Ånäs: empathizing to ideating, prototyping, even testing, you know, just trying out what's there, and maybe that, you know, like the the Plugin would have been a fun fun thing to test in that. That's and if you know that you have the link to the presentation, you also have links to
335
00:57:08.970 --> 00:57:22.209
Susanna Ånäs: planning document and project page, which is actually updated. We just shortened the name from glam plus commons plus AI sound into AI sauna. So it's just more.
336
00:57:22.220 --> 00:57:31.050
Susanna Ånäs: It's it like it will look better in a link. So III would really be happy to see many of you there.
337
00:57:33.190 --> 00:57:51.600
Liam Wyatt: That was lovely, and and I appreciate the quality, the visual quality of those slides. That is the there is one more further question. Child, people can certainly leave. We are over time. There's a question in the chat which deserves
338
00:57:51.780 --> 00:57:59.860
Liam Wyatt: mentioning. Since I have been asking for questions specifically on interactive content and the
339
00:57:59.950 --> 00:58:03.689
Liam Wyatt: capacity of the Wikimedia Foundation to support development, for that
340
00:58:03.970 --> 00:58:25.990
Maryana Pinchuk: was addressed it. Marshall, who is no longer in the call, since we're over time. But, Mariana, perhaps you can speak to that. I can speak to it. Yeah. So thank you, Sandy. It's a really great question. And I think the the main takeaway from what we're trying to do here is that before we can even talk about headcount we need to understand what it is we need to be building
341
00:58:25.990 --> 00:58:49.229
Maryana Pinchuk: whether that's an an amazing AI driven AI powered new reading experience, or a video platform, or a place where people can make interactive graphics and come together and and and collaborate over those, those, each of those things would be a completely different direction. Product and tech wise, and would require completely different resourcing.
342
00:58:49.230 --> 00:59:05.939
Maryana Pinchuk: So before we can even get into understanding, you know how many people we need, or even who we need, we need to understand what is needed for us. As a movement to evolve into. And and we still don't know. There are so many changes happening so rapidly.
343
00:59:05.970 --> 00:59:09.270
So that is the thing that we are trying to do first
344
00:59:09.410 --> 00:59:11.260
Maryana Pinchuk: before anything else.
345
00:59:15.170 --> 00:59:26.590
Liam Wyatt: Sorry, thank you, Mariana. Hope that answers the question. I wanna add in that this being the future audiences team. This isn't a team per se. It's a rag tag bunch of individuals rather than
346
00:59:26.840 --> 00:59:34.970
Liam Wyatt: other parts of the Wikimedia foundation technology and engineering community who are standing
347
00:59:35.040 --> 00:59:42.190
Liam Wyatt: groups who focus on apps or search, etc. This is experimenting in a variety of ways in general.
348
00:59:42.850 --> 00:59:46.050
I will now turn off the recording and upload this
349
00:59:46.240 --> 00:59:49.330
Liam Wyatt: two-week media commons shortly afterwards.
350
00:59:50.280 --> 00:59:54.319
Liam Wyatt: and people are welcome to stay here. Thank you for your time.
351
00:59:56.080 --> 00:59:57.219
Maryana Pinchuk: Thank you.
|