Wikirank/Details

This page considers Wikirank proposal in more detail from various angles.

Prior art

Wikirank differs from previously proposed lists in two ways. Firstly, Wikirank assumes that lists are already easily obtained by querying Wikidata. Wikidata actually has a whole query namespace reserved for lists, just waiting for T67626 to be completed. Wikirank instead focuses on filtering out insignificant items in these lists. Wikidata lists can be sorted too, but all factual sort keys (e.g. sorting things by size) are only trying to approximate desired sorting by relevance to the reader. Wikirank aims to collect and use popularity data, which is a direct measurement of relevance to the average reader. Secondly, Wikirank encourages frequent personalization of lists while showing these personalized lists under the same searchable URL as public lists, which considerably improves usefulness and usability of both public and private lists while at the same time encouraging contribution to the public list.

Wikirank differs from proposed directory projects in that it uses Wikidata Q-items instead of website URLs as item keys. This allows Wikirank to list items that do not have natural URL and to show single item for concepts that have a number of related URLs. It also allows Wikirank to show automatically translated facts from Wikidata whereas plain URLs would need language-specific informal descriptions.

Wikimedia already publishes pageview data (example toplist). This can be used to rank lists with well-defined members (e.g. 2020 movies). It will not work so well when the list is more open-ended like in the case of alternatives. Trying to use global popularity to score alternatives would make Microsoft Office the best alternative for everything.

There's an endless list of websites that incorporate some kind of voting. The list of websites that focus entirely on voting is much shorter, but there are still too many websites to list. Examples below try to highlight interesting examples that are approaching some of the ideals of Wikirank (universality, notability, openness, personalization).

Example sites that focus on voting:

• Ranker - entertainment-focused, a more serious example, upvotes highlighted, manual reordering, crowdsourced ontology, demographic and geographic filters, ad-supported
• AlternativeTo - apps and websites, example, voting is global (not list-specific), upvotes are highlighted, no reordering, all votes are public, crowdsourced P2P ontology (alternatives), ad-supported, lots of similar sites (alternative.me, TopBestAlternatives, ...)
• SaaSHub - apps and websites, example, voting is list-specific (but not fully transparent), upvotes are highlighted, no reordering, ad-supported, contributions are CC-BY-SA
• Slant - products, example, voting is global (not list-specific), upvotes highlighted, no reordering, crowdsourced ontology, ad-supported
• RankedByVotes - entertainment-focused, a more serious example, no vote highlighting or reordering, ad-supported

Sites with voting as an extra feature:

• StackExchange - Q&A, example, upvotes highlighted, no reordering, ad-supported
• GitHub - opensource software, example, upvotes (stars) highlighted, no reordering, all votes are public, freemium
• discussion forums (posts)
• social networks (posts and other content)
• shopping sites (products)

People can of course create offline lists, for example in personal wikis like Zim. The problem with this low-tech approach is that without collaboration, effort spent assembling the list by reviewing a lot of candidate list items has to be invested again and again by each individual user and a lot of time is collectively wasted.

Minimal version

Sections below describe Wikirank in detail, but most of the functionality is not required immediately. This section describes minimal functional version of the project.

Minimal version of Wikirank can be built in Wikispore. Public list pages have names like Wikirank:Some_topic. Private lists are stored in wikitext as subpages of user page, e.g. User:SomeUser/Wikirank:Some_topic. There is therefore zero privacy. Public list is periodically updated by a bot that scans all private lists. Javascript on public list page loads the private list page and merges it into the public list client-side. Another piece of javascript implements voting buttons by performing automated edits of user's private list. Similar script allows adding new items to the list.

The site is English-only. Login is required. There are no temporal or spatial segments. No new Wikirank-specific Q-items are created at this stage. Secondary sort key is used to compensate for low vote count. Spam filtering is manual via standard admin actions.

Estimated cost of the minimal version is under 100 man-hours.

Relationship with Wikidata

Every Wikirank page contains a list. Every list has a topic, which is identified by Wikidata Q-item. Every list item is also identified by Wikidata Q-item. Using Wikidata Q-items is the key to Wikirank's ability to merge private lists of its contributors into popularity-sorted public lists. Presence of list item in the list is interpreted as "instance of" relationship, i.e. list topics are classes in Wikidata parlance while individual list items are their instances. This "instance of" relationship does not have to be recorded in Wikidata. Wikirank just expects list items to have properties compatible with list topic. Users are able to add any item to any private list, but public lists will usually filter out incompatible list items.

Wikidata usually does not have classes like "GMail alternative" or "2020 sci-fi movie". If such lists are created in Wikirank, corresponding class Q-items will be created in Wikidata. There is however no need to manually add "instance of" statements to their potential instances. Class "2020 sci-fi movie" would just declare that it is "subset of" "science fiction film (Q471839)" and that it has "publication date" "2020". All Q-items in Wikidata with compatible properties are automatically accepted by Wikirank as possible list items.

Wikidata has a rich hierarchy of classes. Vote for a particular instance of a class does not extend to subclasses nor superclasses. This is because votes express good fit between the instance and its class. Quality of this fit drops as you move along class hierarchy up or down. Votes from related classes may be however used as a secondary sort key when the list is low on direct votes.

Site and page structure

TODO: UI mocks

Every list is a page. Page URL references Q-item of the list topic (e.g. www.domain.org/wiki/Q12345), so that the URL remains the same regardless of selected language. Page title is a natural language label for the list. Page content is normal wikitext, but it is mostly generated by templates and Lua scripts that exist for every kind of list. The following describes default page layout. Page starts with short summary of list's topic, usually formatted as property-value table. Most space on the page is dedicated to the list. List items are sorted by votes. If votes are scarce, secondary sort key may be used. Every list item has three columns. First column contains vote count, voting buttons, and other Wikirank features. Second column contains summary of the list item, usually formatted as property-value table. Third column contains optional image.

There is no manually added free-form content, because it would make the site language-specific. All information is retrieved from Wikidata and automatically translated using Wikidata language information (lexemes and item labels/descriptions). If there is any free-form text, it must come from Wikidata, for example item title and quotations. Wikirank may synthesize short text from Wikidata statements using techniques similar to Abstract Wikipedia. Additional information may be obtained from Wikipedia and other sources linked from topic and item summaries. Some content, for example Wikipedia links, may be automatically selected to match current language.

List pages may offer several views of the same list. It should be possible to switch between at least two layouts: the above described 3-column layout and plain table view. Table view places vote count and buttons in one of the columns. Further alternative views may include temporal and geographic segments. Subtopics and related topics should generally have their own pages, but current page can link to them prominently.

Upvoting an item moves the item to the top and visually marks it as upvoted. Downvoting an item moves the item to the bottom or completely hides it. Upvoted items are sorted among themselves using public vote count, but there is a way to override that order. Search widget at the end of the list allows users to easily find new items and add them to the list.

Besides list pages, there are item pages and category pages. Item pages, which are linked from list items, contain longer summary, links to lists featuring the item or related to the item (e.g. alternatives), links to sister projects, and additional useful links. Category pages are really just special list pages with list items pointing to other lists. Category pages allow voting on their list items, i.e. other list pages, but they also visually mark and bring to top lists, in which the user has casted a vote. This makes it easier to find one's private lists. There is one all-encompassing category containing all lists. This top-level category can be used by users to find all of their private lists.

Notability

Wikirank will generally allow creating lists of anything that already meets Wikidata notability policy. Quantitative surveys are generally useful for everything, even abstract concepts and unremarkable objects. While voting on such things as exoplanets seems useless at first, these votes are actually likely to point to exoplanets that are in some way interesting. Similar argument can be made about most other topics. There will be only a few exceptions, for example adding people as items in defamatory lists. Wikirank will tend to extend beyond current Wikidata scope, which is largely defined by what Wikipedia considers notable. This is because, in a sense, Wikirank is a way to discover notable items and to separate notable items from uninteresting ones. It is therefore natural that a large fraction of items in Wikirank lists will not be considered notable by other projects.

Data quality

Wikirank will not attempt to censor "wrong" choices. The idea is to reflect public opinion even if public at large is wrong about something. Clarifying notice may be placed above lists that have high risk of misinterpretation. List items may show additional properties pulled from Wikidata that complement Wikirank's popularity data.

All data added to Wikidata in the course of editing Wikirank pages still has to be factually correct and meet Wikidata policies.

Little attention is paid to which item is ranked #1. Wikirank only aims to separate serious contenders from uninteresting alternatives. Precise order near the top of the list is not important. Users are expected to review the highest ranking items and make their own choices.

Collaboration model

The usual collaboration model on wikis is that everyone edits the same content and disputes are resolved via discussion. Wikirank collaboration model is more subtle. Everyone has their own version of every list, but list items preferred by other contributors are shown underneath user's private list, sorted by popularity, which encourages users to add popular items to their own list. In order to keep their private list clean and useful, users are most likely to upvote unreasonably unpopular items buried near the bottom of the public list and downvote unreasonably popular items that they consider to be clutter. This creates self-regulating system that converges toward interests of the majority while still allowing minority users to adjust the list to their liking.

Wikirank should be home to several kinds of contributors, sorted here by amount of time invested:

• Supporters do not build lists. They just drop random upvotes here and there to show support.
• Curators build private lists, mostly by upvoting. They find or create new items to add and cast the first vote for these items.
• Editors create new lists and make other structural changes to the site as well as non-trivial edits in Wikidata.
• Developers create templates, Lua scripts, and special lists, which mostly define new list types.

Login is optional, because supporters don't need it. Curators who use the site only temporarily, for example to build a list for immediate use, don't need login either.

Privacy

Existing voting systems usually do not offer any guarantees of privacy. Social networks either publish the whole list of voters or at least reveal which friends (connected users) voted for the item. Votes are used to target advertising and they are thus indirectly revealed to advertisers. Wikirank sets higher standard of privacy by never showing private lists to anyone other than the user.

Wikimedia servers already store some private data in the form of password hashes and IP addresses. Wikirank is different in that most of its data will be private. This is essential for the project to function, because private lists will end up containing sensitive information, e.g. drug lists, medical condition lists, lists related to intimate life. Leaks of this data would erode trust and undermine the project. High standards of privacy protection and data security are therefore essential to the success of the project.

But before considering data protection, let's first consider why is the data needed in the first place. Why cannot Wikirank perform spam filtering at the moment of voting and then just store vote statistics? If preventing duplicate votes is a concern, why cannot Wikirank use one of the anonymizing counters that perform lossy hashing of IP addresses and/or user names? The reason Wikirank needs to keep detailed vote data is that Wikirank is a curated list service, not just the usual "like" or "thumbs up" service. You cannot curate a list if you cannot see it. You cannot see the list if it is not recorded in database in full detail. Wikirank might offer anonymized vote casting in the future, but its core function will always be to collect full curated lists.

Users would be of course given control over their personal data. Wikirank would allow users to examine, download, and delete their private data or any part of it. Secure and caution-encouraging procedure would allow migration of private data to another service upon user's request. Option to minimize data collection would be also provided. This would cause Wikirank to anonymize IP addresses (by zeroing the least significant bits) and timestamps (by rounding) and to keep only current state of private lists. This option would come with a few downsides: loss of undo and versioning functionality, removal of one's historical votes from older time segments, and slightly higher chance that one's votes would be flagged by spam filters. Some aspects of this opt-out functionality would be also activated automatically where required by privacy laws.

Protecting private data for several decades is challenging. Servers and processes touching raw Wikirank data would have to meet security standards that are not needed in other Wikimedia projects. Developers and administrators would not have access to private lists. Site administrators would be only able to examine summary statistics when flagging suspicious votes. Spam filters would have full access to raw data, but they would run in isolated environment, unable to communicate with outside world.

Private lists of logged out users are tied to browser cookie rather than IP address. Besides being a good match for user's expectations of site behavior, this prevents accidental leaks of private lists when user's IP address is rotated by their ISP.

Since the first votes on a list tend to be correlated with public activities like creating the list page, its Q-item, and Q-items for all list items, the first vote for most items can be deanonymized. This is particularly problematic on fringe lists with naturally low vote count. Wikirank will use two mechanism to protect users against such deanonymization. Firstly, counting of new votes will be randomly delayed for long enough to make it unclear who casted the vote. Secondly, Wikirank will by default hide vote count of list items with too few votes (say, less than 3). Users will be asked for permission before their vote is shown on such items. It is expected that people interested in curation more than privacy will grant the permission. Minimum vote count will be randomized a bit to prevent active deanonymization by casting fake votes.

Spam filters

Early versions of Wikirank will run without spam filter, because the initial site will not be popular enough to attract spammers. Spam filters should be developed in response to actual spamming strategies observed in practice. This section provides some hints on how such spam filters might work.

Spam filters work on user level, not vote level, because single user is either a spammer or regular user. Decisions about individual users however cannot be made, because privacy protection means nobody can look at user's private lists. Users are instead automatically scored using a number of trust signals and votes from users below some trust threshold are ignored. Trust signals, positive and negative, include:

• casting a suspicious vote (see below)
• activities other than voting (editing, activities on sister projects)
• passive technical signals: User-Agent, presence of cookies, etc.
• passing or failing an occasional CAPTCHA (only when trust is low)
• voting patterns similar to other users with high/low trust

Votes can be flagged as suspicious, but this is not done individually per vote. Administrators can instead review suspicious activity, especially temporal and spatial spikes, and flag entire sets of votes as suspicious.

Since no signal is reliable, only total trust score matters. Scoring model is trained on hard signals (CAPTCHA, non-trivial edits, bot User-Agent, known fraud cases). False positives are not of much concern, because accidentally filtered users will see no change in site behavior. Spam filters only affect overall vote counts in public lists.

Bugs and flaws in spam filters do not cause permanent damage. All filtering scores are periodically recalculated in order to reflect improvements in spam filter logic. Some historical data may be discarded for privacy reasons. Such information loss events should leave behind a flag, so that spam filters know that some data is missing.

Extras

The basic concept of Wikirank can be further extended with a number of features.

Shared lists

Users can share their private lists, which is particularly useful for influencers with significant reach and experts with authority in the field. Private lists can be shared under special URL, e.g. www.domain.org/share/6fR3tZhSHPp1tnUjY6ko. This URL is secret, which allows users to provide access only to people who know the URL. Unsharing the list and sharing it again generates new URL. This allows users to revoke access to their private list. Shared lists may be optionally frozen, so that they do not change after publishing even as user's private list continues to change.

Spatial segments

Wikirank will collect location data (either full IP addresses or anonymized IP addresses with last few bits removed) anyway for the purpose of spam filtering. This data can be used to construct spatial segments of popular lists, which would include only votes from people living in certain area. List segments are always shown on the same page as the original list. They can take the form of regional filter, tab-like navigation between preselected regions, or a map. In some cases, especially for inherently regional topics, relevant spatial segment may be the default view of the list.

Temporal segments

Wikirank will record votes with timestamps and it will keep history of voting. Timestamps can be used to group votes by time period. This is useful for spam filtering, but more importantly it allows construction of temporal segments of vote data. Temporal segments enable a number of features: change indicators, "trending" view, charts of change over time. Some of these features can be enabled by default for time-sensitive topics.

Personal notes

Personal notes and tags can be used to annotate list items and to create sketches of new items that are not yet in Wikidata. Personal list may also override any property of any list item. All this information is only present in private lists. Shared lists may include it too. Personal notes can be used as a workaround for any flaw of the public list, especially disputes about notability and representation of new items and new item properties. Wikirank software can use personal notes to suggest Wikidata edits that would make the information public. Frequently occurring tags and personal notes (with some minimum frequency threshold to protect privacy) may be queried and used by Wikirank editors as a suggestion to improve Wikidata.

Experimental items

This is a possible intermediate step between personal notes and Wikidata editing. Since Wikirank tends to explore the space of possibilities, it might include a lot of content that is not particularly notable. While Wikirank is intended to motivate contributions to Wikidata, some of the items that are reasonable on Wikirank might be controversial on Wikidata. Experimental items are like Wikidata items, but they are part of Wikirank project without being exposed to wider community via Wikidata. Experimental items have all the advantages of Wikidata: they allow merging of personal lists and adding item properties. They however do not involve the cost of wide consensus and notability proof associated with Wikidata items. Once any pending issues with experimental items are resolved, they can be moved to Wikidata. When an item in Wikidata is deleted while still in use on Wikirank, it can be recreated on Wikirank as an experimental item.

Imported data

Suitably licensed third party popularity data, e.g. Wikimedia analytics, and other reference data, e.g. lead sentences/paragraphs from Wikipedia, may be shown in Wikirank. This is useful when Wikidata does not include the data for practical reasons (size, frequent change) or licensing reasons (licenses compatible with Wikirank but not Wikidata).

Multiple personal lists

Upvotes and downvotes alone do not always accurately describe how people think of items in the list. For example, people might be tempted to withdraw upvote or even downvote movie they have already seen, because it is no longer relevant to them. When this happens, list pages can always ignore downvotes and sort items by upvotes alone, but this is suboptimal too, because downvotes are useful. One possible solution is to offer users more options than upvotes and downvotes. For example, in the mentioned movie example, a button titled "Watched" would move the film into separate personal list rather than downvoting it. Multiple personal lists may be also created manually by users when list page defaults do not apply. Upvoting the same item in multiple personal lists would of course count as single upvote when constructing the public list.