Objective Revision Evaluation Service/Issues

ORES makes mistakes. It's not as smart as humans are. In order to make sure ORES is as useful as possible despite its limitation, documentation around its issues and mistakes is critical. This page is intended to be a central reference for reporting and discussing issues with ORES.

Edit quality

See /Edit_quality for reporting misclassifications.

Bias against anonymous editors

• T118982 -- hewiki "reverted" model weights strongly against anons
• T129624 -- Investigate nlwiki 'reverted' model seems broken (always ~0.89 for anonymous edits)
• T120138 -- [Epic] Explore disparate impacts of damage detection and goodfaith prediction on anons and newcomers.
• Research talk:Automated classification of edit quality/Work log/2016-04-14 -- Analysis of bias against anons when switching in GradientBoosting model
• Presentation by EpochFail that discusses mitigating bias against anons (slides)

We're working to mitigate the problem by increasing signal from non-user-related features.

• T144636 -- [Epic] Implement PCFG features for editquality and draftquality
• T145812 -- Implement ~100 most important hash vector features in editquality models
• T162617 -- Use 'informals', 'badwords', etc. in Wikidata feature set