Hello. We were read only from 14:02 to 14:47 UTC. After then, between 14:47-15:10 UTC the recent changes list did not work. We have no ability to check vandals in these 23 minutes. Can you restore the data, please?
Thank you.
Description
Details
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | Smalyshev | T133046 WDQS stopped updating during datacenter switch | |||
Resolved | aaron | T133053 Rebuild Recent changes for 2016-04-19 14:48 - 15:10 UTC due to Dallas data centre migration |
Event Timeline
Well, Quarry ignores queries on the lost time in recentchanges table, but has all the data in revision table.
There is a script called rebuildrecentchanges.php, but it would need some adjustments to work on a time range (right now it clears all recentchanges data and rebuilds it from scratch, which takes a long time and also would lose any patrolling information for older changes).
I've updated this task with some of the information from @mark's email to the engineering list.
I'm going to work on the script, ops said they'll review it and run it across the wikis.
Change 284248 had a related patch set uploaded (by Bartosz Dziewoński):
[WIP] rebuildrecentchanges: Allow rebuilding specified time range only
https://gerrit.wikimedia.org/r/284248
Change 284248 merged by jenkins-bot:
rebuildrecentchanges: Allow rebuilding specified time range only
https://gerrit.wikimedia.org/r/284248
Should probably find out what the exact time range is, to avoid missing anything or duplicate RC entries for some actions.
You know, @matmarex, there can be a chance that the range finished in different time on different wikis.
I don't think so, but different mw servers would've got the fix at slightly different times.
Change 284274 had a related patch set uploaded (by Ori.livneh):
rebuildrecentchanges: Allow rebuilding specified time range only
https://gerrit.wikimedia.org/r/284274
Change 284274 merged by Ori.livneh:
rebuildrecentchanges: Allow rebuilding specified time range only
https://gerrit.wikimedia.org/r/284274
Change 284347 had a related patch set uploaded (by Aaron Schulz):
RebuildRecentchanges improvements
https://gerrit.wikimedia.org/r/284347
To actually view these 20 minutes of changes that will need reviewing, try viewing a page like this:
(substitute "commons.wikimedia.org" for your wiki; the additional query parameters are to hide boring changes) …scroll to the bottom, and look at the changes in the affected period (14:48 - 15:10 UTC).
Unfortunately, Special:RecentChanges can show only 5000 changes, and on very active wikis (e.g. English Wikipedia), this might already be outside the limit :( I'm not sure what to do there, they can probably be queried via the API somehow…
(Also, for future reference: what exactly was the config problem and what was the fix? Anyone has a link handy?)
Change 284347 merged by jenkins-bot:
RebuildRecentchanges improvements
https://gerrit.wikimedia.org/r/284347