Page MenuHomePhabricator

Performance regression in wmf15 roll-out
Closed, ResolvedPublic

Assigned To
Authored By
ori
Feb 4 2015, 10:43 AM
Referenced Files
F37636: render2.png
Feb 5 2015, 10:11 PM
F37632: render.png
Feb 5 2015, 10:11 PM
F37309: render.png
Feb 4 2015, 10:35 PM
F37021: pasted_file
Feb 4 2015, 10:43 AM
F37019: pasted_file
Feb 4 2015, 10:43 AM
Tokens
"Haypence" token, awarded by RandomDSdevel.

Description

The roll-out of wmf15 to non-Wikipedias appears to have introduced a significant performance regression, which registered as a 1-second increase in median page load times across the cluster. The regression went away when I reverted https://gerrit.wikimedia.org/r/#/c/188409/.

pasted_file (500×1 px, 121 KB)

pasted_file (371×600 px, 26 KB)

Changelog: https://www.mediawiki.org/wiki/MediaWiki_1.25/wmf15

Event Timeline

ori raised the priority of this task from to Unbreak Now!.
ori updated the task description. (Show Details)
ori added subscribers: ori, greg.

Looks like the rollback messed up ResourceLoader caching somehow, see T88576.

Group1 was put back on 1.25wmf15 at 2015-02-04T20:33Z. mediaWikiLoadComplete.desktop timings rose again, but the rise started a ~19:40Z and has basically held steady since.

If all is well over night, the plan is for @Reedy to push out wmf16 to testwikis (group0) and wmf15 to the wikipedias (group1 and group2).

greg claimed this task.
20:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf16
20:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf15

Back on track.

I think we look good enough, given everything that's happened today/yesterday with outages and loss of graphite data:

render.png (400×800 px, 33 KB)

render2.png (400×800 px, 34 KB)