Page MenuHomePhabricator

Move the train deployment from Thursday to Wednesday for some Wikipedia sites
Closed, ResolvedPublic

Description

TL;DR: MediaWiki and extensions are deployed to all Wikipedias on a train every Thursday. There may be some advantages in moving deployment to some languages to Wednesday. This is an examination of the pros and cons of such move. Actions for particular languages should probably be discussed in separate tickets.

In the current schedule of the weekly deployment train, whatever is merged before the California morning every Tuesday is branched and deployed to mediawiki.org and several testing sites, then on Wednesday it's deployed to non-Wikipedia sites and on Thursday - to Wikipedia sites.

This generally works well, but it has a disadvantage for features that are only deployed on Wikipedia, such as ContentTranslation. Whereas features such as VisualEditor and MobileFrontend, not to mention the core, can be tested on mediawiki.org and other sites before they are live on Thursday and urgent issues can be fixed. testwiki can work, too, but it's not a perfect replication of the live Wikipedias - it doesn't have all of Wikipedias' articles, templates, user load and multilingualism.

I therefore suggest moving Wikipedias in some languages to Wednesday. The the current schedule page says that theoretically it's possible and has already been discussed in relation to the Chinese Wikipedia, but this hasn't actually happened yet. I spoke to @Kippelboy from the Catalan Wikipedia and to @KuboF from the Esperanto Wikipedia. @Kippelboy already started a discussion about this in the Catalan Wikipedia, and the community responses are generally positive - the editors community is willing to try it. (Request for Comment on Esperanto Wikipedia opened; -KuboF)

So, using Catalan just as an example for now, here's how it is supposed to work:

  • The new branch is created on Tuesday and deployed to mediawiki.org and test* sites.
  • It is deployed to non-Wikipedia sites and to the Catalan Wikipedia on Wednesday.
    • If anybody notices very severe ("Unbreak Now") bugs on non-Wikipedia sites or on the Catalan Wikipedia after the train runs, these bugs are fixed and backported to the branch.
  • The branch is deployed to all Wikipedia sites on Thursday; the urgent fixes, if there are any, are deployed to all languages, including Catalan.
  • If this doesn't work well for the community for whatever reason, we revert to the current way - all Wikipedias on Thursday.

Advantages:

  • Ability to test Wikipedia-specific features early in a real Wikipedia environment, with real content, gadgets, templates, and users.
  • Ability to fix urgent bugs before they hit the other languages on Thursday.
    • In particular, this means that it's possible to fix urgent stuff before the weekend. Usually deployments aren't done on Friday, except the super-urgent ones. So all languages, including those that are deployed early, get the chance to enjoy the fixes earlier.
  • No special SWAT deployments should be needed to make this possible. This is supposed to become a part of the deployment train configuration.
  • Rolling deployments are a generally good industry practice.

Possible concerns:

  • This must be compatible with the deployed services, like Parsoid and cxserver.
  • Care must be taken that ContentTranslation, and possibly other features, is compatible across languages. For example, ContentTranslation uses a common database to store translations in progress, suggestion lists, etc., and the database usage must be in sync across all the projects. Usually, this is not supposed to be a blocker, but it may require a bit more diligence in deployment. This may also apply to some other features.
  • Catalan is not necessarily the best first language for testing. It is an advantage to test with a lot of users, but it's a problem if something breaks, so it may be worth picking some languages with less activity for the testing.
  • There's little time to translate new messages that come with Tuesday's latest patches (but it's not new because it affects non-Wikipedias already, and it is not much of problem if the translators are quick).

Of course, there may be more points in both lists. Your further opinions and analysis about pros and cons are welcome.

The actual configuration work to make this happen will probably have to be done by Release Engineering and Ops, but there are a lot of stakeholders, hence the long CC list.

Thanks.

Event Timeline

Amire80 raised the priority of this task from to Needs Triage.
Amire80 updated the task description. (Show Details)
Amire80 added subscribers: Amire80, Kippelboy, greg and 12 others.

(I asked because I do not think a Phabricator task is a good place to discuss such a complex topic.)

It's fine to discuss here, or on wikitech-l, or both.

(NB: releng@ is our private team only list, qa@ is our public list, yes I
don't like the naming)

KuboF set Security to None.

Would this simply mean a change to the dblist files so that the 'canary' wikipedia language sites would be moved into group1? The possible concerns mentioned in the description do concern me a little but overall this isn't a terrible idea.

Just chatted with @santhosh in the office about this and another good wiki to potentially move would be Hebrew Wikipedia.

@dduvall: can you give a heads up to this task, and mention the possibility of Hebrew wiki moving to Wednesday's train day (ie: one day early), in the SoS? Purpose is to see if any other dev teams have any concerns with this from their end (moving a wikipedia up a day, generally, and hewiki specifically). Thanks! :)

I'll have to ask the community, but I like it.

Is it Catalan and Hebrew? cawiki already agrees.

@Amire80 As long as that's ok with everyon, yes :) Sorry, @santhosh and I were just talking about Hebrew in the office.

So, status summary.

  • Catalan looks positive, right?
  • Hewbrew has only supports (and nothing new for a while): link.

We can make this change (moving those two wikipedias to "group 1") starting at any point, technically. Should we just do it?

Yes, both Catalan and Hebrew are positive.

As far as I am concerned let's do it.

Change 253371 had a related patch set uploaded (by Greg Grossmeier):
Move Catalan and Hebrew Wikipedias to group1

https://gerrit.wikimedia.org/r/253371

Per T118212: Take heat off day before the weekly branch-cut? I'd think we want to slow down the train rather than speed it up...

Per T118212: Take heat off day before the weekly branch-cut? I'd think we want to slow down the train rather than speed it up...

That's not about slowing down the train. The fix to that is a real code pipeline where every commit spends the "same" amount of time in testing/beta cluster before being deployed to production. Ironically, the easiest way of doing *that* is by speeding up the train.

Alright, given the next two weeks we won't be having the train anyways (see: https://wikitech.wikimedia.org/wiki/Deployments#Upcoming ), let's plan for this to happen the week of Dec 7th. I've commented as such on the patch and will add it to the Deployments calendar as well. cc @mmodell for visibility :)

Luke081515 triaged this task as Medium priority.Dec 9 2015, 8:43 AM

Change 253371 merged by jenkins-bot:
Move Catalan and Hebrew Wikipedias to group1

https://gerrit.wikimedia.org/r/253371

Looks like this is done and nothing exploded :)

Can this be closed?

greg claimed this task.

Looks like this is done and nothing exploded :)

Can this be closed?

really delayed response, but... yes!