For the potential impact on HTTPS clients Chris Steipp told me on IRC he looked into what I assume is EventLogging data and later told me that Oliver had done some analysis work on that. I wanted to ask Oliver if he could publish his queries (or SQL and R code or whatever he used), but haven't yet done so (feel free to do that). The idea was also to compare before and after deployment. It would be interesting if we could publish an aggregated and anonymized analysis of the before and after comparison.

(In reply to Jan Zerebecki from comment #11)

It would be interesting if we could publish an aggregated
and anonymized analysis of the before and after comparison.

You know about:

• https://gdash.wikimedia.org/dashboards/frontend/ ,
• http://ur1.ca/hn8fi
• and any other graphite graph you may desire,

don't you?

None of those on gdash differentiate between HTTP and HTTPS. I do not have full graphite access, so the ability to create something that might help may exist.

gerrit.wikimedia.org still does not support Forward Secrecy.

• https://www.ssllabs.com/ssltest/analyze.html?d=gerrit.wikimedia.org

wikitech.wikimedia.org also doesn't support Forward Secrecy.

More importantly, SSL Labs says Wikitech server is "vulnerable to the OpenSSL CCS vulnerability (CVE-2014-0224) and exploitable".

• https://www.ssllabs.com/ssltest/analyze.html?d=wikitech.wikimedia.org

(In reply to chmarkine from comment #15)

wikitech.wikimedia.org also doesn't support Forward Secrecy.

More importantly, SSL Labs says Wikitech server is "vulnerable to the
OpenSSL CCS vulnerability (CVE-2014-0224) and exploitable".

• https://www.ssllabs.com/ssltest/analyze.html?d=wikitech.wikimedia.org

F to A- now

Yes and there are more sites that still lack forward secrecy. Now that there is an acceptable configuration with FS we can just apply that one to them. Some like wikitech and gerrit can probably use one that is less backwards compatible (like no SSL3, disable RC4, difficult: disable non-fs ciphers).

I agree with Jan. I think disabling SSL3 and non-fs ciphers is feasible, because only IE 6-8 on XP do not support any FS ciphers, only IE 6 does not support TLS 1.0 or higher, and even IE 7 on Vista supports ECDHE.

Also ticket.wikimedia.org does not support PFS. So all together:

• gerrit.wikimedia.org
• wikitech.wikimedia.org
• ticket.wikimedia.org

https://www.ssllabs.com/ssltest/analyze.html?d=ticket.wikimedia.org

I just find more and more sites with no FS:

• gerrit.wikimedia.org
• wikitech.wikimedia.org
• ticket.wikimedia.org
• lists.wikimedia.org
• dumps.wikimedia.org
• graphite.wikimedia.org
• gdash.wikimedia.org

Again, graphite.wikimedia.org, gdash.wikimedia.org and dumps.wikimedia.org are "vulnerable to the OpenSSL CCS vulnerability (CVE-2014-0224) and exploitable".

lists.wikimedia.org is "vulnerable to the OpenSSL CCS vulnerability (CVE-2014-0224), but probably not exploitable", and lists.wikimedia.org does not support TLS 1.1 and TLS 1.2.

[1] https://www.ssllabs.com/ssltest/analyze.html?d=graphite.wikimedia.org (F)
[2] https://www.ssllabs.com/ssltest/analyze.html?d=gdash.wikimedia.org (F)
[3] https://www.ssllabs.com/ssltest/analyze.html?d=dumps.wikimedia.org (F)
[4] https://www.ssllabs.com/ssltest/analyze.html?d=lists.wikimedia.org (B)

See also: https://bugzilla.wikimedia.org/show_bug.cgi?id=67564

meanwhile dumps and lists have been fixed it seems

dumps.wikimedia.org
Experimental: This server is not vulnerable to the OpenSSL CCS vulnerability (CVE-2014-0224).

lists.wikimedia.org
Experimental: This server is not vulnerable to the OpenSSL CCS vulnerability (CVE-2014-0224).

It's a bit unpractical to have one comment for each domain. Jan and chmarkine, it would be IMHO more useful if you resurrected https://wikitech.wikimedia.org/wiki/Httpsless_domains to make a table of which domains have https but lack PFS.

Change 144731 had a related patch set uploaded by Dzahn:
update SSL cipher list for gerrit to support PFS

https://gerrit.wikimedia.org/r/144731

Change 144734 had a related patch set uploaded by Dzahn:
update SSL cipher list for OTRS to support PFS

https://gerrit.wikimedia.org/r/144734

Change 144736 had a related patch set uploaded by Dzahn:
update SSL cipher list on wikitech to support PFS

https://gerrit.wikimedia.org/r/144736

all services behind the misc. varnish cluster should be fixed now. they were lacking an nginx restart on cp1043/cp1044, which i did now

this should have fixed all these:

doc
git
gdash
graphite
parsoid-tests
performance
integration
releases
legalpad
logstash
scholarships

Change 144731 merged by Dzahn:
update SSL cipher list for gerrit to support PFS

https://gerrit.wikimedia.org/r/144731

(In reply to Nemo from comment #22)

It's a bit unpractical to have one comment for each domain. Jan and
chmarkine, it would be IMHO more useful if you resurrected
https://wikitech.wikimedia.org/wiki/Httpsless_domains to make a table of
which domains have https but lack PFS.

I made such a list: https://wikitech.wikimedia.org/wiki/User:Chmarkine/HTTPS

It summarizes support status for Forward Secrecy and HSTS. It also shows protocol versions, whether HTTP redirects to HTTPS, links to SSL Labs and SSL Labs grades.

It is an incomplete list. Please feel free to update it or move it to main namespace, if you want!

also see the older wiki page that just focused on domains without https

https://wikitech.wikimedia.org/wiki/Httpsless_domains

chmarkine: very nice list, thanks!

I just wanted to add that even though i have those (partly pending) patches to enable it on gerrit,wikitech,otrs ..it will not actually work before Apache is also a 2.4 version. But do you agree i should merge already anyways,based on it being an improvement anyways? Then it would just automatically be supported as soon as Apache will be upgraded.

(In reply to Daniel Zahn from comment #30)

chmarkine: very nice list, thanks!

I just wanted to add that even though i have those (partly pending) patches
to enable it on gerrit,wikitech,otrs ..it will not actually work before
Apache is also a 2.4 version. But do you agree i should merge already
anyways,based on it being an improvement anyways? Then it would just
automatically be supported as soon as Apache will be upgraded.

I agree! I think we should definitely merge them.

Change 144734 merged by Dzahn:
update SSL cipher list for OTRS to support PFS

https://gerrit.wikimedia.org/r/144734

Change 144736 merged by Dzahn:
update SSL cipher list on wikitech to support PFS

https://gerrit.wikimedia.org/r/144736

Change 146510 had a related patch set uploaded by Chmarkine:
update SSL ciphers for contacts.wm.org to support PFS

https://gerrit.wikimedia.org/r/146510

Change 146510 merged by Dzahn:
update SSL ciphers for contacts.wm.org to support PFS

https://gerrit.wikimedia.org/r/146510

Change 147110 had a related patch set uploaded by Chmarkine:
update SSL ciphers for Ganglia to support PFS

https://gerrit.wikimedia.org/r/147110

Change 147123 had a related patch set uploaded by Chmarkine:
update SSL ciphers for noc.wikimedia.org to support PFS

https://gerrit.wikimedia.org/r/147123

Change 147110 merged by Dzahn:
update SSL ciphers for Ganglia to support PFS

https://gerrit.wikimedia.org/r/147110

Why does ganglia still get a B from Qualys SSL Labs after the change, while others are fine?

Change 147123 merged by Dzahn:
update SSL ciphers for noc.wikimedia.org to support PFS

https://gerrit.wikimedia.org/r/147123

It is B for ganglia because that old of an libssl and apache do not support newer TLS versions. ganglia / nickel.wikimedia.org is still on Ubuntu Lucid.

Change 147185 had a related patch set uploaded by JanZerebecki:
racktables - update SSL cipher list

https://gerrit.wikimedia.org/r/147185

Change 147196 had a related patch set uploaded by JanZerebecki:
smokeping - update SSL cipher list

https://gerrit.wikimedia.org/r/147196

Change 147199 had a related patch set uploaded by JanZerebecki:
etherpad - update SSL cipher list

https://gerrit.wikimedia.org/r/147199

Change 147207 had a related patch set uploaded by JanZerebecki:
icinga - update SSL cipher list

https://gerrit.wikimedia.org/r/147207

Change 147208 had a related patch set uploaded by JanZerebecki:
generic_vhost (webserver) - update SSL ciphers

https://gerrit.wikimedia.org/r/147208

Change 147214 had a related patch set uploaded by JanZerebecki:
metrics - update SSL cipher list

https://gerrit.wikimedia.org/r/147214

Change 147196 abandoned by Dzahn:
smokeping - update SSL cipher list

https://gerrit.wikimedia.org/r/147196

Change 147199 merged by Dzahn:
etherpad - update SSL cipher list

https://gerrit.wikimedia.org/r/147199

Change 147185 merged by Dzahn:
racktables - update SSL cipher list

https://gerrit.wikimedia.org/r/147185

Change 147214 merged by Dzahn:
metrics - update SSL cipher list

https://gerrit.wikimedia.org/r/147214

Change 147715 had a related patch set uploaded by Chmarkine:
rt -- update cipher suite list to support PFS

https://gerrit.wikimedia.org/r/147715

Change 147739 had a related patch set uploaded by Chmarkine:
blog -- update cipher suite list to support PFS

https://gerrit.wikimedia.org/r/147739

Change 147740 had a related patch set uploaded by Chmarkine:
ishmael -- update cipher suite list to support PFS

https://gerrit.wikimedia.org/r/147740

Change 147739 abandoned by Chmarkine:
blog -- update cipher suite list to support PFS

https://gerrit.wikimedia.org/r/147739

Change 148618 had a related patch set uploaded by Chmarkine:
tendril -- update cipher suite list to support PFS

https://gerrit.wikimedia.org/r/148618

Change 148624 had a related patch set uploaded by Chmarkine:
planet -- update cipher suite list to support PFS

https://gerrit.wikimedia.org/r/148624

Change 148631 had a related patch set uploaded by Chmarkine:
svn -- update cipher suite list to support PFS

https://gerrit.wikimedia.org/r/148631

Change 149267 had a related patch set uploaded by Chmarkine:
icinga-admin -- update cipher suite list to support PFS

https://gerrit.wikimedia.org/r/149267

Change 149267 merged by Dzahn:
icinga-admin -- update cipher suite list to support PFS

https://gerrit.wikimedia.org/r/149267

I just found that https://payments.wikimedia.org is still using the old cipher suite list:

TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA

https://www.ssllabs.com/ssltest/analyze.html?d=payments.wikimedia.org

Change 178676 had a related patch set uploaded (by JanZerebecki):
Change ru.wikinews.org to HTTPS only.

https://gerrit.wikimedia.org/r/178676

Patch-For-Review

Fundraising-tech should be added for the payments.wm.org comment above

Change 178676 abandoned by JanZerebecki:
Change ru.wikinews.org to HTTPS only.

Reason:
Was done elsewhere.

https://gerrit.wikimedia.org/r/178676

All Gerrit patchsets linked in this ticket are merged or abandoned.

What's left to do here?

A few servers still do not provide forward secrecy because they need to be upgraded to apache 2.4 to support that. See: https://wikitech.wikimedia.org/wiki/HTTPS/domains

Also we still offer non-FS ciphers, but maybe disabling non-forward-secrecy should be a different task.

Guys, while it is true that apache 2.2 doesn’t support ECHD, it supports DHE just fine. And DHE IS a PFS. I know that it is a bit slower, but does it really matter for these web-servers?

We are currently using this:

'compat' => 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-E    CDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-    GCM-SHA256:AES256-GCM-SHA384:AES128:AES256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!DH',

whether it's Apache 2.2 or 2.4

T83471 - seems like a duplicate of this but was resolved and not visibally public, fixed that

T91504 - OTRS SSL config (this is ticket.wikimedia.org in the linked list above)

T86655 - re: SVN (will probably be rejected in favor of decom)

T82698 - blocker for lists.wm

RT - will probably be rejected in favor of decom? needs ticket though?

T94585 - wikitech-static, needs distro upgrade, ticket created

that leaves from the list: gerrit? librenms? icinga?

In this case server load is the least of the concerns. Enabling non-EC DHE ciphers may be acceptable. It is necessary if we want to disable no-FS ciphers, which is desirable. Work on this in general is not blocked by the upgrade.

But there is a big downside to enabling non-EC DHE ciphers on older apaches: they only support something like 1k bit DH parameters. Which means one would decrease key size. I suggest to not enable non-EC DHE on older apaches, but to upgrade them.

In T55259#1166856, @DaBPunkt wrote:

Guys, while it is true that apache 2.2 doesn’t support ECHD, it supports DHE just fine. And DHE IS a PFS. I know that it is a bit slower, but does it really matter for these web-servers?

In T55259#1167839, @JanZerebecki wrote:

In this case server load is the least of the concerns. Enabling non-EC DHE ciphers may be acceptable. It is necessary if we want to disable no-FS ciphers, which is desirable. Work on this in general is not blocked by the upgrade.

But there is a big downside to enabling non-EC DHE ciphers on older apaches: they only support something like 1k bit DH parameters. Which means one would decrease key size. I suggest to not enable non-EC DHE on older apaches, but to upgrade them.

Agreed on all points. Now that the primary production sites are up to speed, we're paying a lot more attention to these issues there as well, and trying to set ever-evolving standards that the other/misc sites can use to follow the primary sites' lead. The ciphersuite lists were recently updated and refactored a few times and there are now 3 options: strong, compat, and compat-dhe. You can read the details in the source file itself: https://github.com/wikimedia/operations-puppet/blob/production/modules/wmflib/lib/puppet/parser/functions/ssl_ciphersuite.rb . In the medium term, we're planning to switch to compat-dhe for the primary prod sites and then slowly remove most (eventually, all) of the non-PFS options from compat-dhe, and then deprecate the use of compat entirely, even for misc sites. There's a lot of auditing and checking that has to happen along the way (and notification of breakage for small minority client platforms!), but we'll eventually get there.

The TL;DR as it applies to random other/misc apache/nginx instances is:

1. All 3 options offer some degree of forward secrecy, and prefer forward secrecy.
2. If it's apache2.2, it needs upgrading to apache2.4 ASAP, but even before the upgrade it can use compat
3. All, regardless of version, need to at least be using compat: that's the safe option and the minimum (vs a custom or defaulted list).
4. compat-dhe is better than compat for secrecy (it enables a common DHE-based option), but comes with a few caveats that may need investigation for some sites:
   1. If apache, needs apache2.4
   2. Absolutely requires that a custom dhparam file of 2048-bit or greater is generated and used in the apache/nginx server config (ssl_ciphersuite doesn't control this at this time)
   3. Must consider that some ancient/un-updated Java 6.x installations out there in the world will be broken for connecting to a site which forces HTTPS + compat-dhe.
5. compat-dhe will eventually, slowly, lose all fallback non-forward-secret ciphers. This may take a very long time as we'll be statistically sampling client accesses to the primary wikis and trying to identify possible breakage before disabling them. Likely the final holdout will be however long we have to wait for IE8/WinXP popularity to fall to sufficiently low levels that we can kill compatibility with it. We may actually do some campaigning on this front along the way to help encourage upgrades.
6. strong is an acceptable option for apache2.4/nginx for security-sensitive sites. All of the ciphers in strong are forward-secret and ECDHE-based, so you don't need to even generate a dhparam file/config like you'd have to for compat-dhe. If you switch a site to strong, it *will* become inaccessible to several insecure and/or legacy client platforms, including: Android 2.x, IE8/XP, Java6, and any automated client code / bots which indirectly use OpenSSL versions < 1.0 (older Linux distros such as Ubuntu Lucid). Those losses aren't acceptable for the primary wikis at this time, but they may be acceptable for some of our smaller and more specialized security-sensitive sites from @Chmarkine's excellent list: https://wikitech.wikimedia.org/wiki/HTTPS/domains

In T55259#1423112, @BBlack wrote:

If you switch a site to strong, it *will* become inaccessible to several insecure and/or legacy client platforms, including: Android 2.x, IE8/XP, Java6, and any automated client code / bots which indirectly use OpenSSL versions < 1.0 (older Linux distros such as Ubuntu Lucid).

There are more. The strong version doesn't allow TLS 1.0, so IE < 11, Android < 4.4, Java < 8, OpenSSL < 1.0, Safari (OS X) < 7 are inaccessible.[1]

[1] https://www.ssllabs.com/ssltest/clients.html

Good point, I wasn't thinking about the -TLSv1.0 part when I wrote that last night (or the other big ticket description). Perhaps we need an intermediate option that's slightly less strong than strong, and just more about ECDHE ciphersuites only.

In T55259#1424380, @Chmarkine wrote:

There are more. The strong version doesn't allow TLS 1.0, so IE < 11, Android < 4.4, Java < 8, OpenSSL < 1.0, Safari (OS X) < 7 are inaccessible.[1]

IE can handle TSL1.2 with IE8+, just not by default-config (you can enable it in the security-config). While that is of course no way for the ordinary users of Wikipedia, in controlled places like the OTRS it would be no problem to tell the few agents that uses such an old IE to enable TSL1.2 in my eyes.

Apparently back on May 28, Ubuntu released an update to their apache2.2 package (2.2.22-1ubuntu1.9) which backported ECC/ECDH/ECDHE support from upstream. Just by upgrading this package (which we should do anyways, as it's a sec update), we'll gain PFS for modern clients with a lot of these misc services. I'll see if I can get the update applied everywhere applicable.

Done, which should get us ECDHE-based Forward Secrecy on any remaining precise+apache2.2 hosts. Tested a few in ssllabs and they look fine. sodium (lists.wikimedia.org) should be the only apache left that doesn't do FS now.

In T55259#1448222, @BBlack wrote:

Done, which should get us ECDHE-based Forward Secrecy on any remaining precise+apache2.2 hosts. Tested a few in ssllabs and they look fine. sodium (lists.wikimedia.org) should be the only apache left that doesn't do FS now.

Works on the OTRS, just tested. Thank you very much :-).

I think this task can finally be closed as resolved, as there're no more domains that lack FS. (T91504 is now about DNSSEC.)

https://wikitech.wikimedia.org/wiki/HTTPS/domains