Page MenuHomePhabricator
Create Task
Maniphest T251480

Normalize pagenames/filenames on save in Wikibase
Closed, ResolvedPublic5 Estimated Story Points
Actions

Assigned To
Ladsgroup
Authored By
Herzi.Pinki
Apr 30 2020, 6:04 AM
Tags
Referenced Files
F34519253: image.png
Jun 21 2021, 8:30 PM
F31788163: Screenshot 2020-04-30 at 14.00.29.png
Apr 30 2020, 1:01 PM
F31788161: Screenshot 2020-04-30 at 13.59.04.png
Apr 30 2020, 1:01 PM
Subscribers
Addshore
Aklapper
Bugreporter
daniel
Herzi.Pinki
hoo
• Jonas
View All 20 Subscribers

Description

As a user, I want to enter pagenames/filenames without triggering a constraint violation because of (missing) underscores.

Problem:
Currently, Wikibase does not normalize pagenames/filenames on save (e.g. underscores in the input for properties of datatype Commons media are allowed).

At the same time, Wikidata’s quality constraints extension triggers a constraint violation after saving, if underscores are used. This is by design as to long-established Community practices.

As a result, this inconsistency leaves users with unnecessary manual work.

Example:
Currently, we don't normalize the file name ("_" not removed on save)

We do already normalize sitelinks ("_" removed on save)

BDD
GIVEN a Wikibase instance like Wikidata
WHEN a new edit is saved via UI or API
AND a pagename/filename is added or changed in that edit
THEN this pagename/filename is normalized on save ("my file_name.jpg" -> "my file name.jpg")

Acceptance criteria:

  • pagenames/filenames are normalized on save, as described in the BDD
  • This should be done considering alignment with the work done for T286047.
  • Wait until community communication has happened, as this could technically be considered a breaking change (planned for the week of July 19-23).

Original:
misleading: "Commons link should be well-formed." warning

If an image name (e.g. P18, P1442) does contain underscores '_', wikidata considers this to be ill-formed. Now

  • Grab fritz jellinek wiener zentralfriedhof 2020-01-30(2).jpg and
  • Grab_fritz_jellinek_wiener_zentralfriedhof_2020-01-30(2).jpg are identical by definition and the warning is an annoyance

So please can you silently do all the necessary underscore <-> space replacements by the underlying software instead of propagating it to the users.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• Manuel subscribed.Jun 22 2021, 12:59 PM
• Manuel moved this task from Needs Wikidata PM Work to Unconnected Stories on the Wikidata-Campsite board.Jun 23 2021, 9:54 AM
Addshore set the point value for this task to 5.Jun 30 2021, 10:21 AM
Addshore moved this task from Unconnected Stories to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.Jun 30 2021, 10:53 AM
Addshore edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Jun 30 2021, 11:15 AM
Ladsgroup subscribed.Jul 1 2021, 8:30 PM

I have been investigating this but I found a weird case, maybe introduced in new refactoring but ValueParsers are totally cut off from the rest of the code and are basically dead code now. For example check for usages of WikibaseRepo::getValueParserFactory is never used. I put die() in it and still managed to save an edit.

Lucas_Werkmeister_WMDE subscribed.Jul 2 2021, 10:47 AM

It is used, just not through the WikibaseRepo getter:

extension-repo.json
		"wbparsevalue": {
			"class": "\\Wikibase\\Repo\\Api\\ParseValue",
			"factory": "\\Wikibase\\Repo\\Api\\ParseValue::factory",
			"services": [
				"StatsdDataFactory",
				"WikibaseRepo.ApiHelperFactory",
				"WikibaseRepo.DataTypeFactory",
				"WikibaseRepo.DataTypeValidatorFactory",
				"WikibaseRepo.ExceptionLocalizer",
				"WikibaseRepo.PropertyDataTypeLookup",
				"WikibaseRepo.ValidatorErrorLocalizer",
				"WikibaseRepo.ValueParserFactory"
			]
		},

But I think the value parser is the wrong point to do this normalization anyways. We want to normalize on save, not on parse – even if someone directly creates the JSON for a commonsMedia value and submits it to the API without using wbparsevalue, and that JSON specifies the file name with an underscore, we still want to save it using a space, just as we do (I assume) for sitelinks.

Lucas_Werkmeister_WMDE added a comment.EditedJul 2 2021, 11:04 AM

For sitelinks, it looks like this happens in SetSiteLink::getChangeOp(), with a call to $site->normalizePageName(); the result ends up in SiteLinkChangeOpFactory::newSetSiteLinkOp().

For Commons media, it looks like we already normalize the page name in CachingCommonsMediaFileNameLookup, but that class is only used by CommonsMediaExistsValidator, which only checks that the normalized name isn’t null before throwing it away. (More or less the same thing seems to be true for geoshape and tabular data validation, too.)

Lucas_Werkmeister_WMDE added a comment.Jul 2 2021, 1:19 PM

But I think the value parser is the wrong point to do this normalization anyways. We want to normalize on save, not on parse – even if someone directly creates the JSON for a commonsMedia value and submits it to the API without using wbparsevalue, and that JSON specifies the file name with an underscore, we still want to save it using a space, just as we do (I assume) for sitelinks.

Hm, let me walk that back a bit. Sitelinks and commonsMedia values aren’t the same thing – I’m not sure if there are currently any places where we normalize data value contents.

Maybe we should instead:

  1. Make the value parser normalize page names. This should be a fairly straightforward change that we can make without breaking anything.
  2. Make the value validator reject non-normalized page names. This would be a breaking change to the API (announce in advance, waiting period, etc.) that we would make afterwards.

(“Normalize” here mainly means replacing underscores with spaces, though I could imagine some Unicode normalization as well.)

• Manuel added a comment.Jul 5 2021, 3:56 PM

@Lucas_Werkmeister_WMDE:

We want to normalize on save, not on parse – even if someone directly creates the JSON for a commonsMedia value and submits it to the API without using wbparsevalue, and that JSON specifies the file name with an underscore, we still want to save it using a space, just as we do (I assume) for sitelinks.

I agree!

Hm, let me walk that back a bit. (...) I’m not sure if there are currently any places where we normalize data value contents.

I would not want to make things harder for API users. I think APIs should ideally be flexible in what they accept, but very standardized in what they give back. For me to better understand your point: Could you please elaborate a bit on why you walked back from your previous comment?

Lucas_Werkmeister_WMDE added a comment.Jul 5 2021, 4:10 PM

Because the data value is in the same format that you get back when you retrieve the entity data. We sometimes allow for some redundancy (mainly for backwards compatibility, I believe) – for example, both of these values

{
  "entity-type": "item",
  "numeric-id": 5
}
{
  "id": "Q5"
}

will get you

{
  "entity-type": "item",
  "numeric-id": 5,
  "id": "Q5"
}

in the entity data. (At least the entity data you get out of the API or Special:EntityData. I don’t know if we actually store the redundant bits internally.) But I’m not aware of any place where we try to “clean up” the value beyond that (e.g. by allowing " Q5 " in the "id"). And I think it would be reasonable for API users to expect that if they pass a data value into the low-level API, and it’s accepted, then they will get it back unmodified. (wbparsevalue would be the higher-level API that’s “allowed” to do normalization.)

Lucas_Werkmeister_WMDE moved this task from To Do (prioritised from top to bottom) to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jul 8 2021, 10:14 AM

Moving to stalled to reflect ongoing internal discussions about how to implement this.

We also looked through the other datatypes, and it looks like _→ normalization is the only kind of normalization we’d implement at the moment (for the Commons media, tabular data, and geoshape datatypes). In the future we could think about normalizing quantities: 0001.0→1.0. (Currently, 0001.0 is rejected as an illegal data value, since DecimalValue::QUANTITY_VALUE_PATTERN doesn’t permit leading zeroes.)

• Manuel updated the task description. (Show Details)Jul 12 2021, 8:53 AM
• Manuel updated the task description. (Show Details)Jul 12 2021, 9:00 AM

Clarified scope of ticket as per discussion with @Lucas_Werkmeister_WMDE.

• Manuel updated the task description. (Show Details)Jul 12 2021, 9:01 AM
• Manuel updated the task description. (Show Details)Jul 12 2021, 10:22 AM
• toan added subscribers: • Samantha_Alipio_WMDE, • toan.Jul 12 2021, 10:35 AM

@Manuel @Lydia_Pintscher @Samantha_Alipio_WMDE

So judging from the open questions here we need a product decision to un-stall this.

Open questions:

  • Standard Mediawiki-normalization for pagenames/filenames in Wikidata uses the opposite normalization ("file_name.jpg" -> "file_name.jpg"). How much of a problem is this inconsistency?
  • Do we want this to happen on all Wikibases or only on Wikidata?
• Manuel added a comment.Jul 12 2021, 11:25 AM

@toan: Yes, I will un-stall it when everything is ready.

• Manuel updated the task description. (Show Details)Jul 13 2021, 6:48 PM
• Manuel renamed this task from Normalize pagenames/filenames on save to Normalize pagenames/filenames on save in Wikibase.Jul 13 2021, 6:53 PM
• Manuel updated the task description. (Show Details)
• Manuel updated the task description. (Show Details)Jul 14 2021, 11:44 AM
• Manuel updated the task description. (Show Details)
Lucas_Werkmeister_WMDE claimed this task.Aug 2 2021, 10:32 AM
Lucas_Werkmeister_WMDE moved this task from Stalled/Waiting to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
gerritbot added a comment.Aug 3 2021, 5:37 PM

Change 709774 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Introduce normalization-related wiring and services

https://gerrit.wikimedia.org/r/709774

gerritbot added a project: Patch-For-Review.Aug 3 2021, 5:37 PM

Change 709775 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Add normalization to StatementChangeOpFactory

https://gerrit.wikimedia.org/r/709775

gerritbot added a comment.Aug 3 2021, 5:37 PM

Change 709776 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Define normalizer for commonsMedia property type

https://gerrit.wikimedia.org/r/709776

Lucas_Werkmeister_WMDE moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 3 2021, 5:38 PM
gerritbot added a comment.Aug 4 2021, 8:56 AM

Change 709958 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseLexeme@master] Temporarily skip LexemeFormsMergerTest

https://gerrit.wikimedia.org/r/709958

gerritbot added a comment.Aug 4 2021, 8:57 AM

Change 709959 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseLexeme@master] Update StatementChangeOpFactory constructor call

https://gerrit.wikimedia.org/r/709959

gerritbot added a comment.Aug 4 2021, 2:04 PM

Change 710044 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Introduce DataValueNormalizer interface

https://gerrit.wikimedia.org/r/710044

gerritbot added a comment.Aug 4 2021, 2:04 PM

Change 710045 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Add data-value-normalizers to data type definitions

https://gerrit.wikimedia.org/r/710045

gerritbot added a comment.Aug 4 2021, 2:04 PM

Change 710046 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Introduce SnakNormalizer

https://gerrit.wikimedia.org/r/710046

gerritbot added a comment.Aug 4 2021, 2:04 PM

Change 710047 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Introduce ReferenceNormalizer

https://gerrit.wikimedia.org/r/710047

gerritbot added a comment.Aug 4 2021, 2:04 PM

Change 710048 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Introduce StatementNormalizer

https://gerrit.wikimedia.org/r/710048

gerritbot added a comment.Aug 4 2021, 2:05 PM

Change 709774 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/extensions/Wikibase@master] Introduce normalization-related wiring and services

Reason:

split into several smaller changes

https://gerrit.wikimedia.org/r/709774

gerritbot added a comment.Aug 4 2021, 5:13 PM

Change 710069 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Add API integration tests for data value normalization

https://gerrit.wikimedia.org/r/710069

Lucas_Werkmeister_WMDE moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 5 2021, 1:21 PM
Lucas_Werkmeister_WMDE moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 5 2021, 1:50 PM
gerritbot added a comment.Aug 10 2021, 10:28 AM

Change 710044 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Introduce DataValueNormalizer interface

https://gerrit.wikimedia.org/r/710044

gerritbot added a comment.Aug 10 2021, 10:28 AM

Change 710045 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Add normalizer-factory-callback to data type definitions

https://gerrit.wikimedia.org/r/710045

gerritbot added a comment.Aug 10 2021, 10:29 AM

Change 710046 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Introduce SnakNormalizer

https://gerrit.wikimedia.org/r/710046

gerritbot added a comment.Aug 10 2021, 10:29 AM

Change 710047 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Introduce ReferenceNormalizer

https://gerrit.wikimedia.org/r/710047

gerritbot added a comment.Aug 10 2021, 10:29 AM

Change 710048 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Introduce StatementNormalizer

https://gerrit.wikimedia.org/r/710048

gerritbot added a comment.Aug 10 2021, 10:29 AM

Change 709958 merged by jenkins-bot:

[mediawiki/extensions/WikibaseLexeme@master] Temporarily skip LexemeFormsMergerTest

https://gerrit.wikimedia.org/r/709958

gerritbot added a comment.Aug 10 2021, 10:31 AM

Change 709775 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Add normalization to StatementChangeOpFactory

https://gerrit.wikimedia.org/r/709775

gerritbot added a comment.Aug 10 2021, 10:34 AM

Change 709776 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Define normalizer for commonsMedia property type

https://gerrit.wikimedia.org/r/709776

gerritbot added a comment.Aug 10 2021, 10:39 AM

Change 710069 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Add API integration tests for data value normalization

https://gerrit.wikimedia.org/r/710069

gerritbot added a comment.Aug 10 2021, 10:58 AM

Change 709959 merged by jenkins-bot:

[mediawiki/extensions/WikibaseLexeme@master] Update StatementChangeOpFactory constructor call

https://gerrit.wikimedia.org/r/709959

ReleaseTaggerBot added a project: MW-1.37-notes (1.37.0-wmf.19; 2021-08-16).Aug 10 2021, 11:00 AM
Lucas_Werkmeister_WMDE moved this task from Peer Review to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 10 2021, 12:25 PM

Stalled until next week’s train, then we can enable this on Test Wikidata and announce it.

Lucas_Werkmeister_WMDE moved this task from Stalled/Waiting to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 23 2021, 9:56 AM
gerritbot added a comment.Aug 23 2021, 10:17 AM

Change 714334 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[operations/mediawiki-config@master] Set $wgWBRepoSettings['tmpNormalizeDataValues'] on test wikis

https://gerrit.wikimedia.org/r/714334

Lucas_Werkmeister_WMDE moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 23 2021, 10:19 AM
gerritbot added a comment.Aug 23 2021, 11:08 AM

Change 714334 merged by jenkins-bot:

[operations/mediawiki-config@master] Set $wgWBRepoSettings['tmpNormalizeDataValues'] on test wikis

https://gerrit.wikimedia.org/r/714334

Stashbot added a comment.Aug 23 2021, 11:31 AM

Mentioned in SAL (#wikimedia-operations) [2021-08-23T11:31:58Z] <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:714334|Set $wgWBRepoSettings['tmpNormalizeDataValues'] on test wikis (T251480)]] (1/2) (duration: 00m 58s)

Stashbot added a comment.Aug 23 2021, 11:33 AM

Mentioned in SAL (#wikimedia-operations) [2021-08-23T11:33:39Z] <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/Wikibase.php: Config: [[gerrit:714334|Set $wgWBRepoSettings['tmpNormalizeDataValues'] on test wikis (T251480)]] (2/2) (duration: 00m 57s)

Lucas_Werkmeister_WMDE added a comment.Aug 23 2021, 11:37 AM

I noticed one issue while testing the deployment above: if you add an image statement with a non-normalized value, it gets saved normalized, but if you then edit the statement again, you’ll still see the non-normalized value (with underscores). Only a reload solves this. We probably want to apply the data value returned by the API, I assume.

Mike_Peel subscribed.Aug 23 2021, 12:46 PM

A related issue that is quite frustrating when adding images to Wikidata: you need to remove the 'File:' part, otherwise you get a pull-down list of possible matches where it's easy to then click on the wrong file. Could it be possible to submit 'File:' (and maybe also 'Image:' for legacy reasons), and for the code to then strip it out in the same way as underscores?

Bugreporter merged a task: T168515: Normalize commonsMedia values (titles).Aug 23 2021, 2:23 PM
Bugreporter added subscribers: hoo, daniel, • Jonas, thiemowmde.
Bugreporter merged a task: T204723: Normalize Wikimedia Commons file names via a ValueParser.Aug 23 2021, 2:25 PM
Bugreporter added subscribers: Saamlegend, WMDE-leszek.
Bugreporter subscribed.Aug 23 2021, 2:27 PM

See also:
T47282: [Bug] Normalize titles before lookup in the SiteLinksTable
T47925: [Task] Apply normalization to string values in statements

Lucas_Werkmeister_WMDE mentioned this in T289501: wbsetclaim and wbsetreference return unnormalized data.Aug 23 2021, 3:24 PM
Lucas_Werkmeister_WMDE moved this task from Peer Review to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Stalled until 6 September (per announcement).

gerritbot added a comment.Aug 26 2021, 1:35 PM

Change 715018 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[operations/mediawiki-config@master] Set $wgWBRepoSettings['tmpNormalizeDataValues'] on all wikis

https://gerrit.wikimedia.org/r/715018

Lucas_Werkmeister_WMDE removed Lucas_Werkmeister_WMDE as the assignee of this task.Aug 27 2021, 3:30 PM

Unassigning myself since I won’t be around on 6 September, someone else will have to deploy that config change.

Addshore closed subtask T289501: wbsetclaim and wbsetreference return unnormalized data as Resolved.Aug 31 2021, 5:12 PM
gerritbot added a comment.Sep 7 2021, 8:37 PM

Change 715018 merged by jenkins-bot:

[operations/mediawiki-config@master] Set $wgWBRepoSettings['tmpNormalizeDataValues'] on all wikis

https://gerrit.wikimedia.org/r/715018

Ladsgroup claimed this task.Sep 7 2021, 8:40 PM
Ladsgroup moved this task from Stalled/Waiting to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 7 2021, 8:40 PM
Stashbot added a comment.Sep 7 2021, 8:41 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-07T20:41:15Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:715018|Set $wgWBRepoSettings['tmpNormalizeDataValues'] on all wikis (T251480)]] (duration: 00m 59s)

Addshore closed this task as Resolved.Sep 21 2021, 2:10 PM

Verified on test via UI and API

Lucas_Werkmeister_WMDE mentioned this in T292181: Enable data value normalization by default (remove tmpNormalizeDataValues repo setting).Sep 30 2021, 12:02 PM
Michael mentioned this in T47282: [Bug] Normalize titles before lookup in the SiteLinksTable.Mar 6 2024, 5:06 PM
Maintenance_bot moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 6 2024, 5:29 PM
Maintenance_bot removed a project: Patch-For-Review.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits