Commons search seems to have stopped indexing statements since 30 October 2019
Closed, ResolvedPublic
Actions

Description

Yesterday I noticed some of my edits didn't show up in the search engine. For example https://commons.wikimedia.org/w/index.php?title=File:Harderwijk_Kleine_Marktstraat_1.JPG&action=history , the new statements don't show up in https://commons.wikimedia.org/w/index.php?title=File:Harderwijk_Kleine_Marktstraat_1.JPG&action=cirrusdump . Another example https://commons.wikimedia.org/w/index.php?title=File:Aert_van_der_Neer_-_A_Cattle_Ferry.jpg&action=history and https://commons.wikimedia.org/w/index.php?title=File:Aert_van_der_Neer_-_A_Cattle_Ferry.jpg&action=cirrusdump . Doing a manual edit on https://commons.wikimedia.org/w/index.php?title=File:Group_of_Hetepheres_II_and_Meresankh_III-30.1456-IMG_4559-gradient.jpg&action=history doesn't seem to update https://commons.wikimedia.org/w/index.php?title=File:Group_of_Hetepheres_II_and_Meresankh_III-30.1456-IMG_4559-gradient.jpg&action=cirrusdump either.

Judging from https://commons.wikimedia.org/w/index.php?sort=create_timestamp_desc&search=haswbstatement%3A*&title=Special%3ASearch&profile=advanced&fulltext=1&advancedSearch-current=%7B%7D&ns0=1&ns6=1&ns9=1&ns12=1&ns14=1&ns100=1&ns106=1 it seems to have last worked on 30 October

Details

	Subject	Repo	Branch	Lines +/-
	Restore CirrusSearchBuildDocumentParse hook	mediawiki/extensions/CirrusSearch	wmf/1.35.0-wmf.5	+15 -0
	Restore CirrusSearchBuildDocumentParse hook	mediawiki/extensions/CirrusSearch	master	+15 -0

Customize query in gerrit

Related Objects

Mentioned In: rECIR21af6448ec59: Restore CirrusSearchBuildDocumentParse hook
rECIRc3178010382e: Restore CirrusSearchBuildDocumentParse hook
Mentioned Here: P6216 (An Untitled Masterwork)
P180 Error trace encountered working on Bug T74747

Event Timeline

Multichill created this task.Nov 10 2019, 11:25 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 10 2019, 11:25 AM

EBernhardson claimed this task.Nov 12 2019, 9:53 PM

EBernhardson edited projects, added Discovery-Search (Current work); removed Discovery-Search.

Hmm, in all of the linked pages the version numbers in elasticsearch match the revision id of the latest page, so updates are making it through.

Using one of the example pages: https://commons.wikimedia.org/wiki/File:Group_of_Hetepheres_II_and_Meresankh_III-30.1456-IMG_4559-gradient.jpg

We can ask cirrus to build a new document without writing it to elastic (test the pipeline essentially): https://commons.wikimedia.org/w/api.php?action=query&prop=cirrusbuilddoc&titles=File:Group_of_Hetepheres_II_and_Meresankh_III-30.1456-IMG_4559-gradient.jpg

Indeed this does not contain statement_keywords, even though the file has P180=Q74377458. Will look closer into this and see where it's gone haywire.

Change 550578 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/CirrusSearch@master] Restore CirrusSearchBuildDocumentParser hook

https://gerrit.wikimedia.org/r/550578

gerritbot added a project: Patch-For-Review.Nov 13 2019, 1:18 AM

EBernhardson moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.Nov 13 2019, 4:30 PM

Change 550578 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Restore CirrusSearchBuildDocumentParse hook

https://gerrit.wikimedia.org/r/550578

EBernhardson mentioned this in rECIRc3178010382e: Restore CirrusSearchBuildDocumentParse hook.Nov 13 2019, 4:49 PM

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.8; 2019-11-26).Nov 13 2019, 5:00 PM

Maintenance_bot removed a project: Patch-For-Review.Nov 13 2019, 5:10 PM

Change 550769 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/CirrusSearch@wmf/1.35.0-wmf.5] Restore CirrusSearchBuildDocumentParse hook

https://gerrit.wikimedia.org/r/550769

gerritbot added a project: Patch-For-Review.Nov 13 2019, 11:01 PM

EBernhardson moved this task from Needs review to Waiting on the Discovery-Search (Current work) board.Nov 13 2019, 11:19 PM

Change 550769 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@wmf/1.35.0-wmf.5] Restore CirrusSearchBuildDocumentParse hook

https://gerrit.wikimedia.org/r/550769

EBernhardson mentioned this in rECIR21af6448ec59: Restore CirrusSearchBuildDocumentParse hook.Nov 14 2019, 12:34 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-14T00:36:47Z] <ebernhardson@deploy1001> Synchronized php-1.35.0-wmf.5/extensions/CirrusSearch/includes/BuildDocument/BuildDocument.php: T237849: Restore CirrusSearchBuildDocumentParse hook (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2019-11-14T00:41:06Z] <ebernhardson> T237849 Start CirrusSearch forceSearchIndex.php commonswiki 2019-10-20T00:00:00 - 2019-11-14T01:00:00 pushing into jobqueue