I made a very simple edit (added (and then deleted before save, I think?) Template:Notice, and then added a section header) and after save, I got this:
Apparently an edit from a completely different wikipedia? I've never visited or seen that article before, so it was not a copy/paste error.
[Gabe, I cc'd you on this because Roan told me to; set priority to unbreak now because it's a brutal data corruption bug - edit lost and resulting article incomprehensible.]
From IRC:
<RoanKattouw> Maybe they have the same pageid or revid or something
That's correct (oldid):
https://meta.wikimedia.org/w/index.php?title=User:LuisV_(WMF)/High-Volume_Email&oldid=12014207
https://nl.wikipedia.org/w/index.php?title=Kane_(band)&oldid=12014207 (from April 2008)
https://www.mediawiki.org/w/index.php?title=User:Krinkle/1.25&veaction=edit
Requests https://rest.wikimedia.org/www.mediawiki.org/v1/page/html/User%3AKrinkle%2F1.25
(note no revision or page id present)
This request is returning:
<!DOCTYPE html>
<html prefix="dc: http://purl.org/dc/terms/ mw: http://mediawiki.org/rdf/" about="http://www.mediawiki.org/wiki/Special:Redirect/revision/1623324"><head prefix="mwr: http://www.mediawiki.org/wiki/Special:Redirect/"><meta property="mw:TimeUuid" content="335ee9a0-eab3-11e4-a389-a7aafc7e4885"/><meta property="mw:articleNamespace" content="0"/><link rel="dc:replaces" resource="mwr:revision/0"/><meta property="dc:modified" content="2003-10-25T15:17:31.000Z"/><meta about="mwr:user/0" property="dc:title" content="166.102.201.16"/><link rel="dc:contributor" resource="mwr:user/0"/><meta property="mw:revisionSHA1" content="52123dd50f51d0ce9dd8fb7cd51306900bd8de8e"/><meta property="dc:description" content=""/><meta property="mw:parsoidVersion" content="0"/><link rel="dc:isVersionOf" href="www.mediawiki.org/wiki/User%3AKrinkle/1.25"/><title>User:Krinkle/1.25</title><base href="www.mediawiki.org/wiki/"/><link rel="stylesheet" href="//en.wikipedia.org/w/load.php?modules=mediawiki.legacy.commonPrint,shared|mediawiki.skinning.elements|mediawiki.skinning.content|mediawiki.skinning.interface|skins.vector.styles|site|mediawiki.skinning.content.parsoid&only=styles&skin=vector"/></head><body id="mwAA" lang="en" class="mw-content-ltr sitedir-ltr ltr mw-body mw-body-content mediawiki" dir="ltr"><p id="mwAQ">The signal recognition particle is a protein-RNA complex that recognizes and transports specific proteins to the ER in Eukaryotes and the plasma membrane in Prokaryotes. The core is "universal," being conserved in all three Kingdoms.</p></body></html>
Which contains http://www.mediawiki.org/wiki/Special:Redirect/revision/1623324 (correct) in the header, but contains the document of https://en.wikipedia.org/w/index.php?oldid=1623324 (the revision on en.wikipedia.org)
This issue started with an unplanned Parsoid deploy last night that also pushed out several changes from master. That deploy is now reverted, and the issue seems to be gone.
The current working theory is that some changes in config loading / caching are responsible for the API mix-up, but this is currently being investigated.
Ok, we found the issue and will fix it. But, can the incident report wait till the weekend? :)
There was at least one occurrence of this issue at cswiki, too. One user reported this version of one article got intermingled with this one from itwiki.
I and at least one other user are now still seeing this issue at cswiki (he has reported it while I was writing the last comment). This revision of article Zdzislaw Krzyszkowia is still linked with that one from itwiki if one tries to edit it via VisualEditor. Both those intermingled articles have the same revision id, though that one from itwiki is for a revision from like 2007.
I am reopening this as I am used from other bugtrackers; if Phabricator has other ways for this situation I do not see them in help on reporting.
Looks like RESTBase has bad HTML stored as well from that corruption. https://rest.wikimedia.org/cs.wikipedia.org/v1/page/html/Zdzis%C5%82aw_Krzyszkowiak/12536477 .. need gwicke to weigh in how to handle this on the RB end.
A good case by case solution for these pages (where the entire content that shows up in VE is from the wrong wiki), is to force reload the page in VE that then gets RB to reparse the page from scratch. Alternatively, any wikitext edit on the page would have the safe effect. That is a good stopgap fix till we figure out how to handle this on the RB end. @Utar: FYI in case that helps.
I did shift-reload http://rest.wikimedia.org/cs.wikipedia.org/v1/page/html/Zdzis%C5%82aw_Krzyszkowiak/12536477 in Chrome, which fixed the problem for that particular revision. Under the hood, that sends a request with 'Cache-Control: no-cache', which forces RESTBase to re-render the page from Parsoid.
We should do this systematically for all articles rendered while the broken Parsoid version was deployed. We have a timeuuid for each that encodes the render time, so should be able to create a script that performs these re-render for all revisions within the time window.
@ssastry: Are you saying any wikicode edit after this corrupted revision should have fixed it too? Note Bazi has already done one nearly day after the corrupted revision. Or was that even before this issue was fixed in the Parsoid code?
@GWicke: Yes, now it no more shows itwiki for me. That revision seems to be fixed.
@Utar: Ah .. you were loading an old revision. I thought you were trying to edit the latest revision of that article. Yes, old revisions aren't reparsed on edits. Only the newest revision is which is what you would normally edit in VE.
https://en.wikipedia.org/w/index.php?title=Wikipedia:VisualEditor/Feedback&oldid=659511196#Weird_bug_when_adding_photo is the other known corruption report.
I have started a script that re-renders all revisions in RESTBase that were stored while Parsoid was emitting corrupted output.
This now looks like it's fixed, including the caching. Does someone want to confirm?
@Jdforrester-WMF, the re-render script is not completely done yet, but the probability of still encountering one of the corrupted renders is already pretty low at this point.
@GWicke: what time period are you using for the RESTbase re-render? I should probably purge that set of renders from OCG's PDF cache as well.
(rowDate > new Date('2015-04-23T23:30-0700')
&& rowDate < new Date('2015-04-24T13:00-0700'))
Thanks for the nice machine-readable dates. My Lazy Programmer Copy-And-Paste Fingers (tm) thank you.