for the specific reqID:

from /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/Database.php(1719)
#0 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/Database.php(1703): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/Database.php(1678): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/Database.php(1244): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/Database.php(1929): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/Database.php(2029): Wikimedia\Rdbms\Database->select(string, array, array, string, array, array)
#5 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->selectRow(string, array, array, string, array)
#6 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/DBConnRef.php(331): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.36.0-wmf.35/includes/Title.php(3610): Wikimedia\Rdbms\DBConnRef->selectRow(string, array, array, string, array)
#8 /srv/mediawiki/php-1.36.0-wmf.35/includes/MovePage.php(350): Title->isSingleRevRedirect()
#9 /srv/mediawiki/php-1.36.0-wmf.35/includes/MovePage.php(235): MovePage->isValidMoveTarget()
#10 /srv/mediawiki/php-1.36.0-wmf.35/includes/MovePage.php(418): MovePage->isValidMove()
#11 /srv/mediawiki/php-1.36.0-wmf.35/includes/specials/SpecialMovepage.php(718): MovePage->moveIfAllowed(User, string, boolean)
#12 /srv/mediawiki/php-1.36.0-wmf.35/includes/specials/SpecialMovepage.php(204): MovePageForm->doSubmit()
#13 /srv/mediawiki/php-1.36.0-wmf.35/includes/specialpage/SpecialPage.php(646): MovePageForm->execute(NULL)
#14 /srv/mediawiki/php-1.36.0-wmf.35/includes/specialpage/SpecialPageFactory.php(1375): SpecialPage->run(NULL)
#15 /srv/mediawiki/php-1.36.0-wmf.35/includes/MediaWiki.php(309): MediaWiki\SpecialPage\SpecialPageFactory->executePath(Title, RequestContext)
#16 /srv/mediawiki/php-1.36.0-wmf.35/includes/MediaWiki.php(925): MediaWiki->performRequest()
#17 /srv/mediawiki/php-1.36.0-wmf.35/includes/MediaWiki.php(547): MediaWiki->main()
#18 /srv/mediawiki/php-1.36.0-wmf.35/index.php(53): MediaWiki->run()
#19 /srv/mediawiki/php-1.36.0-wmf.35/index.php(46): wfIndexMain()
#20 /srv/mediawiki/w/index.php(3): require(string)
#21 {main}

[d5b63e4b-44b2-49cb-a8d3-9baf4152f738] /w/index.php?title=Special:MovePage&action=submit   Wikimedia\Rdbms\DBQueryError: Error 1205: Lock wait timeout exceeded; try restarting transaction (10.64.16.101)
Function: Title::isSingleRevRedirect
Query: SELECT  page_namespace,page_title,page_id,page_len,page_is_redirect,page_latest,page_content_model  FROM `page`    WHERE page_id = 67200171  LIMIT 1   FOR UPDATE

I just checked, and the master is still heavily processing the move because that page apparently had lots of watchers.

Number of page watchers 18,777

The jobqueue is stuck (in a good way) running database writes:

REPLACE /* WatchedItemStore::duplicateEntry */ INTO `watchlist` (wl_user, wl_namespace, wl_title, wl_notificationtimestamp) VALUES (' ... , 13, 'Getting_started', '...') <very large query>

I also heard there is a delay on processing events from the job queue due to unrelated maintenance -which may had made the issue worse-, so the advice is to wait until all watchlist and other background task complete and retry again (hopefully in some hours).

For people with an NDA, this is the current progress: https://tendril.wikimedia.org/report/slow_queries_checksum?checksum=14679c1d8d752b67412db97567ba6cb6&host=%5Edb1083&user=wikiuser&schema=wik&hours=24 It seems to be running quite slowly:

If it doesn't finish in a few hours we should escalate to the devels, as something may be very broken (indefinitely stuck).

@jcrespo am I correct in summarizing that the issue is that moving pages with many watchers results in the job queue being used to update watchlists, and that can take a while, and while its being updated that move cannot be reverted? It may make sense to add restrictions on moving pages with lots of watchers, just like we restrict deleting pages with lots of revisions, to prevent this?

Please note, the original page history is now split between Help_talk:Getting_Started and Special:Undelete/User_talk:TheAafi/Help_talk after further undiscussed attempts to revert this. This needs repairing now and restored back to Help_talk:Getting_started. Thank you for your assistance in this matter.

In T278350#6943253, @DannyS712 wrote:

@jcrespo am I correct in summarizing that the issue is that moving pages with many watchers results in the job queue being used to update watchlists, and that can take a while, and while its being updated that move cannot be reverted?

That seems right, except for the last part, as I do not know enough about how mediawiki logic works. You should ask a developer, maybe someone from the Editing-team or Growth-Team/MediaWiki-Watchlist developers or even MediaWiki-libs-Rdbms?

When you get a database error like that (locked) the database is making sure that no two actions happen at the same time, to make sure things are consistent. This is normally a "good thing", even if it can be annoying- although ideally it should show a nice error message of why it failed, and not a stacktrace. E.g. trying to move the same page with lot of edits can take time, so it fails while the process is ongoing to make sure it doesn't end up in an inconsistent state. Ideally it should say why exactly the error happened/at what step.

However, why it is taking so long to process and if that is intended or a bug, I cannot say, hopefully someone with better mediawiki knowledge can answer/help you.

In T278350#6943578, @Polyamorph wrote:

Please note, the original page history is now split between Help_talk:Getting_Started and Special:Undelete/User_talk:TheAafi/Help_talk after further undiscussed attempts to revert this. This needs repairing now and restored back to Help_talk:Getting_started. Thank you for your assistance in this matter.

I don't have access to deleted history on enwiki, is it still split?

At database level, I saw no more long-running processes since 2021-03-24 22:48:23 UTC.

I don't have access to deleted history on enwiki, is it still split?

Yes, it is still split

At database level, I saw no more long-running processes since 2021-03-24 22:48:23 UTC.

An admin HJ Mitchell attempted the move again, it still throws up an Exception : [7a5f5030-ba4e-4df4-84da-d0acf4182fee] 2021-03-26 08:13:02: Fatal exception of type "Wikimedia\Rdbms\DBTransactionSizeError"

stacktrace for the most recent error is different:

from /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1747)
#0 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2255): Wikimedia\Rdbms\LoadBalancer::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\DatabaseMysqli)
#1 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1761): Wikimedia\Rdbms\LoadBalancer->forEachOpenMasterConnection(Closure)
#2 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/lbfactory/LBFactory.php(247): Wikimedia\Rdbms\LoadBalancer->approveMasterChanges(array, string, integer)
#3 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(236): Wikimedia\Rdbms\LBFactory::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer, string, array)
#4 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/lbfactory/LBFactory.php(249): Wikimedia\Rdbms\LBFactoryMulti->forEachLB(Closure, array)
#5 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/lbfactory/LBFactory.php(308): Wikimedia\Rdbms\LBFactory->forEachLBCallMethod(string, array)
#6 /srv/mediawiki/php-1.36.0-wmf.35/includes/MediaWiki.php(680): Wikimedia\Rdbms\LBFactory->commitMasterChanges(string, array)
#7 /srv/mediawiki/php-1.36.0-wmf.35/includes/MediaWiki.php(650): MediaWiki::preOutputCommit(RequestContext, Closure)
#8 /srv/mediawiki/php-1.36.0-wmf.35/includes/MediaWiki.php(940): MediaWiki->doPreOutputCommit(Closure)
#9 /srv/mediawiki/php-1.36.0-wmf.35/includes/MediaWiki.php(547): MediaWiki->main()
#10 /srv/mediawiki/php-1.36.0-wmf.35/index.php(53): MediaWiki->run()
#11 /srv/mediawiki/php-1.36.0-wmf.35/index.php(46): wfIndexMain()
#12 /srv/mediawiki/w/index.php(3): require(string)
#13 {main}

with error

Transaction spent 5.7038812637329 second(s) in writes, exceeding the limit of 3

That limit comes from

CommonSettings.php:$wgMaxUserDBWriteDuration = 3;

In T278350#6948205, @ArielGlenn wrote:

Transaction spent 5.7038812637329 second(s) in writes, exceeding the limit of 3

Is the query being executed (and timing out) stored anywhere?

It is possible that we could fix the specific instance via the moveBatch.php maintenance script, running on a batch of just the one page. Adding @Urbanecm who has used that script in the past.

In T278350#6948215, @Majavah wrote:

In T278350#6948205, @ArielGlenn wrote:

Transaction spent 5.7038812637329 second(s) in writes, exceeding the limit of 3

Is the query being executed (and timing out) stored anywhere?

No, the url is /w/index.php?title=Special:MovePage&action=submit and the referrer is https://en.wikipedia.org/wiki/Special:MovePage/User_talk:TheAafi/Help_talk but we already knew that.

I just saw 1 single run in the last 24 hours on 2021-03-26 08:12:13 UTC:

REPLACE /* WatchedItemStore::duplicateEntry */ INTO `watchlist` (wl_user, wl_namespace, wl_title, wl_notificationtimestamp) VALUES ('XXXX', 12, 'Getting_started',

It is new renames, or is the jobqueue continuously trying to work on something and fail?

In T278350#6948348, @jcrespo wrote:

I just saw 1 single run in the last 24 hours on 2021-03-26 08:12:13 UTC:

REPLACE /* WatchedItemStore::duplicateEntry */ INTO `watchlist` (wl_user, wl_namespace, wl_title, wl_notificationtimestamp) VALUES ('XXXX', 12, 'Getting_started',

It is new renames, or is the jobqueue continuously trying to work on something and fail?

I just tried to move the page (seconds ago), so 08:12:13 UTC would not be me. But if it happened now, that's most likely me.

For further reference, this is what I did:

[urbanecm@mwmaint1002 ~]$ cat batch.txt
User talk:TheAafi/Help talk|Help talk:Getting Started
[urbanecm@mwmaint1002 ~]$ mwscript moveBatch.php --wiki=enwiki -r 'sysadmin action: fixing [[:phab:T278350]]' -u 'Martin Urbanec' batch.txt
User talk:TheAafi/Help talk --> Help talk:Getting Started
[urbanecm@mwmaint1002 ~]$

Mentioned in SAL (#wikimedia-operations) [2021-03-26T10:49:25Z] <Urbanecm> Move User talk:TheAafi/Help talk to Help talk:Getting Started via [urbanecm@mwmaint1002 ~]$ mwscript moveBatch.php --wiki=enwiki -r 'sysadmin action: fixing [[:phab:T278350]]' -u 'Martin Urbanec' batch.txt to fix an UBN task (T278350)

In T278350#6948370, @Urbanecm wrote:

https://en.wikipedia.org/w/index.php?title=Special:Log&logid=116619886: 11:45, 26 March 2021 Martin Urbanec talk contribs block moved page User talk:TheAafi/Help talk to Help talk:Getting Started (sysadmin action: fixing phab:T278350) (revert). moveBatch.php appears to did the trick! Closing, we should probably have a follow-up task to fix the cause of this.

Unfortunately that is not the correct page, it should be moved to Help talk:Getting started - note the lowercase "s"

Mentioned in SAL (#wikimedia-operations) [2021-03-26T10:55:42Z] <Urbanecm> Move Help talk:Getting Started --> Help talk:Getting started on enwiki with [urbanecm@mwmaint1002 ~]$ mwscript moveBatch.php --wiki=enwiki -r 'sysadmin action: fixing [[:phab:T278350]]' -u 'Martin Urbanec' batch.txt (T278350)

In T278350#6948384, @Polyamorph wrote:

In T278350#6948370, @Urbanecm wrote:

https://en.wikipedia.org/w/index.php?title=Special:Log&logid=116619886: 11:45, 26 March 2021 Martin Urbanec talk contribs block moved page User talk:TheAafi/Help talk to Help talk:Getting Started (sysadmin action: fixing phab:T278350) (revert). moveBatch.php appears to did the trick! Closing, we should probably have a follow-up task to fix the cause of this.

Unfortunately that is not the correct page, it should be moved to Help talk:Getting started - note the lowercase "s"

Damn, I swear I copy pasted the title. Fixed with https://en.wikipedia.org/w/index.php?title=Special:Log&logid=116619997.

In T278350#6948386, @Urbanecm wrote:

In T278350#6948384, @Polyamorph wrote:

In T278350#6948370, @Urbanecm wrote:

https://en.wikipedia.org/w/index.php?title=Special:Log&logid=116619886: 11:45, 26 March 2021 Martin Urbanec talk contribs block moved page User talk:TheAafi/Help talk to Help talk:Getting Started (sysadmin action: fixing phab:T278350) (revert). moveBatch.php appears to did the trick! Closing, we should probably have a follow-up task to fix the cause of this.

Unfortunately that is not the correct page, it should be moved to Help talk:Getting started - note the lowercase "s"

Damn, I swear I copy pasted the title. Fixed with https://en.wikipedia.org/w/index.php?title=Special:Log&logid=116619997.

But Help talk:Getting started shows as a redirect currently

In T278350#6948387, @AaqibAnjum wrote:

In T278350#6948386, @Urbanecm wrote:

In T278350#6948384, @Polyamorph wrote:

In T278350#6948370, @Urbanecm wrote:

https://en.wikipedia.org/w/index.php?title=Special:Log&logid=116619886: 11:45, 26 March 2021 Martin Urbanec talk contribs block moved page User talk:TheAafi/Help talk to Help talk:Getting Started (sysadmin action: fixing phab:T278350) (revert). moveBatch.php appears to did the trick! Closing, we should probably have a follow-up task to fix the cause of this.

Unfortunately that is not the correct page, it should be moved to Help talk:Getting started - note the lowercase "s"

Damn, I swear I copy pasted the title. Fixed with https://en.wikipedia.org/w/index.php?title=Special:Log&logid=116619997.

But Help talk:Getting started shows as a redirect currently

Reverted via https://en.wikipedia.org/w/index.php?title=Help_talk:Getting_started&diff=1014309378&oldid=1014309092, but you should be able to do it too :-).

Damn, I swear I copy pasted the title. Fixed with https://en.wikipedia.org/w/index.php?title=Special:Log&logid=116619997.

Brilliant! Just for completeness, does this also include the split deleted history at Special:Undelete/User_talk:TheAafi/Help_talk?

In T278350#6948395, @Polyamorph wrote:

Damn, I swear I copy pasted the title. Fixed with https://en.wikipedia.org/w/index.php?title=Special:Log&logid=116619997.

Brilliant! Just for completeness, does this also include the split deleted history at Special:Undelete/User_talk:TheAafi/Help_talk?

I don't see any deleted revisions at Special:Undelete/User_talk:TheAafi/Help_talk. It's possible I missed something, in that case, please do ping me and I'll gladly fix it :).

OK, I will check with the admins, thanks for your assistance!

In T278350#6942481, @jcrespo wrote:

I just checked, and the master is still heavily processing the move because that page apparently had lots of watchers.

Number of page watchers 18,777

The jobqueue is stuck (in a good way) running database writes:

REPLACE /* WatchedItemStore::duplicateEntry */ INTO `watchlist` (wl_user, wl_namespace, wl_title, wl_notificationtimestamp) VALUES (' ... , 13, 'Getting_started', '...') <very large query>

I also heard there is a delay on processing events from the job queue due to unrelated maintenance -which may had made the issue worse-, so the advice is to wait until all watchlist and other background task complete and retry again (hopefully in some hours).

I checked the actual number of watchers to be sure this is also fine:

wikiadmin@10.64.16.185(enwiki)> select count(*) from watchlist where wl_title='Getting_started' and wl_namespace=12;
+----------+
| count(*) |
+----------+
|    18794 |
+----------+
1 row in set (0.01 sec)

That sounds to be greater than the previously reported number, so watchers were inserted back as well.

User_talk:TheAafi/Help_talk has 18767 watchers, so some deletes might happen, but hopefully it won't be an issue for us.

This is the full trace- easy to locate thanks to the user reporting the request Id https://logstash.wikimedia.org/goto/d1e2ffabc0987ddb3395cf2ec4cdbda4:

from /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/TransactionProfiler.php(430)
#0 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/TransactionProfiler.php(258): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, Wikimedia\Rdbms\GeneralizedSql, integer)
#1 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/Database.php(1432): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion(Wikimedia\Rdbms\GeneralizedSql, double, boolean, integer)
#2 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/Database.php(1311): Wikimedia\Rdbms\Database->executeQueryAttempt(string, string, boolean, string, integer)
#3 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/Database.php(1240): Wikimedia\Rdbms\Database->executeQuery(string, string, integer)
#4 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/DatabaseMysqlBase.php(1367): Wikimedia\Rdbms\Database->query(string, string, integer)
#5 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/Database.php(3272): Wikimedia\Rdbms\DatabaseMysqlBase->doReplace(string, array, array, string)
#6 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->replace(string, array, array, string)
#7 /srv/mediawiki/php-1.36.0-wmf.35/includes/libs/rdbms/database/DBConnRef.php(496): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#8 /srv/mediawiki/php-1.36.0-wmf.35/includes/watcheditem/WatchedItemStore.php(1646): Wikimedia\Rdbms\DBConnRef->replace(string, array, array, string)
#9 /srv/mediawiki/php-1.36.0-wmf.35/includes/watcheditem/WatchedItemStore.php(1600): WatchedItemStore->duplicateEntry(TitleValue, TitleValue)
#10 /srv/mediawiki/php-1.36.0-wmf.35/includes/MovePage.php(712): WatchedItemStore->duplicateAllAssociatedEntries(Title, Title)
#11 /srv/mediawiki/php-1.36.0-wmf.35/includes/MovePage.php(435): MovePage->moveUnsafe(User, string, boolean, array)
#12 /srv/mediawiki/php-1.36.0-wmf.35/includes/specials/SpecialMovepage.php(718): MovePage->moveIfAllowed(User, string, boolean)
#13 /srv/mediawiki/php-1.36.0-wmf.35/includes/specials/SpecialMovepage.php(204): MovePageForm->doSubmit()
#14 /srv/mediawiki/php-1.36.0-wmf.35/includes/specialpage/SpecialPage.php(646): MovePageForm->execute(NULL)
#15 /srv/mediawiki/php-1.36.0-wmf.35/includes/specialpage/SpecialPageFactory.php(1375): SpecialPage->run(NULL)
#16 /srv/mediawiki/php-1.36.0-wmf.35/includes/MediaWiki.php(309): MediaWiki\SpecialPage\SpecialPageFactory->executePath(Title, RequestContext)
#17 /srv/mediawiki/php-1.36.0-wmf.35/includes/MediaWiki.php(925): MediaWiki->performRequest()
#18 /srv/mediawiki/php-1.36.0-wmf.35/includes/MediaWiki.php(547): MediaWiki->main()
#19 /srv/mediawiki/php-1.36.0-wmf.35/index.php(53): MediaWiki->run()
#20 /srv/mediawiki/php-1.36.0-wmf.35/index.php(46): wfIndexMain()
#21 /srv/mediawiki/w/index.php(3): require(string)
#22 {main}

Mentioned in SAL (#wikimedia-operations) [2021-03-30T14:58:40Z] <Urbanecm> Move Help talk:Help talk:Getting started --> Help talk:Getting started via moveBatch.php on enwiki (T278350)