Page MenuHomePhabricator
Create Task
Maniphest T281515

June 2021 Datacenter switchover
Closed, ResolvedPublic
Actions

Description

This is the meta task for the June 2021 Datacenter switchover (eqiad -> codfw).

Schedule:

Services: Monday, June 28th, 2021 14:00 UTC
Traffic: Monday, June 28th, 2021 15:00 UTC
MediaWiki: Tuesday, June 29th, 2021 14:00 UTC

Switching back: TBD, but at least 1 month later

See also: https://wikitech.wikimedia.org/wiki/Switch_Datacenter - section Schedule

Details

Subject Repo Branch Lines +/-
admin_state: depool eqiad for datacenter switchover (June 2021) operations/dns master +1 -0
wmnet: Change masters cnames operations/dns master +18 -18
Customize query in gerrit

Related Objects
Search...

Status Subtype Assigned Task
Unknown Object (Task)
Resolved Marostegui T258361 Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers)
Declined None T258386 db1080-95 batch possibly suffering BBU issues
Resolved Marostegui T258336 db1082 crashed
Resolved Jclark-ctr T258910 Remove db1082's BBU on-site
Resolved Marostegui T258360 db1085 crashed
Resolved Marostegui T264755 db1076 crashed - BBU failure
Resolved Request wiki_willy T267088 decommission db1091.eqiad.wmnet
Restricted Task
Resolved Legoktm T281515 June 2021 Datacenter switchover
Resolved sgrabarczuk T281209 CommRel support for June 2021 Switchover
Resolved Marostegui T284897 Pre DC switchover eqiad -> codfw DB work
Resolved Marostegui T285139 DB maintenance work to do while eqiad is passive (June 2021)
Resolved Marostegui T167973 Move database for wikitech (labswiki) to a main cluster section
Resolved Andrew T188029 Move labswiki database to m5
Resolved Marostegui T282074 Audit labswiki grants
Resolved Andrew T282209 Enable connection from labweb1001/labweb1002 to s6
Declined Andrew T282573 Move labweb/cloudweb hosts to private IPs
Resolved Andrew T284106 Replicate & sanitize wikitech data
Resolved Andrew T284108 Bring labswiki database tables up to date
Resolved Ladsgroup T269348 wikitech database has almost all of its varbinary fields wrong
Duplicate None T284736 Create views on labswiki on clouddb* hosts
Resolved • Bstorm T287442 Create views for labswiki (wikitech)
Resolved Marostegui T283093 Schema change for making cuc_id in cu_changes unsigned
Restricted Task
Restricted Task
Restricted Task
Resolved BTullis T298940 Reimage WMCS db proxies to Bullseye
Resolved Marostegui T287852 Failover m2 master (db1107) to a different host to upgrade its kernel
Resolved Marostegui T288197 Failover m3 (phabricator) master (db1132) to a different host to upgrade its kernel
Resolved Marostegui T266483 Enable report_host for mariadb
Resolved • Kormat T271106 Enable report_host on candidate masters
Resolved Marostegui T271427 Switchover s4 (commonswiki) from db1081 to db1138
Resolved Trizek-WMF T271791 Database primary master failover on s4 (commonswiki)
Resolved • Kormat T272954 Fix db-switchover update zarcillo part
Resolved Marostegui T272568 Add m* and es4/es5 sections to Orchestrator
Resolved Marostegui T271540 Upgrade and restart m1 master (db1080)
Resolved Marostegui T272596 Restart m3 (phabricator) database master db1132
Resolved Marostegui T272388 Restart m5 master (db1128)
Resolved Marostegui T272964 Restart m2 database master (db1107)
Resolved Marostegui T273758 Restart x1 database master (db1103)
Resolved sgrabarczuk T273762 Read only time for extension 1 (x1) primary database on 2021-02-17
Resolved Marostegui T274336 Switchover s7 from db1086 to db1136
Resolved sgrabarczuk T276899 Read only time window needed for s7
Resolved • Cmjohnson T277007 Upgrade firmware on db1136
Resolved Marostegui T278214 Switchover s1 from db1083 to db1163
Resolved Marostegui T276150 Schema change to make rc_id unsigned and rc_timestamp BINARY
Resolved sgrabarczuk T279505 Read only time window needed for s1 (enwiki)
Resolved Request • Cmjohnson T281445 decommission db1083.eqiad.wmnet
Resolved Marostegui T280492 Upgrade all sanitarium masters to 10.4 and Buster
Resolved Request wiki_willy T281794 decommission db1082.eqiad.wmnet
Resolved Request • Cmjohnson T281959 decommission db1074.eqiad.wmnet
Resolved Request • Cmjohnson T282079 decommission db1079.eqiad.wmnet
Resolved Request • Cmjohnson T282093 decommission db1087.eqiad.wmnet
Resolved Request • Cmjohnson T282096 decommission db1085.eqiad.wmnet
Resolved Marostegui T284375 Rename name_title index on page to page_name_title
Resolved Marostegui T284619 Schema change for renaming several indexes in change_tag table
Resolved Marostegui T273360 Schema change for dropping default of img_timestamp and making it binary(14)
Resolved Marostegui T268392 Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14)
Resolved Marostegui T266486 Schema change to turn user_last_timestamp.user_newtalk to binary(14)
Resolved • Kormat T283131 Upgrade s3 to Debian Buster and MariaDB 10.4
Resolved • Kormat T284648 Switchover s3 from db1123 to db1157
Resolved Trizek-WMF T284650 Read-only window needed for s3 (~900 wikis)
Resolved Marostegui T283235 Upgrade s5 to Debian Buster and MariaDB 10.4
Resolved Marostegui T284529 Switchover s5 from db1100 to db1130
Resolved sgrabarczuk T284530 Read only time window needed for s5
Resolved Marostegui T277123 Extend iwlinks.iwl_prefix to VARBINARY(32)
Resolved • Kormat T277118 iw_url in interwiki is varbinary(127) in production but blob in code
Resolved Marostegui T283499 Schema change for renaming page_timestamp index on revision table to rev_page_timestamp
Resolved Marostegui T278621 job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007
Resolved • Kormat T277116 fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production
Resolved Marostegui T287344 Please optimize logging table in dewiki
Resolved Marostegui T288273 Please optimize image table in commonswiki
Resolved Marostegui T290057 Optimize flaggedtemplates tables in production
Resolved Marostegui T286102 Please optimize logging table in ruwiki
Resolved Marostegui T270620 Schema change for renaming several indexes in logging table

Event Timeline

Legoktm created this task.Apr 29 2021, 7:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 29 2021, 7:04 PM
Legoktm added a subtask: T281209: CommRel support for June 2021 Switchover.Apr 29 2021, 7:04 PM
taavi subscribed.Apr 29 2021, 7:04 PM
Krinkle added a project: Performance-Team (Radar).Apr 29 2021, 8:18 PM
Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.
Marostegui subscribed.Apr 29 2021, 8:19 PM
LSobanski subscribed.Apr 29 2021, 9:04 PM
Peachey88 subscribed.Apr 29 2021, 9:14 PM
Volans subscribed.Apr 29 2021, 10:10 PM
Marostegui mentioned this in T167973: Move database for wikitech (labswiki) to a main cluster section.Apr 30 2021, 12:47 PM
EBernhardson subscribed.May 3 2021, 3:04 PM
Dzahn triaged this task as Medium priority.May 3 2021, 7:04 PM
Aklapper added a parent task: Restricted Task.May 17 2021, 9:43 AM
• wkandek subscribed.May 19 2021, 4:16 PM
• Kormat subscribed.Jun 9 2021, 12:19 PM
Kaartic subscribed.Jun 15 2021, 8:53 AM
Legoktm added a subtask: T284897: Pre DC switchover eqiad -> codfw DB work.Jun 17 2021, 4:47 PM
Legoktm removed a subtask: T285069: Check "Days in advance preparation" for databases before DC switchover.
Ladsgroup subscribed.Jun 18 2021, 3:06 AM
Marostegui added a subtask: T285139: DB maintenance work to do while eqiad is passive (June 2021).Jun 18 2021, 11:21 AM
• wkandek added a project: serviceops.Jun 21 2021, 4:39 AM
• wkandek moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.
lmata subscribed.Jun 21 2021, 5:28 AM
Marostegui closed subtask T284897: Pre DC switchover eqiad -> codfw DB work as Resolved.Jun 24 2021, 9:24 AM
JMeybohm subscribed.Jun 24 2021, 11:27 AM
• wkandek updated the task description. (Show Details)Jun 24 2021, 5:46 PM
ssingh subscribed.Jun 25 2021, 8:23 PM
gerritbot added a comment.Jun 25 2021, 8:32 PM

Change 701610 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] admin_state: depool eqiad for datacenter switchover (June 2021)

https://gerrit.wikimedia.org/r/701610
gerritbot added a project: Patch-For-Review.Jun 25 2021, 8:32 PM
ReaperDawn subscribed.Jun 28 2021, 4:37 AM
gerritbot added a comment.Jun 28 2021, 3:43 PM

Change 701610 merged by Ssingh:

[operations/dns@master] admin_state: depool eqiad for datacenter switchover (June 2021)

https://gerrit.wikimedia.org/r/701610
Maintenance_bot removed a project: Patch-For-Review.Jun 28 2021, 4:10 PM
Stashbot added a comment.Jun 28 2021, 6:40 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-28T18:40:34Z] <ebernhardson@deploy1002> Synchronized wmf-config/: T281515: Prepare Cirrus more_like for dc switchover (duration: 01m 02s)
Legoktm added a comment.Jun 28 2021, 7:35 PM

I posted an update on today's switchover to wikitech-l: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/XI57Z6T2DK7IC345VFTENM5RLTQBQDEQ/
Legoktm added a comment.Jun 28 2021, 10:56 PM

I did a successful run through of the live-test mode just now, where we "switch" from codfw -> eqiad. The only issue I ran into is T285519#7182377, which I live-hacked a fix for.
R4356th subscribed.Jun 29 2021, 5:32 AM
gerritbot added a comment.Jun 29 2021, 1:38 PM

Change 702128 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Change masters cnames

https://gerrit.wikimedia.org/r/702128
gerritbot added a project: Patch-For-Review.Jun 29 2021, 1:38 PM
Agusbou2015 subscribed.Jun 29 2021, 1:44 PM
gerritbot added a comment.Jun 29 2021, 2:47 PM

Change 702128 merged by Marostegui:

[operations/dns@master] wmnet: Change masters cnames

https://gerrit.wikimedia.org/r/702128
Maintenance_bot removed a project: Patch-For-Review.Jun 29 2021, 3:10 PM
Legoktm added a comment.Jun 29 2021, 3:42 PM

The switchover is mostly complete now, we were read only from 2021-06-29 14:21:26.671853 to 2021-06-29 14:23:23.504447, or 1m57s.

The raw notes of the issues we encountered are at https://etherpad.wikimedia.org/p/2021-switchdc-notes, later today I'll distill those into actionable Phabricator tasks and write up a report for how it went and what should be improved.
sgrabarczuk closed subtask T281209: CommRel support for June 2021 Switchover as Resolved.Jun 29 2021, 4:44 PM
Zabe subscribed.Jun 29 2021, 6:05 PM
Legoktm added a comment.Jun 30 2021, 1:52 AM

Bugs filed as a result of today's switchover:
Legoktm added a comment.Jun 30 2021, 2:20 AM

Today's summary: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/ENL3P5SA7RSOHPN4ILMXQ2BGBF5XR776/
Legoktm mentioned this in T287539: September 2021 Datacenter switchover (codfw -> eqiad).Jul 27 2021, 11:45 PM
Legoktm closed this task as Resolved.Jul 28 2021, 12:59 AM
Legoktm claimed this task.

A recap blog post was published a few days ago: https://techblog.wikimedia.org/2021/07/23/june-2021-data-center-switchover/

T287539: September 2021 Datacenter switchover (codfw -> eqiad) tracks switching back to eqiad in September, closing this as resolved accordingly.
Marostegui closed subtask T285139: DB maintenance work to do while eqiad is passive (June 2021) as Resolved.Oct 20 2021, 3:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL