Page MenuHomePhabricator
Create Task
Maniphest T327920

March 2023 Datacenter Switchover
Closed, ResolvedPublic
Actions

Description

This is the meta task for the March 2023 Datacenter switchover (eqiad -> codfw).

Switchover

Schedule

Services: Tuesday, February 28th, 2023 14:00 UTC
Traffic: Tuesday, February 28th, 2023 15:00 UTC
MediaWiki: Wednesday, March 1st, 2023 14:00 UTC

Repooling

Traffic repooling of eqiad: Wednesday, March 8th, 2023
restbase-async eqiad pooling: Wednesday, March 8th, 2023
Services/mediawiki-RO eqiad pooling: Tuesday, March 14th, 2023

Checklist

See also:

Switchback

Services: Tuesday, April 25th, 2023 14:00 UTC
Switching back: Wednesday, April 26th, 2023 14:00 UTC

Checklist

Details

Subject Repo Branch Lines +/-
wmnet: Update maintenance.eqiad.wmnet to point to mwmaint2002 operations/dns master +1 -1
wmnet: Update parsercache CNAME operations/dns master +3 -3
wmnet: Update parsercache CNAME operations/dns master +3 -3
db: Switch dns master alias to eqiad operations/dns master +15 -15
debug.json: List primary DC servers first operations/mediawiki-config master +2 -2
sre.switchdc.mediawiki: Add mw-api-int to mediawiki services operations/cookbooks master +2 -2
wmnet: Update pcX DNS operations/dns master +3 -3
db: Switch dns master alias to codfw operations/dns master +15 -15
Customize query in gerrit

Related Objects
Search...

Status Subtype Assigned Task
Resolved Clement_Goubert T327920 March 2023 Datacenter Switchover
Resolved Trizek-WMF T328287 CommRel support for March 2023 Datacenter Switchover
Resolved Clement_Goubert T328770 March 2023 Datacenter Switchover Blockers
Resolved RLazarus T285803 switchdc should automatically downtime "Read only" checks on DB masters being switched
Resolved Clement_Goubert T285711 Figure out plan for restbase-async w/r database switchover
Resolved Clement_Goubert T290919 switchdc services cookbook should allow pooling services in both DCs (active/active)
Resolved Clement_Goubert T328768 Wikitech issues for datacentre switchover (March 2023)
Resolved Clement_Goubert T329020 Remove outdated invert-redis-sessions cookbook
Resolved Clement_Goubert T327924 Prepare mw-on-k8s for March 2023 Datacenter Switchover
Resolved Clement_Goubert T329065 Ensure sre.switchdc.mediawiki live test multi-DC compatibility
Resolved BUG REPORT Clement_Goubert T329319 What should happen to Toolhub during the 2023 DC switch?
Resolved Clement_Goubert T329533 spicerack.mysql_legacy errors on get_core_masters_heartbeats when checking x2
Resolved Clement_Goubert T328903 March 2023 Datacenter Switchover eqiad pooling schedule
Resolved Clement_Goubert T330651 28 February 2023 Service Switchover checklist
Resolved Clement_Goubert T331285 March 2023 Traffic Repool checklist
Open Clement_Goubert T328907 Post March 2023 Datacenter Switchover Tasks
Open None T328908 Migrate sre.switchdc.mediawiki to spicerack class API
Open None T328911 Expose hosts from MysqlLegacyRemoteHosts in spicerack
Open None T237773 Move Wikitech onto the production MW cluster
Resolved Marostegui T167973 Move database for wikitech (labswiki) to a main cluster section
Resolved Andrew T188029 Move labswiki database to m5
Resolved Marostegui T282074 Audit labswiki grants
Resolved Andrew T282209 Enable connection from labweb1001/labweb1002 to s6
Declined Andrew T282573 Move labweb/cloudweb hosts to private IPs
Resolved Andrew T284106 Replicate & sanitize wikitech data
Resolved Andrew T284108 Bring labswiki database tables up to date
Resolved Ladsgroup T269348 wikitech database has almost all of its varbinary fields wrong
Duplicate None T284736 Create views on labswiki on clouddb* hosts
Resolved • Bstorm T287442 Create views for labswiki (wikitech)
Declined None T237889 Install php-ldap on all MW appservers
Resolved taavi T237890 Remove and empty useless user groups
Resolved Jdforrester-WMF T196466 Remove 'shell user' right on wikitech
Declined None T246405 On wikitech, make sysops unable to edit site JS, and then retire contentadmin
Open jijiki T292707 Migrate Wikitech to Kubernetes
Open None T330304 Globalize mwconfig ReadOnly
Resolved jbond T330985 apt.wikimedia.org post-switchover
Open None T330996 Add scap lock/unlock steps to sre.switchdc.mediawiki cookbook
Open None T330997 Support locking cookbooks run except for switchover related cookbooks
Resolved BUG REPORT Clement_Goubert T335341 sre.discovery.datacenter breaks on services not in "production" state
In Progress Clement_Goubert T335364 sre.discovery.datacenter should support switching the active/passive services to the other datacenter
Open None T335409 Investigate failed maintenance jobs discovered during DC switchback
Resolved Clement_Goubert T329042 March 2023 Datacenter Switchover SRE-side Communication
Resolved Clement_Goubert T329193 March 2023 Datacenter Switchover Excluded services
Resolved eoghan T329930 Switchover gitlab-replica (gitlab2002 -> gitlab1003)
Resolved eoghan T329931 Switchover gitlab (gitlab1004 -> gitlab2002)
Resolved jbond T330744 gitlab backup timer failing
Resolved Dzahn T330091 Switchover People and Planet services to codfw
Resolved Dzahn T330090 Switchover static miscweb services to codfw
Resolved Clement_Goubert T330271 March 2023 Datacenter Switchover live test
Resolved Clement_Goubert T330300 sre.switchdc.mediawiki.07-set-readwrite doesn't reset both datacenter to rw
Resolved Clement_Goubert T330302 sre.switchdc.mediawiki.03-set-db-readonly fails in live-test mode
Resolved Marostegui T330619 Enable DB replication codfw -> eqiad before the switchover
Resolved Clement_Goubert T330650 28 February 2023 Traffic Switchover checklist
Declined Dzahn T330960 switch releases.wikimedia.org from eqiad to codfw
Resolved ayounsi T330165 eqiad row B switches upgrade
Invalid Marostegui T330977 Move db1183 to m1
Resolved Marostegui T331510 Switchover m1 master (db1164 -> db1101)
Resolved Marostegui T331511 Move db1101 to m1
Resolved Marostegui T330847 Switchover m5 master (db1183 -> db1176)
Resolved Clement_Goubert T331634 switch noc.wikimedia.org from eqiad to codfw
Resolved Clement_Goubert T331541 14 March 2023 eqiad Service repooling
Resolved Clement_Goubert T331981 Warmup script does not warm memcached enough
Open None T332009 Cookbooks that do DNS discovery change should check recdns
Resolved cmooney T333377 eqiad row D switches upgrade
Resolved cmooney T329073 eqiad row A switches upgrade
Resolved Marostegui T329141 Switchover m3 master db1164 -> db1159
Resolved Marostegui T329143 Move db1164 to m1
Resolved Marostegui T329259 Switchover m1 master (db1176 -> db1164)
Resolved Marostegui T331382 Move db1101 to m3 and make it master
Resolved Marostegui T331384 Switchover m3 master db1159 -> db1101
Resolved Marostegui T331387 Switchover m3 master db1101 -> db1159
Resolved Marostegui T331626 Mailman hasn't delivered emails since 2023-03-07 14 UTC (was: reviewer-bot is not working)
Resolved ayounsi T331882 eqiad row C switches upgrade
Resolved Ladsgroup T333123 Switchover m1 master (db1101 -> db1164)
Resolved herron T333478 failover alert1001 to alert2001
Resolved herron T333837 failover alert2001 to alert1001
Declined herron T333838 alerting_host: Reduced availability for job icinga-am after failover event
Open herron T333855 vopsbot needed manual restart after alerting hosts failover
Resolved Jclark-ctr T308339 eqiad: move non WMCS servers out of rack C8
Resolved Jclark-ctr T306162 Decommission mw13[07-48]
Resolved Joe T313327 Put mw14[57-98] in production
Resolved • Cmjohnson T306121 Q4: (Need By: TBD) rack/setup/install mw14[57-98]
Resolved Jclark-ctr T349587 deploy1002 lost connectivity
Resolved Request Jclark-ctr T350141 decommission db1131.eqiad.wmnet
Resolved sgrabarczuk T334671 CommRel support for April 2023 Datacenter Switchback
Resolved Clement_Goubert T335015 25 April 2023 Service Switchback checklist
Resolved Marostegui T335266 Enable DB replication eqiad -> codfw before the switchover
Resolved Marostegui T335267 Disconnect replication codfw -> eqiad

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Clement_Goubert updated the task description. (Show Details)Feb 28 2023, 11:32 AM
akosiaris added a comment.Feb 28 2023, 11:54 AM
In T327920#8647481, @Tgr wrote:

MwHttpRequest (that is, Guzzle/php-curl) and the URLs from https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews. I don't know if RESTBase is involved in that in some way. If you mean the VirtualRESTServiceClient in MediaWiki, that's not used (so there is no parallelism). See WikimediaPageViewService for the code.

Those URLs are RESTBase alright. And if the are being used as is from that page, that is.

  • e.g. GET https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/all-agents/Albert_Einstein/daily/2015100100/2015103100

vs the internal service-mesh

  • e.g. http://localhost:6011/wikimedia.org/v1/metrics/pageviews/per-article/en.wikipedia/all-access/all-agents/Albert_Einstein/daily/2015100100/2015103100

then MediaWiki goes via the edge caches, incurring some extra latency since there are another 3 hops before getting the actual serving cluster. In any case, something to fix after the switchover, thanks for the heads up, we 'll keep it in mind in case we have big issues.
Clement_Goubert closed subtask T330650: 28 February 2023 Traffic Switchover checklist as Resolved.Feb 28 2023, 4:41 PM
Clement_Goubert closed subtask T330651: 28 February 2023 Service Switchover checklist as Resolved.
Dzahn closed subtask T330090: Switchover static miscweb services to codfw as Resolved.Mar 1 2023, 2:11 AM
Clement_Goubert updated the task description. (Show Details)Mar 1 2023, 11:35 AM
Clement_Goubert updated the task description. (Show Details)Mar 1 2023, 12:00 PM
Stashbot added a comment.Mar 1 2023, 1:10 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-01T13:10:04Z] <claime> Adding scheduled maintenance for switchover to statuspage - T327920
Clement_Goubert updated the task description. (Show Details)Mar 1 2023, 1:11 PM
Clement_Goubert updated the task description. (Show Details)
Stashbot added a comment.Mar 1 2023, 1:31 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-01T13:31:04Z] <claime> Locking scap deployments for datacenter switchover - T327920
Clement_Goubert updated the task description. (Show Details)Mar 1 2023, 1:33 PM
Stashbot added a comment.Mar 1 2023, 1:40 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-01T13:40:10Z] <claime> Starting mediawiki datacenter switchover step 0 - T327920
gerritbot added a comment.Mar 1 2023, 2:08 PM

Change 891552 merged by Clément Goubert:

[operations/dns@master] db: Switch dns master alias to codfw

https://gerrit.wikimedia.org/r/891552
Stashbot added a comment.Mar 1 2023, 2:08 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-01T14:08:18Z] <claime> Phase 9.5 Update DNS records for new database masters - T327920
Stashbot added a comment.Mar 1 2023, 2:09 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-01T14:09:56Z] <claime> Phase 9.5 DNS records for new database masters updated - T327920
Stashbot added a comment.Mar 1 2023, 2:16 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-01T14:16:04Z] <claime> Removing scap lock - T327920
gerritbot added a comment.Mar 1 2023, 2:16 PM

Change 893479 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Update pcX DNS

https://gerrit.wikimedia.org/r/893479
gerritbot added a comment.Mar 1 2023, 2:17 PM

Change 892428 merged by jenkins-bot:

[operations/mediawiki-config@master] debug.json: List primary DC servers first

https://gerrit.wikimedia.org/r/892428
Stashbot added a comment.Mar 1 2023, 2:18 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-01T14:18:24Z] <cgoubert@deploy2002> Started scap: Backport for [[gerrit:892428|debug.json: List primary DC servers first (T327920)]]
Stashbot added a comment.Mar 1 2023, 2:20 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-01T14:20:30Z] <cgoubert@deploy2002> cgoubert: Backport for [[gerrit:892428|debug.json: List primary DC servers first (T327920)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
gerritbot added a comment.Mar 1 2023, 2:23 PM

Change 893479 merged by Marostegui:

[operations/dns@master] wmnet: Update pcX DNS

https://gerrit.wikimedia.org/r/893479
Stashbot added a comment.Mar 1 2023, 2:26 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-01T14:26:18Z] <cgoubert@deploy2002> Finished scap: Backport for [[gerrit:892428|debug.json: List primary DC servers first (T327920)]] (duration: 07m 54s)
Clement_Goubert updated the task description. (Show Details)Mar 1 2023, 2:26 PM
Stashbot added a comment.Mar 1 2023, 2:27 PM

Mentioned in SAL (#wikimedia-operations) [2023-03-01T14:27:02Z] <claime> End mediawiki datacenter switchover - T327920
Dzahn added a subtask: T330960: switch releases.wikimedia.org from eqiad to codfw.Mar 2 2023, 1:02 AM
Dzahn added a subtask: T330963: switch doc.wikimedia.org from eqiad to codfw.Mar 2 2023, 1:18 AM
gerritbot added a comment.Mar 2 2023, 10:09 AM

Change 893675 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/dns@master] wmnet: Update maintenance.eqiad.wmnet to point to mwmaint2002

https://gerrit.wikimedia.org/r/893675
gerritbot added a comment.Mar 2 2023, 10:41 AM

Change 893675 merged by Clément Goubert:

[operations/dns@master] wmnet: Update maintenance.eqiad.wmnet to point to mwmaint2002

https://gerrit.wikimedia.org/r/893675
Dzahn subscribed.Mar 3 2023, 2:05 AM

https://releases.wikimedia.org switched to codfw (T330960)
Dzahn changed the status of subtask T330960: switch releases.wikimedia.org from eqiad to codfw from Open to Stalled.Mar 3 2023, 3:12 PM
Tgr added a comment.Mar 4 2023, 2:20 AM
In T327920#8652181, @akosiaris wrote:

And if the are being used as is from that page, that is.

  • e.g. GET https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/all-agents/Albert_Einstein/daily/2015100100/2015103100

vs the internal service-mesh

  • e.g. http://localhost:6011/wikimedia.org/v1/metrics/pageviews/per-article/en.wikipedia/all-access/all-agents/Albert_Einstein/daily/2015100100/2015103100

then MediaWiki goes via the edge caches, incurring some extra latency since there are another 3 hops before getting the actual serving cluster.

Yeah, it uses the public URL. Ping me when it is a good time to fix that, it should be trivial.
Clement_Goubert updated the task description. (Show Details)Mar 6 2023, 11:57 AM
Clement_Goubert added a subtask: T331285: March 2023 Traffic Repool checklist.Mar 6 2023, 12:05 PM
eoghan closed subtask T329931: Switchover gitlab (gitlab1004 -> gitlab2002) as Resolved.Mar 6 2023, 2:57 PM
Dzahn reopened subtask T330091: Switchover People and Planet services to codfw as In Progress.Mar 6 2023, 11:28 PM
Dzahn closed subtask T330091: Switchover People and Planet services to codfw as Resolved.Mar 7 2023, 12:13 AM
Clement_Goubert added a subtask: T330165: eqiad row B switches upgrade.Mar 8 2023, 10:02 AM
Clement_Goubert closed subtask T331285: March 2023 Traffic Repool checklist as Resolved.Mar 8 2023, 11:45 AM
Clement_Goubert reopened subtask T330651: 28 February 2023 Service Switchover checklist as In Progress.
Clement_Goubert closed subtask T330651: 28 February 2023 Service Switchover checklist as Resolved.Mar 8 2023, 12:02 PM
Clement_Goubert closed subtask T328903: March 2023 Datacenter Switchover eqiad pooling schedule as Resolved.
Clement_Goubert reopened subtask T328903: March 2023 Datacenter Switchover eqiad pooling schedule as In Progress.Mar 8 2023, 2:23 PM
Kappakayala subscribed.Mar 8 2023, 2:50 PM
Clement_Goubert updated the task description. (Show Details)Mar 8 2023, 3:20 PM
Clement_Goubert changed the status of subtask T331634: switch noc.wikimedia.org from eqiad to codfw from Open to In Progress.Mar 13 2023, 2:21 PM
Clement_Goubert closed subtask T331634: switch noc.wikimedia.org from eqiad to codfw as Resolved.Mar 13 2023, 2:25 PM
Clement_Goubert added a subtask: T331541: 14 March 2023 eqiad Service repooling.Mar 14 2023, 11:38 AM
Clement_Goubert closed subtask T331541: 14 March 2023 eqiad Service repooling as Resolved.Mar 14 2023, 2:19 PM
Dzahn closed subtask T330960: switch releases.wikimedia.org from eqiad to codfw as Declined.Mar 14 2023, 6:23 PM
Dzahn removed a subtask: T330963: switch doc.wikimedia.org from eqiad to codfw.
Trizek-WMF closed subtask T328287: CommRel support for March 2023 Datacenter Switchover as Resolved.Mar 27 2023, 2:53 PM
ayounsi closed subtask T330165: eqiad row B switches upgrade as Resolved.Mar 29 2023, 7:35 AM
ayounsi edited subtasks, added: T333377: eqiad row D switches upgrade, T329073: eqiad row A switches upgrade, T331882: eqiad row C switches upgrade, T308339: eqiad: move non WMCS servers out of rack C8; removed: T327248: eqiad/codfw virtual-chassis upgrades.Apr 12 2023, 12:30 PM
cmooney closed subtask T333377: eqiad row D switches upgrade as Resolved.Apr 18 2023, 5:02 PM
Clement_Goubert updated the task description. (Show Details)Apr 19 2023, 8:03 AM
Marostegui updated the task description. (Show Details)Apr 19 2023, 8:10 AM
Trizek-WMF changed the status of subtask T334671: CommRel support for April 2023 Datacenter Switchback from Open to In Progress.Apr 19 2023, 5:49 PM
kostajh mentioned this in T335125: Account creation attempt on mobile Wikipedia domain leads user to desktop Special:CentralLogin/complete, often in logged-out state.Apr 20 2023, 12:37 PM
Clement_Goubert updated the task description. (Show Details)Apr 24 2023, 8:48 AM
Stashbot added a comment.Apr 24 2023, 10:27 AM

Mentioned in SAL (#wikimedia-operations) [2023-04-24T10:27:40Z] <claime> Datacenter switchover live testing setting db to read-only and back in eqiad - T327920
Stashbot added a comment.Apr 24 2023, 10:29 AM

Mentioned in SAL (#wikimedia-operations) [2023-04-24T10:29:58Z] <claime> Datacenter switchover live testing setting db to read-only and back in eqiad successful - T327920
Marostegui closed subtask T335266: Enable DB replication eqiad -> codfw before the switchover as Resolved.Apr 25 2023, 6:12 AM
gerritbot added a comment.Apr 25 2023, 10:42 AM

Change 911780 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/cookbooks@master] sre.switchdc.mediawiki: Add mw-api-int to mediawiki services

https://gerrit.wikimedia.org/r/911780
gerritbot added a comment.Apr 25 2023, 10:48 AM

Change 911780 merged by jenkins-bot:

[operations/cookbooks@master] sre.switchdc.mediawiki: Add mw-api-int to mediawiki services

https://gerrit.wikimedia.org/r/911780
Clement_Goubert changed the status of subtask T335015: 25 April 2023 Service Switchback checklist from Open to In Progress.Apr 25 2023, 1:33 PM
Clement_Goubert closed subtask T335015: 25 April 2023 Service Switchback checklist as Resolved.Apr 25 2023, 3:22 PM
Clement_Goubert closed subtask T328903: March 2023 Datacenter Switchover eqiad pooling schedule as Resolved.Apr 25 2023, 3:41 PM
Clement_Goubert closed subtask T329193: March 2023 Datacenter Switchover Excluded services as Resolved.
gerritbot added a comment.Apr 26 2023, 5:53 AM

Change 912171 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Update parsercache CNAME

https://gerrit.wikimedia.org/r/912171
Marostegui updated the task description. (Show Details)Apr 26 2023, 5:53 AM
gerritbot added a comment.Apr 26 2023, 8:40 AM

Change 912235 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/dns@master] db: Switch dns master alias to eqiad

https://gerrit.wikimedia.org/r/912235
Clement_Goubert updated the task description. (Show Details)Apr 26 2023, 8:41 AM
Clement_Goubert updated the task description. (Show Details)
Stashbot added a comment.Apr 26 2023, 1:14 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-26T13:13:59Z] <claime> Locking scap for datacenter switchback - T327920
Stashbot added a comment.Apr 26 2023, 1:15 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-26T13:15:43Z] <cgoubert@deploy1002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchback - T327920
Clement_Goubert updated the task description. (Show Details)Apr 26 2023, 1:16 PM
Stashbot added a comment.Apr 26 2023, 1:23 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-26T13:23:27Z] <claime> Starting mediawiki datacenter switchback preparation - T327920
Clement_Goubert updated the task description. (Show Details)Apr 26 2023, 1:28 PM
Clement_Goubert updated the task description. (Show Details)Apr 26 2023, 1:38 PM
Stashbot added a comment.Apr 26 2023, 1:45 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-26T13:45:16Z] <claime> Stopping maintenance scripts for datacenter switchback - T327920
Stashbot added a comment.Apr 26 2023, 1:59 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-26T13:59:49Z] <claime> Going to read-only for mediawiki datacenter switchback - T327920
Stashbot added a comment.Apr 26 2023, 2:05 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-26T14:05:00Z] <claime> Restarting maintenance jobs - T327920
gerritbot added a comment.Apr 26 2023, 2:08 PM

Change 912235 merged by Clément Goubert:

[operations/dns@master] db: Switch dns master alias to eqiad

https://gerrit.wikimedia.org/r/912235
Clement_Goubert updated the task description. (Show Details)Apr 26 2023, 2:13 PM
gerritbot added a comment.Apr 26 2023, 2:14 PM

Change 912302 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Update parsercache CNAME

https://gerrit.wikimedia.org/r/912302
gerritbot added a comment.Apr 26 2023, 2:14 PM

Change 912171 abandoned by Marostegui:

[operations/dns@master] wmnet: Update parsercache CNAME

Reason:

I messed up the rebase, so abandoning in favour of 912302

https://gerrit.wikimedia.org/r/912171
RLazarus added a subtask: T335409: Investigate failed maintenance jobs discovered during DC switchback.Apr 26 2023, 2:15 PM
gerritbot added a comment.Apr 26 2023, 2:15 PM

Change 912302 merged by Marostegui:

[operations/dns@master] wmnet: Update parsercache CNAME

https://gerrit.wikimedia.org/r/912302
Stashbot added a comment.Apr 26 2023, 2:16 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-26T14:16:00Z] <marostegui> Update dns for parsercache T327920
Marostegui updated the task description. (Show Details)Apr 26 2023, 2:16 PM
Stashbot added a comment.Apr 26 2023, 2:24 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-26T14:24:46Z] <cgoubert@deploy1002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchback - T327920 (duration: 69m 03s)
Clement_Goubert closed this task as Resolved.Apr 26 2023, 2:37 PM
Clement_Goubert updated the task description. (Show Details)
Ladsgroup awarded a token.Apr 26 2023, 2:39 PM
Clement_Goubert closed subtask T329042: March 2023 Datacenter Switchover SRE-side Communication as Resolved.Apr 26 2023, 2:46 PM
Trizek-WMF closed subtask T334671: CommRel support for April 2023 Datacenter Switchback as Resolved.May 30 2023, 3:45 PM
Jclark-ctr closed subtask T308339: eqiad: move non WMCS servers out of rack C8 as Resolved.Nov 1 2023, 5:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL