Page MenuHomePhabricator
Create Task
Maniphest T282787

Configure dns and puppet repositories for new drmrs datacenter
Closed, ResolvedPublic
Actions

Description

Due: Q4 FY2021

drmrs is the codename selected for our new edge DC in Marseille.
185.15.58.0/24 is the public subnet for the new site.
The datacenter numeric code is 6 (vs e.g. 3 for esams or 5 for eqsin).

https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions should probably be updated for drmrs as well!

Details

Subject Repo Branch Lines +/-
Add dummy private keys for drmrs ganeti labs/private master +6 -0
hieradata/cloud: Add drmrs to ntp peers list operations/puppet production +2 -0
hieradata: Add drmrs domain to puppet master allow list operations/puppet production +1 -0
conftool-data/node: Add drmrs nodes operations/puppet production +25 -0
Add drmrs site instances operations/puppet production +92 -5
cumin: Add drmrs DC site operations/puppet production +3 -0
drmrs: add vk delivery error alerting operations/puppet production +10 -1
Add ops-drmrs to alertmanager config operations/puppet production +6 -0
Add netflow6001 to kafka custom ferm operations/puppet production +1 -0
drmrs: various minor global config operations/puppet production +7 -2
drmrs: ncredir puppetization operations/puppet production +14 -18
install_server: Add drmrs ncredir second instance operations/puppet production +8 -0
install_server: Add drmrs ncredir first instance operations/puppet production +5 -0
site: Add drmrs ncredir host operations/puppet production +17 -1
bird anycast: fix defaulting to local gateway operations/puppet production +1 -1
drmrs host bgp fixups operations/puppet production +2 -5
drmrs: lvs/cp puppetization operations/puppet production +112 -3
reverse-proxy: add drmrs ranges operations/mediawiki-config master +7 -1
Ferm: allow dhcp request from infra IPs operations/puppet production +1 -1
Add drmrs addresses operations/homer/public master +5 -1
cloudgw: add newly-allocated drmrs IPs operations/puppet production +8 -0
drmrs: configure ats-tls params operations/puppet production +15 -16
drmrs: include Netbox files for LVS subnets operations/dns master +13 -0
Add drmrs prometheus to various global config operations/puppet production +13 -0
Add prometheus.svc.drmrs.wmnet alias operations/dns master +1 -1
prometheus6001: add to global node list operations/puppet production +1 -1
prometheus6001: macaddr and site.pp operations/puppet production +6 -1
drmrs: remove fake prometheus6001 dns entry operations/dns master +0 -4
install6001: use as proxy for drmrs operations/dns master +1 -2
install6001: use for drmrs installs operations/puppet production +4 -4
install6001: dhcp entry operations/puppet production +6 -0
install6001: add site.pp entry operations/puppet production +1 -1
bast6001: add to bastion_hosts operations/puppet production +2 -1
bast6001: set dhcp macaddr for ganeti vm operations/puppet production +5 -0
site: Add drmrs bastion host operations/puppet production +6 -1
site: Add drmrs lvs instances operations/puppet production +5 -0
drmrs: include netbox svc file operations/dns master +2 -2
ganeti6: switch to ganeti role operations/puppet production +1 -1
drmrs ganeti: add cluster cert public keys operations/puppet production +44 -0
drmrs: define dual ganeti clusters operations/puppet production +26 -0
Add drmrs networks to webproxy squid config operations/puppet production +2 -0
install_server: Add instance hardware category operations/puppet production +21 -2
install_server: Add instance hardware category operations/puppet production +50 -2
site: Add drmrs cache instances operations/puppet production +7 -0
constants: add new drmrs datacenter operations/software/pywmflib master +2 -1
drmrs: configure dns servers for general use operations/puppet production +4 -4
site: Add drmrs ganeti instances operations/puppet production +5 -0
site: Add drmrs ganeti instances operations/puppet production +24 -0
site: Add dns instances for drmrs DC site operations/puppet production +11 -3
install_server: Add drmrs nodes to partman configs operations/puppet production +2 -2
install_server: Add drmrs DC site subnets operations/puppet production +108 -0
Add ntp.drmrs.wm.o for initial installs operations/dns master +3 -0
discovery-map: add drmrs ranges operations/puppet production +4 -0
ntp: Add drmrs DC site operations/puppet production +6 -3
realm: Add drmrs DC site IP regex operations/puppet production +2 -0
hiera::drmrs Add drmrs DC site instances operations/puppet production +24 -0
Add temporary junk IP for prometheus6001 operations/dns master +5 -1
drmrs common.yaml: comment out dc for now operations/puppet production +1 -1
hiera::common: Add drmrs DC site operations/puppet production +56 -0
Remove max_core_rtt variables operations/puppet production +0 -24
network: add drmrs prefixes operations/puppet production +22 -0
wmnet: make edge DC layout more-obvious operations/dns master +12 -12
Add drmrs to DNS (Netbox generated records) operations/dns master +106 -1
Add drmrs to DNS (Netbox generated records) operations/dns master +95 -1
Add drmrs to DNS (Netbox generated records) operations/dns master +78 -1
grafana: Add drmrs DC site operations/puppet production +2 -1
Revert "ntp: Add drmrs DC site" operations/puppet production +1 -4
ntp: Add drmrs DC site operations/puppet production +4 -1
exim: Add drmrs DC site operations/puppet production +1 -0
prometheus: Add drmrs DC site operations/puppet production +5 -0
resolving: Add drmrs DC site operations/puppet production +1 -0
rsyslog: Add drmrs DC site operations/puppet production +1 -0
prometheus: Add drmrs DC site operations/puppet production +1 -1
bastionhost: Add drmrs DC site operations/puppet production +1 -0
acme_chief: Add drmrs DC site operations/puppet production +3 -1
puppetmaster: Add drmrs DC Site operations/puppet production +1 -0
wmcs::monitoring: Add drmrs DC Site operations/puppet production +1 -0
pontoon: Add drmrs DC Site operations/puppet production +1 -0
cloud: Add drmrs DC Site operations/puppet production +5 -0
thanos: Add drmrs DC Site operations/puppet production +1 -0
prometheus: Add drmrs DC Site operations/puppet production +2 -0
puppetmaster: Add drmrs DC site operations/puppet production +2 -0
puppetmaster: Add drmrs DC Site operations/puppet production +3 -1
configmaster: Add drmrs DC site operations/puppet production +1 -0
rake_modules: Add drmrs DC site operations/puppet production +2 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Status Subtype Assigned Task
Unknown Object (Task)
Resolved BBlack T282788 drmrs: primary software task
Resolved • MMandere T282787 Configure dns and puppet repositories for new drmrs datacenter
Resolved cmooney T301447 Reverse DNS zones includes for drmrs

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Nov 17 2021, 3:31 PM

Change 739553 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] drmrs: define dual ganeti clusters

https://gerrit.wikimedia.org/r/739553
gerritbot added a comment.Nov 17 2021, 3:59 PM

Change 739553 merged by BBlack:

[operations/puppet@production] drmrs: define dual ganeti clusters

https://gerrit.wikimedia.org/r/739553
gerritbot added a comment.Nov 17 2021, 5:52 PM

Change 739584 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] drmrs ganeti: add cluster cert public keys

https://gerrit.wikimedia.org/r/739584
gerritbot added a comment.Nov 17 2021, 5:53 PM

Change 739584 merged by BBlack:

[operations/puppet@production] drmrs ganeti: add cluster cert public keys

https://gerrit.wikimedia.org/r/739584
gerritbot added a comment.Nov 17 2021, 5:55 PM

Change 739586 had a related patch set uploaded (by BBlack; author: BBlack):

[labs/private@master] Add dummy private keys for drmrs ganeti

https://gerrit.wikimedia.org/r/739586
gerritbot added a comment.Nov 17 2021, 6:08 PM

Change 739588 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] ganeti6: switch to ganeti role

https://gerrit.wikimedia.org/r/739588
gerritbot added a comment.Nov 17 2021, 6:11 PM

Change 739588 merged by BBlack:

[operations/puppet@production] ganeti6: switch to ganeti role

https://gerrit.wikimedia.org/r/739588
gerritbot added a comment.Nov 17 2021, 6:20 PM

Change 739594 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/dns@master] drmrs: include netbox svc file

https://gerrit.wikimedia.org/r/739594
gerritbot added a comment.Nov 17 2021, 6:22 PM

Change 739594 merged by BBlack:

[operations/dns@master] drmrs: include netbox svc file

https://gerrit.wikimedia.org/r/739594
gerritbot added a comment.Nov 18 2021, 9:14 AM

Change 739757 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] site: Add drmrs lvs instances

https://gerrit.wikimedia.org/r/739757
gerritbot added a comment.Nov 18 2021, 10:01 AM

Change 739757 merged by MMandere:

[operations/puppet@production] site: Add drmrs lvs instances

https://gerrit.wikimedia.org/r/739757
ops-monitoring-bot added a comment.Nov 18 2021, 10:17 AM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host lvs6001.drmrs.wmnet with OS buster
ops-monitoring-bot added a comment.Nov 18 2021, 10:57 AM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host lvs6001.drmrs.wmnet with OS buster completed:

  • lvs6001 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111181017_mmandere_20302_lvs6001.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Nov 18 2021, 11:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host lvs6002.drmrs.wmnet with OS buster
ops-monitoring-bot added a comment.Nov 18 2021, 11:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host lvs6003.drmrs.wmnet with OS buster
ops-monitoring-bot added a comment.Nov 18 2021, 11:45 AM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host lvs6002.drmrs.wmnet with OS buster completed:

  • lvs6002 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111181105_mmandere_29884_lvs6002.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Nov 18 2021, 12:06 PM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host lvs6003.drmrs.wmnet with OS buster completed:

  • lvs6003 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111181127_mmandere_31664_lvs6003.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
gerritbot added a comment.Dec 16 2021, 2:46 PM

Change 747856 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] site: Add drmrs bastion host

https://gerrit.wikimedia.org/r/747856
gerritbot added a comment.Dec 16 2021, 3:24 PM

Change 747856 merged by MMandere:

[operations/puppet@production] site: Add drmrs bastion host

https://gerrit.wikimedia.org/r/747856
gerritbot added a comment.Dec 17 2021, 1:15 PM

Change 748125 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] bast6001: set dhcp macaddr for ganeti vm

https://gerrit.wikimedia.org/r/748125
gerritbot added a comment.Dec 17 2021, 1:16 PM

Change 748125 merged by BBlack:

[operations/puppet@production] bast6001: set dhcp macaddr for ganeti vm

https://gerrit.wikimedia.org/r/748125
gerritbot added a comment.Dec 17 2021, 4:15 PM

Change 748151 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] bast6001: add to bastion_hosts

https://gerrit.wikimedia.org/r/748151
gerritbot added a comment.Dec 17 2021, 4:17 PM

Change 748151 merged by BBlack:

[operations/puppet@production] bast6001: add to bastion_hosts

https://gerrit.wikimedia.org/r/748151
gerritbot added a comment.Dec 17 2021, 5:46 PM

Change 748174 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] install6001: add site.pp entry

https://gerrit.wikimedia.org/r/748174
gerritbot added a comment.Dec 17 2021, 5:49 PM

Change 748175 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] install6001: use for drmrs installs

https://gerrit.wikimedia.org/r/748175
gerritbot added a comment.Dec 17 2021, 5:51 PM

Change 748178 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/dns@master] install6001: use as proxy for drmrs

https://gerrit.wikimedia.org/r/748178
gerritbot added a comment.Dec 17 2021, 6:00 PM

Change 748174 merged by BBlack:

[operations/puppet@production] install6001: add site.pp entry

https://gerrit.wikimedia.org/r/748174
gerritbot added a comment.Dec 17 2021, 6:07 PM

Change 748182 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] install6001: dhcp entry

https://gerrit.wikimedia.org/r/748182
gerritbot added a comment.Dec 17 2021, 6:08 PM

Change 748182 merged by BBlack:

[operations/puppet@production] install6001: dhcp entry

https://gerrit.wikimedia.org/r/748182
gerritbot added a comment.Dec 17 2021, 6:15 PM

Change 748175 merged by BBlack:

[operations/puppet@production] install6001: use for drmrs installs

https://gerrit.wikimedia.org/r/748175
gerritbot added a comment.Dec 17 2021, 7:23 PM

Change 748178 merged by BBlack:

[operations/dns@master] install6001: use as proxy for drmrs

https://gerrit.wikimedia.org/r/748178
gerritbot added a comment.Dec 17 2021, 9:26 PM

Change 748215 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/dns@master] drmrs: remove fake prometheus6001 dns entry

https://gerrit.wikimedia.org/r/748215
gerritbot added a comment.Dec 17 2021, 9:27 PM

Change 748215 merged by BBlack:

[operations/dns@master] drmrs: remove fake prometheus6001 dns entry

https://gerrit.wikimedia.org/r/748215
gerritbot added a comment.Dec 17 2021, 11:21 PM

Change 748224 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] prometheus6001: macaddr and site.pp

https://gerrit.wikimedia.org/r/748224
gerritbot added a comment.Dec 17 2021, 11:21 PM

Change 748225 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] prometheus6001: add to global node list

https://gerrit.wikimedia.org/r/748225
gerritbot added a comment.Dec 17 2021, 11:22 PM

Change 748224 merged by BBlack:

[operations/puppet@production] prometheus6001: macaddr and site.pp

https://gerrit.wikimedia.org/r/748224
gerritbot added a comment.Dec 18 2021, 12:08 AM

Change 748225 merged by BBlack:

[operations/puppet@production] prometheus6001: add to global node list

https://gerrit.wikimedia.org/r/748225
gerritbot added a comment.Dec 18 2021, 12:22 AM

Change 748227 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/dns@master] Add prometheus.svc.drmrs.wmnet alias

https://gerrit.wikimedia.org/r/748227
gerritbot added a comment.Dec 18 2021, 12:25 AM

Change 748227 merged by BBlack:

[operations/dns@master] Add prometheus.svc.drmrs.wmnet alias

https://gerrit.wikimedia.org/r/748227
gerritbot added a comment.Dec 18 2021, 12:26 AM

Change 748228 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] Add drmrs prometheus to various global config

https://gerrit.wikimedia.org/r/748228
gerritbot added a comment.Dec 18 2021, 12:28 AM

Change 748228 merged by BBlack:

[operations/puppet@production] Add drmrs prometheus to various global config

https://gerrit.wikimedia.org/r/748228
gerritbot added a comment.Dec 20 2021, 2:52 PM

Change 748728 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/dns@master] drmrs: include Netbox files for LVS subnets

https://gerrit.wikimedia.org/r/748728
gerritbot added a comment.Dec 20 2021, 2:53 PM

Change 748728 merged by BBlack:

[operations/dns@master] drmrs: include Netbox files for LVS subnets

https://gerrit.wikimedia.org/r/748728
gerritbot added a comment.Dec 20 2021, 4:54 PM

Change 748746 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] drmrs: configure ats-tls params

https://gerrit.wikimedia.org/r/748746
gerritbot added a comment.Dec 20 2021, 4:54 PM

Change 748747 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] cloudgw: add newly-allocated drmrs IPs

https://gerrit.wikimedia.org/r/748747
gerritbot added a comment.Dec 20 2021, 5:15 PM

Change 748752 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] drmrs: configure lvs and public IPs

https://gerrit.wikimedia.org/r/748752
gerritbot added a comment.Dec 20 2021, 5:23 PM

Change 748757 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] drmrs: add to global datacenter list

https://gerrit.wikimedia.org/r/748757
gerritbot added a comment.Dec 20 2021, 5:25 PM

Change 748746 merged by BBlack:

[operations/puppet@production] drmrs: configure ats-tls params

https://gerrit.wikimedia.org/r/748746
gerritbot added a comment.Dec 20 2021, 5:25 PM

Change 748747 merged by BBlack:

[operations/puppet@production] cloudgw: add newly-allocated drmrs IPs

https://gerrit.wikimedia.org/r/748747
gerritbot added a comment.Dec 20 2021, 6:12 PM

Change 748775 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/homer/public@master] Add drmrs addresses

https://gerrit.wikimedia.org/r/748775
gerritbot added a comment.Dec 20 2021, 7:21 PM

Change 748790 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] drmrs: ncredir puppetization

https://gerrit.wikimedia.org/r/748790
gerritbot added a comment.Dec 21 2021, 9:24 AM

Change 748775 merged by jenkins-bot:

[operations/homer/public@master] Add drmrs addresses

https://gerrit.wikimedia.org/r/748775
gerritbot added a comment.Jan 6 2022, 3:36 PM

Change 751952 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/mediawiki-config@master] reverse-proxy: add drmrs ranges

https://gerrit.wikimedia.org/r/751952
gerritbot added a comment.Jan 11 2022, 2:22 PM

Change 751952 merged by jenkins-bot:

[operations/mediawiki-config@master] reverse-proxy: add drmrs ranges

https://gerrit.wikimedia.org/r/751952
Stashbot added a comment.Jan 11 2022, 2:25 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-11T14:25:36Z] <taavi@deploy1002> Synchronized wmf-config/reverse-proxy.php: Config: [[gerrit:751952|reverse-proxy: add drmrs ranges (T282787)]] (duration: 01m 36s)
gerritbot added a comment.Jan 13 2022, 12:55 PM

Change 748752 merged by MMandere:

[operations/puppet@production] drmrs: lvs/cp puppetization

https://gerrit.wikimedia.org/r/748752
gerritbot added a comment.Jan 24 2022, 3:23 PM

Change 756613 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] site: Add drmrs ncredir host

https://gerrit.wikimedia.org/r/756613
gerritbot added a comment.Jan 24 2022, 4:47 PM

Change 756627 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] drmrs: connect lvs bgp to switches

https://gerrit.wikimedia.org/r/756627
gerritbot added a comment.Jan 24 2022, 5:59 PM

Change 756627 merged by BBlack:

[operations/puppet@production] drmrs host bgp fixups

https://gerrit.wikimedia.org/r/756627
gerritbot added a comment.Jan 24 2022, 6:17 PM

Change 756639 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] bird anycast: fix defaulting to local gateway

https://gerrit.wikimedia.org/r/756639
gerritbot added a comment.Jan 24 2022, 6:30 PM

Change 756639 merged by BBlack:

[operations/puppet@production] bird anycast: fix defaulting to local gateway

https://gerrit.wikimedia.org/r/756639
gerritbot added a comment.Jan 25 2022, 9:18 AM

Change 756613 merged by MMandere:

[operations/puppet@production] site: Add drmrs ncredir host

https://gerrit.wikimedia.org/r/756613
gerritbot added a comment.Jan 25 2022, 10:20 AM

Change 756953 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] install_server: Add drmrs ncredir first instance

https://gerrit.wikimedia.org/r/756953
gerritbot added a comment.Jan 25 2022, 10:31 AM

Change 756953 merged by MMandere:

[operations/puppet@production] install_server: Add drmrs ncredir first instance

https://gerrit.wikimedia.org/r/756953
gerritbot added a comment.Jan 25 2022, 4:11 PM

Change 757024 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] install_server: Add drmrs ncredir second instance

https://gerrit.wikimedia.org/r/757024
gerritbot added a comment.Jan 25 2022, 4:22 PM

Change 757024 merged by MMandere:

[operations/puppet@production] install_server: Add drmrs ncredir second instance

https://gerrit.wikimedia.org/r/757024
gerritbot added a comment.Jan 25 2022, 8:12 PM

Change 748790 abandoned by BBlack:

[operations/puppet@production] drmrs: ncredir puppetization

Reason:

already done elsewhere

https://gerrit.wikimedia.org/r/748790
gerritbot added a comment.Jan 25 2022, 8:42 PM

Change 748757 merged by BBlack:

[operations/puppet@production] drmrs: various minor global config

https://gerrit.wikimedia.org/r/748757
gerritbot added a comment.Feb 7 2022, 4:54 PM

Change 760613 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] Add netflow6001 to kafka custom ferm

https://gerrit.wikimedia.org/r/760613
gerritbot added a comment.Feb 7 2022, 4:54 PM

Change 760614 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] Add ops-drmrs to alertmanager config

https://gerrit.wikimedia.org/r/760614
gerritbot added a comment.Feb 7 2022, 4:54 PM

Change 760615 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] drmrs: add vk delivery error alerting

https://gerrit.wikimedia.org/r/760615
ReaperDawn subscribed.Feb 8 2022, 7:18 AM
cmooney closed subtask T301447: Reverse DNS zones includes for drmrs as Resolved.Feb 10 2022, 6:23 PM
gerritbot added a comment.Feb 15 2022, 7:34 PM

Change 760613 merged by BBlack:

[operations/puppet@production] Add netflow6001 to kafka custom ferm

https://gerrit.wikimedia.org/r/760613
gerritbot added a comment.Feb 15 2022, 7:40 PM

Change 760614 merged by BBlack:

[operations/puppet@production] Add ops-drmrs to alertmanager config

https://gerrit.wikimedia.org/r/760614
gerritbot added a comment.Feb 15 2022, 7:40 PM

Change 760615 merged by BBlack:

[operations/puppet@production] drmrs: add vk delivery error alerting

https://gerrit.wikimedia.org/r/760615
ops-monitoring-bot added a comment.Mar 15 2022, 6:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6009.drmrs.wmnet with OS buster
ops-monitoring-bot added a comment.Mar 15 2022, 7:12 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6009.drmrs.wmnet with OS buster completed:

  • cp6009 (WARN)
    • Downtimed on Icinga
    • Set pooled=inactive for the following services on confctl:

{"cp6009.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6009.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6009.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203151810_sukhe_1359834_cp6009.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes

  • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 15 2022, 8:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6010.drmrs.wmnet with OS buster
ops-monitoring-bot added a comment.Mar 15 2022, 9:35 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6010.drmrs.wmnet with OS buster completed:

  • cp6010 (WARN)
    • Downtimed on Icinga
    • Set pooled=inactive for the following services on confctl:

{"cp6010.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6010.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6010.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203152026_sukhe_1375326_cp6010.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes

  • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 16 2022, 12:12 AM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6011.drmrs.wmnet with OS buster
ops-monitoring-bot added a comment.Mar 16 2022, 1:28 AM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6011.drmrs.wmnet with OS buster completed:

  • cp6011 (WARN)
    • Downtimed on Icinga
    • Set pooled=inactive for the following services on confctl:

{"cp6011.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6011.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6011.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203160011_sukhe_1403513_cp6011.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes

  • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 16 2022, 11:09 AM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6012.drmrs.wmnet with OS buster
ops-monitoring-bot added a comment.Mar 16 2022, 12:14 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6012.drmrs.wmnet with OS buster completed:

  • cp6012 (WARN)
    • Downtimed on Icinga
    • Set pooled=inactive for the following services on confctl:

{"cp6012.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6012.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6012.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203161108_sukhe_1482638_cp6012.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes

  • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 16 2022, 12:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6013.drmrs.wmnet with OS buster
ops-monitoring-bot added a comment.Mar 16 2022, 1:25 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6013.drmrs.wmnet with OS buster completed:

  • cp6013 (WARN)
    • Downtimed on Icinga
    • Set pooled=inactive for the following services on confctl:

{"cp6013.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6013.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6013.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203161227_sukhe_1494418_cp6013.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes

  • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 16 2022, 1:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6014.drmrs.wmnet with OS buster
ops-monitoring-bot added a comment.Mar 16 2022, 2:43 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6014.drmrs.wmnet with OS buster completed:

  • cp6014 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Set pooled=inactive for the following services on confctl:

{"cp6014.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6014.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6014.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203161357_sukhe_1509133_cp6014.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes

  • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 16 2022, 2:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6015.drmrs.wmnet with OS buster
AntiCompositeNumber unsubscribed.Mar 16 2022, 3:26 PM
ops-monitoring-bot added a comment.Mar 16 2022, 3:52 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6015.drmrs.wmnet with OS buster completed:

  • cp6015 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Set pooled=inactive for the following services on confctl:

{"cp6015.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6015.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6015.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203161446_sukhe_1518579_cp6015.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes

  • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 16 2022, 4:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6016.drmrs.wmnet with OS buster
ops-monitoring-bot added a comment.Mar 16 2022, 5:11 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6016.drmrs.wmnet with OS buster completed:

  • cp6016 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Set pooled=inactive for the following services on confctl:

{"cp6016.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6016.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6016.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203161607_sukhe_1531550_cp6016.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes

  • Updated Netbox data from PuppetDB
BBlack closed this task as Resolved.Mar 17 2022, 3:47 PM

With the addition of the drmrs to the dns config in https://gerrit.wikimedia.org/r/c/operations/dns/+/771342 we're basically done with the task work here. There may be further commits, but they're in the normal production flow, not initial site config!

Nice work all!
gerritbot added a comment.Sep 29 2022, 10:43 AM

Change 734245 abandoned by Muehlenhoff:

[operations/puppet@production] cumin: Add drmrs DC site

Reason:

Obsolete

https://gerrit.wikimedia.org/r/734245
gerritbot added a comment.Nov 3 2022, 12:17 PM

Change 692869 abandoned by BBlack:

[operations/puppet@production] Add drmrs site instances

Reason:

Superseded by other patches months ago

https://gerrit.wikimedia.org/r/692869
gerritbot added a comment.Nov 3 2022, 12:17 PM

Change 692331 abandoned by BBlack:

[operations/puppet@production] conftool-data/node: Add drmrs nodes

Reason:

Superseded by other patches months ago

https://gerrit.wikimedia.org/r/692331
gerritbot added a comment.Nov 3 2022, 12:17 PM

Change 692332 abandoned by BBlack:

[operations/puppet@production] hieradata: Add drmrs domain to puppet master allow list

Reason:

Superseded by other patches months ago

https://gerrit.wikimedia.org/r/692332
gerritbot added a comment.Nov 3 2022, 12:17 PM

Change 692333 abandoned by BBlack:

[operations/puppet@production] hieradata/cloud: Add drmrs to ntp peers list

Reason:

Superseded by other patches months ago

https://gerrit.wikimedia.org/r/692333
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL