Migrating to new host#
Prepare salt configuration for migration#
Ensure that salt-master, loadbalancer, and host in question can be brought up with vagrant locally, and that their health check for the relevant service is failing in haproxy after the host is fully up
laptop:psf-salt user$ vagrant up salt-master
laptop:psf-salt user$ vagrant up loadbalancer
laptop:psf-salt user$ vagrant up host
To view haproxy status:
vagrant up the salt-master, loadbalancer, and host in question (
vagrant up
)Prepare an ssh configuration file to access the host with native ssh commands:
vagrant ssh-config salt-master loadbalancer >> vagrant-ssh
Open an ssh session with port forwarding to the haproxy status page:
ssh -L 4646:127.0.0.1:4646 -F vagrant-ssh loadbalancer
view the haproxy status page in your browser
http://localhost:4646/haproxy?stats
Edit pillar data for roles.sls to include both old and new hostnames (ex. hostname*)
diff --git a/pillar/prod/roles.sls b/pillar/prod/roles.sls
index 68387c9..7a8ace1 100644
--- a/pillar/prod/roles.sls
+++ b/pillar/prod/roles.sls
@@ -35,7 +35,7 @@ roles:
purpose: "Builds and serves CPython's documentation"
contact: "mdk"
downloads:
- pattern: "downloads.nyc1.psf.io"
+ pattern: "downloads*.nyc1.psf.io"
purpose: "Serves python.org downloads"
contact: "CPython Release Managers"
hg:
Migrate the host#
Update Salt Master with latest config including prep from above#
ssh into the salt-master server
ssh salt.nyc1.psf.io
Navigate to srv/psf-salt
user@salt:~$ cd /srv/psf-salt
Run
user@salt:/srv/psf-salt$ sudo git pull
Run highstate to update the roles settings to reflect the new matchng pattern, as well as additional changes to support migration:
user@salt:/srv/psf-salt$ sudo salt-call state.highstate
Ensure new configuration doesn’t impact host being migrated#
ssh into the old-host
laptop:psf-salt user$ ssh old-host
Run
user@old-host:~$ sudo salt-call state.highstate
Create new host#
Start a new droplet in digital ocean, and check resources being used on old host to see if we are over or under spending on resources
Create a new droplet with new version of ubuntu, appropriate resources, and name it according to hostname + 2004
Provision new host for migration#
ssh into new-host via the IP address provided by DigitalOcean
ssh root@NNN.NNN.NNN.NNN
Add Salt repositories for our current target version (add the apt-repo and install salt-minion package)
user@new-host:~$ wget --quiet -O /usr/share/keyrings/salt-archive-keyring.gpg https://repo.saltproject.io/py3/ubuntu/20.04/$(dpkg --print-architecture)/3004/salt-archive-keyring.gpg
user@new-host:~$ echo "deb [signed-by=/usr/share/keyrings/salt-archive-keyring.gpg arch=$(dpkg --print-architecture)] https://repo.saltproject.io/py3/ubuntu/20.04/$(dpkg --print-architecture)/3004 focal main" > /etc/apt/sources.list.d/salt.list
Install and configure the salt-minion. On new-host, run the command
user@new-host:~$ apt-get update -y && apt-get install -y --no-install-recommends salt-minion
On the old-host, look through
/etc/salt/minion.d*
to set up salt-minion configuration files to match on new-host:run
user@old-host:~$ for file in /etc/salt/minion.d/*; do echo -e "cat > $file <<EOF"; sudo cat $file; echo "EOF"; done
to generate bash that will create these files
Restart the salt-minion service on the new host to pickup the configuration and register with salt-master:
user@new-host:~$ sudo salt-call service.restart
On salt-master, accept the key for the new-host:
user@new-host:~$ sudo salt-key -a new-host
On the new-host, run highstate
sudo salt-call sate.highstate
Log out of root session
Ensure that the new host is not passing health checks in the loadbalancer:
ssh -L 4646:127.0.0.1:4646 lb-a.nyc1.psf.io
then open http://localhost:4646/haproxy?stats in your browser.Run hightstate on the salt-master to create a public dns record for the new-host
user@salt:/srv/psf-salt$ sudo salt-call state.highstate
Begin data migration#
laptop:psf-salt user$ ssh -A new-host
into new host to enable forwarding of ssh-agentStop cron jobs
user@new-host:~$ sudo service cron stop
Stop public-facing services, like nginx, or the service the health check is looking for. Use this command as an example:
user@new-host:~$ sudo service nginx stop
Ensure that any additional volumes are mounted and in the correct location:
Check what disks are currently mounted and where:
df
Determine where any additional disks should be mounted (based on salt configuration of services, for example
docs
anddownloads
roles need a big/srv
for their data storageEnsure mounting of any external disks are in the right location using
mount
command with appropriate argumentsEnsure that the volumes will be remounted on startup by configuring them in
/etc/fstab
Run rsync once to move bulk of data and as necessary to watch for changes
user@new-host:~$ sudo -E -s rsync -av --rsync-path="sudo rsync" username@old-host:/pathname/ /pathname/
The
/pathname/
can be determined by looking at the pillar data for backups,pillar/prod/backup
using the source_directory path for the given host (example: the downloads host uses/srv/
)
Stop services on old host#
ssh into old-host (
laptop:psf-salt user$ ssh old-host
)Stop cron jobs
user@old-host:~$ sudo service cron stop
Stop public-facing services, like nginx, or the service the health check is looking for ex)
user@old-host:~$ sudo service nginx stop
Finish data migration and restart cron/public-facing services#
Run rsync once more to finalize data migration
user@new-host:~$ sudo -E -s rsync -av --rsync-path="sudo rsync" username@hostname: /pathname/ /pathname/
Start cron jobs
user@new-host:~$ sudo service cron start
Start public-facing services involved with healthcheck, like nginx,
user@new-host:~$ sudo service nginx start
Ensure that the new-host is live and serving traffic by viewing loadbalancer page:
view the haproxy status page in your browser
http://localhost:4646/haproxy?stats
Check if users have any files on old-host and transfer accordingly:
user@new-host:~$ for user in /home/psf-users/*; do sudo -E -s rsync --delete -av --progress --rsync-path="sudo rsync" user@old-host:$user/ $user/migrated-from-ubuntu-1804-lts-host/; done
Shutdown and reclaim hostname#
On old-host, stop the old-host by running,
user@old-host:~$ sudo shutdown -h now
Destroy the old-host in DigitalOcean
Change the new-host name in DigitalOcean by removing the suffix or similar that was used to differentiate it from the old-host.
Run
user@salt:~$ sudo salt-key -L
to list out anduser@salt:~$ sudo salt-key -d old-host
to remove old keysOn new-host, run
user@new-host:~$ sudo hostname new-host
to renameUpdate new-host name in
/etc/hostname
,/etc/salt/minion_id
, and/etc/hosts
Restart the salt minion
user@new-host:~$ sudo salt-call service.restart
Restart datadog
user@new-host:~$ sudo service datadog-agent restart
Run
user@salt:~$ sudo salt-key -a new-host
to accept new keysRun highstate
user@salt:~$ sudo salt-call state.highstate
on salt-master to update domain name as well as known_hosts file