Search Documentation
< All Topics

Disaster Recovery Plan

Posted15 July 2021
Updated15 July 2021
ByAndrew

Loss of data from a server

CS3793/5 & CS3311 (Mojohost)

If a file or files are lost from these servers, e.g. they are accidentally deleted, then a ticket should be raised with Mojohost with a request to restore the file(s) either to the original location or to a temporary location on the appropriate server(s), whichever is appropriate.

CS2671 (Mojohost)

If data is accidentally deleted from the MySQL database on CS2671 then a ticket should be raised with Mojohost asking them to extract the data from the most recent database backup (or older if appropriate) and placed into a temporary database and table(s) on CS2671. The data can then be copied across to the “live” database(s) and/or table(s) manually from the temporary location. Once restored, the data should be removed from the temporary location.

OT Image Processing (AWS)

There isn’t really anything on here to be deleted that isn’t either part of the OS or the OT Image Processing source code, that is in a Git repo on BitBucket. If worse comes to worst, restore a snapshot as per the description above in the “Loss of single server” section.

OT Blogs (AWS)

Boot up a new instance from the latest snapshot available, find the file(s) and copy them across to the live server and then terminate the new instance.

Loss of single server

CS3793/5 – Main Load Balanced Web Servers (Mojohost)

Mojohost to either replicate from the other or to restore from their backup.

CS3311 – Miscellaneous Web Server (Mojohost)

Mojohost to restore from their backup.

CS2671 – Database Server (Mojohost)

Switch DB to use CS3793 as master and update ProxySQL setting to use CS3793 for everything, Mojohost to restore CS2671 from their backup and then set as the slave to CS3793 and sync up. Once in sync, switch CS2671 back to being the master and CS3793 to being the slave and update the ProxySQL setting as required.

OT Registration (AWS)

Boot up a new instance from the latest snapshot available and switch the Elastic IP from the old instance to the new old. Daily snapshots from the past 7 days should be available.

OT Image Processing (AWS)

Boot up a new instance from the latest snapshot available and switch the Elastic IP from the old instance to the new old. Daily snapshots from the past 7 days should be available.

OT Blogs (AWS)

Boot up a new instance from the latest snapshot available and switch the Elastic IP from the old instance to the new old. Daily snapshots from the past 7 days should be available.

Total Hosting Loss

If Mojohost is completely lost, e.g. data centre destroyed by fire, explosion, etc… then we will need to do the following to get the sites up and running again:

  1. Create an RDS instance and restore the most recent database backup in ot-mojo-db-backup to this new RDS instance.
  2. Create an EC2 instance \- Install Lucee \- Add sites from Bitbucket repo – OAS (tours and members), OAS Auth, AL (tour and members) and LN (tour and members) from Bitbucket installed on it. \- Add sites vpndetect.oas.me and hooks.oas.me from backup \- Configure Apache with the sites (backup of httpd configuration from CS3793 is on S3 backup \- Configure Lucee with DSN’s pointing the RDS created in step 1. \- Map the content to the S3 bucket tss-s3-storage-backup using goofys \- Add scheduled tasks (list here)
  3. Create a second EC2 instance based off an AMI from the first.
  4. Change the DB details for registration scripts to point to the RDS instance.
  5. Temporarily remove the country check against the CRM.
  6. Create an Elastic Load Balancer (ELB) and configure it (SSL can be obtained via AWS Certificate Manager and DNS entries).
  7. Change the DNS points in Cloudflare for the OAS sites, AL and LN to point at the ELB.
  8. Set up tour. sites from S3 backup.
  9. Change the DNS points in Cloudflare for the tours
  10. Set up portal. sites from S3 backup.
  11. Change the DNS points in Cloudflare for the portals.
  12. Set up sites from “miscellaneous” server using the backup of that server stored on the S3 backup. This will include the CRM, onlycarla, onlymelanie, OTCash and so on.
  13. Change the DNS points in Cloudflare for sites on “miscellaneous” server.
  14. Put the country check against the CRM, remove in step 5, back in place.
Table of Contents