PLANNED DOWNTIME (2012-11-10 2AM - 4AM): Transition to New Hosting Platform

This site may earn a commission from merchant affiliate links, including eBay, Amazon, and others.

vvvvvvv

Sharpshooter
Special Hen
Joined
Nov 18, 2008
Messages
12,284
Reaction score
65
Location
Nowhere
---KNOWN ISSUES---

Blue Steel Style (and perhaps a couple of others) are not working properly - http://www.okshooters.com/forum.php?styleid=8
Search is currently broken.

---PROGRESS UPDATES---

2012-11-10 2:22PM

Search is now working.

2012-11-10 12:57PM

Login issues should all be fixed. Working on fixing theming issues.

If you're seeing "Array" in a lot of places, go here: http://www.okshooters.com/forum.php?styleid=8

2012-11-10 8:08AM

So nothing went as planned. However, we did get upgraded to 4.2.0.

2012-11-09 7:40PM

Around 1PM today while I was partaking in my lunch, I got the final issues on the new webservers fixed. You have been using them since. If there was a hiccough, it's because they are still pointed to the old server's database.

At approximately 2AM, I will be migrating the database to the new servers. Additionally, an upgrade to the software will be attempted. This will result in 30-45 minutes of downtime. However, in case of unforeseen issues, I'm reserving a window from 2AM to 4AM for the migration and upgrade.

Thank you for your patience!

2012-11-06 1:52AM

And back on track - should be point back to the new load balancer. The world's DNS network sure is getting a workout with this one.

Issue: vBulletin doesn't process HTTP headers properly out of the box, and apparently hasn't since 2005. I should really note that core patch before actually doing the upgrade process.

I'll probably do the webservers later this week.

2012-11-06 12:34AM

Everyone should be pointing back on the old server. vBulletin didn't like having several hundred users from the same IP and was getting confused resulting in password failures that should not have been failures. Since the load balancer that was being used can't pass the source IP, we'll have to put up our own load balancer that can handle it.

2012-11-05 8:56PM

The transition process to the new hosting platform has already begun. In an effort to minimize the downtime associated directly with the transfer, the load balancer at the new place has been setup to use the old server as the backend. This has allowed us to go ahead and update the DNS to point to the new location with no interruption. (At this time, about a third of the traffic is going through the load balancer. I've actually be posting through it for a couple of weeks now to make sure it worked properly.)

A side effect of this setup may be some minor slowdowns during peak times as more users are going through the load balancer instead of directly to the current server. Those slowdowns will go away when the entire system is at the new location.

Once we're confident that the DNS records have adequately propagated and nearly all of the traffic is coming through the new load balancer, we'll switch the load balancer to use the new webservers. Depending on where we are in the upgrade process, we'll either still be making database hits on the current server or we might also migrate the database to the new cluster at the same time.

There should not be any downtime during the transition other than when the database is migrated and vBulletin is upgraded. The downtime goal for the database migration and vBulletin upgrade is 30 minutes total, but there will be a two-hour window reserved as planned downtime just in case. I'm not sure if those two events will happen together or at separate times.

The new hosting platform is a high-availability cluster and should rid us of the downtime issues and database crashes we've been having.
 
Last edited:

vvvvvvv

Sharpshooter
Special Hen
Joined
Nov 18, 2008
Messages
12,284
Reaction score
65
Location
Nowhere
Is any of this going to effect the election or hula hoop girl videos?

No. It will probably be at least a week before the minimal amount of downtime.

Nice, good luck! Just out of curiosity, what sort of cluster you running?

*hint* starts with "F" j/k.

Lurker66 is right. (just kidding)

And to please n8thegr8, here's the basics from the geek world. (If you got lost on the first post, skip this one.)

In front is a load balancer. Normally, I run Varnish (and Nginx for SSL termination) as dual caching front end load balancers with a floating IP so that if one dies the other picks up and answers that IP within a few seconds. But for this site, it looks more cost-effective to run Rackspace's Cloud Load Balancer option. I did do some performance testing first and found them satisfactory. I've got a few of them running for other "special" sites and have found that they usually have a replacement provisioned within 30 seconds in the event of a device failure.

Next will be a pair of webservers running Nginx, PHP-FPM, APC, and Memcache. (I am considering putting a small Varnish cache on them to help with current-event spikes where we get a sudden boost of anonymous users for only a small part of the site - like when Open Carry passed.) The webservers also run HAProxy for connecting to the MySQL cluster. In the event of high load detected by Zenoss, an additional server is provisioned and the load balancer is told about it automatically. When the load dies down below a certain threshold, the server gets nuked. Most of the time, I'm getting webservers provisioned within minutes thanks to deploying from images and ensuring configuration with Puppet.

Finally, the meat is in the cluster. I'm a Galera nut, and the MySQL servers will be running Percona XtraDB Cluster (which is MySQL patched with Galera and other enhancements). There will be a minimum of three MySQL servers at all times, and since Galera is synchronous replication we don't have to worry about the issues created by having a single master take writes and slaves answering reads - every server is a master that handles reads and writes without the headaches of MySQL's normal multi-master implementation. Just like the webservers, the MySQL servers will also be scaled in and out horizontally in response to load. Also, there will always be one server in the cluster that is the designated donor for new nodes and is also where backups will be made from. This way, backups can be made and new nodes synced without having an effect on everyone else's use of the site.

Oh yeah, I'm a Debian man.
 

Latest posts

Top Bottom