Monday, April 5, 2010

Woes of a Big Site: How to Prepare Your Site for Growth Tips on creating a scaleable site that can deal with growth

I often read posts in forums and discussion lists by newcomers asking how their site can become big and well-known. Well, naturally this is every webmaster's dream - who ever creates a site hoping it will become obscure? However, what many newcomers fail to appreciate is that big and well-known sites face a number of problems. Problems which my other site, thefreecountry.com, has begun to face as a result of its rising popularity and size. This article mentions a few of those problems and some things that can be done to pre-empt them (which alas I did not do).

1. Traffic (Bandwidth) Problems

It's interesting how everyone faces traffic problems. The small site faces traffic problems - few people visit them. When the site increases in popularity, it too faces traffic problems, albeit a different kind: the expensive kind.
When a site is starting out, the typical data transfer (sometimes loosely, and incorrectly, called "bandwidth") allocation of (say) 5GB seems generous and a distant target that looks impossible to hit. But wait till it becomes popular. Suddenly that 5GB looks very skimpy, and you'll be frantically searching your web host's documentation for how they charge for "overages" (the amount by which you exceed your allocation).
One of thefreecountry.com's previous web hosts had a rather pathetic traffic "overage" policy. If the site exceeded its traffic allocation, it would be shut-down for a month. (Gee, how did I miss that in the fine print?) To prevent that, the owner has to pre-pay for additional blocks of data transfer. The trouble is, the host had no system of warning you in advance when you were nearing your limit, which meant that the webmaster had to monitor the traffic of the site closely. To add insult to injury, the additional data transfer you had to pre-pay for came only in blocks of 10GB. Let's face it: when you initially exceed the "bandwidth" limit, it is unlikely that you'll exceed by 10GB. The initial "overage" is likely to be at most 1 to 2GB (unless of course you had just embarked on an aggressive advertising campaign).
My recommendation: when scouting for web hosts - think ahead. Check out the "bandwidth" or data transfer policy. It should bill you according to the amount you actually exceed and you should not be required to have to pre-pay it. And read the fine print in all their policy statements carefully. Oh yes. For reasons mentioned in my article on "How to Choose a Web Host", it is probably best to stay clear of hosts that advertise "unlimited bandwidth". If you have not read that article, you can find it at http://www.thesitewizard.com/archive/findhost.shtml

2. Too Many Pages to Update

When I first designed thefreecountry.com, I did not plan for it to be the large site it is today. As such, I made no provision for keeping certain common features found on every page in a single location. Instead, things like the navigation menu, logo, etc, were inserted into every single page.
This was fine when thefreecountry.com was small. But it became a nightmare to maintain as the site grew. Just imagine - each time I wanted to change the design of the site, I had to manually go through the two hundred odd pages to modify them. I could not even use the search and replace feature of my editor to simplify my task - the changes often span multiple lines and are interspersed with page-specific information.
My suggestion is that even if you think your site is a small site in terms of number of pages, plan for its eventual growth. Put common design elements of your pages in a central location. There are many ways to do this: using Server Side Includes (SSI), using frames, dynamically generating it from a template, etc.
Don't think this will not happen to you. When I started the predecessor of thefreecountry.com in 1996, I only had 3 pages. Today it's a monster. True it is not as big as those sites with thousands of pages (shudder) - but I can tell you from experience that once you hit a three digit number for the number of pages, updating the design of the site isn't quite so enjoyable anymore.

3. Reliability

Usually when a site is new, the owners consider it little more than a hobby horse. As such they tend to simply find a cheap web host to dump it on, since in its infancy, the site is unlikely to generate much income anyway. This is what I did originally with thefreecountry.com.
It would have been fine except that I kept thefreecountry.com on the cheap web host far longer than I should have.
Cheap web hosts seldom have the margin they need to hire good, competent help. Now I don't mean technical support that responds to you fast. Anyone can do that - even that cheap host I was on. Their responses each time however display their ignorance. When I say competent help, I mean the kind that knows the server hardware, the various software that run on it, security issues, and how to fix software and hardware problems. You need that kind of competent help so that they can pre-empt (as far as possible) potential problems as well as handle any true crisis that arise (eg fixing new security holes in BIND or Sendmail, etc, or troubleshooting hardware problems).
When your site grows and gains more visitors, the "uptime" of the site becomes increasingly important. If the server goes "down" for a couple of hours when your site is new, you have lost at most a few visitors. But when your site is well established, every hour that your site is down loses you thousands of visitors (and the income associated with them).
If, like me, you placed your site on a cheap host when you started out, you should keep an eye out for good, reputable web hosts so that when your site grows, you'll know where to move it to. Such hosts may cost slightly more than a budget web host, but the stability you get is well worth the extra expense, especially when your site has a lot of traffic.
Unfortunately though, keeping up to date with the web hosting scene is not a one-time affair. If you have been following the changes in my "Which Web Hosts Do You Recommend?" FAQ page, found athttp://www.thesitewizard.com/archive/webhosting.shtml you would have seen that good hosts do turn bad over time. This usually happens when they get too big, and have to hire additional help that do not have the commitment and competence of the original owner.
The verdict: budget hosts are fine when your site is small. But always be prepared to move. And you definitely should move when your site grows bigger and becomes a viable commercial venture.
(For the curious, thefreecountry.com is no longer hosted at the cheap web host I mentioned.)

Planning for Growth

This article is really about planning for growth. The best time to do that is when your site is young. The changes that you need to make to prepare for the future will be smaller when it is young. Don't wait till your site is too big. I had to learn the hard way from hindsight. You don't have to.

Basic Intro to AdSense (for beginners)
Adsense Earns
Adsense Tips
Adsense Alternatives

No comments:

Post a Comment