Website Security (Security Blog #2)

in, but not too far!

Securing a website is essentially a three step process. Firstly you have to decide on your security model, what services you want people to access, whether there are different access levels (e.g. your staff to update stuff, rest of the world to view stuff) and whether you need to have special access for your support staff.

Seconldy you need to lock the server down that the website is running on so only the bits you want are available. You can do this just on the server, but I recommend running a seperate firewall in front of the server (it can be shared) as this provides better security. All our websites run behind dedicated or shared firewalls.

Finally you need to make sure the website itself is secure, and doesn't have any vulnerabilities on it (I'll dicuss the common ones below and explain how they work, and how you can avoid them).

With regards to roles, the first one is your job (in cahoots with your website designer). The second one is down to your web host, or your tech team if you are doing it. The third one is down to your software developer or the providers of the product you bought. Do NOT ASSUME that off the shelf software is secure - I've tested plenty of off-the-shelf packages and found them vulnerable to simple web exploits (see below for details).

Lets start at the beginning:

Security Model

The simplest website (static content, never updated) says 'let anyone view web pages'. And that's it. No FTP access, no mail, no updates, no database, no other access to the server. If the content is all static (i.e. no one can add to it or change it) then this is pretty secure. It's also not very exciting.

Typically a simple business site will have some kind of content management system (CMS) that users can update. This can be open source, bespoke or paid for, and may even be a managed service (i.e. you are renting it). Either way you need to ensure that however you access the CMS it is secured with a good password (see previous blog) that is regularly changed.

Please don't assume that calling the update folder 'slartibardfast' or similar will prevent people finding it - security through obscurity is not security.

As you get more complex with the site, the security becomes more complex. If you require Role Based Access Control (RBAC) then you need to verify that each role can only do what it says - login as a high level user, copy a shortcut from Internet Explorer for something only you have access to and then log in as a low level user and paste the shortcut back in - you should get a nice clean error page, or thrown back to the login screen. You shouldn't get the page that only the admin can see. There is more to it than this, but it's a good simple test!

Lock the server down.

Next we come to locking down a server. This is really best left to the techie people like me, but the basic principle is all about attack surface again (remember that from the last blog?). Turn off anything you don't need, lock down anything you do need. Sample code? Remove it. Some of the most common exploits use things like the standard code Windows used to put into it's Internet Information Server to attack the server - leave it there and you are vulnerable.

Again a hardware firewall helps out at this stage - you may have turned stuff of, but if you only let the relevant ports through (i.e. only let people use a web browser to connect) then you are more secure.

Tools like Retina, Microsoft Baseline Security Analyzer and their compatriots will tell you if your server has any issues - run these tools, do what it says, run them again until the tool is happy or you are happy with the remaining risk.

A particular note at this point about PCI DSS (Payment Card Industry Data Security Standard - you can see why it gets abbreviated). This is something banks are enforcing more and more to ensure that your data is secure, and it covers the whole gamut from network security, website security through to storage and use of the card information itself. If you are storing or handling credit card information expect to encounter this at some point. It even applies to printed copies and manual forms! Particular key points to note are that you CANNOT STORE the three numbers on the back of the card (aka CVV code) in anyway after the transaction (you can keep it whilst the transaction is processing). Not even on paper. The same applies to certain information on the magnetic strip and the PIN and PIN blocks. If you have a database record with their personal details, card number and CVV code all unencrypted then seek help now!

Secure the site itself

There are three primary things you have to worry about (plus a whole bunch of other minor stuff). You already have a secure server from steps one and two, which is only letting people access the stuff you want.

There are plenty of other minor exploits, but the three that always get press are SQL Injection, Cross Site Scripting (XSS) and Dumb Coding.

SQL Injection

SQL Injection is where lazy programmers haven't bothered to validate the data going into their database. *** WARNING: Geek Speak coming up ***

Whenever you access stuff in a database on the internet, it usually gets translated into something called SQL (Structured Query Language) which is fired at a database.

For instance, you click login on your account with your username and password (in this case joe and bloggs) and it sends a command like:

SELECT * FROM USERS WHERE Username='joe' and password='bloggs';

All happy, a bit of data comes back and you are logged in. In English you have just asked the database to give you all the users whose username and password match what you have typed in - if it gets a row of data back, then your details matched and you are logged in. No data back means failed login.

However if you log in as joe with a password of

' or 1=1; --

You get this:

SELECT * FROM USERS Where Username='joe' and Password='' or 1=1; --';

Spot the difference? The Database ignores anything after --, so it processes the first bit of the command. Which in English logs you in as joe without knowing the password! It selects all users whose username is joe AND whose password is '' or 1=1 (which it always true for you maths people).

This would work for any username.

The above exploit also works inside the address in internet explorer. So for instance:

http://mysitename.com/viewproduct.aspx?productid=4

would fire:

select * from products where productid=4

at the database, which can be turned into:

http://mysitename.com/viewproduct.aspx?productid=4 and 1 in (select top 1 convert(int, username + '/' + password) from users);

would fire:

select * from products where productid=4 and 1 in (select top 1 convert(int, username + '/' + password) from users);

Which depending on how the site is setup, either gives a nasty error message or something like 'Error, unable to convert 'joe/bloggs' to an integer'.

Which gives you a username and password to the database...

It gets more complex than this (as you can do funny stuff to enumerate every single item in the database, or if the site is REALLY poorly written, execute any command on the server as the highest level user). All from the form you thought was just there to log people into your site and/or show then a product.

So how can you defend against this? Either you can scan for single quotes and remove them (and hope you don't have many foreigners, especially Irish (O'Brien)). Or you can write your code properly and use parameters (this turns the above statement into [SELECT * FROM USERS WHERE Username=@Username and Password=@Password;] and provides the bits with the @ sign alongside). Note that the latter option is by far the best.

XSS

The second exploit, XSS (Short for Cross Site Scripting - we use an X so we don't confuse it with CSS, which is Cascading Style Sheets which are useful!), works by people updating your site with stuff. If you have a forum, for instance, where people can post messages they can post a message that says something like:

<script language=javascript>windows.open('http://mysite.com/fakepage.html');</script>Hi mum.

When someone else views that page, a window pops up which can look dramatically like your site, but is in fact fake.

More importantly, they can do anything they like inside the script tags, as long as they know a bit of javascript. This includes moving stuff around on screen, redirecting you to other sites and if their browser is a little old or not fully patched, potentially download nasty code to the person viewing the page! Note: that's your customer. Visiting your site, and getting infected by something because your site is poorly written. Not exactly good for business...

This one is easier to defend against, any time you allow user entry you simply replace any > or < sign with the www equivalent. Most pages do this nowadays, and most forums provide this option. However anywhere where a user can add content you should beware of this exploit.

Dumb Coding

Lastly we come to Dumb Coding. This is personal hate of mine, and can represent itself in many ways. It is basically where the developers of the software haven't thought about security whilst writing the site, either because they aren't aware of it (particularly true with older software).

For instance, your users should be seperated from each other and only able to view stuff relevant to themselves - this isn't just common sense, it's also the law - the Data Protection Act requires you to make 'reasonable effort' to protect data, especially if that data is about an individual.

Take for example a site where you can upload a document as evidence (e.g. proofs for a bank loan). If you can view that site at a URL like:

http://mysitename.com/users/viewdocument.aspx?iDocID=523

then someone shouldn't be able to change the 523 at the end to 524 and view someone elses documents(!!) You may think this is obvious, I've tested sites where this is the case (and one of them was in the finance industry...). I've also tested sites where you don't actually even have to be logged in as a user to get to these pages...

Whoever wrote the site didn't bother to put in a bit of code to check that the person downloading the document was the owner/uploader of that document. The same applies anywhere where there is a number or reference point in the URL. This could apply to other users data, stuf you wanted hidden (e.g. retired / deleted products in your ecommerce page) or anything else references through the query string (the bit after the ?).

Other classics are checkouts where boundaries haven't been put in - so you can add -1000 items to the basket and get a credit, or where the payment screen has a box which you can edit for the amount you are going to pay.

Please don't hesitate to drop me a line if you have any questions (or if you feel your site is in need of some checking over!).

Next time we leave the realms of dumb coding and website security and move into something even more embarassing - often called the weakest link in IT Security...People!

---

Russell Henley
Managing Director
Henley Software Limited
T: 01628 550030 | M: 07770 380004 | email | web | profile | twitter | blog

IT Security for the non-technical user

IT Security for the non-techie.

People often ask me why we should worry about IT Security, particularly as an SME.

The simple fact is that the illegal hacking industry is now starting to target SME business more and more. This is because blue-chips are wising up to where their security holes are, and shoring up their defences. Therefore the next easy targets are SMEs because they don’t have the budget or know how. The so-called ‘low hanging fruit’.

So what can you do (other than paying people like me a lot of money)?

Well the aim of this blog is to give you an overview of what you can do in simple terms. Obviously you'll need some idea of how to use a computer (or you can opt for the most secure method - leave it in the box switched off!).

Your security all stems down to what we call 'attack surface'. Which is just a fancy way of describing your exposure to risk (you must know by now that us techy types have to have our own language so everyone else has no idea what we are talking about! – it’s the modern day equivalent of Thieves Cant).

In English this means the amount of exposure your IT system has to an attacker (aka hacker). IT Security is, in essence, about reducing this 'attack surface' to the smallest possible size that still allows you to do your business, and for the remaining 'attack surface' to be as secure and robust as possible.

If you think about your office building, all your doors and windows are locked and you have keys for those. You might also have CCTV and an alarm system. The same applies to your IT systems. You don't leave the door open, and you give each staff member that needs it a key. They might not get a key to the Stationery Cupboard or the Safe.

You can easily draw parallels with IT Security - your keys represent passwords, doors and windows represent ways into the network, CCTV and alarms represent monitoring software and intrusion detection systems, spyware is unauthorised personnel.

Before we move on a warning about TLA's (Three Letter Abbreviations). For some reason the IT industry is awash with these. I've tried very hard to make sure I expand them on first use, and I'll put a glossary at the bottom as well. Nothing is more annoying to a non-techie than to have no idea what WPA stands for. Unfortunately you can’t avoid these in IT…

Moving away from the metaphorical world, usually there are several key points in your network you need to protect:

1. Any wireless devices you have
2. Any internet or public network facing devices you have
3. Defend against viruses and spyware that have already got in

Externally you have to worry about:

1. Your website
2. Mobile users

However there are other attack points in your network - also known as your staff. These are usually exploited through:

1. social engineering
2. malicious use
3. stupidity

For this first article I'll look at your network security - where the weak points are and how you can easily secure them. At this point we are operating system independent - it doesn't matter if you are Mac, PC, Linux user or ZX Spectrums (!), you'll still need to use the same rules.

Let’s now look at each of these in turn, describe how it's usually weak and then how you can secure it.

Wireless

Wireless Technology is fantastic as it means your laptop / Mac / PDA users can sit where they want and can roam the office without having to worry about cumbersome cables. It also means you can improve the look of offices by removing ugly cables, sockets etc. You can now get wireless enabled printers, wireless access points that only require a single network cable to work and your wireless can extend to support mobiles, iPhones and PDAs.

However, this is a common access point for a hacker (just ask TK Maxx…), although does require proximity (i.e. you need to be close to the office/warehouse/whatever to gain access). Almost all access points are shipped with no security or filtering enabled, so a common mistake is to just plug them in and hey-presto, wireless access!

I've been to countless security audits for customers to find an access point still in this configuration. What it means is that anyone can park in your car park, switch on their laptop and connect to your network. Sure, they might not be able to log into your servers at this point, but they've done the hard work already (getting into your network).

The next most common mistake is to use WEP (Wireless Encryption Protocol). This is one of the first security standards for wireless networks and it's close to useless. There is a flaw in the way it works that means that anyone with the right software can crack the password in a matter of time - that time depends on the level of traffic on the network.

Even WPA (WEP's successor - Wi-Fi Protected Access) isn't that much better as this can also be cracked (it takes a little longer, and requires more know how, but it’s still possible)

So what can you do?

You have a few options - either you can use WPA2 which is a more secure version of WPA and the password can't be cracked. For most users this is reasonable unless you have certain types of data you need to keep really secure (e.g. credit cards).

You can also filter devices by their MAC address (each network device has a unique Media Access Code (MAC) which you can filter by). This is like having a guest list of computers allowed in (and your wireless router is the large bloke on the door...).

The best option is to treat your wireless network the same way as internet users - i.e. totally untrusted. That means they will need authenticate into the network somehow (e.g. VPN (virtual private network)), and anyone who gets into the wireless network is no closer to hacking in that being plugged into the internet.

Internet Connection

You need to be connected to the internet to browse it, send/receive email and do other line of business work. So what is the risk?

To put it simply, the internet is akin to the wild west - it's full of people trying to crack into your network. This is not an exaggeration.

I've seen reports of people plugging a fresh server into the internet without anyone know who or what it is and it being attacked several hundred times within a few hours. That's without people knowing what’s behind it!

Your public internet address may be unchanging (a.k.a. static), which means once someone knows it, it's always there. And every time you send an email you are probably giving it away (it’s usually in the email route information contained in the email headers).

So how can you be safe?

Firstly, you need to NOT connect a computer direct to the internet via internal or external modems unless you know what you are doing. You should use some kind of router/firewall (hardware device) to do this. Technically routers route traffic (i.e. act like a little police person with the white gloves pointing each little bit of traffic in the correct direction), and the firewall is the bouncer - only letting in traffic you want to know about.

You can get both in one box, however they tend to have less features than separate boxes (we tend to decide based on the customers size - larger customers want the added security of a separate router and firewall).

Most firewalls and/or routers support something called NAT (Native Address Translation). This is like a one-way valve that lets traffic from your network out to the internet but doesn't let it back in (unless you tell it to). Generally this makes everything a lot safer.

Secondly if you need to publish services onto the internet you should be very careful how you do this - work on the principle of least privilege where you only allow access to certain addresses at certain times. For instance if you receive mail from an internet server, restrict the traffic to only being from those addresses (this will also cut out spam). Don't just allow full access to one server from the firewall - there is no point having the firewall. This bit gets technical so it's best getting someone who knows what they are doing for this!

Thirdly, if you have budget for it, make sure your firewall has some kind of IDP (Intrusion Detection and Prevention). This will look for suspicious activity and cut off any communication from that address regardless of innocence level or other rules.

This doesn't require massive budgets or a dedicated IT team - there are mid-range products from manufacturers such as DLink, Linksys and ZyXEL that will do this. Publishing services (e.g. mobile email, inbound email, web servers, VPN) is a little more complex.

You probably want home workers to be able to dial in - so how do you do this securely? Well you need them to use a Virtual Private Network which is fairly secure. Again if you have budget you use two factor authentication (e.g. a password and a token - a key fob that gives you a number (also called a one-time password (OTP)) every 60 seconds).

This is the same principle behind chip and pin (something you know, something you have), with the something you know being your PIN number and the something you have being the token (or credit card).

Don't give each user the same username and password, make sure the password is complex (see more below on this) and make them change it regularly!

DO NOT expose services such as VNC (Virtual Network Client - a 'freeware' remote access tool), Remote Desktop, PC Anywhere or telnet if you can possibly avoid it - some of these only require a single password to gain access to the computer. It might be convenient for you to use VNC to connect to your office server from home, but it's really insecure.

A quick word about desktop or software firewalls – these do have a use, which is to block access from inside the network, however these aren’t best at defending from internet facing attacks – a hardware firewall does a much better job. Also software firewalls can fail (unusual, but possible) or be disabled by users (more on this in the next issue!).

Passwords

People often talk about having a 'strong password policy' but a lot of people have no idea what this means.

Complex passwords need to be long (at least 7/8 characters), not dictionary words ('password' is not a good password...) and also contain uppercase letters, symbols or numbers.
Ideally all of the above.

Complex passwords are simple if you educate the users in how to create them. I usually offer two methods - character replacements and acronyms.

Character Replacement is where you take a memorable word or two (at least seven characters) and replace a few letters with symbols. E.g. 'software' becomes 's0ftw@re'. You can combine words for this approach.

Acronyms are where you take a favourite movie, song track or whatever and encode it somehow. For instance, if your favourite single is Stronger by Britney Spears then you can have a password of O!idia.2 (for Oops! I did it again track 2 - which just happens to be Stronger for the non-Britney fans out there).

Glossary

For all the people terrified of TLA's - here's a list again with some more detail:

IDP - Intrusion Detection and Prevention - the ability of a firewall to pick up dodgy looking traffic and stop it. Comes in various forms, some of which look inside the network protocol to see what is going on and prevent unauthorised commands.

NAT - Native Address Translation - a technology that lets lots of private computers share one public address and also acts as a valve - requests can go out for data, but requests can't come in

OTP - A One-Time password - generated password that is only valid once, and for a certain time period

RSA - A company name and also Random Sequential Algorithm - the technology that generates OTP's

TLA - Three Letter Abbreviation - one of these!

VNC - Virtual Network Client - a freeware tool to control your PC from another location

VPN - Virtual Private Network - a 'secret tunnel' between your office and another computer/office

WPA - Wi-Fi Protected Access - a security model for making wireless networks secure. Version 2 is the best at the moment

WEP - Wireless Encryption Protocol - an old security model for making wireless networks secure that is flawed.

For the next article I'll be discussing making your website secure, and then what you can do to make your users secure!

Regards,





Russell.

One of the biggest questions I always get asked as a consultant is what/how/why should I backup in my business, and how should I plan my Disaster Recovery and Business continuity around that backup?

The specific answers to these questions depend on a number of factors, but this blog should give you some clear guidelines and techniques to solve them.

A quick few definitions:

Business Continuity (BC) - this is the approach that says 'How can I keep my business going should something happen?'
Disaster Recovery (DR) - this is the approach that says 'What do I do if a disaster happens?'

The two are closely intertwined, but separate issues. Business Continuity is about what is in place to maintain business functions - and it includes plenty of non-IT related considerations (office space, stock, telephony, staff etc.). Disaster Recovery is about the short term plan to get from a failure state (i.e. servers stolen) back to a better state. DR is short term, BC is long term.

There is one important caveat with this blog - these are GUIDELINES. Individual businesses have individual requirements and these may not be correct for you.

To start with we'll discuss the first question - what to backup?

What?

Firstly, the old rule of 'backup it all up' is somewhat flawed - data grows exponentially in all companies, and will overflow your backup method at some point. You may not simply have enough room to backup everything, or a big enough time window.

The simplest approach is to first have a good filing system, then break all your data into one of four categories.

The filing system is key to this - lots of SME business rely on Windows, Outlook and some kind of server yet a lot of them don't have it setup to store everything in one place. This means each laptop, desktop and server has its own little set of critical data that isn't backed up. The same can happen in a more corporate environment if the IT guys aren't on top of this. This means your PC crashes and dies and you've just lost your business forecast. And then you are on the phone to a data recovery company and spending thousands of pounds (more on this later), or worse it's been stolen and it's now gone forever.

Centralising this is simple, but requires a bit of knowledge (we aren't talking computer genius knowledge here). Technically (skip this bit if IT scares you), you need to enable My Documents redirection for everyone and offline folders for laptops. And use Exchange Server with caching enabled/use an IMAP based server product/backup PST files somehow for email. These two techniques will ensure users still think their data is local on their PCs, but actually keep it all centralised on the server. Backing up one location is much simpler than backing up 25.

Once you've centralised it, make sure it's organised well. A simple system is one area per department and/or location, and a 'temp' area for stuff you don't need to keep (see categories below). These can be shares, directories, separate servers or whatever depending on the organisation. You can keep user profiles somewhere else and user documents in another location. You can then put permissions onto everything if necessary (so non-Accounts people can't look at accounts for instance).

The rough categories of data (in non-technical speech):

1. Stuff I have on disks (e.g. installations, operating systems, applications, music files - note that having a copy of the disks is probably a good idea!)
   
2. Stuff I don't need to keep (temporary records, stuff you've downloaded from the internet)
   
3. Stuff that isn't business critical (copies of documents you have hardcopies of, documents with no retention requirements, some email data, some pictures, stuff that has been sent to you by email that you know is being kept, user profiles, personal files such as music and photos)
   
4. Everything else

Note that these are broad categories and there will be exceptions (for instance files downloaded after paying for them should go into Category 3 or 4, some installations might ask you to back themselves up to preserve state).

You NEED to backup category 4. You SHOULD backup Category 3. The rest doesn't matter. No really, it doesn't. You can backup category 2 data if you have capacity. Make sure it gets done last, so when you do run out of room, it doesn't matter. Category one data is too often backed up. You don't need to backup the 400+Mb installation of your favourite office program - it's on the disk, it can be reinstalled. Your profile settings should all be stored elsewhere nowadays (i.e. in your user profile, which is backed up).

Next we look at when you will need the data - this forms part of your business continuity (BC) and disaster recovery (DR) plan:

Let's assume your worst case scenario - all your servers get stolen. Which applications do you need access to NOW, which do you need access to in a day or two, and which do you need access to in a week or so? Using this 1-2-3 or Gold/Silver/Bronze approach lets you gestate your disaster recovery plan, lets you know what kind of business continuity plan you need and the lets you know where your critical systems are.

You may find you don't need anything for a week and can survive accordingly. You may find that without your stock system, email or whatever else your business is dead in the water. You need to adjust your backup, DR and BC plans accordingly. Note that this DOESN'T have to be expensive - if you know you can survive without stuff for a few days you have time to buy hardware, to get it rebuilt and to get data shipped/restored. If one system is critical you can have standby equipment offsite just to run that system - it could just be a standby PC stored at a Directors house preconfigured and ready to go.

Some of the biggest mistakes in planning BC and DR is that you assume it's an expensive process - more often than not it's about stabilising the business until you can get back into your offices and/or find somewhere else to go. Planning for it is the best thing you can do, as doing it on the fly is much more expensive and finding out that you don't have a critical piece of data after your server has caught fire isn't going to keep your board happy.

Some key points to review:

1. Ensure ALL the stuff you need backed up is centralised. For dispersed offices this means you either need to backup each office, or replicate each office back to a head office location for backup.
   
2. Think about all the data you have - your emails are often overlooked, my documents directories on local computers are often overlooked
   
3. If you have the install disks, don't back it up. If you downloaded it from the internet, don't back it up unless you have room. There are exceptions, but these rules generally apply
   
4. Figure out what systems need to be back up and running in what order - make sure you have planned for that to happen.

How?

This is a complex decision based on many factors.

Your key criteria for HOW is based on:

1. How often do you NEED to backup (hourly, daily, weekly, less regular)
   
2. How much data do you NEED to keep?
   
3. How long do you need to keep a version for?
   
4. What time window do you have for backup?
   
5. (Crucially) how much does your data change between backups (expressed as a percentage)
   
6. How quickly do you need access, and what kinds of access?

Question one is based on your recovery window - how long can you afford to be not working. If your business has mission critical, time sensitive data then your backup window might be hourly (or even lower). For instance an online, high volume commercial website might required live replication of data to a secondary data centre. Whereas a painting and decorating company might only require a backup once a week after their invoice run. Certain types of backups can be ruled in or out by this process (the former is unlikely to use tape except as a tertiary backup, the latter is unlikely to use replication).

Your recovery window is how long you can afford to repeat - if you only key in a few invoices a day and send/receive a few emails then this can be quite long. If you have twenty staff bashing in orders every hour it can be quite short!

Question two is about volume of data. Some simple sizing rules can be applied here. Most normal data will compress to a 2:1 ratio on average, however this doesn't apply for anything which is already compressed. This INCLUDES things like most images files (tiff, jpeg), music (aac and mp3) and most movie formats. These won't compress, so if your business is about making video trailers then you shouldn't expect much compression. On the other hand text data will compress more and Microsoft Access Databases are notorious for being mostly empty and compressing sometimes ten-fold.

Some simple guidelines for common low-end storage media.

(Storage Sizes are expressed in Gb. 1Gb = 1024Kb. Typical files are around 50-100Kb, although the bigger the file the larger the file size. Anything graphical in nature tends to be bigger. JPEG Pictures @ 4Megapixels are 3-500Kb, music files are 5-15Mb, 60 mins of movie data is ~600Mb. These all depend on the compression technology used to store them.)

CD's hold around 0.7Gb, or 1.4Gb compressed.
Single-layer DVD's hold around 4.5Gb of data (9Gb compressed). Dual-layer doubles this.
USB Sticks are now available in 32Gb versions, with USB disks going well over a terabyte (1024Gb).
DDS-5 (or DAT-72), the largest size of 'cheap' tape formats holds 36Gb (72Gb compressed).
SDLT (mid-range expense tape) holds 160Gb or 320Gb compressed.

You must allow for data growth - if you have 60-70Gb to backup up now, don't go for DDS-5 tapes as you'll run out in a year or so (data growth is hard to estimate, but 10% per annum is not unusual). Otherwise you simply add up the total size of the data you need to backup, do a quick sum on how much of it is already compressed and there is your backup set size.

Question three is about retention. If you might need to refer back to data from 6 months ago that will be overwritten (for instance transactional database records back to a point in time) you need to plan for that storage. This is where traditional backup methods such as tape have benefit - you can keep a tape for each month for a very low cost. Online backup usually has a recovery window of a month or two (although you can usually get longer at additional cost).

Most businesses tend to only need the most recent or possibly a week or two of history. However your individual requirements may differ.

Question four is about when you can backup. Most systems do allow for interactive backup (i.e. backup whilst data is being worked on) in some fashion, however this notoriously slows down the systems. Some systems can export incremental data so you can backup the changes since the last backup, or differential data which is the difference since the last full backup. These shorten your backup size dramatically. If you are running a 24x7 internet facing business, you don't HAVE a backup window - you've just got to accept your systems will run slow at some point. Pick a point in time when transactions are lower. Often you combine incremental (log) backups with a full backup in a maintenance/slow time window (especially true for databases).

9-5 business have a decent backup window (17:00 - 09:00!). Your backup needs to be able to complete inside this window (sounds obvious, but is worth checking!). You may be forced to do incremental/differential backups on week nights and a full backup on the weekend.

Question five forces options closed for you. If your data is volatile (i.e. changes a LOT every day) then you may not be able to use systems like online backup, and incremental/differential backups might not work for you.

Typically data changes less that one percent per day.

Question six depends on your recovery plans and if you need to use small scale restores a lot.

If you have issues with some systems that require frequent restores then this needs to be included in your backup strategy. Your disaster recovery plan and/or business continuity plan will tell you how quickly you need the backup (this is one of the major draw backs with online backup - it's great having cheap online backup in China, but if you need your 2Tb of data downloaded by tomorrow you'd better have a fast internet connection!).

So now you've answered those questions, what kinds of backup are there?

Without investigating specific technologies, they can loosely be grouped into:

1. Live backup. This is about replicating data to a standby or clustered system. It's VERY expensive, complex and hard to manage and maintain, but when your primary system fails you either have instant or triggered failover to a standby system. This can be locally clustered or geographically dispersed. Windows includes forms of live backup for free (e.g. Distributed File System (DFS)). Virtualisation supports excellent failover. Products such as SQL Server clustering, DoubleTake and the like allow disperse systems to failover successfully. Recovery windows are either instantaneous or very short.
   
2. Replicated backup. This is where the data is replicated to another server. This includes most online backup services. The difference between the above is that there isn't a failover necessarily (there may be partial failover). You usually need to do something to get the data back. Database log shipping does this (where you backup the transaction logs and then 'ship' them to another server). Recovery windows for this kind of backup are usually low, but depend on the speed of copying back the data.
   
3. Near-line backup. This is where the data is physically copied to some kind of removable device (e.g. tape, disk cartridge, USB stick, CD/DVD). Recovery windows here are higher as tape is slower to transfer (plus you have to find it!), however can be a lot quicker to restore than online backup.

Note that you can (and sometimes should) mix the above. There is nothing wrong with having clustered servers that also have a tape backup - this gives you the retention you need (question three above) without using tons of disk space. Or you might backup your Email data from Microsoft Exchange using an online agent but backup your file data onto tape as one is more critical.

Why?

Well there are two primary reason to backup.

Firstly is to recover in the case of a disaster (server failure, fire, flood, theft). This means you aren't out of business if this happens. The usual statistics thrown around here is that 90% of businesses without backup or DR planning are out of business within a year of a major incident.

The second reason is so you can go BACK in time to recover something important (e.g. if you changed your sales forecast and want to compare it to last weeks version).

There are additional reasons - you may have to for legislative reasons, your customers and/or suppliers may require you to have an active DR plan, which backup forms a key aspect of.

The online vs. onsite argument?

Both sides of the coin will always argue in favour of their solution, but cutting through the sales pitch reveals some key truths:

1. Online backup WORKS. It is more reliable than tape backup, and doesn't require any intervention to tape (the human element of tape backup can be unreliable, tape drives can fail, tapes and drives get dirty etc. etc.)


2. Online backup depends on you having reasonable internet connectivity, but more importantly on the 'delta' (the change in data each night) being transmittable during your backup window. If not, it won't work as it will get out of sync. Most of the time the company internet connection is fast enough to achieve this (as companies with more data will tend to have faster connections!)


3. Online backup supports excellent small-scale recovery. If you need one file, one email etc. then recovery is often simpler than onsite backup (as you don't need to find the right tape etc.) Onsite backup can support small-scale recovery, as you can create a tape and disk backup using most common backup tools. Otherwise you have to hunt for your tapes.


4. Online backup doesn't necessarily support full-scale recovery. If you need to recover ALL your data you may need to get it burnt onto disk/media and couriered over. Downloading it may take some time which you need to plan for in your DR plan. Onsite backup is easier for a full recovery. Remember you need to factor in media burn times into this (your online backup company may take 2 hours to burn the media plus 2 hours to courier it over, followed by 2 hours to upload it - that means you've lost most of a working day - near-line means you start off 4 hours ahead).


5. Online backup tends to cost a regular fee that isn't a capital investment - usually charged per Gb per month (sometimes with additional cost per service that you backup, e.g. database, email). Onsite backup tends to have an up-front capital infrastructure costs but less ongoing cost. If you are starting out, online is cheaper. If you already have tape, it's cheaper to stay with tape. However cost isn't the only choice!

Some golden rules:

1. Tape wears out. Replace them. 12-18 months for normal use.
   
2. TEST the backup! Restore stuff regularly and make sure it works.
   
3. MONITOR the backup. Don't assume it works, check the log files/emails/whatever
   
4. Give someone ownership of the backup - don't assume someone will do it
   
5. Data recovery is expensive, backup is cheap in comparison
   
6. Business continuity plans are for everyone, not just massive blue chips. It's all about planning ahead and thinking what would happen.
   
7. Untested disaster recovery plans aren't worth the paper they are printed on. Only by testing them do you find out they don't work, and fix them.
   
8. It isn't always expensive to create a DR/BC plan.
   
9. DR/BC isn't just an IT issue - it's an HR issue, it's a logistics issue, it's a financial issue - every department should be involved in some way.

If you want to receive updates on my blogs or contact me please either check out my Ecademy profile here or follow me on twitter here.

Here are my top tips on how to cut your IT budget in a recession. You can't eliminate usage of IT unfortunately - it's always something you have to spend money on, however there are a lot of simple ways you can make better use of your existing infrastructure rather than spending out!I've grouped this into some approximate categories.

Firstly, there is hardware itself:

Upgrade

Plenty of IT managers work on the 'replace it in 3 years' principle with hardware assets - this makes sense in a boom period as you get the Total Cost of Ownership (TCO) benefit of having warranties on all your equipment, however if you have equipment that is approaching it's replacement point, it makes more sense to carry out smaller scale upgrades rather than replacing. Simply increasing the memory in most modern PC's and servers will give them an additional lifetime, and this costs tens of pounds instead of hundreds. You would be surprised how much quicker a PC or laptop runs with extra memory.

Extend

You can also extend warranties for most hardware as well - certainly a lot cheaper than replacing in most circumstances. Unlike your home appliances where the extended warranty might not be worth the money (and have been branded a rip-off by certain watchdogs), IT warranties often include guaranteed response times or even replacement hardware within a timeframe, which is business critical.

Virtualise

Rather than buying new servers, you can use virtualisation software to 'create' additional servers on your existing hardware - this can save massive amounts of money in terms of hardware assets, and if you are running Enterprise licenses the software is also free. Even the software need to virtualise is free (VMWare Server or ESX Server - check out www.vmware.com).

Read more on my blog here.

Sell your old stuff

Almost every company I've ever been involved with in has a cupboard full of old IT stuff - it doesn't have much value, but you might be able to recoup something for it. It's also environmentally friendly. (You can't actually throw it away, and recycling it normally costs you money). Ebay is your friend.

Lease, don't buy

Nowadays you can lease everything you need - Exchange Servers are available per user online, you can lease your Windows / Office licenses, lease hardware, lease printers.

Secondly, software:

Go open-source

This is a big change to look at, but if you really must buy lots of new software this year, look at products like OpenOffice, Google Apps or other online providers. You might not want to switch servers to Linux (unless you have the skill base to do this), but changing the desktop apps instead of upgrading them can be a good idea.

Software Assurance and Open Value/Volume

If you do go down the Microsoft Route, get the best deal from your licenses by ensuring you are on the right plan (hint: don't buy boxed products!) and look at software assurance as it's much cheaper than upgrading to the next version when it comes out! Be aware of the pitfalls of buying OEM software (you can't move the licenses around) and make sure you buy the right version of Office (costs can vary by 300% depending on version and license type).

Skip a version

Never more pertinent that now with Windows Vista and Windows 7 - skipping a version of the software (if you don't have software assurance obviously!) saves you a whole upgrade cycle of licensing, rollout and downtime. Windows 7 will be released either end of this year or early 2010 (probably the latter...) so why upgrade now?

Next, IT Services in general

Review contracts

Something you should always be doing, but more important now. If you are using external backup services, web hosting, internet connections, antivirus software, SaaS providers, outsourced support etc. review the costs of those and shop around a little. The market place has changed dramatically over the past few years - a lot of traditional internal services (e.g. anti-spam) are better served online, costs for internet connectivity are a lot cheaper now. We have done this for some suppliers and are looking at reducing costs by 50% or more on some services.

Hotdesk

If you have 10 sales people and half of them aren't in the office all the time, you don't need 10 desks! Downsize the office, sub-let space if your contracts allow and reduce your expenditure.

Telework

With fast home broadband and small (and usually cheap) changes to infrastructure you can have your staff working from home, or mobile working with mobile broadband cards. This is so simple to setup, and you probably already have most of the software and hardware to do it! Imagine not having to pay rent, or having to pay a lot less for a virtual office? This also helps improve staff morale, plus is greener as people aren't driving around all the time.

Review your Backup and DR plans

If you don't have them, get them done. If you do have them, check that they include key suppliers disappearing. Think about all your key IT services - support, connectivity, backup - do you know what would happen if one of those providers disappeared! Also a good time to make sure everything is up to date (have you changed/added anything recently etc.?) This ties in with points above as well - do you really need mission-critical hosting for your website? Could you drop to a lower-scale plan?

Outsource

If you are employing full-time IT staff, look at outsourcing them - this can save you a lot of money, plus you'll often end up with additional cover. If you have already outsourced, review the costs of your outsourcing - is it working? Are you getting Value for Money?

Something everyone does - printing:

Investigate pay-per-copy printing/copying

This can save significant outlay - if you are looking at new copiers/printers there are lots of companies that will supply you with a fully maintained copier/printer and you just pay per print/copy. This saves the capital outlay and lets you pull the ongoing cost from different budgets.

Stick to one printer type

This is a simple and powerful trick - if you only buy the same kind of printer across your organisation (obviously depending on your needs - you might need two or three types) then you only need to keep one stock of consumables. If you buy whatever printer is on special that month you'll need a warehouse to store the consumables in which is just tying up capital.

Go Laser

Inkjet printers look cheap to buy, but the cost per page is horrendous (it can be 1,000% more expensive to print a page of inkjet print - especially if you are using inkjet paper as well!) Lasers are more expensive to buy but much cheaper to run.

Do you need a hardcopy?

This is another easy win - introduce a policy of NOT printing stuff - people print emails in order to file them (don't laugh...), people print PDF's to read them and people print several copies of a document to review each draft. Don't. Or print duplex, print multiple pages per sheet. We print meeting copies of our reports 2 pages per page, duplex. It uses 1/4 of the paper, and less ink, but still looks professional.

Ecofont!

Saw this one on the internet - it's a font with holes! The theory is that you use 20% less ink per page (saves you a lot more money with inkjets, but not a reason to keep them!).

Finally a warning:

Don't compromise on quality

The biggest mistake people make when looking at saving money is buying cheap stuff. Don't make this mistake - it will cost you more in the longer term. You might think that you can save money buy purchasing cheaper hardware, unbranded ink cartridges or internet services from less reputable agencies/countries. However this may end up costing you far more in the longer term from downtime, blocked printers and missing websites! It is a sliding scale, and finding the right balance is a difficult skill.

Hopefully these all make sense to people - please get in touch if you need help with any of them!

You'll also note that a lot of these policies are quite green as well (recycling, saving energy, saving paper) so you increase your companies green credentials whilst saving money!

Virtual Machines

So this is my first blog post and I thought I'd talk about what I've been looking into for HSL for 2009.

First on the list is virtualisation - for those that don't understand this, the basic principle is to take one physical computer (i.e. big metal box, or 'tin' as we call it (even thought it's not made of tin!)) and use some funky software to convert it into several 'virtual' machines. Each virtual machine looks and feels like a real computer (on the network at least), but is actually just a file on the server. They share the resources of the 'host' computer (disks, memory, processors etc.).

The obvious benefit is that you don't need to buy so many servers, which has various benefits (reduced overhead, lower running costs, greener IT as you aren't using excess power etc. etc.). However it doesn't necessary save on licensing costs (unless you are open source of course). Most servers are primarily idle so performance isn't a massive issue.

The good news is that the nice people at VMWare are currently giving away ESX Server for free - which lets you try out running virtual machines yourself for free. And it's a doddle to do.

We've used this kind of software before for infrastructure testing and software testing, but now we are venturing into running live servers using this technology - hosting Exchange Servers and providing standby recovery environments for our customers. None of this is live yet, but I'll keep you posted with our experiences into how this is going.

I have already used VMWare Server (latest version) to create a virtual Exchange Server for a customer which will be going live in early 2009, and plan to upgrade this to using ESX server in the future (to support running more virtual machines) as well as upgrading memory on the relevant hardware at the same time.

For those that don't know the difference, VMWare Server (or Microsoft Virtual Server) lets you run a virtual server 'inside' another server operating system (so you run Windows 2003 Server as normal, inside which runs VMWare Server, inside which runs another copy of Windows 2003 Server, Linux or whatever your favourite is). ESX Server is a 'hypervisor' which is basically a very small operating system that JUST runs the virtual machine environment, it's management software and not a lot else. This means there is more processing power, memory etc. left to run Virtual Servers, almost doubling the capacity to do this.

The benefits to us are that we don't have to buy a spare server for every customer we want to have a recovery setup for (which probably keeps them happy, as they'd be paying for it in one way or another), plus the setup time is much quicker (we can create a basic server config as a file, then just copy it around).

The other real benefit is portability - you can move a virtual server into totally different hardware and it still works. You can even do clever stuff with Storage Area Networks (SAN) or Direct Attached Storage (DAS) and higher end software that lets you move the virtual machines around whilst they are still running. This means we can start off recoving to an online VM and then download / transport the relevant files to a new set of hardware in a new office location.

We are using Dell PowerEdge 860 Servers with 8Gb RAM to handle this and are planning to implement a DAS/SAN behind this to handle the volumes of data and support cold/warm failover to similar hardware. This should allow us to run 4 VMs on each machine comfortably, potentially more if memory requirements are lower. We are only planning on running one or two 'live' servers with a few disaster recovery (DR) servers setup for customers.

Coupling this with an online backup solution will enable us to recover a customer to their last backup in a matter of minutes or hours rather than days or weeks, without having to invest heavily in additional licenses or in hardware - thereby keeping the costs down for our customers (and us!), whilst providing an excellent service. This forms a core part of our business continuity/disaster recovery strategy for both ourselves and our customers.

If anyone wants more detail about how we are planning to do this, or is interested in talking more about any of these elements please don't hesitate to get in touch.