April 5th, 2010
I have been playing around with Openfiler for the past few weeks. Basically Openfiler is an open source, customised Linux operating system specifically designed to be a file server, or more specifically, an “Open Source Storage Management Appliance” . It has far more functionality than simple file storage though, it can be an FTP server, a Network Attached Storage (NAS) server, and even an iSCSI SAN if you need one (I used it while testing a VMWare vSphere infrastructure system). It will run on pretty much anything x86/x64 based (min spec 1Ghz processor with 512MB RAM), can interface with a Windows domain and its web based interface is pretty simple to use so anyone looking for a simple and cost effective bulk storage solution should definitely have a look at it.
I had a test server (2x Xeon 2.8, 4GB RAM and 4x200GB SATA drives in RAID 5) to try it on. As I already had hardware RAID, I didnt need to implement software RAID but as it supports software RAID 0,1,5,6 or 10, I could have. One thing that caught me out was a limitation of 4 primary partitions on the drives. Apparently a normal implementation would have the Openfiler system (by default this is 4 partitions) on a single drive or array and it would be separate from the data storage. It is not recommended to have the OS and the data on the same disks as a restoration may be more challenging. As I already had a 4 disk RAID array ready to use, and this was for testing only I just installed to that and therefore I could not use any of my drives for data which kind of defeated the purpose. A reinstallation on the same array but with a manual partition creating an extended fourth partition rather than a primary gave me over 550GB of usage storage. Following the basic installation instructions, I found it relatively simple to create a usable NAS box. I did not add it to a Windows domain but think that it would actually be easier than having to configure the Openfiler device as its own LDAP server. FTP was also pretty easy to get up and running. You dont need to know any Linux at all, the initial installation is graphical (unless you want console) and after the initial installation, all configuration is done via a web browser.
Apparently if you plan to use it for production block level storage (iSCSI, SAN), you apparently should use a second network interface for management although in testing, I have not bothered and simply use it across my network with only minor performance issues. It is actually easier to set up as an iSCSI target than it was for NFS or FTP and is simple to connect to VMWare ESX (although I did need to reboot it after re-mapping LUN’s before ESX could connect to it as an iSCSI target even though it could see it). I also had no problems connecting my Windows 7 laptop to it either using the built in software iSCSI initiator with pretty good performance (30-50MB/s over gigabit ethernet)
To set up openfiler as an iSCSI SAN:
1a. Create Physical volume on a single disk OR
1b. Create RAID volumes on multiple disks and create array
2. Add volumes from #1 into a Volume Group
3. Create an iSCSI volume in the VG from #2
4. Start the “iSCSI target server” service
5. Add a network entry for the client machine (or local subnet if private) at the bottom of “System > Network Setup”
6. Click “iSCSI Targets” on the Volumes page
7. Click “Add” to create a new target.
8. Click “Lun Mapping”
9. Click “Map”
10. Click “Network ACL”
11. Change the combo box for your network to “Allow”.
I am not sure if I would roll this into production just yet but for a backup storage system, or bulk storage of non-critical files (I used to run a 250GB iTunes server at a music publishing company that could definitely benefit from this type of flexible storage), it could be very useful. There are purely commercial alternatives for production use such as Datacore SAN Melody but there is an active userbase of Openfiler which should be able to assist with and commercial support options are available if required.
December 2nd, 2009
An alternative to expensive Microsoft server licences and higher hardware requirements is to use Linux as your server platform. Most server functions can easily be performed by free (yes free, really) Linux servers. This includes company firewalls, file and print servers, web servers, email servers, authentication, database servers, proxy servers, storage servers (such as Openfiler) etc. If it can be done on a server, more than likely, the function can be done for free with Linux. There are commercial Linux server releases such as Suse and Red Hat which are not free but these come with support etc. Most of the high end server manufacturers offer Linux out of the box (HP, IBM, Dell and others)
Most Linux servers have far smaller hardware requirements than Windows servers and will happily run on older hardware. Most of them also only have a terminal interface though rather than a nice graphical interface so configuration and maintenance can be difficult for staff familiar with Windows servers only. There are tools available to make configuration easier. Interaction with Windows PC’s is made possible with a system called SAMBA so the end user does not even realise their servers are not Windows.
If you find the idea of using something a bit unknown worries you, you may be interested to know that (according to a Wikipedia article), over 85% of the worlds super computers run Linux distribution of some description and the big movies studios all use Linux servers for movie production. A lot of web servers that serve you up your internet content (including this site) are also all hosted with Linux servers.
If you want to set up a free Linux server, stick with one of the main distributions such as OpenSuse, Ubuntu or Fedora to make sure you have a wide user base to draw support from. My development Linux server is Ubuntu running under VMWare.
October 18th, 2009
Regular backups are very important for anyone, especially businesses. While a home user may lose some music or photos if their backups are not up to date, a business may lose invoices, orders, emails etc which cannot be replaced and may have a long term effect on the business.
You really cannot go too far with backups, depending on your risk profile and budget, your backups may be a simple file copy once per day or real time backups pushed to multiple locations. Obviously the more you do the more it will cost but the lower the risk of data loss if something happens.
Before I go into the options, don’t think that it wont happen. It will. Hardware fails, computers die, laptops get dropped or stolen, power spikes occur etc. If you go into this expecting the worst, you are usually in better shape when it happens than those who are not ready for it. Once you have had a significant failure and your backups are not good enough, you tend to take it more seriously in the future.
* Hard Drive Backups
As a bare minimum, and I mean bare minimum, a portable hard drive is a cheap and convenient option to back up your data from one or more locations. It can be a bit manual and does require some discipline but is much easier than burning to CD/DVD. Many come with backup tools but having seen some of them in action, I recommend using a simple backup script to maintain full control over the backup process. Windows (from Vista onwards) ships with a utility called Robocopy which, while small, is one of the best free applications ever to come out of Microsoft. Older versions of Windows can also use it but it needs to be downloaded as part of the Windows 2000 or Windows XP “Resource Kit”. To use it, simply create a folder (call it “scripts”) and create a blank text file, call it “backup.bat”. If you have Windows XP or earlier, you need to put the robocopy.exe file into the same folder. You need to edit backup.bat (right click and edit or it will try to run it). The way you use it is to call robocopy, give it a source location and a destination location and tell it what you want it to do, one command per line. eg robocopy “c:\email” “f:\email” /MIR will use robocopy to “MIRror” the c:\email folder to f:\email assuming that your portable hard drive is allocated drive F:. The /MIR or “mirror” option will delete target files if they no longer exist in the source. This is useful to ensure your backup drive doesnt grow bigger than your data drive but you run the risk of data being lost if the source file get accidently deleted then a backup is run. A better option for a portable hard drive is to have two backups pushed to it, one with the /MIR switch to mirror it and one without which will copy changed files and new files but will not delete anything.
Other Robocopy options can make your backups work better or be a bit more flexible. eg
- robocopy “source folder” “destination folder” /MIR /w:2 /r:2 will wait for two seconds (/w:2) and retry twice (/r:1) if a file is in use and cannot be copied. The defaults are wait 30 seconds and retry 1,000,000 times which will not always be useful.
- robocopy “source folder” “destination folder” /S /log:logfile.txt will copy from the source to the destination including subdirectories (/S) (but not empty subdirectories use “/E” if you want empty subdirectories as well) and will log everything it does to logfile.txt
- robocopy “source folder” “destination folder” /S /XF *.txt *.tmp will copy but will exclude files (/XF) that end with “txt” or “tmp”
Other useful switches are /XD (eXclude Directory), /MOVE (MOVE files and folders, ie delete from source after copying) and /PURGE (delete destination files that no longer exist in the source – used with /E has same effect as /MIR)
A full list of Robocopy options can be found by opening a command prompt and typing “robocopy /?”
The next step up from a USB/eSATA hard drive for disk based backups is a NAS (Network Attached Storage) device. This can be a single drive like the USB connected on or can be a RAID array connecting via NAS, FTP, iSCSI etc depending on your needs and budget. Openfiler can convert pretty much any hardware to a NAS device that Robocopy or another backup system can access for backup storage.
* CD/DVD backups
If you need or want archives of your data for long term storage, DVD backups are cheap and relatively reliable (CD’s as well but since DVD burners and blank DVD’s are so cheap, there is little point persevering with CD’s). A standard blank DVD holds 4.3GB of data which should cover most of your important stuff (documents and emails) for some time. They take up very little space and are readily readable. They do, however, require more work to create, the process cannot be as automated. DVD burning software like Infrarecorder is required (most DVD burners and PC’s will come with some form of burning software which will usually suffice). You will need to know where your data is stored and how much space it takes up.
* Tape Backups
If you have a lot of data and need archiving, the most cost effective solution is a tape backup unit. They are relatively expensive to buy but in Dollars per Megabyte, they are very cheap. The tapes are also very portable which makes it easy for you to transport your data if required (having the most recent tape in your bag each night is better than leaving your tapes onsite if there is a fire!). Tape drives run from a few hundred dollars for slow DAT format tape drives which will do 20GB or so, up to a few thousand for a high speed LTO 4 format tape drive that can hold over 1000GB of data on a single tape (the tapes are more expensive too). They also go much, much higher than this if you decide to opt for a tape library where the backups can span multiple tapes and tape changes are done automatically but I am not going to go into Enterprise class tape libraries here. My rule of thumb is to calculate the storage space you need now, at least triple it and buy a tape system accordingly. While it is possible, I strongly recommend ensuring that your backups dont run longer than a single tape over the lifespan of the tape unit and tapes (you should be able to assume that a DAT drive will last at least 3 years and LTO 4-5 years, the tapes will last longer than this).
* Offsite Backups
There are a number of backup services which, for a fee, provide a quantity of space on the internet where you can upload your files to keep a copy offiste where you can access them as you need them. While they are generally considered reasonably secure, if you are uploading sensitive information, your data should be secured before uploading. Zip archives can be secured with powerful encryption, 7Zip has this functionality built in, simply select the encryption option and put in a secure password and the file will be both compressed (for easier upload) and securely password protected.
