Media storage & sorting

It’s been a long time (over a year in fact) since I last blogged about my home AV setup, and I left the last post on a real cliffhanger – how do I sort and store all these videos? Well, wait in suspense no longer, because here’s the answer!

Firstly, a bit about the storage – for a while, I have crammed more and more hard-drives into a single old desktop machine that I re-purposed to be my media centre/SVN repository/Tomcat/Apache/seedbox/etc, but it was getting to the point where I had files spread across 3 different drives, and no redundancy in the event of drive failure. Given I’ve had more than a few drives die on me recently, some sort of recovery strategy was becoming very necessary. We’re not talking small size requirements here either – somehow I’ve managed to accrue over 4TB of videos, music, pictures and other files. With these requirements in mind, I went shopping for some sort of Network Attached Storage style of device, but with more of a DIY edge to it. Having looked closely at Drobo and the like, I decided I’d rather have full control of the OS and RAID setup; the idea of not knowing what’s going on under-the-hood, and not being able to recover the data apart from on another Drobo, was not appealing at all. I ended up getting a HP ProLiant N40L Microserver, sadly well after the £100 rebate ended, but still at a very good price for a decent specification of machine and a nicely engineered case. I moved the 250GB system drive to the optical drive bay as documented here, in went three 3TB hard drives, on went a fresh copy of Ubuntu 12.04.2 server, and I was away. The fourth drive port on the ProLiant is left empty for now, with a view to putting in another 3TB drive when I eventually run out of space again.

I’ve heard interesting things about ZFS recently, but decided to stick with what I know, and set up a RAID5 array across the three disks, giving me 6TB usable space, and allowing for the failure of any single disk. This arrayis then shared over NFS so that my other server can access to add files, and my Mac Mini (running XBMC) can access the content to display on the TV.

So, travelling back to the end of my last post on my home media setup – how do we sort the files we’ve downloaded to keep them in an OCD-friendly manner? The answer – SortTV. This program is one that I wish I had found years ago, it does a fantastic job of looking up program metadata on various sites (IMDB and TheTVDB I believe) and then moving files into sub-folders per-programme and -series, or per-movie, and also downloading artwork for XBMC and the like to show. I’m not going to go into much more detail on how I set this up, there are great instructions available on configuring it for your system. One recommendation I will make is to use something like trickle to limit the bandwidth available for moving when you’re using NFS, otherwise it can grind things to a halt if moving several large files. I ended up with the following in my crontab:

30 * * * * trickle -s -d 100 -u 500 perl /path/to/sorttv/sorttv.pl

So that’s it! I think I’ve finally reached the end of this mini-series on how my home AV setup works, more than 2 years after I first set out writing it. I’m sure I’ll be back on the subject at some point soon though, something is bound to break…