EverCrest Message Forums
You are not logged in. Login or Register.
Author
Topic: When is it "too much data"?
Random Insanity Generator
Condom Ninja El Supremo
posted 11-06-2010 09:01:26 PM
Been migrating my collection of crap from a server with ~4tb of storage (and almost out) that was random disks to a system that has a proper raid5 setup and management tools to handle backup/recovery operations. Initial salvo of data migrated over took ~30 hours or so (2.2tb) due to the fact that it's reading from single disks on the old server. I'm close to finishing up the rest of the data migration and it kinda hit me... I think I might have too much crap. New array is 5.4tb of usable disk, expandable as long as I can keep buying 2tb disks to add to the array (and don't mind waiting to perform rebuild), but damn... why do I, as a single person, have this much crap floating around?

I could probably kill around 100gb in random dupes of crap (move data around to organize but forget to delete the original), but that's not gonna make a dent. Anyone else with this kind of stupid shit? If so how are you managing/controlling the spread?

* NullDevice kicks the server. "Floggings will continue until processing power improves!"
-----------------------------------
"That was black magic, and it was easy to use. Easy and fun. Like Legos." -- Harry Dresden
-----------------------------------
That's what playing Ragnarok Online taught me: There's no problem in the universe that can't be resolved by the proper application of daggers to faces.
Damnati
Filthy
posted 11-07-2010 12:05:43 AM
I personally don't have that much data but I've got a friend who has something like 2tb of video alone, to say nothing of his collection of pictures, music, PDFs, etc. He big on Windows Home Server and has managed to get MVP awards for his work on it for several years now.
Love is hard, harder than steel and thrice as cruel. It is as inexorable as the tides and life and death alike follow in its wake. -Phèdre nó Delaunay, Kushiel's Chosen

It is by caffeine alone I set my mind in motion. It is by the beans of Java the thoughts aquire speed, the teeth acquire stains, the stains become a warning. It is by caffeine alone I set my mind in motion.

Random Insanity Generator
Condom Ninja El Supremo
posted 11-07-2010 12:40:52 AM
Problem is, I don't want to deal with Windows Home Server, I'd like to have a real server.

Home Server is ok if you're in a primarily Windows environment in a "normal home" environment. If I were going to go with a MS solution I'd be looking at 2k3 server or 2k8 server. However since there's only one Windows box on this network (my dad is a Mac user, everything else I have BUT my WoW box is Linux) it seems kinda useless in that respect. Moving to FreeNAS (FreeBSD based system) I can do "native" AFP file serving for my dad's systems (as well as setup a TimeMachine share, etc) and still cover SMB/CIFS for my windows box and NFS or FTP or RSync for my linux systems.

3.2tb of Video data, then there's the MP3's, pr0n, game patches, backup images, and development data in my home directory, my dad's pr0n stash and his photoshop work in his home directory, not to mention the random "off site data storage" I do for a few places.... And these data sets will most likely grow over time. At this volume there's no 'reasonable' way to perform backups if I needed to other than to determine "most important" and just do snapshots. Last tape drive I had was a DLT-4 unit, which is a 40g native, 80g best compression and $30 a tape, IF I still had the drive. Going with the assumption that vid data compresses like ass, that's 40gb per tape so ~80 tapes to get the video data set backed up... time to swap the tapes (or buy a robotic stacker... owww) and about 2400 for the media itself every 6 to 9 months, assuming the data set doesn't increase.

How is your friend handling backup/data security with his collection? What are his "worst case scenario" recovery options?

* NullDevice kicks the server. "Floggings will continue until processing power improves!"
-----------------------------------
"That was black magic, and it was easy to use. Easy and fun. Like Legos." -- Harry Dresden
-----------------------------------
That's what playing Ragnarok Online taught me: There's no problem in the universe that can't be resolved by the proper application of daggers to faces.
Damnati
Filthy
posted 11-07-2010 01:32:19 AM
quote:
Random Insanity Generator thought this was the Ricky Martin Fan Club Forum and wrote:
Problem is, I don't want to deal with Windows Home Server, I'd like to have a real server.

Home Server is ok if you're in a primarily Windows environment in a "normal home" environment. If I were going to go with a MS solution I'd be looking at 2k3 server or 2k8 server. However since there's only one Windows box on this network (my dad is a Mac user, everything else I have BUT my WoW box is Linux) it seems kinda useless in that respect. Moving to FreeNAS (FreeBSD based system) I can do "native" AFP file serving for my dad's systems (as well as setup a TimeMachine share, etc) and still cover SMB/CIFS for my windows box and NFS or FTP or RSync for my linux systems.

3.2tb of Video data, then there's the MP3's, pr0n, game patches, backup images, and development data in my home directory, my dad's pr0n stash and his photoshop work in his home directory, not to mention the random "off site data storage" I do for a few places.... And these data sets will most likely grow over time. At this volume there's no 'reasonable' way to perform backups if I needed to other than to determine "most important" and just do snapshots. Last tape drive I had was a DLT-4 unit, which is a 40g native, 80g best compression and $30 a tape, IF I still had the drive. Going with the assumption that vid data compresses like ass, that's 40gb per tape so ~80 tapes to get the video data set backed up... time to swap the tapes (or buy a robotic stacker... owww) and about 2400 for the media itself every 6 to 9 months, assuming the data set doesn't increase.

How is your friend handling backup/data security with his collection? What are his "worst case scenario" recovery options?


WHS has a native file duplication feature for backup. I'm know little about its configuration options but I gather it's pretty good, if space-consuming.

Love is hard, harder than steel and thrice as cruel. It is as inexorable as the tides and life and death alike follow in its wake. -Phèdre nó Delaunay, Kushiel's Chosen

It is by caffeine alone I set my mind in motion. It is by the beans of Java the thoughts aquire speed, the teeth acquire stains, the stains become a warning. It is by caffeine alone I set my mind in motion.

Random Insanity Generator
Condom Ninja El Supremo
posted 11-07-2010 01:36:39 AM
Nice if you have an external HD to backup to and your data set will fit on one.
* NullDevice kicks the server. "Floggings will continue until processing power improves!"
-----------------------------------
"That was black magic, and it was easy to use. Easy and fun. Like Legos." -- Harry Dresden
-----------------------------------
That's what playing Ragnarok Online taught me: There's no problem in the universe that can't be resolved by the proper application of daggers to faces.
Mortious
Gluttonous Overlard
posted 11-07-2010 04:48:44 AM
I don't even fill up a 30GB backup drive with all my crap.

What the hell are you people hoarding? A few thousand full resolution blu-ray rips?

Vorago
A completely different kind of Buckethead
posted 11-07-2010 11:45:49 AM
I can fit all my backup onto a DVD, mostly just pictures of the wife and baby. Everything else I just periodically reformat then just re-DL what I need, helps keep the junk to a minimum
Greenlit
posted 11-07-2010 11:53:03 AM
Vorago has a wife and a baby?
Random Insanity Generator
Condom Ninja El Supremo
posted 11-07-2010 12:40:06 PM
quote:
Mortious was naked while typing this:
I don't even fill up a 30GB backup drive with all my crap.

What the hell are you people hoarding? A few thousand full resolution blu-ray rips?


Hell no.. a few thousand of them (given they're between 8 and 12gb each on average) would be crazy.

However television series that are between 6 and 10gb per season... bit more "reasonable".

* NullDevice kicks the server. "Floggings will continue until processing power improves!"
-----------------------------------
"That was black magic, and it was easy to use. Easy and fun. Like Legos." -- Harry Dresden
-----------------------------------
That's what playing Ragnarok Online taught me: There's no problem in the universe that can't be resolved by the proper application of daggers to faces.
Vallo, the Second Coming
Pancake
posted 11-07-2010 01:57:53 PM
quote:
Mortious had this to say about Optimus Prime:
I don't even fill up a 30GB backup drive with all my crap.

What the hell are you people hoarding? A few thousand full resolution blu-ray rips?


More or less, yeah

"We are just an advanced breed of monkeys on a minor planet of a very average star. But we can understand the Universe. That makes us something very special."
All times are US/Eastern
Hop To: