http://dsanalytics.com/dsblog/mining-terabytes-on-the-desktop_103 Data Analytics- the art and science of analyzing data Sun, 06 Jul 2008 04:21:47 +0000 http://wordpress.org/?v=2.0.1 http://dsanalytics.com/dsblog/mining-terabytes-on-the-desktop_103#comment-70 Wed, 08 Aug 2007 08:16:24 +0000 http://dsanalytics.com/dsblog/mining-terabytes-on-the-desktop_103#comment-70 thanks for the comment Joe. I did at various stages through the Netflix analysis end up keeping everything in core, but at the end of the day it seems to work quite well if you keep it on disk and load the needed bits as required, making sure that the OS caches it for you. Disk caching == Virtual Memory, more or less. And yes, I do have 2gb of memory but never need to use more than about 1gb. thanks for the comment Joe. I did at various stages through the Netflix analysis end up keeping everything in core, but at the end of the day it seems to work quite well if you keep it on disk and load the needed bits as required, making sure that the OS caches it for you. Disk caching == Virtual Memory, more or less. And yes, I do have 2gb of memory but never need to use more than about 1gb.

]]> http://dsanalytics.com/dsblog/mining-terabytes-on-the-desktop_103#comment-69 Mon, 06 Aug 2007 22:35:05 +0000 http://dsanalytics.com/dsblog/mining-terabytes-on-the-desktop_103#comment-69 If you browse around the Netflix site you will find people putting the data into structures measured in a few hundred megabytes so it is possible (for some algorithms) to fit all of the data into the RAM on a modern desktop. Simon Funk says you should have 2 gigabytes but there are obviously people doing it in less. If you browse around the Netflix site you will find people putting the data into structures measured in a few hundred megabytes so it is possible (for some algorithms) to fit all of the data into the RAM on a modern desktop. Simon Funk says you should have 2 gigabytes but there are obviously people doing it in less.

]]> http://dsanalytics.com/dsblog/mining-terabytes-on-the-desktop_103#comment-62 Sat, 21 Jul 2007 01:51:13 +0000 http://dsanalytics.com/dsblog/mining-terabytes-on-the-desktop_103#comment-62 [...] And lots of other uses. But beware .. you will need your system set up for mining terabytes on the desktop .. the corpus comes on 6 DVD’s [...] […] And lots of other uses. But beware .. you will need your system set up for mining terabytes on the desktop .. the corpus comes on 6 DVD’s […]

]]> http://dsanalytics.com/dsblog/mining-terabytes-on-the-desktop_103#comment-38 Mon, 28 May 2007 09:13:28 +0000 http://dsanalytics.com/dsblog/mining-terabytes-on-the-desktop_103#comment-38 [...] I have previously written about mining terabytes of data with a desktop machine.. some of the strategies that work for me. [...] […] I have previously written about mining terabytes of data with a desktop machine.. some of the strategies that work for me. […]

]]>