In the provocatively titled, Space is Time: How Your CS Theory Class Lied to You, Greg Parker explains why hogging memory to speed up your program is a bad idea on consumer-level devices.
Space is time. An optimization that makes your program faster may make the user’s system slower overall. Play well with others.
I think it’s excellent advice. There’s only one point I want to make (and it must be made now, before silent SSD hard disks become more popular).
Every time your whole computer is nonresponsive, it makes that grrnrrnrnrr sound, right? That’s because it ran out of RAM, and the virtual memory system is thrashing to the disk. In the real word, on computers like the one you are on now, low memory plays a part in every system-wide slowdown.
Hogging the CPU can make your application slow. (Although if you pay attention to threading, and the run loop, using 100% of every core shouldn’t kill responsiveness and usability.) But hogging memory makes the whole damn world slow, including your application. And the worst case scenario of bad memory management kills the system deader then a kernel panic.
This isn’t to say that caching never makes sense. But a healthy respect for memory usage generally trumps worrying about the CPU.