I've recently been having this issue myself. Same deal as you. Small wordPress blog, using a cacheing plugin, WordPress 3, separate user etc. It gotten so bad that I decided to put Google Analytics on my Internal Server Error page to track how frequently this is happening. The results are scary. I don't know what to do to reduce the amount of memory WordPress uses.
Trending Articles
More Pages to Explore .....