Wow thanks James. That plugin is the perfect tool for diagnosing memory usage.
And because of it I noticed a simple omission on my part. In the memory usage report I noticed
PHP Memory Limit: 90M
WordPress Memory Limit: 32M
Well that looks like it could be causing a problem. So I went to my wp-config file and added
define('WP_MEMORY_LIMIT', '90M');
I'll be monitoring this for a couple of days to see what my memory usage is. Hopefully that simple tweak does the trick.