- Connect your iPhone/iPod to your Mac.
- This step is probably unnecessary but I'm paranoid. Back up the photos from your iPhone with the Image Capture utility. Not a perfect tool but it allows you to pull photos from your iPhone to a plain old directory and organize them your self which I like better than using iPhoto.
- Navigate to the "Photos" tab in iTunes.
- Check "Sync photos from" and expand the drop down box next to that text. Apparently the photo cache gets re-created in the folder chosen here every time you connect your phone even after you disable photo syncing.
- Select some other out of the way folder, even an empty one. I created an empty folder "~/Pictures/iPhone Sync".
- Apply the changes and Sync the empty folder once and then deactivate Photo syncing.
This cured the issue for me.