If you ever need to remove a troublesome web part that is preventing you getting to the page, try adding "?contents=1" to the querystring. That should take you to the Web Parts Maintenance Page where you can manually remove the web part.

Thanks to my colleague Harry for this one.