My take on the difference is not that a prototype necessarily needs to be more high fidelity. The difference to me is that a prototype is always built to test (or verify) something. There are several dimensions to this(coverage in terms of breadth and width, data richness and authenticity, interaction depth, and visual fidelity). A prototype can therefore be very crude and basic (“low fidelity”). A wireframe doesn’t need to be testable per se, and it’s purpose is often to communicate something (often layout and interaction flow) to clients, stakeholders, or even users (but then it often turns into some kind of prototype :)
See Buxton’s book Sketching User Experiences for a good discussion on the role of the prototype.