If you have an appointment scheduled, simply call and say you would like to cancel it. They'll probably ask if you would like to reschedule. Say, "No, not at this time, thank you." Then never call them again, unless you need your records transfered to a new psychiatrist.
BTW, the way in which you end the relationship should in no way affect your ability to release your records or transfer them to a different office, or what they'll say. For instance, they will not write, "Bad patient! Never called to reschedule!!" Doctors of all kinds routinely have patients who stop making appointments - they will neither be shocked nor offended.