He was averse to physical contact, at least to those not close to him.
He was an extremely private person.
This from Wiki: "Gould was averse to cold and wore heavy clothing (including gloves), even in warm places. He was once arrested, presumably mistaken for a vagrant, while sitting on a park bench in Sarasota, Florida, dressed in his standard all-climate attire of coat(s), hat and mittens.[50] He also disliked social functions. He hated being touched, and in later life he limited personal contact, relying on the telephone and letters for communication. On one visit to Steinway Hall in New York City in 1959, the chief piano technician at the time, William Hupfer, greeted Gould by giving him a slap on the back. Gould was shocked by this, and complained of aching, lack of coordination, and fatigue because of the incident. He went on to explore the possibility of litigation against Steinway & Sons if his apparent injuries were permanent."
He insisted on playing using the same four legged chair his father had given him and would travel with that chair.
Sometimes studio recording technicians were maddened by his vocalizations during play since they couldn't strip the sounds out from the piano's music.