Less is More in Listening