negative regulation of protein localization to endoplasmic reticulum
Any process that stops, prevents or reduces the frequency, rate or extent of protein localization to endoplasmic reticulum.