I understand that the abstracting header is trivial to spoof using standard http. But when using https, can you trust the referent or is it potentially faked?
No. Using HTTPS does not change anything; the referent can be trivially tampered with; eg:
wget --referer=http://whitehouse.gov/ https://example.com/