The short answer is that you do not need to dispose of it, although it is a good exercise to place any IDisposable objects that you own.
In fact, trying to remove WebException.Response or the thread returned from it will cause the problems you mentioned, as you may encounter code trying to read its properties in the top call chain exception handler.
The reason it is safe not to dispose of it is because HttpWebRequest internally creates a memory stream from the network stream before it throws a WebException and the main network stream is already closed / deleted. Thus, at the moment he does not have an unmanaged resource. I assume this solution will facilitate exception handling.
Unfortunately, the MSDN documentation has no explanation for this. Technically, the implementation may change in the future, creating a problem for your code, not eliminating the HttpWebResponse and / or the associated stream that you receive from WebException , but it is unlikely that this behavior would change the implementation, given that many applications depend on the current behavior.
I must add that good practice is that you should dispose of IDisposable objects that you own. If possible, use the HttpClient class so you don't have to deal with this situation at all. If you cannot, consider handling WebException yourself and throw a new type of exception that will not WebException caller of your code so that you do not encounter a situation where the caller tries to access WebException.Response after removing it.
Disclaimer: I work at Microsoft, but this does not reflect the opinion of my employer or the .NET Framework. No warranty is implied.