If you detach the EF object from the DbContext before it is serialized, this disables lazy loading, so your custom serializer will not try to serialize anything that is not yet part of the entity graph.
Then, when you return from the cache, if you attach it to a new (identical) DbContext, which should reuse lazy loading.
(Caution: after disconnecting an object from the context, any new requests containing the same object will create a new, attached copy, so you will need to be careful to encode different versions of the same object to avoid problems with several potentially dangerous objects, but it means that you should do what you want.
source share