If my query in the DynamoDB table returns an empty result set, is it calculated in reserve capacity? If this is not the case, does this mean that I can query as often as I want, without throttling, while there is no record?
You will still use read bandwidth if there is an empty result set.
Relevant section from the File Size Size Documentation :
If you are performing a read operation on an element that does not exist, DynamoDB will still consume the provided read throughput: a highly consistent read request consumes one reader module, while a consistent read request consumes 0.5 units of read capacity.