Why does annotation_raster not work?

Question

There is one example in the documentation showing usage annotation_rasteras shown below.

ggplot(aes(x=mpg, y=wt), data=mtcars) +
  annotation_raster('red', -Inf, Inf, -Inf, Inf) +
  geom_point()

This works great, however, when I move the data and ah into the layer, it suddenly does not work:

ggplot() +
  annotation_raster('red', -Inf, Inf, -Inf, Inf) +
  geom_point(aes(x=mpg, y=wt), data=mtcars)  # doesn't work

This is bewildering, because for me these two seem to be semantically identical.

Is there a reason why the second line will not work, and is there a way to use annotation_rasterwithout specifying data and aes at an early stage of the base layer?