If you offer Guava, then new MapMaker().weakKeys().makeMap() will do the job directly, since weakKeys uses referential equality for keys.
The weakKeys documentation says:
Indicates that each key (not value) stored on the map should be wrapped in WeakReference (strong links are used by default). Warning: when this method is used, the resulting map will use an identity comparison (==) to determine key equality, which is a technical violation of the map specification and may not be what you expect.
Louis Wasserman
source share