<C> not a return type. This is a declaration of a new generic type variable that can only be used by the map method.
Since the map method appears to have additional generic variables of type — V and K — that are not declared the same as C , we can assume that V and K are declared as generic variables of type at the class level (the class that contains this method). If they are absent, V and K will be treated as regular identifiers (i.e., the Compiler will expect classes to have the names V and K ).
Based on the signature of your method and its return type ( CustomMap ), I can assume that this method belongs to some class that implements the Map<K,V> interface. It takes a Function , which receives an instance of type V and returns an instance of type C , so it is reasonable to assume that it converts a Map<K,V> to Map<K,C> (that is, the keys remain unchanged and the values are converted).