You can only register JAX-RS components such as Providers or Features . You need to register a provider that carries ObjectMapper, which carries a module with your registrar registered.
ResteasyJacksonProvider resteasyJacksonProvider = new ResteasyJacksonProvider();
ObjectMapper mapper = new ObjectMapper();
SimpleModule myModule = new SimpleModule("myModule", new Version(1, 0, 0, null));
myModule.addSerializer(Custom.class, new CustomSerializer());
mapper.registerModule(myModule);
resteasyJacksonProvider.setMapper(mapper);
ResteasyClient client = new ResteasyClientBuilder().register(resteasyJacksonProvider).build();
Note that this example uses classes from org.codehaus.jackson. The API from com.fasterxml.jacksonlooks a bit different.