With client REQUEST_ENTITY_PROCESSING set to CHUNKED I am losing documents

Question

I have a REST web service running on Jetty. I want to write a Java client that splits a huge batch of documents into this leisure service using the same web connection.

I managed to establish an Iterator-based streaming approach:

Send document flow to Jersey @POST endpoint

This does not work unless you set clientConfig.property(ClientProperties.REQUEST_ENTITY_PROCESSING, RequestEntityProcessing.CHUNKED); because Content-length unknown.

Although it works somewhat, the transfer seems to have lost several documents. For example:

 num_docs 500000 numFound 499249 

Perhaps it sends pieces like:

{some:doc}, {some:doc}, {some:doc}, {some:doc}, {some:doc}, {some:doc}, {some:do

So, do I lose every time at the ends? UPDATE: I was wrong.

How do I do this wrong? Any ideas what else could happen?

  ClientConfig clientConfig = new ClientConfig(); clientConfig.property(ClientProperties.CONNECT_TIMEOUT, (int)TimeUnit.SECONDS.toMillis(60)); clientConfig.property(ClientProperties.REQUEST_ENTITY_PROCESSING, RequestEntityProcessing.CHUNKED); clientConfig.property(ClientProperties.ASYNC_THREADPOOL_SIZE, 100); clientConfig.property(ApacheClientProperties.CONNECTION_MANAGER, HttpClientFactory.createConnectionManager(name, metricRegistry, configuration)); ApacheConnectorProvider connector = new ApacheConnectorProvider(); clientConfig.connectorProvider(connector); clientConfig.register(new ClientRequestFilter() { @Override public void filter(ClientRequestContext requestContext) throws IOException { List<Object> orig = requestContext.getHeaders().remove(HttpHeaders.CONTENT_LENGTH); if (orig != null && !orig.isEmpty()) { requestContext.getHeaders().addAll("Length", orig); } } }); clientConfig.register(new ClientRequestFilter() { @Override public void filter(ClientRequestContext requestContext) throws IOException { if (requestContext.getMediaType() != null && requestContext.getMediaType().getType() != null && requestContext.getMediaType().getType().equalsIgnoreCase("multipart")) { final MediaType boundaryMediaType = Boundary.addBoundary(requestContext.getMediaType()); if (boundaryMediaType != requestContext.getMediaType()) { requestContext.getHeaders().putSingle(HttpHeaders.CONTENT_TYPE, boundaryMediaType.toString()); } if (!requestContext.getHeaders().containsKey("MIME-Version")) { requestContext.getHeaders().putSingle("MIME-Version", "1.0"); } } } }); 

Answer 1

closing this - I accidentally closed the stream earlier, so in the end I didn’t have enough documents at the end, which gave me a hint to wait until the lock queue was empty before closing the artists.