How to control the number of concurrent Spring batch jobs

Question

I have a reporting application. Since preparing such reports is heavy, they are prepared asynchronously with a couple of Spring. Requests for such reports are created through the REST interface using HTTP.

The goal is that the REST resource simply queues the report and completes ( as described in the documentation ). Thus, a TaskExecutor was provided for JobLauncher:

<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher"> <property name="jobRepository" ref="jobRepository" /> <property name="taskExecutor"> <bean class="org.springframework.core.task.SimpleAsyncTaskExecutor"/> </property> </bean> 

Since the reports are really heavyweights, only a certain number of them can be produced at a given time. Hoping that you can configure Spring Batch to create only 2 instances at a time, the concurrencyLimit parameter is specified:

  <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher"> <property name="jobRepository" ref="jobRepository" /> <property name="taskExecutor"> <bean class="org.springframework.core.task.SimpleAsyncTaskExecutor"> <property name="concurrencyLimit" value="2" /> </bean> </property> </bean> 

Unfortunately, when 2 jobs are already running, the start task call is blocked: jobLauncher.run (job, builder.toJobParameters ());

Obviously, jobLauncher immediately tries to complete the job. I would suggest that this is more likely to be done as soon as the thread is available. That way, I could scale my application by simply adding additional processing instances, all of which use the same repository database.

A similar question was here . I'm going to start exploring Spring batch integration , but I'm not sure if this is the right direction.

It seems to me that I do not like that for me there is no widely discussed template that I apparently cannot find?

Thanks e

Answer 1

SimpleAsyncTaskExecutor not recommended for heavy use, as it creates a new thread with each task. It also does not support more robust concepts such as thread pooling and task ordering.

If you look at ThreadPoolTaskExecutor , it supports a more robust task execution paradigm with things like the task queue and the use of a thread pool instead of spawning random, unused threads.

Read more about ThreadPoolTaskExecutor in javadoc here: http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/scheduling/concurrent/ThreadPoolTaskExecutor.html

Answer 2

It helped, thank you very much. After replacing SimpleAsyncTaskExecutor, I have exactly what I need. The code:

 @Bean public TaskExecutor jobLauncherTaskExecutor() { ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor(); executor.setMaxPoolSize(executorsPoolSize); executor.setCorePoolSize(executorsPoolSize); return executor; } 

Thanks e