Split LongStream into substreams with maximum length

Question

There are several SQL statements in my program that contain IN-classes with the given identifiers. The problem is that in some cases there may be more than 1000 identifiers, which causes Oracle to crash with ORA-01795. Too many elements.

So, I want to split this list into several subscriptions.

Example: I have 2403 identifiers

The result will be three lists:

• 0 - 999
• 1000 - 1999
• 2000 - 2402

I wrote a piece of code that works, but looks awful. Is there a better solution to this problem? Maybe something with Collectors and a group or something like that?

My code is:

    Map<Integer, List<Long>> result = new HashMap<>();
    ArrayList<Long> asList = new ArrayList<Long>(listOfIds);
    IntStream.range(0, (listOfIds.size() / 1000) + 1)
             .forEach(partGroup -> result.put(partGroup, asList.subList(partGroup * 1000, (partGroup * 1000) + Math.min(1000, 
             asList.size() - partGroup * 1000))));

Answer 1

, , . , , :

public static <T> Stream<List<T>> splitListStream(List<T> input, int batchSize) {
  if (batchSize <= 0)
    throw new IllegalArgumentException("batchSize must be positive (" + batchSize + ")");
  if (input.size() <= batchSize) return Stream.of(input);
  return IntStream.range(0, (input.size() + batchSize - 1) / batchSize)
          .mapToObj(i -> {
            int from = i * batchSize;
            int to = Math.min((i + 1) * batchSize, input.size());
            return input.subList(from, to);
          });
}

Answer 2

- . .

 private static <T> Collector<T, ?, List<List<T>>> partitioning(int size) {
    class Acc {
        int count = 0;

        List<List<T>> list = new ArrayList<>();

        void add(T elem) {
            int index = count++ / size;
            if (index == list.size()) {
                list.add(new ArrayList<>());
            }
            list.get(index).add(elem);
        }

        Acc merge(Acc right) {

            List<T> lastLeftList = list.get(list.size() - 1);
            List<T> firstRightList = right.list.get(0);
            int lastLeftSize = lastLeftList.size();
            int firstRightSize = firstRightList.size();

            // they are both size, simply addAll will work
            if (lastLeftSize + firstRightSize == 2 * size) {
                list.addAll(right.list);
                return this;
            }

            // last and first from each chunk are merged "perfectly"
            if (lastLeftSize + firstRightSize == size) {
                int x = 0;
                while (x < firstRightSize) {
                    lastLeftList.add(firstRightList.remove(x));
                    --firstRightSize;
                }
                right.list.remove(0);
                list.addAll(right.list);
                return this;
            }

            right.list.stream().flatMap(List::stream).forEach(this::add);
            return this;
        }

        public List<List<T>> finisher() {
            return list;
        }

    }
    return Collector.of(Acc::new, Acc::add, Acc::merge, Acc::finisher);
}

:

List<List<Integer>> list = Arrays.asList(1, 3, 4, 5, 9, 8, 7)
            .stream()
            .parallel()
            .collect(partitioning(3));

Answer 3

As an alternative to your own camera, you can consider jOOL or Guava ( Iterators.partition(stream.iterator(), batchSize)).