forkjoinpool parallel stream foreach

Custom thread pool in Java 8 parallel stream

stackoverflow.com › questions › 21163108 › custom-thread-pool-in-java-8-parallel-stream

There actually is a trick how to execute a parallel operation in a specific fork-join pool. If you execute it as a task in a fork-join pool, it stays there and does not use the common one.

final int parallelism = 4;
ForkJoinPool forkJoinPool = null;
try {
    forkJoinPool = new ForkJoinPool(parallelism);
    final List<Integer> primes = forkJoinPool.submit(() ->
        // Parallel task here, for example
        IntStream.range(1, 1_000_000).parallel()
                .filter(PrimesPrint::isPrime)
                .boxed().collect(Collectors.toList())
    ).get();
    System.out.println(primes);
} catch (InterruptedException | ExecutionException e) {
    throw new RuntimeException(e);
} finally {
    if (forkJoinPool != null) {
        forkJoinPool.shutdown();
    }
}

The trick is based on ForkJoinTask.fork which specifies: "Arranges to asynchronously execute this task in the pool the current task is running in, if applicable, or using the ForkJoinPool.commonPool() if not inForkJoinPool()"

Answer from Lukáš Křečan on Stack Overflow

Stack Overflow

stackoverflow.com › questions › 21163108 › custom-thread-pool-in-java-8-parallel-stream

concurrency - Custom thread pool in Java 8 parallel stream - Stack Overflow

Top answer

1 of 16

520

There actually is a trick how to execute a parallel operation in a specific fork-join pool. If you execute it as a task in a fork-join pool, it stays there and does not use the common one.

final int parallelism = 4;
ForkJoinPool forkJoinPool = null;
try {
    forkJoinPool = new ForkJoinPool(parallelism);
    final List<Integer> primes = forkJoinPool.submit(() ->
        // Parallel task here, for example
        IntStream.range(1, 1_000_000).parallel()
                .filter(PrimesPrint::isPrime)
                .boxed().collect(Collectors.toList())
    ).get();
    System.out.println(primes);
} catch (InterruptedException | ExecutionException e) {
    throw new RuntimeException(e);
} finally {
    if (forkJoinPool != null) {
        forkJoinPool.shutdown();
    }
}

2 of 16

251

The parallel streams use the default ForkJoinPool.commonPool which by default has one less threads as you have processors, as returned by Runtime.getRuntime().availableProcessors() (This means that parallel streams leave one processor for the calling thread).

For applications that require separate or custom pools, a ForkJoinPool may be constructed with a given target parallelism level; by default, equal to the number of available processors.

This also means if you have nested parallel streams or multiple parallel streams started concurrently, they will all share the same pool. Advantage: you will never use more than the default (number of available processors). Disadvantage: you may not get "all the processors" assigned to each parallel stream you initiate (if you happen to have more than one). (Apparently you can use a ManagedBlocker to circumvent that.)

To change the way parallel streams are executed, you can either

submit the parallel stream execution to your own ForkJoinPool: yourFJP.submit(() -> stream.parallel().forEach(soSomething)).get(); or
you can change the size of the common pool using system properties: System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "20") for a target parallelism of 20 threads.

Example of the latter on my machine which has 8 processors. If I run the following program:

long start = System.currentTimeMillis();
IntStream s = IntStream.range(0, 20);
//System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "20");
s.parallel().forEach(i -> {
    try { Thread.sleep(100); } catch (Exception ignore) {}
    System.out.print((System.currentTimeMillis() - start) + " ");
});

The output is:

215 216 216 216 216 216 216 216 315 316 316 316 316 316 316 316 415 416 416 416

So you can see that the parallel stream processes 8 items at a time, i.e. it uses 8 threads. However, if I uncomment the commented line, the output is:

215 215 215 215 215 216 216 216 216 216 216 216 216 216 216 216 216 216 216 216

This time, the parallel stream has used 20 threads and all 20 elements in the stream have been processed concurrently.

Lankydan

lankydan.dev › 2017 › 02 › 01 › common-fork-join-pool-and-streams

Common Fork Join Pool and Streams | Lanky Dan Blog

February 1, 2017 - Thread [main] [Daemon Thread ... into the forEach code it shows that when a parallel stream is ran it uses the main thread and the threads in the Common Fork Join Pool....

Discussions

ForkJoinPool and Nested (Parallel) Streams - Or why are inner streams faster with new pools ?

Please ensure that: Your code is properly formatted as code block - see the sidebar (About on mobile) for instructions You include any and all error messages in full You ask clear questions You demonstrate effort in solving your question/problem - plain posting your assignments is forbidden (and such posts will be removed) as is asking for or giving solutions. Trying to solve problems on your own is a very important skill. Also, see Learn to help yourself in the sidebar If any of the above points is not met, your post can and will be removed without further warning. Code is to be formatted as code block (old reddit: empty line before the code, each code line indented by 4 spaces, new reddit: https://i.imgur.com/EJ7tqek.png ) or linked via an external code hoster, like pastebin.com, github gist, github, bitbucket, gitlab, etc. Please, do not use triple backticks (```) as they will only render properly on new reddit, not on old reddit. Code blocks look like this: public class HelloWorld { public static void main(String[] args) { System.out.println("Hello World!"); } } You do not need to repost unless your post has been removed by a moderator. Just use the edit function of reddit to make sure your post complies with the above. If your post has remained in violation of these rules for a prolonged period of time (at least an hour), a moderator may remove it at their discretion. In this case, they will comment with an explanation on why it has been removed, and you will be required to resubmit the entire post following the proper procedures. To potential helpers Please, do not help if any of the above points are not met, rather report the post. We are trying to improve the quality of posts here. In helping people who can't be bothered to comply with the above points, you are doing the community a disservice. I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns. More on reddit.com

r/javahelp

December 3, 2024

java - Why does the parallel stream not use all the threads of the ForkJoinPool? - Stack Overflow

Originally this answer was an elaborate ... parallelism level, because there are always idle workers available to process the stream. That's incorrect. The actual answer is provided in the original question to which this was marked as duplicate - using a custom ForkJoinPool for stream processing is not officially supported, and when using forEach, the default ... More on stackoverflow.com

stackoverflow.com

java - StreamEx.parallel().forEach() does not run in parallel after .map() - Stack Overflow

I noticed that if I use StreamEx lib to parallel out my streams with a custom ForkJoinPool as below - the subsequent actions do run in parallel threads from that pool. However, if I add a map() ope... More on stackoverflow.com

stackoverflow.com

concurrency - Why does parallelStream use a ForkJoinPool, not a normal thread pool? - Stack Overflow

Here's where the ForkJoinPool shines with its multiple queues. Every worker just takes tasks from its own queue, which doesn't need to be synchronized by blocking most of the time, and if it's empty, it can steal a task from another worker, but from the other end of the queue, which also leads rarely to synchronization overhead as work-stealing is supposed to be rather rare. Now what does that have to do with parallel streams... More on stackoverflow.com

stackoverflow.com

Videos