tokio-rs/tokio
1
0
Fork 0
mirror of https://github.com/tokio-rs/tokio.git synced 2026-09-01 00:00:10 +02:00
Code Issues Packages Projects Releases Wiki Activity

executor: rewrite the work-stealing thread pool (#1657)

Browse Source 
This patch is a ground up rewrite of the existing work-stealing thread
pool. The goal is to reduce overhead while simplifying code when
possible.

At a high level, the following architectural changes were made:

- The local run queues were switched for bounded circle buffer queues.
- Reduce cross-thread synchronization.
- Refactor task constructs to use a single allocation and always include
  a join handle (#887).
- Simplify logic around putting workers to sleep and waking them up.

**Local run queues**

Move away from crossbeam's implementation of the Chase-Lev deque. This
implementation included unnecessary overhead as it supported
capabilities that are not needed for the work-stealing thread pool.
Instead, a fixed size circle buffer is used for the local queue. When
the local queue is full, half of the tasks contained in it are moved to
the global run queue.

**Reduce cross-thread synchronization**

This is done via many small improvements. Primarily, an upper bound is
placed on the number of concurrent stealers. Limiting the number of
stealers results in lower contention. Secondly, the rate at which
workers are notified and woken up is throttled. This also reduces
contention by preventing many threads from racing to steal work.

**Refactor task structure**

Now that Tokio is able to target a rust version that supports
`std::alloc` as well as `std::task`, the pool is able to optimize how
the task structure is laid out. Now, a single allocation per task is
required and a join handle is always provided enabling the spawner to
retrieve the result of the task (#887).

**Simplifying logic**

When possible, complexity is reduced in the implementation. This is done
by using locks and other simpler constructs in cold paths. The set of
sleeping workers is now represented as a `Mutex<VecDeque<usize>>`.
Instead of optimizing access to this structure, we reduce the amount the
pool must access this structure.

Secondly, we have (temporarily) removed `threadpool::blocking`. This
capability will come back later, but the original implementation was way
more complicated than necessary.

**Results**

The thread pool benchmarks have improved significantly:

Old thread pool:

```
test chained_spawn ... bench:   2,019,796 ns/iter (+/- 302,168)
test ping_pong     ... bench:   1,279,948 ns/iter (+/- 154,365)
test spawn_many    ... bench:  10,283,608 ns/iter (+/- 1,284,275)
test yield_many    ... bench:  21,450,748 ns/iter (+/- 1,201,337)
```

New thread pool:

```
test chained_spawn ... bench:     147,943 ns/iter (+/- 6,673)
test ping_pong     ... bench:     537,744 ns/iter (+/- 20,928)
test spawn_many    ... bench:   7,454,898 ns/iter (+/- 283,449)
test yield_many    ... bench:  16,771,113 ns/iter (+/- 733,424)
```

Real-world benchmarks improve significantly as well. This is testing the hyper hello
world server using: `wrk -t1 -c50 -d10`:

Old scheduler:

```
Running 10s test @ http://127.0.0.1:3000
  1 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   371.53us   99.05us   1.97ms   60.53%
    Req/Sec   114.61k     8.45k  133.85k    67.00%
  1139307 requests in 10.00s, 95.61MB read
Requests/sec: 113923.19
Transfer/sec:      9.56MB
```

New scheduler:

```
Running 10s test @ http://127.0.0.1:3000
  1 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   275.05us   69.81us   1.09ms   73.57%
    Req/Sec   153.17k    10.68k  171.51k    71.00%
  1522671 requests in 10.00s, 127.79MB read
Requests/sec: 152258.70
Transfer/sec:     12.78MB
```
This commit is contained in:
Carl Lerche
2019-10-19 11:09:40 -07:00
committed by GitHub
parent 2a181320b7
commit ed5a94eb2d
100 changed files with 7409 additions and 6796 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tokio-executor/benches/blocking.rs
-133
View File
@@ -1,133 +0,0 @@
#![cfg(feature = "broken")]
#![feature(test)]
#![warn(rust_2018_idioms)]
extern crate test;
const ITER: usize = 1_000;
mod blocking {
use super::*;
use futures::future::*;
use tokio_executor::threadpool::{blocking, Builder};
#[bench]
fn cpu_bound(b: &mut test::Bencher) {
let pool = Builder::new().pool_size(2).max_blocking(20).build();
b.iter(|| {
let count_down = Arc::new(CountDown::new(ITER));
for _ in 0..ITER {
let count_down = count_down.clone();
pool.spawn(lazy(move || {
poll_fn(|| blocking(|| perform_complex_computation()).map_err(|_| panic!()))
.and_then(move |_| {
// Do something with the value
count_down.dec();
Ok(())
})
}));
}
count_down.wait();
})
}
}
mod message_passing {
use super::*;
use futures::future::*;
use futures::sync::oneshot;
use tokio_executor::threadpool::Builder;
#[bench]
fn cpu_bound(b: &mut test::Bencher) {
let pool = Builder::new().pool_size(2).max_blocking(20).build();
let blocking = threadpool::ThreadPool::new(20);
b.iter(|| {
let count_down = Arc::new(CountDown::new(ITER));
for _ in 0..ITER {
let count_down = count_down.clone();
let blocking = blocking.clone();
pool.spawn(lazy(move || {
// Create a channel to receive the return value.
let (tx, rx) = oneshot::channel();
// Spawn a task on the blocking thread pool to process the
// computation.
blocking.execute(move || {
let res = perform_complex_computation();
tx.send(res).unwrap();
});
rx.and_then(move |_| {
count_down.dec();
Ok(())
})
.map_err(|_| panic!())
}));
}
count_down.wait();
})
}
}
fn perform_complex_computation() -> usize {
use rand::*;
// Simulate a CPU heavy computation
let mut rng = rand::thread_rng();
rng.gen()
}
// Util for waiting until the tasks complete
use std::sync::atomic::AtomicUsize;
use std::sync::atomic::Ordering::*;
use std::sync::*;
struct CountDown {
rem: AtomicUsize,
mutex: Mutex<()>,
condvar: Condvar,
}
impl CountDown {
fn new(rem: usize) -> Self {
CountDown {
rem: AtomicUsize::new(rem),
mutex: Mutex::new(()),
condvar: Condvar::new(),
}
}
fn dec(&self) {
let prev = self.rem.fetch_sub(1, AcqRel);
if prev != 1 {
return;
}
let _lock = self.mutex.lock().unwrap();
self.condvar.notify_all();
}
fn wait(&self) {
let mut lock = self.mutex.lock().unwrap();
loop {
if self.rem.load(Acquire) == 0 {
return;
}
lock = self.condvar.wait(lock).unwrap();
}
}
}
tokio-executor/benches/thread_pool.rs
+161
View File
@@ -0,0 +1,161 @@
#![feature(test)]
extern crate test;
use tokio_executor::thread_pool::{Builder, Spawner, ThreadPool};
use tokio_sync::oneshot;
use std::future::Future;
use std::pin::Pin;
use std::sync::atomic::AtomicUsize;
use std::sync::atomic::Ordering::Relaxed;
use std::sync::{mpsc, Arc};
use std::task::{Context, Poll};
struct Backoff(usize);
impl Future for Backoff {
type Output = ();
fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<()> {
if self.0 == 0 {
Poll::Ready(())
} else {
self.0 -= 1;
cx.waker().wake_by_ref();
Poll::Pending
}
}
}
const NUM_THREADS: usize = 6;
#[bench]
fn spawn_many(b: &mut test::Bencher) {
const NUM_SPAWN: usize = 10_000;
let threadpool = Builder::new().num_threads(NUM_THREADS).build();
let (tx, rx) = mpsc::sync_channel(1000);
let rem = Arc::new(AtomicUsize::new(0));
b.iter(|| {
rem.store(NUM_SPAWN, Relaxed);
for _ in 0..NUM_SPAWN {
let tx = tx.clone();
let rem = rem.clone();
threadpool.spawn(async move {
if 1 == rem.fetch_sub(1, Relaxed) {
tx.send(()).unwrap();
}
});
}
let _ = rx.recv().unwrap();
});
}
#[bench]
fn yield_many(b: &mut test::Bencher) {
const NUM_YIELD: usize = 1_000;
const TASKS_PER_CPU: usize = 50;
let threadpool = Builder::new().num_threads(NUM_THREADS).build();
let tasks = TASKS_PER_CPU * num_cpus::get_physical();
let (tx, rx) = mpsc::sync_channel(tasks);
b.iter(move || {
for _ in 0..tasks {
let tx = tx.clone();
threadpool.spawn(async move {
let backoff = Backoff(NUM_YIELD);
backoff.await;
tx.send(()).unwrap();
});
}
for _ in 0..tasks {
let _ = rx.recv().unwrap();
}
});
}
#[bench]
fn ping_pong(b: &mut test::Bencher) {
const NUM_PINGS: usize = 1_000;
let threadpool = Builder::new().num_threads(NUM_THREADS).build();
let (done_tx, done_rx) = mpsc::sync_channel(1000);
let rem = Arc::new(AtomicUsize::new(0));
b.iter(|| {
let done_tx = done_tx.clone();
let rem = rem.clone();
rem.store(NUM_PINGS, Relaxed);
let spawner = threadpool.spawner().clone();
threadpool.spawn(async move {
for _ in 0..NUM_PINGS {
let rem = rem.clone();
let done_tx = done_tx.clone();
let spawner2 = spawner.clone();
spawner.spawn(async move {
let (tx1, rx1) = oneshot::channel();
let (tx2, rx2) = oneshot::channel();
spawner2.spawn(async move {
rx1.await.unwrap();
tx2.send(()).unwrap();
});
tx1.send(()).unwrap();
rx2.await.unwrap();
if 1 == rem.fetch_sub(1, Relaxed) {
done_tx.send(()).unwrap();
}
});
}
});
done_rx.recv().unwrap();
});
}
#[bench]
fn chained_spawn(b: &mut test::Bencher) {
const ITER: usize = 1_000;
let threadpool = Builder::new().num_threads(NUM_THREADS).build();
fn iter(spawner: Spawner, done_tx: mpsc::SyncSender<()>, n: usize) {
if n == 0 {
done_tx.send(()).unwrap();
} else {
let s2 = spawner.clone();
spawner.spawn(async move {
iter(s2, done_tx, n - 1);
});
}
}
let (done_tx, done_rx) = mpsc::sync_channel(1000);
b.iter(move || {
let done_tx = done_tx.clone();
let spawner = threadpool.spawner().clone();
threadpool.spawn(async move {
iter(spawner, done_tx, ITER);
});
done_rx.recv().unwrap();
});
}
tokio-executor/benches/threadpool.rs
-161
View File
@@ -1,161 +0,0 @@
#![cfg(feature = "broken")]
#![feature(test)]
#![warn(rust_2018_idioms)]
extern crate test;
const NUM_SPAWN: usize = 10_000;
const NUM_YIELD: usize = 1_000;
const TASKS_PER_CPU: usize = 50;
mod threadpool {
use futures::{future, task, Async};
use num_cpus;
use std::sync::atomic::AtomicUsize;
use std::sync::atomic::Ordering::SeqCst;
use std::sync::{mpsc, Arc};
use tokio_executor::threadpool::*;
#[bench]
fn spawn_many(b: &mut test::Bencher) {
let threadpool = ThreadPool::new();
let (tx, rx) = mpsc::sync_channel(10);
let rem = Arc::new(AtomicUsize::new(0));
b.iter(move || {
rem.store(super::NUM_SPAWN, SeqCst);
for _ in 0..super::NUM_SPAWN {
let tx = tx.clone();
let rem = rem.clone();
threadpool.spawn(future::lazy(move || {
if 1 == rem.fetch_sub(1, SeqCst) {
tx.send(()).unwrap();
}
Ok(())
}));
}
let _ = rx.recv().unwrap();
});
}
#[bench]
fn yield_many(b: &mut test::Bencher) {
let threadpool = ThreadPool::new();
let tasks = super::TASKS_PER_CPU * num_cpus::get();
let (tx, rx) = mpsc::sync_channel(tasks);
b.iter(move || {
for _ in 0..tasks {
let mut rem = super::NUM_YIELD;
let tx = tx.clone();
threadpool.spawn(future::poll_fn(move || {
rem -= 1;
if rem == 0 {
tx.send(()).unwrap();
Ok(Async::Ready(()))
} else {
// Notify the current task
task::current().notify();
// Not ready
Ok(Async::NotReady)
}
}));
}
for _ in 0..tasks {
let _ = rx.recv().unwrap();
}
});
}
}
// In this case, CPU pool completes the benchmark faster, but this is due to how
// CpuPool currently behaves, starving other futures. This completes the
// benchmark quickly but results in poor runtime characteristics for a thread
// pool.
//
// See rust-lang-nursery/futures-rs#617
//
mod cpupool {
use futures::future::{self, Executor};
use futures::{task, Async};
use futures_cpupool::*;
use num_cpus;
use std::sync::atomic::AtomicUsize;
use std::sync::atomic::Ordering::SeqCst;
use std::sync::{mpsc, Arc};
#[bench]
fn spawn_many(b: &mut test::Bencher) {
let pool = CpuPool::new(num_cpus::get());
let (tx, rx) = mpsc::sync_channel(10);
let rem = Arc::new(AtomicUsize::new(0));
b.iter(move || {
rem.store(super::NUM_SPAWN, SeqCst);
for _ in 0..super::NUM_SPAWN {
let tx = tx.clone();
let rem = rem.clone();
pool.execute(future::lazy(move || {
if 1 == rem.fetch_sub(1, SeqCst) {
tx.send(()).unwrap();
}
Ok(())
}))
.ok()
.unwrap();
}
let _ = rx.recv().unwrap();
});
}
#[bench]
fn yield_many(b: &mut test::Bencher) {
let pool = CpuPool::new(num_cpus::get());
let tasks = super::TASKS_PER_CPU * num_cpus::get();
let (tx, rx) = mpsc::sync_channel(tasks);
b.iter(move || {
for _ in 0..tasks {
let mut rem = super::NUM_YIELD;
let tx = tx.clone();
pool.execute(future::poll_fn(move || {
rem -= 1;
if rem == 0 {
tx.send(()).unwrap();
Ok(Async::Ready(()))
} else {
// Notify the current task
task::current().notify();
// Not ready
Ok(Async::NotReady)
}
}))
.ok()
.unwrap();
}
for _ in 0..tasks {
let _ = rx.recv().unwrap();
}
});
}
}
tokio-executor/benches/threadpool_depth.rs
-72
View File
@@ -1,72 +0,0 @@
#![cfg(feature = "broken")]
#![feature(test)]
#![warn(rust_2018_idioms)]
extern crate test;
const ITER: usize = 20_000;
mod us {
use futures::future;
use std::sync::mpsc;
use tokio_executor::threadpool::*;
#[bench]
fn chained_spawn(b: &mut test::Bencher) {
let threadpool = ThreadPool::new();
fn spawn(pool_tx: Sender, res_tx: mpsc::Sender<()>, n: usize) {
if n == 0 {
res_tx.send(()).unwrap();
} else {
let pool_tx2 = pool_tx.clone();
pool_tx
.spawn(future::lazy(move || {
spawn(pool_tx2, res_tx, n - 1);
Ok(())
}))
.unwrap();
}
}
b.iter(move || {
let (res_tx, res_rx) = mpsc::channel();
spawn(threadpool.sender().clone(), res_tx, super::ITER);
res_rx.recv().unwrap();
});
}
}
mod cpupool {
use futures::future::{self, Executor};
use futures_cpupool::*;
use num_cpus;
use std::sync::mpsc;
#[bench]
fn chained_spawn(b: &mut test::Bencher) {
let pool = CpuPool::new(num_cpus::get());
fn spawn(pool: CpuPool, res_tx: mpsc::Sender<()>, n: usize) {
if n == 0 {
res_tx.send(()).unwrap();
} else {
let pool2 = pool.clone();
pool.execute(future::lazy(move || {
spawn(pool2, res_tx, n - 1);
Ok(())
}))
.ok()
.unwrap();
}
}
b.iter(move || {
let (res_tx, res_rx) = mpsc::channel();
spawn(pool.clone(), res_tx, super::ITER);
res_rx.recv().unwrap();
});
}
}