The work on sharding the timer implementation has caused a measurable performance regression due to increased contention. This patch reverts the current work on sharding. The next step will be to work on a per-worker timer wheel.
The alternative multi-threaded runtime started as an experiment. We have been
unable to find real-world benefit. Work has halted on this effort, so lets get
rid of it.
The broadcast channel does not require values to implement `Sync` yet it calls
the `.clone()` method without synchronizing. This is unsound logic. This patch
adds per-value synchronization on receive to handle this case. It is unlikely
any usage of the broadcast channel is currently at risk of the unsoundeness
issue as it requires accessing a `!Sync` type during `.clone()`, which would be
very unusual when using the broadcast channel.
This patch includes an initial implementation of a new multi-threaded
runtime. The new runtime aims to increase the scheduler throughput by
speeding up how it dispatches work to peer worker threads. This
implementation improves most benchmarks by about ~10% when the number of
threads is below 16. As threads increase, mutex contention deteriorates
performance.
Because the new scheduler is not yet ready to replace the old one, the
patch introduces it as an unstable runtime flavor with a warning that it
isn't production ready. Work to improve the scalability of the runtime
will most likely require more intrusive changes across Tokio, so I am
opting to merge with master to avoid larger conflicts.
This patch fixes a bug where nesting `block_in_place` with a `block_on`
between could lead to a panic. This happened because the nested
`block_in_place` would try to acquire a core on return when it should
not attempt to do so. The `block_on` between the two nested
`block_in_place` altered the thread-local state to lead to the incorrect
behavior.
The fix is for each call to `block_in_place` to track if it needs to try
to steal a core back.
Fixes#5239
This patch fixes a bug where nesting `block_in_place` with a `block_on`
between could lead to a panic. This happened because the nested
`block_in_place` would try to acquire a core on return when it should
not attempt to do so. The `block_on` between the two nested
`block_in_place` altered the thread-local state to lead to the incorrect
behavior.
The fix is for each call to `block_in_place` to track if it needs to try
to steal a core back.
Fixes#5239
This patch removes the custom slab in favor of regular allocations an
`Arc`. Originally, the slab was used to be able to pass indexes as
tokens to the I/O driver when registering I/O resources. However, this
has the downside of having a more expensive token lookup path. It also
pins a `ScheduledIo` to a specific I/O driver. Additionally, the slab is
approaching custom allocator territory.
We plan to explore migrating I/O resources between I/O drivers. As a
step towards that, we need to decouple `ScheduledIo` from the I/O
driver. To do this, the patch uses plain-old allocation to allocate the
`ScheduledIo` and we use the pointer as the token. To use the token, we
need to be very careful about releasing the `ScheduledIo`. We need to
make sure that the associated I/O handle is deregistered from the I/O
driver **and** there are no polls. The strategy in this PR is to let the
I/O driver do the final release between polls, but I expect this
strategy to evolve over time.
The `drain_filter` method on the internal `LinkedList` type passes a
`&mut` reference to the node type. However, the `LinkedList` is intended
to be used with nodes that are shared in other ways. For example
`task::Header` is accessible concurrently from multiple threads.
Currently, the only usage of `drain_filter` is in a case where `&mut`
access is safe, so this change is to help prevent future bugs and
tighten up the safety of internal utilities.
Calling `Handle::enter()` returns a `EnterGuard` value, which resets the
thread-local context on drop. The drop implementation assumes that
guards from nested `enter()` calls are dropped in reverse order.
However, there is no static enforcement of this requirement.
This patch checks that the guards are dropped in reverse order and
panics otherwise. A future PR will deprecate `Handle::enter()` in favor
of a method that takes a closure, ensuring the guard is dropped
appropriately.
The tuning test relies on a predictable execution environment. It
assumes that spawning a new task can complete reasonably fast. When
running tests with ASAN, the tuning test will spurriously fail. After
investigating, I believe this is due to running tests with ASAN enabled
and without `release` in a low resource environment (CI) results in an
execution environment that is too slow for the tuning test to succeed.
This PR restructures `runtime::context` into multiple files by component and feature flag. The goal is to reduce code defined in macros and make each context component more manageable.
There should be no behavior changes except tweaking how the RNG seed is set. Instead of putting it in `set_current`, we set it when entering the runtime. This aligns better with the feature's original intent, enabling users to make a runtime's RNG deterministic. The seed should not be changed by `Handle::enter()`, so there is no need to have the code in `context::set_current`.
Removes `Send` from `EnterGuard` (returned by `Handle::enter()`. The
guard type changes a thread-local variable on drop. If the guard is
moved to a different thread, it would modify the wrong thread-local.
This is a **breaking change** but it fixes a bug and prevents incorrect
user behavior. If user code breaks because of this, it is because they
(most likely) have a bug in their code.
This commit is a step towards the ongoing effort to unify the mutex in
the multi-threaded scheduler. The Inject queue is split into two
structs. `Shared` holds fields that are concurrently accessed, and
`Synced` holds fields that must be locked to access. The multi-threaded
scheduler is responsible for locking `Synced` and passing it in when
needed.
The commit also splits `inject` into multiple files to help reduce the
amount of code defined in macros.
PR #5720 introduced runtime self-tuning. It included a test that
attempts to verify self-tuning logic. The test is heavily reliant on
timing details. This patch attempts to make the test a bit more reliable
by not assuming tuning will converge within a set amount of time.
Previously, `Inject` was defined in `runtime::task`. This was because it
used some internal fns as part of the intrusive linked-list
implementation.
In the future, we want to remove the mutex from Inject and move it to
the scheduler proper (to reduce mutex ops). To set this up, this commit
moves `Inject` to `runtime::scheduler`. To make this work, we have to
`pub(crate)` `task::RawTask` and use that as the interface to access the
next / previous pointers.
Previously, the deferred task list (list of tasks that yielded and are
waiting to be woken) was stored on the global runtime context. Because
the scheduler is responsible for waking these tasks, it took additional
TLS reads to perform the wake operation.
Instead, this commit moves the list of deferred tasks into the scheduler
context. This makes it easily accessible from the scheduler itself.
In order to reduce the number of mutex operations in the multi-threaded
scheduler hot path, we need to unify the various mutexes into a single
one. To start this work, this commit splits up `Idle` into `Idle` and
`Synced`. The `Synced` component is stored separately in the scheduler's
`Shared` structure.
Each multi-threaded runtime worker prioritizes pulling tasks off of its
local queue. Every so often, it checks the injection (global) queue for
work submitted there. Previously, "every so often," was a constant
"number of tasks polled" value. Tokio sets a default of 61, but allows
users to configure this value.
If workers are under load with tasks that are slow to poll, the
injection queue can be starved. To prevent starvation in this case, this
commit implements some basic self-tuning. The multi-threaded scheduler
tracks the mean task poll time using an exponentially-weighted moving
average. It then uses this value to pick an interval at which to check
the injection queue.
This commit is a first pass at adding self-tuning to the scheduler.
There are other values in the scheduler that could benefit from
self-tuning (e.g. the maintenance interval). Additionally, the
current-thread scheduler could also benfit from self-tuning. However, we
have reached the point where we should start investigating ways to unify
logic in both schedulers. Adding self-tuning to the current-thread
scheduler will be punted until after this unification.
This commit eliminates the current_thread::CURRENT and multi_thread::current
thread-local variables in favor of using `runtime::context`. This is another step
towards reducing the total number of thread-local variables used by Tokio.
As an optimization to improve locality, the multi-threaded scheduler
maintains a single slot (LIFO slot). When a task is scheduled, it goes
into the LIFO slot. The scheduler will run tasks in the LIFO slot first
before checking the local queue.
Ping-ping style workloads where task A notifies task B, which
notifies task A again, can cause starvation as these two tasks
repeatedly schedule the other in the LIFO slot. #5686, a first
attempt at solving this problem, consumes a unit of budget each time a
task is scheduled from the LIFO slot. However, at the time of this
commit, the scheduler allocates 128 units of budget for each chunk of
work. This is relatively high in situations where tasks do not perform many
async operations yet have meaningful poll times (even 5-10 microsecond
poll times can have an outsized impact on the scheduler).
In an ideal world, the scheduler would adapt to the workload it is
executing. However, as a stopgap, this commit limits the times
the LIFO slot is prioritized per scheduler tick.
In the multi-threaded scheduler, when there are no tasks on the local
queue, a worker will attempt to pull tasks from the injection queue.
Previously, the worker would only attempt to poll one task from the
injection queue then continue trying to find work from other sources.
This can result in the injection queue backing up when there are many
tasks being scheduled from outside of the runtime.
This patch updates the worker to try to poll more than one task from the
injection queue when it has no more local work. Note that we also don't
want a single worker to poll **all** tasks on the injection queue as
that would result in work becoming unbalanced.
Previously, the current_thread scheduler used its own injection queue
instead of sharing the same one as the multi-threaded scheduler. This
patch updates the current_thread scheduler to use the same injection
queue as the multi-threaded one (`task::Inject`).
`task::Inject` includes an optimization where it does not need to
acquire the mutex if the queue is empty.
These counters are enabled using the `tokio_internal_mt_counters` and
are intended to help with debugging performance issues.
Whenever I work on the threaded runtime, I often find myself adding
these counters, then removing them before submitting a PR. I think
keeping them in will save time in the long run and shouldn't impact dev
much.
Adds support for instrumenting the poll times of all spawned tasks. Data is tracked in a histogram. The user must specify the histogram scale and bucket ranges. Implementation-wise, the same strategy is used in the runtime where we are just using atomic counters. Because instrumenting each poll duration will result in frequent calls to `Instant::now()`, I think it should be an opt-in metric.
The test-util feature flag is only intended to be used with tests.
However, it is possible to enable it in release mode accidentally. This
patch reduces the overhead of `Instant::now()` when the `test-util`
feature flag is enabled but `time::pause()` is not called.
The optimization is implemented by adding a static atomic flag that
tracks if `time::pause()` has ever been called. In `Instant::now()`, the
atomic flag is first checked before the thread-local and mutex are
accessed.
This patch removes `Arc` from Tokio's internal clock source. Instead of
cloning `Clock` when needed, a reference is passed into functions that
need to get the current instant.
The `pipe_mode` function would erase any previously set configuration
option that is specified using the pipe_mode fit field. This patch fixes
the builder to maintain the bit field when changing the pipe mode.
This patch simplifies the implementation of `task::Id` by moving
conditional compilation into the `AtomicU64` definition. To handle
platforms that do not include `const fn Mutex::new()`, `StaticAtomicU64`
is defined as always having a `const fn new()`. `StaticAtomicU64` is
implemented with `OnceCell` when needed.
This adds CI coverage for a couple of code paths that are not currently
hit in CI:
* no `const fn Mutex::new`
* no `AtomicU64`
This is done by adding some new CFG flags used only for tests in order
to force those code paths.
PR #5223 changed the behavior of `yield_now()` to store yielded tasks
and notify them *after* polling the resource drivers. This PR fixes a
couple of bugs with this new behavior when combined with
`block_in_place()`.
First, we need to avoid freeing the deferred task queue when exiting a
runtime if it is *not* the root runtime. Because `block_in_place()`
allows a user to start a new runtime from within an existing task, this
check is necessary.
Second, when a worker core is stolen from a thread during a
`block_in_place()` call, we need to ensure that deferred tasks are
notified anyway.
Previously, calling `task::yield_now().await` would yield the current
task to the scheduler, but the scheduler would poll it again before
polling the resource drivers. This behavior can result in starving the
resource drivers.
This patch creates a queue tracking yielded tasks. The scheduler
notifies those tasks **after** polling the resource drivers.
Refs: #5209
This patch updates CI to use `cross` to run Tokio tests on virtualized
ARM and i686 VMs. Because ipv6 doesn't work on Github action running in
a docker instance, those tests are disabled
`LocalSet` cleans up any tasks that have not yet been completed when it is
dropped. Previously, this cleanup process required access to a thread-local.
Suppose a `LocalSet` is stored in a thread-local itself. In that case, when it is
dropped, there is no guarantee the drop implementation will be able to
access the internal `LocalSet` thread-local as it may already have been
destroyed.
The internal `LocalSet` thread local is mainly used to avoid writing unsafe
code. All `LocalState` that cannot be moved across threads is stored in the
thread-local and accessed on demand.
This patch moves this local-only state into the `LocalSet`'s "shared" struct.
Because this struct *is* `Send`, the local-only state is stored in `UnsafeCell`,
and callers must ensure not to touch it from other threads.
A debug assertion is added to enforce this requirement in tests.
Fixes#5162
An earlier change updated `enter_runtime` to also set the current
handle. However, the change did not store the `SetCurrentGuard`, so the
"current handle" was immediately unset. This patch stores the
`SetCurrentGuard` in the `EnterRuntimeGuard`.
No existing test exposed this bug because all tests went via `Runtime`
instead of `Handle`. Currently, `Runtime` is still explicitly setting
the handle before entering runtime, so all tests still passed. A new
test is added that covers the case of calling `Handle::block_on` and
accessing the current handle.
A previous patch moved code related to entering a runtime into the
context module but did not change anything. This patch combines both
thread-local variables.
This moves the functions, types, and thread-local related to entering a
runtime into the context module. This does not yet unify the thread-local
variables, as that, will be done in a follow-up PR.
Instead of each scheduler flavor holding a reference to the scheduler
handle, the scheduler handle is passed in as needed. This removes a
duplicate handle reference in the `Runtime` struct and lays the
groundwork for further handle struct tweaks.
This is a first step towards unifying the concepts of "entering a
runtime" and setting `Handle::current`.
Previously, these two operations were performed separately at each call
site (runtime block_on, ...). This is error-prone and also requires
multiple accesses to the thread-local variable. Additionally, "entering
the runtime" conflated the concept of entering a blocking region. For
example, calling `mpsc::Receiver::recv_blocking` performed the "enter
the runtime" step. This was done to prevent blocking a runtime, as the
operation will panic when called from an existing runtime.
To untangle these concepts, the patch splits out each logical operation
into functions. In total, there are three "enter" operations:
* `set_current_handle`
* `enter_runtime`
* `enter_blocking_region`
There are some behavior changes with each function, but they
should not translate to public behavior changes. The most significant is
`enter_blocking_region` does not change the value of the thread-local
variable, which means the function can be re-entered. Since
`enter_blocking_region` is an internal-only function and we do not
re-enter, this has no public-facing impact.
Because `enter_runtime` takes a `&Handle` to combine the
`set_current_handle` operation with entering a runtime, the patch
exposes an annoyance with the current `scheduler::Handle` struct layout.
A new instance of `scheduler::Handle` must be constructed at each call
to `enter_runtime`. We can explore cleaning this up later.
This patch also does not combine the "entered runtime" thread-local
variable with the "context" thread-local variable. To keep the patch
smaller, this has been punted to a follow-up change.
This patch consolidates the budget thread-local state with the
`runtime::context` thread local. This reduces the number of thread-local
variables used by Tokio by one.
The `LocalSet::run_until` future is just a "plain" future that should
run on a runtime that already has a coop budget. In other words, the
`run_until` future should not get its own budget but should inherit the
calling task's budget. Getting this behavior is done by removing the
call to `budget` in `run_until`.
