Calling `Handle::enter()` returns a `EnterGuard` value, which resets the
thread-local context on drop. The drop implementation assumes that
guards from nested `enter()` calls are dropped in reverse order.
However, there is no static enforcement of this requirement.
This patch checks that the guards are dropped in reverse order and
panics otherwise. A future PR will deprecate `Handle::enter()` in favor
of a method that takes a closure, ensuring the guard is dropped
appropriately.
The tuning test relies on a predictable execution environment. It
assumes that spawning a new task can complete reasonably fast. When
running tests with ASAN, the tuning test will spurriously fail. After
investigating, I believe this is due to running tests with ASAN enabled
and without `release` in a low resource environment (CI) results in an
execution environment that is too slow for the tuning test to succeed.
This PR restructures `runtime::context` into multiple files by component and feature flag. The goal is to reduce code defined in macros and make each context component more manageable.
There should be no behavior changes except tweaking how the RNG seed is set. Instead of putting it in `set_current`, we set it when entering the runtime. This aligns better with the feature's original intent, enabling users to make a runtime's RNG deterministic. The seed should not be changed by `Handle::enter()`, so there is no need to have the code in `context::set_current`.
Removes `Send` from `EnterGuard` (returned by `Handle::enter()`. The
guard type changes a thread-local variable on drop. If the guard is
moved to a different thread, it would modify the wrong thread-local.
This is a **breaking change** but it fixes a bug and prevents incorrect
user behavior. If user code breaks because of this, it is because they
(most likely) have a bug in their code.
If the `Scoped` type is `Sync`, then you can call `set` from two threads in parallel. Since it accesses `inner` without synchronization, this is a data race.
This is a soundness issue for the `Scoped` type, but since this is an internal API and we don't use it incorrectly anywhere, no harm is done.
This commit is a step towards the ongoing effort to unify the mutex in
the multi-threaded scheduler. The Inject queue is split into two
structs. `Shared` holds fields that are concurrently accessed, and
`Synced` holds fields that must be locked to access. The multi-threaded
scheduler is responsible for locking `Synced` and passing it in when
needed.
The commit also splits `inject` into multiple files to help reduce the
amount of code defined in macros.
PR #5720 introduced runtime self-tuning. It included a test that
attempts to verify self-tuning logic. The test is heavily reliant on
timing details. This patch attempts to make the test a bit more reliable
by not assuming tuning will converge within a set amount of time.
Previously, `Inject` was defined in `runtime::task`. This was because it
used some internal fns as part of the intrusive linked-list
implementation.
In the future, we want to remove the mutex from Inject and move it to
the scheduler proper (to reduce mutex ops). To set this up, this commit
moves `Inject` to `runtime::scheduler`. To make this work, we have to
`pub(crate)` `task::RawTask` and use that as the interface to access the
next / previous pointers.
Previously, the deferred task list (list of tasks that yielded and are
waiting to be woken) was stored on the global runtime context. Because
the scheduler is responsible for waking these tasks, it took additional
TLS reads to perform the wake operation.
Instead, this commit moves the list of deferred tasks into the scheduler
context. This makes it easily accessible from the scheduler itself.
In order to reduce the number of mutex operations in the multi-threaded
scheduler hot path, we need to unify the various mutexes into a single
one. To start this work, this commit splits up `Idle` into `Idle` and
`Synced`. The `Synced` component is stored separately in the scheduler's
`Shared` structure.
Each multi-threaded runtime worker prioritizes pulling tasks off of its
local queue. Every so often, it checks the injection (global) queue for
work submitted there. Previously, "every so often," was a constant
"number of tasks polled" value. Tokio sets a default of 61, but allows
users to configure this value.
If workers are under load with tasks that are slow to poll, the
injection queue can be starved. To prevent starvation in this case, this
commit implements some basic self-tuning. The multi-threaded scheduler
tracks the mean task poll time using an exponentially-weighted moving
average. It then uses this value to pick an interval at which to check
the injection queue.
This commit is a first pass at adding self-tuning to the scheduler.
There are other values in the scheduler that could benefit from
self-tuning (e.g. the maintenance interval). Additionally, the
current-thread scheduler could also benfit from self-tuning. However, we
have reached the point where we should start investigating ways to unify
logic in both schedulers. Adding self-tuning to the current-thread
scheduler will be punted until after this unification.
This commit eliminates the current_thread::CURRENT and multi_thread::current
thread-local variables in favor of using `runtime::context`. This is another step
towards reducing the total number of thread-local variables used by Tokio.
As an optimization to improve locality, the multi-threaded scheduler
maintains a single slot (LIFO slot). When a task is scheduled, it goes
into the LIFO slot. The scheduler will run tasks in the LIFO slot first
before checking the local queue.
Ping-ping style workloads where task A notifies task B, which
notifies task A again, can cause starvation as these two tasks
repeatedly schedule the other in the LIFO slot. #5686, a first
attempt at solving this problem, consumes a unit of budget each time a
task is scheduled from the LIFO slot. However, at the time of this
commit, the scheduler allocates 128 units of budget for each chunk of
work. This is relatively high in situations where tasks do not perform many
async operations yet have meaningful poll times (even 5-10 microsecond
poll times can have an outsized impact on the scheduler).
In an ideal world, the scheduler would adapt to the workload it is
executing. However, as a stopgap, this commit limits the times
the LIFO slot is prioritized per scheduler tick.
In the multi-threaded scheduler, when there are no tasks on the local
queue, a worker will attempt to pull tasks from the injection queue.
Previously, the worker would only attempt to poll one task from the
injection queue then continue trying to find work from other sources.
This can result in the injection queue backing up when there are many
tasks being scheduled from outside of the runtime.
This patch updates the worker to try to poll more than one task from the
injection queue when it has no more local work. Note that we also don't
want a single worker to poll **all** tasks on the injection queue as
that would result in work becoming unbalanced.
Previously, the current_thread scheduler used its own injection queue
instead of sharing the same one as the multi-threaded scheduler. This
patch updates the current_thread scheduler to use the same injection
queue as the multi-threaded one (`task::Inject`).
`task::Inject` includes an optimization where it does not need to
acquire the mutex if the queue is empty.
These counters are enabled using the `tokio_internal_mt_counters` and
are intended to help with debugging performance issues.
Whenever I work on the threaded runtime, I often find myself adding
these counters, then removing them before submitting a PR. I think
keeping them in will save time in the long run and shouldn't impact dev
much.
Adds support for instrumenting the poll times of all spawned tasks. Data is tracked in a histogram. The user must specify the histogram scale and bucket ranges. Implementation-wise, the same strategy is used in the runtime where we are just using atomic counters. Because instrumenting each poll duration will result in frequent calls to `Instant::now()`, I think it should be an opt-in metric.
Task dumps are snapshots of runtime state. Taskdumps are collected by
instrumenting Tokio's leaves to conditionally collect backtraces, which
are then coalesced per-task into execution tree traces.
This initial implementation only supports collecting taskdumps from
within the context of a current-thread runtime, and only `yield_now()`
is instrumented.
In CI, we are using a newer version of Clippy than what is stated in the
contributions guide.
Additionally, it is no longer necessary to use Clippy from the MSRV. As
of Clippy 1.64, the `rust-version` field in Cargo.toml is respected.
The text and the command have been updated to reflect the current state
of CI and best practices.
When `tokio::time::Interval::poll_tick()` returns `Poll::Pending`, it
schedules itself for being woken up again through the waker of the
passed context, which is correct behavior.
However when `Poll::Ready(_)` is returned, the interval timer should be
reset but not scheduled to be woken up again as this is up to the
caller.
This commit fixes the bug by introducing a `reset_without_reregister`
method on `TimerEntry` which is called by `Intervall::poll_tick(cx)` in
case the delay poll returns `Poll::Ready(_)`.
Co-authored-by: Simon B. Gasse <[email protected]>
The condition checked in the and_then() call is the same as is checked
in the match below, so we can clean it up by just matching on
next_expiration() directly.
In a previous PR (#4128), the `spawn.location` field on task spans was
structured into 3 separate fields for the `file`, `line`, and `col`.
There is a separately created span for blocking tasks which was missed.
This caused tasks created with `spawn_blocking` to appear in
`tokio-console` without a location, but with an additional "free form"
field containing the formatted source code location.
This change modifies this span to use the same format. The span creation
needs to be separate from the other task spans because it records the
function name. This information is useful in the `spawn_blocking` case,
but can be "catastrophically long" in the `async fn` case and was
removed in #3074.
Implementation of Tokio's ReadDir assumes that ReadDir from std is
fused, but that's not the case on Windows. This change wraps the std
iterator in std::iter::Fuse to make its usage correct.
* Add links to fundamental testing concepts in Rust
* Add information about conditional compilation attributes
and how to use them to run tests with cargo
The test-util feature flag is only intended to be used with tests.
However, it is possible to enable it in release mode accidentally. This
patch reduces the overhead of `Instant::now()` when the `test-util`
feature flag is enabled but `time::pause()` is not called.
The optimization is implemented by adding a static atomic flag that
tracks if `time::pause()` has ever been called. In `Instant::now()`, the
atomic flag is first checked before the thread-local and mutex are
accessed.
Some users mentioned that the behavior of a channel when the receivers
and/or senders are dropped isn't explicitly documented.
This change adds wording to the documentation for each channel in the
sync module, explaining under which conditions messages in a channel are
dropped with respect to dropping the senders and the receivers.
Refs: #5490
These flags were previously only needed due to a bug in the `cargo-semver-checks` CLI logic.
The correct behavior (available as of v0.18.3) for `cargo-semver-checks` is to ignore `publish = false` crates when scanning a workspace, *unless* those crates are specifically selected for checking.
All the crates being excluded here are `publish = false` so they are already excluded by the default behavior, so all `--exclude` flags are no-ops.
This additional benchmark exercises a common request/reply pattern using an MPSC for requests along with a oneshot payload as a reply mechanism. When used in a current threaded scenario, the bench is 17 times faster on my machine than when using the multi-threaded runtime and one worker thread. Not only that, but if I increase the number of worker threads to 6, performance degrades further.
Does this suggest a scheduling problem with the multi-threaded runtime?
No matter what, hopefully the benchmarks are a useful addition.
This patch removes `Arc` from Tokio's internal clock source. Instead of
cloning `Clock` when needed, a reference is passed into functions that
need to get the current instant.
Fixes#5373Closes#5358
- Add check for no_atomic_u64 & no_const_mutex_new (condition to atomic_u64_static_once_cell.rs is compiled)
- Allow unused_imports in TARGET_ATOMIC_U64_PROBE. I also tested other *_PROBE and found no other errors triggered by -D warning.
- Fix cfg of util::once_cell module
The `pipe_mode` function would erase any previously set configuration
option that is specified using the pipe_mode fit field. This patch fixes
the builder to maintain the bit field when changing the pipe mode.
The global flag remains and is used to prevent duplicated shutdowns and
new io operations after shutdown.
On shutdown, the driver flips the shutdown flag of every pending io
operations and wake them to fail with a shutdown error.
Fixes: #5227
- Use dtolnay/rust-toolchain instead of actions-rs/toolchain
- Use cargo/cross directly instead of actions-rs/cargo
- Use rustsec/audit-check instead of actions-rs/audit-check
This patch simplifies the implementation of `task::Id` by moving
conditional compilation into the `AtomicU64` definition. To handle
platforms that do not include `const fn Mutex::new()`, `StaticAtomicU64`
is defined as always having a `const fn new()`. `StaticAtomicU64` is
implemented with `OnceCell` when needed.
This adds CI coverage for a couple of code paths that are not currently
hit in CI:
* no `const fn Mutex::new`
* no `AtomicU64`
This is done by adding some new CFG flags used only for tests in order
to force those code paths.
Adds a method `Builder::max_io_events_per_tick()` to the runtime builder. This can be used to configure the capacity of events that may be processed per OS poll.
Raise the mio-aio dev dependency, which transitively brings in Nix, to
ensure that the tests will continue to compile if libc switches from a
FreeBSD 11 ABI to a FreeBSD 12 one.
Readiness futures inconsistently return the current readiness of an I/O resource if it is immediately available, or all readiness relevant for the given `Interest`, if a future needs to wait. In particular, it always returns `read_closed` for `Interest::READABLE` and `write_closed` for `Interest::WRITABLE`, which often is not true. Tokio should not tolerate false positives for `*_closed` events because they are considered final states and are not cleared internally.
In the case of an `io_resource.ready(Interest::READABLE | Interest::WRITABLE)` call, this behavior may also lead to false positives of other events.
## Solution
Follow the same strategy as `poll_ready` and return the current resource's readiness.
Closes: #5098
There's already an existing warning about this combo in the
documentation for `Handle::block_on`. This commit adds a summarized
version in `Runtime::handle`.
PR #5223 changed the behavior of `yield_now()` to store yielded tasks
and notify them *after* polling the resource drivers. This PR fixes a
couple of bugs with this new behavior when combined with
`block_in_place()`.
First, we need to avoid freeing the deferred task queue when exiting a
runtime if it is *not* the root runtime. Because `block_in_place()`
allows a user to start a new runtime from within an existing task, this
check is necessary.
Second, when a worker core is stolen from a thread during a
`block_in_place()` call, we need to ensure that deferred tasks are
notified anyway.
Previously, calling `task::yield_now().await` would yield the current
task to the scheduler, but the scheduler would poll it again before
polling the resource drivers. This behavior can result in starving the
resource drivers.
This patch creates a queue tracking yielded tasks. The scheduler
notifies those tasks **after** polling the resource drivers.
Refs: #5209
This PR should hopefully reduce the amount of code generated per
future-type spawned on the runtime. The following methods are no longer generic:
* `try_set_join_waker`
* `remote_abort`
* `clone_waker`
* `drop_waker`
* `wake_by_ref`
* `wake_by_val`
A new method is added to the vtable called schedule, which is used when a task
should be scheduled on the runtime. E.g. wake_by_ref will call it if the state change
says that the task needs to be scheduled. However, this method is only generic over
the scheduler, and not the future type, so it also isn't generated for every task.
Additionally, one of the changes involved in the above makes it possible to remove
the id field from JoinHandle and AbortHandle.
This patch updates CI to use `cross` to run Tokio tests on virtualized
ARM and i686 VMs. Because ipv6 doesn't work on Github action running in
a docker instance, those tests are disabled
`LocalSet` cleans up any tasks that have not yet been completed when it is
dropped. Previously, this cleanup process required access to a thread-local.
Suppose a `LocalSet` is stored in a thread-local itself. In that case, when it is
dropped, there is no guarantee the drop implementation will be able to
access the internal `LocalSet` thread-local as it may already have been
destroyed.
The internal `LocalSet` thread local is mainly used to avoid writing unsafe
code. All `LocalState` that cannot be moved across threads is stored in the
thread-local and accessed on demand.
This patch moves this local-only state into the `LocalSet`'s "shared" struct.
Because this struct *is* `Send`, the local-only state is stored in `UnsafeCell`,
and callers must ensure not to touch it from other threads.
A debug assertion is added to enforce this requirement in tests.
Fixes#5162
An earlier change updated `enter_runtime` to also set the current
handle. However, the change did not store the `SetCurrentGuard`, so the
"current handle" was immediately unset. This patch stores the
`SetCurrentGuard` in the `EnterRuntimeGuard`.
No existing test exposed this bug because all tests went via `Runtime`
instead of `Handle`. Currently, `Runtime` is still explicitly setting
the handle before entering runtime, so all tests still passed. A new
test is added that covers the case of calling `Handle::block_on` and
accessing the current handle.
A previous patch moved code related to entering a runtime into the
context module but did not change anything. This patch combines both
thread-local variables.
This moves the functions, types, and thread-local related to entering a
runtime into the context module. This does not yet unify the thread-local
variables, as that, will be done in a follow-up PR.
Instead of each scheduler flavor holding a reference to the scheduler
handle, the scheduler handle is passed in as needed. This removes a
duplicate handle reference in the `Runtime` struct and lays the
groundwork for further handle struct tweaks.
Publish the blocking thread pool metrics as thread-safe values, written
under the blocking thread pool's lock and able to be read in a lock-free
fashion by any reader.
Fixes#5156
This is a first step towards unifying the concepts of "entering a
runtime" and setting `Handle::current`.
Previously, these two operations were performed separately at each call
site (runtime block_on, ...). This is error-prone and also requires
multiple accesses to the thread-local variable. Additionally, "entering
the runtime" conflated the concept of entering a blocking region. For
example, calling `mpsc::Receiver::recv_blocking` performed the "enter
the runtime" step. This was done to prevent blocking a runtime, as the
operation will panic when called from an existing runtime.
To untangle these concepts, the patch splits out each logical operation
into functions. In total, there are three "enter" operations:
* `set_current_handle`
* `enter_runtime`
* `enter_blocking_region`
There are some behavior changes with each function, but they
should not translate to public behavior changes. The most significant is
`enter_blocking_region` does not change the value of the thread-local
variable, which means the function can be re-entered. Since
`enter_blocking_region` is an internal-only function and we do not
re-enter, this has no public-facing impact.
Because `enter_runtime` takes a `&Handle` to combine the
`set_current_handle` operation with entering a runtime, the patch
exposes an annoyance with the current `scheduler::Handle` struct layout.
A new instance of `scheduler::Handle` must be constructed at each call
to `enter_runtime`. We can explore cleaning this up later.
This patch also does not combine the "entered runtime" thread-local
variable with the "context" thread-local variable. To keep the patch
smaller, this has been punted to a follow-up change.
This patch consolidates the budget thread-local state with the
`runtime::context` thread local. This reduces the number of thread-local
variables used by Tokio by one.
The `LocalSet::run_until` future is just a "plain" future that should
run on a runtime that already has a coop budget. In other words, the
`run_until` future should not get its own budget but should inherit the
calling task's budget. Getting this behavior is done by removing the
call to `budget` in `run_until`.
This is a step towards unifying thread-local variables. In the future,
`coop` will be updated to use the runtime context thread-local to store
its state.
This patch does some internal renames to remove some confusion.
* `allow_blocking` is renamed to `allow_block_in_place` to indicate that
the variable only impacts the `block_in_place()` function.
* `context::try_enter` is renamed to `context::try_set_current` to
disambiguate between the various "enter" functions. This function only
sets the runtime handle used by Tokio's public APIs. Entering a runtime
is a separate operation. # Please enter the commit message for your
changes.
* `scheduler::Handle::enter()` is removed to consolidate methods that
set the current context.
This patch is the first step towards unifying all the thread-local
variables spread out across Tokio. A new `Context` struct is added which
will be used to replace the various thread-locals that exist today.
Initially, `Context` only holds the current runtime handle and the
random number generator. Further PRs will add other thread-local state.
A previous PR removed `runtime::context`. At that time,
`runtime::context` was used as an extra layer to access the various
runtime driver handles. This version of `runtime::context` serves a
different purpose (unifying all the thread-locals).
While setting the random number generator seed with the multi-threaded
scheduler does result in deterministic behavior related to the random
number generator, threads still introduce non-determinism, making it
hard (impossible?) to test this. There also is little value in doing so.
This patch also updates the docs to remove mention of work stealing.
The signal driver still uses an `Arc` internally to track if the driver
is still running, however, using the `scheduler::Handle` to access the
signal driver handle lets us delete some code.
The improvement to the `rng_seed` tests added in #5075 missed a case in
the `rt_threaded` tests which was still checking for a specific value.
As described in that PR, this makes the tests fragile and changing tokio
internals may require updating the test.
This change fixes that half-implemented improvement so that the tests no
longer depend on the exact internal ordering, but rather compare two
runs of separate runtimes built with the same seed to check that the
results are the same.
This is the start of applying a similar treatment to the I/O driver as
the time driver. The I/O driver will no longer hold its own reference to
the I/O handle. Instead, the handle is passed in when needed.
This patch also moves the process driver to the `runtime` module.
The signal driver uses a `UnixStream` to receive signal events.
Previously, the signal driver used `PollEvented` internally to receive
events on the `UnixStream`. However, using `PollEvented` from
within a runtime driver created a circular link between the runtime and
the `PollEvented` instance.
This patch replaces `PollEvented` usage in favor of accessing the I/O
driver directly. The I/O driver now reserves a token for signal-related
events and tracks signal readiness internally. The signal driver queries
the I/O driver to check for signal-related readiness.
The signal feature only requires a driver with unix platforms. The
unix signal driver uses the I/O driver. A future refactor would like to
make the signal driver use internal APIs of the I/O driver. To do this,
the signal driver must be moved to the runtime module.
This patch removes a handle to the internal runtime driver handle held
by the runtime. This is another step towards reducing the number of Arc
refs across the runtime internals. Specifically, this change is part of
an effort to remove an Arc in the time driver itself.
This excludes the initial let statement of the proc-macro expansion
from receiving the last statement spans to aid completions in rust-analyzer.
The current span confuses rust-analyzer as it will map the tail expression
tokens to the let keyword (as this is the first token it finds with the
same span) which currently breaks completions. This commit should not
degrade the initial intent of the span reusages, as the type mismatch
parts are still spanned appropriately.
The `LocalSet` implementation includes a conditional compilation clause
that removes the `const` statement from the `thread_local` definition.
However, there already is an internal macro that does this:
`tokio_thread_local`.
This patch removes the conditional compilation in favor of using the
`tokio_thread_local` macro. This also fixes a conditional compilation
issue with an internal utility (`RcCell`).
Tokio maintains an internal thread_local macro that abstracts some
conditional build logic. Before this patch, the macro was named the same
as the `std` macro (`thread_local`). This resulted in confusion as to
whether or not the internal macro or the std macro was being called.
This patch renames the internal macro to `tokio_thread_local` making it
more obvious.
Motivation
Currently, when a task spawned on a `LocalSet` is woken by an I/O driver
or time driver running on the same thread as the `LocalSet`, the task is
pushed to the `LocalSet`'s locked remote run queue rather than to its
unsynchronized local run queue. This is unfortunate, as it
negates some of the performance benefits of having an unsynchronized
local run queue. Instead, tasks are only woken to the local queue when
they are woken by other tasks also running on the local set.
This occurs because the local queue is only used when the `CONTEXT`
thread-local contains a Context that's the same as the task's
`Schedule` instance (an `Arc<Shared>`)'s Context. When the `LocalSet`
is not being polled, the thread-local is unset, and the local run queue
cannot be accessed by the `Schedule` implementation for `Arc<Shared>`.
Solution
This branch fixes this by moving the local run queue into Shared along
with the remote run queue. When an `Arc<Shared>`'s Schedule impl wakes
a task and the `CONTEXT` thread-local is None (indicating we are not
currently polling the LocalSet on this thread), we now check if the
current thread's `ThreadId` matches that of the thread the `LocalSet`
was created on, and push the woken task to the local queue if it was.
Moving the local run queue into `Shared` is somewhat unfortunate, as it
means we now have a single field on the `Shared` type, which must not be
accessed from other threads and must add an unsafe impl `Sync` for `Shared`.
However, it's the only viable way to wake to the local queue
from the Schedule impl for `Arc<Shared>`, so I figured it was worth
the additional unsafe code. I added a debug assertion to check that the
local queue is only accessed from the thread that owns the `LocalSet`.
## Motivation
It turns out that `LocalSet` has...never actually passed Miri. This is
because all of the tests for `LocalSet` are defined as integration tests
in `tokio/tests/task_local_set.rs`, rather than lib tests in
`tokio/src`, and we never run Miri for integration tests.
PR #5095 added a new test reproducing an unrelated bug in `LocalSet`,
which had to be implemented as a lib test, as it must make internal
assertions about the state of the `LocalSet` that cannot be tested with
just the public API. This test failed under Miri, and it turns out that
the reason for this is unrelated to the change in #5095 --- `LocalSet`
uses the `LocalOwnedTasks` type, which turns out to contain a stacked
borrows violation due to converting an `&Header` into an
`NonNull<Header>` when removing a task from the `LocalOwnedTasks` list.
## Solution
Fortunately, this was actually quite easy to fix. The non-`Local`
version of `OwnedTasks` already uses `Task::header_ptr()` to get a
`NonNull<Header>` in its version of `remove`, which avoids this issue.
Therefore, the fix was as simple as updating `LocalOwnedTasks` to do the
same.
I've also added a very simple lib test in `src/task/local_set.rs` that
just creates a `LocalSet` and spawns a single task on it under a
current-thread runtime. This test fails under Miri without the fix in
this PR: https://github.com/tokio-rs/tokio/actions/runs/3237072694/jobs/5303682530
We should probably keep the test so that we're always testing at least
the most trivial `LocalSet` usage under Miri...
The original tests for the `Builder::rng_seed` added in #4910 were a bit
fragile. There have already been a couple of instances where internal
refactoring caused the tests to fail and need to be modified. While it
is expected that internal refactoring may cause the random values to
change, this shouldn't cause the tests to break.
The tests should be more robust and not be affected by internal
refactoring or changes in the Rust compiler version. The tests are
changed to perform the same operation in 2 runtimes created with the
same seed, the expectation is that the values that result from each
runtime are the same.
When 64-bit atomics are supported, use 32-bit queue indices. This
greatly improves resilience to ABA and has no impact on performance on
64-bit platforms.
Fixes: #5041
Functions that may panic can be annotated with `#[track_caller]` so that
in the event of a panic, the function where the user called the
panicking function is shown instead of the file and line within Tokio
source.
This change adds `#[track_caller]` to two public APIs in tokio task
module which weren't added in #4848.
* `tokio::task::block_in_place`
* `tokio::task::spawn_local`
These APIs had call stacks that went through closures, which is a use
case not supported by `#[track_caller]`. These two cases have been
refactored so that the `panic!` call is no longer inside a closure.
Tests have been added for these two new cases.
Refs: #4413
This patch removes all internal `Unpark` runtime types. The runtime now
uses the `Handle` types (`runtime::io::Handle` and
`runtime::time::Handle`) to signal threads to unpark.
Without separate `Unpark` types, future patches will be able to remove
more `Arc`s used inside various drivers.
This patch removes the driver Unpark handle out of the multi-thread
parker and passes a reference in when it is needed. This is a first step
towards getting rid of the separate driver unpark handle in favor of
just using the regular driver handle. Because the regular driver handle
is owned at a higher level (at the top of the worker struct).
This is the equivalent as tokio-rs/tokio#5023, but for the MultiThread
scheduler. This patch updates `MultiThread` to use `Arc<Handle>` for all
of its cross-thread needs instead of `Arc<Shared>`.
The effect of this change is that the multi-thread scheduler only has a
single `Arc` type, which includes the driver handles, keeping all
"shared state" together.
This is the next step in the internal runtime refactoring. This patch
updates `CurrentThread` to use `Arc<Handle>` for all of its cross-thread
needs instead of `Arc<Shared>`.
The effect of this change is that the current-thread scheduler only has
a single `Arc` type, which includes the driver handles, keeping all
"shared state" together.
The `runtime::Handle` struct is part of the public API but is also used
internally. This has created a bit of tension. An earlier patch made
defined Handle as a private struct in some cases when `rt` is not
enabled.
This patch splits out internal handle concerns into a new
`scheduler::Handle` type, which will only be internal. This also defines
a `Handle` type for each scheduler variant. Eventually, the
per-scheduler `Handle` types will replace the per-scheduler `Spawner`
types, but more work is needed before we can make that change.
The `tokio::select!` macro polls branches in a random order. While this
is desirable in production, for testing purposes a more deterministic
approach can be useul.
This change adds an additional parameter `rng_seed` to the runtime
`Builder` to set the random number generator seed. This value is then
used to reset the seed on the current thread when the runtime is entered
into (restoring the previous value when the thread leaves the runtime). All
threads created explicitly by the runtime also have a seed set as the
runtime is built. Each thread is set with a seed from a deterministic
sequence.
This guarantees that calls to the `tokio::select!` macro which are
performed in the same order on the same thread will poll branches in the
same order.
Additionally, the peer chosen to attempt to steal work from also uses a
deterministic sequence if `rng_seed` is set.
Both the builder parameter as well as the `RngSeed` struct are marked
unstable initially.
The `runtime::Kind` enum doesn't really represent the runtime flavor,
but is an enumeration of the different scheduler types. This patch
renames the enum to reflect this.
At a later time, it is likely the `Scheduler` enum will be moved to
`tokio::runtime::scheduler`, but that is punted to a later PR.
This rename is to make space for other enums
This patch removes println statements from tests. Some of these were
used to test that types implement `fmt::Debug`. These println statements
have been replaced with an equivalent test that does not output to
STDOUT. The rest of the println statements are most likely left over
from debugging sessions.
Currently, drivers are responsible for clean up when they are dropped.
This is possible today because each driver struct holds a reference to
its internal handle.
As part of an effort to decouple drivers from their handles, this patch
removes the drop implementations for the time and IO driver in favor of
having the scheduler explicitly call `shutdown()` as part of its
shutdown process. The scheduler will hold a reference to both the driver
and the driver handles, so in the future, it will be able to pass in the
driver handles as part of the shutdown process.
Currently, the various resource drivers use a layered approach where
each driver contains the next one. When the scheduler calls park on the
top-most driver, it does work then calls park on the inner driver it
holds. The handles do the same with unparking.
This patch is a step towards refactoring the runtime to move away from
the nested approach towards keeping all drivers and handles together in
a single runtime driver/handle. The unparker is removed from the time
handle and placed in the runtime handle and is passed into the time
driver as needed.
The `TimerEntry` struct is the internal integration point for public
time APIs (`sleep`, `interval`, ...) with the time driver. Currently,
`TimerEntry` holds an ref-counted reference to the time driver handle.
This patch replaces the reference to the time driver handle with a
reference to the runtime handle. This is part of a larger effort to
consolate internal handles across the runtime.
In an earlier PR (#4629), driver handles were moved into the scheduler
handle (`Spawner`). This was done to let the multi-threaded scheduler
have direct access to the thread pool spawner.
However, we are now working on a greater decoupling of the runtime
internals. All drivers and schedulers will be peers, stored in a single
thread-local variable, and the scheduler will be passed the full
runtime::Handle. This will achieve the original goal of giving the
scheduler access to the thread-pool while also (hopefully) simplifying
other aspects of the code.
As has been pointed out a few times in the past (e.g., #4036 (comment)), each-feature
is not sufficient for features check.
Ideally, we'd like to check all combinations of features, but there are too many
combinations. So limit the max number of simultaneous feature flags to 2 by --depth
option. I think this should be sufficient in most cases as @carllerche said in
taiki-e/cargo-hack#58.
The docs for `Registration::new_with_interest_and_handle` (internal
function) included a reference to two old functions `new_with_interest`
and `new` on the same struct. These functions no longer exist, which
could be confusing.
Since there is currently only a single constructor function on
`Registration`, the misleading text has simply been removed.
A few spawn_blocking methods were implemented on a Handle struct that is
intended to hold all runtime driver handles. These methods make more
sense in the blocking module.
A step towards the goal of consolidating thread-local variables, this
patch removes the `time::Handle` reference from the time driver struct.
The reference is moved a level up and passed into methods that need it.
The end goal is to have a single `Driver` struct that holds each
sub-driver and a single `Handle` ref-counted struct that holds all the
misc handles. Further work on the time driver is blocked by some other
refactoring that will happen in follow-up patches.
Years ago, Tokio was organized as a cluster of separate crates. This
architecture required a trait to represent "park the thread and do
work.". When all crates were combined into a single crate, the `Park`
and `Unpark` traits remained as internal details.
This patch removes these traits as they are no longer needed. This is in
service of a future refactor that will decouple the various resource
drivers and store them in a single struct instead of nesting them.
However, in the mean time, removing the Park/Unpark traits adds a bit of
messy code to make the conditional compilation work. This code will
(hopefully) be short-lived.
This patch makes some minor refactors. It renames `ClockTime` to
`TimeSource` since that is how all variables refer to it. It also moves
the type into a new file.
Finally, it moves the `unpark` handle out of the mutex as it does not
need to be there. Note, the call to `unpark` is still called while the
mutex is held, so there is no functional change. Moving it out of the
mutex is in preparation for moving the unpark handle completely out of
the time driver.
This patch moves the time driver into the runtime module. The time driver
is a runtime concern and is only used by the runtime. Moving the drivers
is the first step to cleaning up some Tokio internals. There will be
follow-up patches that integrate the drivers and other runtime concerns
more closely.
This is an internal refactor and should not impact any public APIs.
Current documentation of `sync::watch::Sender::send` may be intepreted
as the method failing if at some point in past the channel has been
closed because every receiver has been dropped. This isn't true,
however, as the channel could have been reopened by using
`sync::watch::Sender::subscribe`.
This fix clarifies the behavior. Moreover, it is noted that on failure,
the value isn't made available to future subscribers (but returned as
part of the `SendError`).
Fixes#4957.
For historical reasons, the current-thread runtime's scheduler was named
BasicScheduler and the multi-thread runtime's scheduler were named
ThreadPool. This patch renames the schedulers to mirror the runtime
names to increase consistency. This patch also moves the scheduler
implementations into a new `runtime::scheduler` module.
This patch moves the I/O driver into the runtime module. The I/O driver
is a runtime concern and is only used by the runtime. Moving the driver
is the first step to cleaning up some Tokio internals. There will be
follow-up patches that integrate the I/O driver and other runtime concerns
more closely.
Because `Interest` and `Ready` are public APIs, they were moved to the
top-level `io` module instead of moving the types to `runtime`.
This is an internal refactor and should not impact any public APIs.
The multi-threaded scheduler includes a per-worker LIFO slot to
store the last scheduled task. This can improve certain usage patterns,
especially message passing between tasks. However, this LIFO slot is not
currently stealable.
Eventually, the LIFO slot **will** become stealable. However, as a
stop-gap, this unstable option lets users disable the LIFO task when
doing so improves their application's overall performance.
Refs: #4941
The command to build the documentation locally provided in the
Contribution Guide did not work for all crates in the project workspace.
Specifically, to build the docs for `tokio-stream` the flag `--cfg
docsrs` needs to be in the environment variable `RUSTFLAGS` in addition
to being in `RUSTDOCFLAGS`.
Additionally, there was text describing that the docs cannot be built
from the root of the workspace with a link to rust-lang/cargo#9274. That
issue has since been closed as complete and the listed commands do now
work from the root of the workspace. As such, that text has been
removed.
The documentation for`PollEvented` had a single remaining reference to
`Registration::clear_read_ready`, which no longer exists. This change
updates the documentation to refer to `Registration::clear_readiness`
instead.
Upgrade the nightly to a version that works with all of miri,
cargo-hack, minimal-versions, and cargo-check-external-types.
This is a prerequisite for #4902.
The broadcast channel allows multiple senders to send messages to
multiple receivers, where each receiver receives messages starting from
when it subscribes. After all senders are dropped, the receivers will
continue to receive all waiting messages in the buffer and then receive
a `Closed` error.
To mark that a channel has closed, it stores two closed flags, one on
the channel level and another in the buffer slot *after* the last used
slot (this may also be the earliest entry being kept for lagged
receivers, see #2425).
However, we don't need both closed flags, keeping the channel level
closed flag is sufficient.
Without the slot level closed flag, each receiver receives each message
until it is up to date and for that receiver the channel is empty. Then,
the actual return message is chosen depending on the channel level
closed flag; if the channel is NOT closed, then `Empty` is returned, if
the channel is closed then `Closed` is returned instead.
With the modified logic, there is no longer a need to append a closed
token to the internal buffer (by setting the slot level closed flag on
the next slot). This fixes the off by one error described in #4814,
which caused a receiver which was created after the channel was already
closed to get `Empty` from `try_recv` (or hang forever when calling
`recv`) instead of receiving `Closed`.
As a bonus, we save a single `bool` on each buffer slot.
Refs: #4814
The contribution guide describes the Tokio code of conduct and gives a
list of steps to follow when contributing in different ways. In the
guide to contributing a pull request, commands are listed to be run by
the contributor locally to help ensure that the PR will pass on CI.
The command to run clippy isn't the same as the one run on CI,
specifically the command doesn't check the tests. This commit changes
the command to match the one on CI, so that tests are checked by clippy
as well.
As the [epoll documentation points out](https://man7.org/linux/man-pages/man7/epoll.7.html), a read that only partially fills a buffer is sufficient to show that the socket buffer has been drained.
We can take advantage of this by clearing readiness in this case, which seems to significantly improve performance under certain conditions.
Co-authored-by: Carl Lerche <[email protected]>
The documentation for the runtime `Builder::worker_threads` function
incorrectly stated that it would panic if used when constructing a
`current_thread` runtime. In truth, the call to the function has no
effect.
Since adding the described panic to the code could cause new panics in
existing code using tokio, the documentation has been modified to
describe the existing behavior.
Refs: #4773
This change adds a case that was missing from the original PR, #4793.
The `io::driver::Handle::current` function was only covered by
`#[track_caller]` in the case that the `rt` feature is enabled, however
it was missing in the case that the `rt` feture isn't enabled (in which
case a panic would be more common).
This particular case cannot be tested in the tokio tests as they always
run with all features enabled.
Refs: #4413
Functions that may panic can be annotated with `#[track_caller]` so that
in the event of a panic, the function where the user called the
panicking function is shown instead of the file and line within Tokio
source.
This change adds `#[track_caller]` to all the public APIs in the task
module of the tokio crate where the documentation describes how the
function may panic due to incorrect context or inputs.
In cases where `#[track_caller]` does not work, it has been left out.
For example, it currently does not work on async functions, blocks, or
closures. So any call stack that passes through one of these before
reaching the actual panic is not able to show the calling site outside
of tokio as the panic location.
The following functions have call stacks that pass through closures:
* `task::block_in_place`
* `task::local::spawn_local`
Tests are included to cover each potentially panicking function.
The following functions already had `#[track_caller]` applied everywhere
it was needed and only tests have been added:
* `task::spawn`
* `task::LocalKey::sync_scope`
Refs: #4413
Functions that may panic can be annotated with #[track_caller] so that
in the event of a panic, the function where the user called the
panicking function is shown instead of the file and line within Tokio
source.
This change adds #[track_caller] to all the non-unstable public net APIs
in the main tokio crate where the documentation describes how the
function may panic due to incorrect context or inputs. Not all panic
cases can have #[track_caller] applied fully as the callstack passes
through a closure which isn't yet supported by the annotation (e.g. net
functions called from outside a tokio runtime).
Additionally, the documentation was updated to indicate additional cases
in which the public net functions may panic (the same as the io
functions).
Tests are included to cover each potentially panicking function.
Refs: #4413
This adds initial, unstable, support for the wasm32-wasi target. Not all of Tokio's
features are supported yet as WASI's non-blocking APIs are still limited.
Refs: tokio-rs/tokio#4827
Currently, we only have WASM regression tests that run without WASI.
However, rust provides a WASI specific target which enables code to
special case WASI. This PR adds a basic test to cover that case.
This is an initial addition to help land tokio-rs/tokio#4716.
Functions that may panic can be annotated with `#[track_caller]` so that
in the event of a panic, the function where the user called the
panicking function is shown instead of the file and line within Tokio
source.
This change adds `#[track_caller]` to all the public APIs in the sync
module of the tokio crate where the documentation describes how the
function may panic due to incorrect context or inputs.
In cases where `#[track_caller]` does not work, it has been left out.
For example, it currently does not work on async functions, blocks, or
closures. So any call stack that passes through one of these before
reaching the actual panic is not able to show the calling site outside
of tokio as the panic location.
The following functions have call stacks that pass through closures:
* `sync::watch::Sender::send_modify`
* `sync::watch::Sender::send_if_modified`
Additionally, in the above functions it is a panic inside the supplied
closure which causes the function to panic, and so showing the location
of the panic itself is desirable.
The following functions are async:
* `sync::mpsc::bounded::Sender::send_timeout`
Tests are included to cover each potentially panicking function.
Refs: #4413
Functions that may panic can be annotated with #[track_caller] so that
in the event of a panic, the function where the user called the
panicking function is shown instead of the file and line within Tokio
source.
This change adds #[track_caller] to the signal() function which is the
only function in the signal public API which can panic. Documentation
was added to this function to indicate that it may panic.
Not all panic cases can have #[track_caller] applied fully as the
callstack passes through a closure which isn't yet supported by the
annotation (e.g. signal() called from outside a tokio runtime).
Tests are included to cover the case where signal() is called from a
runtime without IO enabled.
Refs: #4413
This fixes#4801, where, as a result of https://github.com/rust-lang/rust/pull/95469, our implementation of cat used for this test no longer works, as stdio functions on windows now can abort the process if the pipe is set to nonblocking mode.
Unfortunately in windows, setting one end of the pipe to be nonblocking makes the whole thing nonblocking, so when, in tokio::process we set the child pipes to nonblocking mode, it causes serious problems for any rust program at the other end.
Fixing this issue is for another day, but fixing the tests is for today.
Functions that may panic can be annotated with #[track_caller] so that
in the event of a panic, the function where the user called the
panicking function is shown instead of the file and line within Tokio
source.
This change adds #[track_caller] to all the non-unstable public io APIs
in the main tokio crate where the documentation describes how the
function may panic due to incorrect context or inputs.
Additionally, the documentation for `AsyncFd` was updated to indicate
that the functions `new` and `with_intent` can panic.
Tests are included to cover each potentially panicking function. The
logic to test the location of a panic (which is a little complex), has
been moved to a test support module.
Refs: #4413
Functions that may panic can be annotated with #[track_caller] so that
in the event of a panic, the function where the user called the
panicking function is shown instead of the file and line within Tokio
source.
This change adds #[track_caller] to all the non-unstable public APIs in
tokio-util where the documentation describes how the function may panic
due to incorrect context or inputs.
In cases where `#[track_caller]` does not work, it has been left out. For
example, it currently does not work on async functions, blocks, or
closures. So any call stack that passes through one of these before
reaching the actual panic is not able to show the calling site outside
of tokio as the panic location.
The public functions that could not have `#[track_caller]` added for
this reason are:
* `time::advance`
Tests are included to cover each potentially panicking function. In the
following cases, `#[track_caller]` had already been added, and only
tests have been added:
* `time::interval`
* `time::interval_at`
Refs: #4413
* util: add track_caller to public APIs
Functions that may panic can be annotated with `#[track_caller]` so that
in the event of a panic, the function where the user called the
panicking function is shown instead of the file and line within Tokio
source.
This change adds `#[track_caller]` to all the non-unstable public APIs in
tokio-util where the documentation describes how the function may panic
due to incorrect context or inputs.
In one place, an assert was added where the described behavior appeared
not to be implemented. The documentation for `DelayQueue::reserve`
states that the function will panic if the new capacity exceeds the
maximum number of entries the queue can contain. However, the function
didn't panic until a higher number caused by an allocation failure. This
is inconsistent with `DelayQueue::insert_at` which will panic if the
number of entries were to go over MAX_ENTRIES.
Tests are included to cover each potentially panicking function.
Refs: #4413
* fix tests on FreeBSD 32-bit (I hope)
Some tests were failing on FreeBSD 32-bit because the "times too far in
the future" for DelayQueue were also too far in the future for the OS.
Fixed by copying the MAX_DURATION value from where it's defined and
using it to create a duration that is just 1 more than the maximum. This
will start to break once we get close (within 2 and a bit years) of the
Epochalypse (19 Jan, 2038) - but a lot of other things are going to be
breaking on FreeBSD 32-bit by then anyway.
Functions that may panic can be annotated with `#[track_caller]` so that
in the event of a panic, the function where the user called the
panicking function is shown instead of the file and line within Tokio
source.
This change adds `#[track_caller]` to all the non-unstable public APIs
in tokio-stream (only `chunks_timeout` in `StreamExt`) where the
documentation describes how this function may panic due to incorrect
input.
A test has been included to cover the panic.
Refs: #4413
Functions that may panic can be annotated with `#[track_caller]` so that
in the event of a panic, the function where the user called the
panicking function is shown instead of the file and line within Tokio
source.
This change adds track caller to all the non-unstable public APIs in
Tokio core where the documentation describes how the function may panic
due to incorrect context or inputs. Since each internal function needs
to be annotated down to the actual panic, it makes sense to start in
Tokio core functionality.
Tests are needed to ensure that all the annotations remain in place in
case internal refactoring occurs.
The test installs a panic hook to extract the file location from the
`PanicInfo` struct and clone it up to the outer scope to check that the
panic was indeed reported from within the test file. The downside to
this approach is that the panic hook is global while set and so we need
a lot of extra functionality to effectively serialize the tests so that
only a single panic can occur at a time.
The annotation of `block_on` was removed as it did not work. It appears
to be impossible to correctly chain track caller when the call stack to
the panic passes through clojures, as the track caller annotation can
only be applied to functions. Also, the panic message itself is very
descriptive.
Allows the user to configure the runtime's behavior when a spawned task
panics. Currently, the panic is propagated to the JoinHandle and the
runtime resumes. This patch lets the user set the runtime to shutdown on
unhandled panic.
So far, this is only implemented for the current-thread runtime.
Refs: #4516
Extracts the refactor from #4518.
The basic scheduler takes many configuration options as arguments to the
constructor. This cleans it up a bit by defining a `Config` struct and
using that to pass arguments to the constructor.
Found some `if let` statements that don't actually bind any variables.
This can make it somewhat confusing as to whether the pattern is binding
the value or being matched against. Switching to `matches!` and equality
comparisons allow these to be removed.
Co-authored-by: David Koloski <[email protected]>
This doesn't cause any issues in practice because this is a private API
that is only used in ways that cannot trigger UB. Indexing into `slots`
is not sound until after we've asserted that the page is allocated,
since that aliases the first slot which may not be allocated. This PR
also switches to using `as_ptr` to obtain the base pointer for clarity.
Co-authored-by: David Koloski <[email protected]>
* task: add consume_budget for cooperative scheduling
For cpu-only computations that do not use any Tokio resources,
budgeting does not really kick in in order to yield and prevent
other tasks from starvation. The new mechanism - consume_budget,
performs a budget check, consumes a unit of it, and yields only
if the task exceeded the budget. That allows cpu-intenstive
computations to define points in the program which indicate that
some significant work was performed. It will yield only if the budget
is gone, which is a much better alternative to unconditional yielding,
which is a potentially heavy operation.
* tests: add a test case for task::consume_budget
The test case ensures that the task::consume_budget utility
actually burns budget and makes the task yield once the whole
budget is gone.
Updating an atomic variable with a load followed by a store is racy. It
defeats the entire purpose of using atomic variables, and can result in
"lost" updates. Instead, use fetch_add .
Adds knobs to the runtime builder to control the number of ticks
between polling the global task queue (fairness) and the event driver
(I/O prioritization).
Both varieties of scheduler already supported these intervals, but they
were defined by private constants. Some workloads benefit from
customizing these values.
Closes#4651
## Motivation
Currently, the various spawning methods on `tokio::task::JoinSet` and
`tokio_util::task::JoinMap` lack `#[track_caller]` attributes, so the
`tracing` spans generated for tasks spawned on `JoinSet`s/`JoinMap`s
will have the `JoinSet` spawning method as their spawn location, rather
than the user code that called that method.
## Solution
This PR fixes that by...adding a bunch of `#[track_caller]` attributes.
## Future Work
In the future, we may also want to consider adding additional data to
the task span indicating that the task is spawned on a `JoinSet`...
Signed-off-by: Eliza Weisman <[email protected]>
## Motivation
`task::JoinSet` currently has both `spawn`/`spawn_local` methods,
and `spawn_on`/`spawn_local_on` variants of these methods that take a
reference to a runtime `Handle` or to a `LocalSet`, and spawn tasks on
the provided runtime/`LocalSet`, rather than the current one. The
`task::Builder` type is _also_ an API type that can spawn tasks, but it
doesn't have `spawn_on` variants of its methods. It occurred to me that
it would be nice to have similar APIs on `task::Builder`.
## Solution
This branch adds `task::Builder::spawn_on`,
`task::Builder::spawn_local_on`, and `task::Builder::spawn_blocking_on`
methods, similar to those on `JoinSet`. In addition, I did some
refactoring of the internal spawning APIs --- there was a bit of
duplicated code that this PR reduces.
Signed-off-by: Eliza Weisman <[email protected]>
This documents why it is safe to convert `bytes::UninitSlice` to `&mut
[MaybeUninit<u8>]`, and shrinks one of the unsafe blocks to make these
functions easier to audit.
Add a function that is more versatile than send_modify(). The result of
the passed closure indicates if the mutably borrowed value has actually
been modified or not. Receivers are only notified if the value has been
modified as indicated by the sender.
Signed-off-by: Uwe Klotz <[email protected]>
## Motivation
In many cases, it is desirable to spawn a set of tasks associated with
keys, with the ability to cancel them by key. As an example use case for
this sort of thing, see Tower's [`ReadyCache` type][1].
Now that PR #4530 adds a way of cancelling tasks in a
`tokio::task::JoinSet`, we can implement a map-like API based on the
same `IdleNotifiedSet` primitive.
## Solution
This PR adds an implementation of a `JoinMap` type to
`tokio_util::task`, using the `JoinSet` type from `tokio::task`, the
`AbortHandle` type added in #4530, and the new task IDs added in #4630.
Individual tasks can be aborted by key using the `JoinMap::abort`
method, and a set of tasks whose key match a given predicate can be
aborted using `JoinMap::abort_matching`.
When tasks complete, `JoinMap::join_one` returns their associated key
alongside the output from the spawned future, or the key and the
`JoinError` if the task did not complete successfully.
Overall, I think the way this works is pretty straightforward; much of
this PR is just API boilerplate to implement the union of applicable
APIs from `JoinSet` and `HashMap`. Unlike previous iterations on the
`JoinMap` API (e.g. #4538), this version is implemented entirely in
`tokio_util`, using only public APIs from the `tokio` crate. Currently,
the required `tokio` APIs are unstable, but implementing `JoinMap` in
`tokio-util` means we will never have to make stability commitments for
the `JoinMap` API itself.
[1]: https://github.com/tower-rs/tower/blob/master/tower/src/ready_cache/cache.rs
Signed-off-by: Eliza Weisman <[email protected]>
## Motivation
PR #4538 adds a prototype implementation of a `JoinMap` API in
`tokio::task`. In [this comment][1] on that PR, @carllerche pointed out
that a much simpler `JoinMap` type could be implemented outside of
`tokio` (either in `tokio-util` or in user code) if we just modified
`JoinSet` to return a task ID type when spawning new tasks, and when
tasks complete. This seems like a better approach for the following
reasons:
* A `JoinMap`-like type need not become a permanent part of `tokio`'s
stable API
* Task IDs seem like something that could be generally useful outside of
a `JoinMap` implementation
## Solution
This branch adds a `tokio::task::Id` type that uniquely identifies a
task relative to all other spawned tasks. Task IDs are assigned
sequentially based on an atomic `usize` counter of spawned tasks.
In addition, I modified `JoinSet` to add a `join_with_id` method that
behaves identically to `join_one` but also returns an ID. This can be
used to implement a `JoinMap` type.
Note that because `join_with_id` must return a task ID regardless of
whether the task completes successfully or returns a `JoinError`, I've
also changed `JoinError` to carry the ID of the task that errored, and
added a `JoinError::id` method for accessing it. Alternatively, we could
have done one of the following:
* have `join_with_id` return `Option<(Id, Result<T, JoinError>)>`, which
would be inconsistent with the return type of `join_one` (which we've
[already bikeshedded over once][2]...)
* have `join_with_id` return `Result<Option<(Id, T)>, (Id, JoinError)>>`,
which just feels gross.
I thought adding the task ID to `JoinError` was the nicest option, and
is potentially useful for other stuff as well, so it's probably a good API to
have anyway.
[1]: https://github.com/tokio-rs/tokio/pull/4538#issuecomment-1065614755
[2]: https://github.com/tokio-rs/tokio/pull/4335#discussion_r773377901Closes#4538
Signed-off-by: Eliza Weisman <[email protected]>
Previously, `runtime::Handle` was a single struct composed of the
internal handles for each runtime component. This patch splits the
`Handle` struct into a `HandleInner` which contains everything
**except** the task scheduler handle. Now, `HandleInner` is passed to
the task scheduler during creation and the task scheduler is responsible
for storing it. `Handle` only needs to hold the scheduler handle and
can access the rest of the component handles by querying the task
scheduler.
The motivation for this change is it now enables the multi-threaded
scheduler to have direct access to the blocking spawner handle.
Previously, when spawning a new thread, the multi-threaded scheduler had
to access the blocking spawner by accessing a thread-local variable.
Now, in theory, the multi-threaded scheduler can use `HandleInner`
directly. However, this change hasn't been done in this PR yet.
Also, now the `Handle` struct is much smaller.
This change is intended to make it easier for the multi-threaded
scheduler to shutdown idle threads and respawn them on demand.
Handle::current docs say it's not possible to call it on any non-runtime thread, but you can call it from a runtime context created by an EnterGuard. This updates the docs to mention EnterGuard as a way to avoid this panic.
It turns out that the CI job for testing `tokio_unstable` features isn't
actually running doctests for `tokio_unstable`, just lib and integration
tests. This is because RustDoc is responsible for running doctests, and
it needs the unstable cfg passed to it separately from `RUSTFLAGS`.
This means that if the examples for unstable APIs are broken, CI won't
catch this, which is not great!
This commit changes the `test-unstable` CI job to pass `--cfg
tokio_unstable` in `RUSTDOCFLAGS` as well as `RUSTFLAGS`. This way,
doctests for unstable APIs should actually run.
I also fixed a typo in one of the runtime metrics doctests that was
causing a compilation error, which was caught as a result of actually
testing the unstable API docs on CI. :)
Signed-off-by: Eliza Weisman <[email protected]>
## Motivation
Recent Clippy releases have added some new lints that trigger on some
code in Tokio. These aren't a big deal, but seeing them in my editor is
mildly annoying.
## Solution
This branch fixes the following issues flagged by Clippy:
* manual `Option::map` implementation
* use of `.map(...).flatten(...)` that could be replaced with
`.and_then(...)`
* manual implementation of saturating arithmetic on `Duration`s
* simplify some boolean expressions in assertions (`!res.is_ok()` can be
`res.is_err()`)
* fix redundant field names in initializers
* replace an unnecessary cast to `usize` with an explicitly typed
integer literal
Signed-off-by: Eliza Weisman <[email protected]>
## Motivation
Before we stabilize the `JoinSet` API, we intend to add a method for
individual tasks in the `JoinSet` to be aborted. Because the
`JoinHandle`s for the tasks spawned on a `JoinSet` are owned by the
`JoinSet`, the user can no longer use them to abort tasks on the
`JoinSet`. Therefore, we need another way to cause a remote abort of a
task on a `JoinSet` without holding its `JoinHandle`.
## Solution
This branch adds a new `AbortHandle` type in `tokio::task`, which
represents the owned permission to remotely cancel a task, but _not_ to
await its output. The `AbortHandle` type holds an additional reference
to the task cell.
A crate-private method is added to `JoinHandle` that returns an
`AbortHandle` for the same task, incrementing its ref count.
`AbortHandle` provides a single method, `AbortHandle::abort(self)`, that
remotely cancels the task. Dropping an `AbortHandle` decrements the
task's ref count but does not cancel it. The `AbortHandle` type is
currently marked as unstable.
The spawning methods on `JoinSet` are modified to return an
`AbortHandle` that can be used to cancel the spawned task.
## Future Work
- Currently, the `AbortHandle` type is _only_ available in the public
API through a `JoinSet`. We could also make the
`JoinHandle::abort_handle` method public, to allow users to use the
`AbortHandle` type in other contexts. I didn't do that in this PR,
because I wanted to make the API addition as minimal as possible, but we
could make this method public later.
- Currently, `AbortHandle` is not `Clone`. We could easily make it
`Clone` by incrementing the task's ref count. Since this adds more trait
impls to the API, we may want to be cautious about this, but I see no
obvious reason we would need to remove a `Clone` implementation if one
was added...
- There's been some discussion of adding a `JoinMap` type that allows
aborting tasks by key, and manages a hash map of keys to `AbortHandle`s,
and removes the tasks from the map when they complete. This would make
aborting by key much easier, since the user wouldn't have to worry about
keeping the state of the map of abort handles and the tasks actually
active on the `JoinSet` in sync. After thinking about it a bit, I
thought this is probably best as a `tokio-util` API --- it can currently
be implemented in `tokio-util` with the APIs added in `tokio` in this
PR.
- I noticed while working on this that `JoinSet::join_one` and
`JoinSet::poll_join_one` return a cancelled `JoinError` when a task is
cancelled. I'm not sure if I love this behavior --- it seems like it
would be nicer to just skip cancelled tasks and continue polling. But,
there are currently tests that expect a cancelled `JoinError` to be
returned for each cancelled task, so I didn't want to change it in
_this_ PR. I think this is worth revisiting before stabilizing the API,
though?
Signed-off-by: Eliza Weisman <[email protected]>
## Motivation
Rust error messages seem to have changed a bit in 1.59.0
## Solution
Update the `trybuild` stderr output.
This should unbreak the tests on the latest Rust (and fix CI).
## Motivation
The `JoinSet` type is currently missing the `tokio_unstable` and
`feature = "rt"` `doc(cfg(...))` attributes, making it erroneously
appear to be available without the required feature and without unstable
features enabled. This is incorrect.
I believe this is because `doc(cfg(...))` on a re-export doesn't
actually add the required cfgs to the type itself, and the
`cfg_unstable!` is currently only guarding a re-export and module.
## Solution
This PR fixes the missing attributes.
## Motivation
PR #4499 made the `JoinSet` API unstable, but did not add a
documentation note explaining unstable features. In general, since the
docs.rs build includes unstable APIs, it's probably worth including
these notes so that users understand what it means for an API to be
unstable.
## Solution
This branch adds a note on unstable APIs to the `JoinSet` type-level
documentation, similar to the notes for `task::Builder` and the runtime
metrics APIs.
Also, I noticed that there was a broken link to the top-level
documentation on unstable APIs in the docs for `task::Builder`, so I
fixed that as well.
time: resolve TODO
## Motivation
Existing `TODO` comment in `src/time/driver/wheel/level.rs`.
## Solution
`level_range()` always return a strictly positive power of 2. If `b` is a
strictly positive power of 2, `a - (a % b)` is equal to `a & !(b - 1)`.
The major breaking change in Mio v0.8 is TcpSocket type being removed.
Replacing Mio's TcpSocket we switch to the socket2 library which
provides a similar type Socket, as well as SockRef, which provide all
options TcpSocket provided (and more!).
Tokio's TcpSocket type is now backed by Socket2 instead of Mio's
TcpSocket. The main pitfall here is that socket2 isn't non-blocking by
default, which Mio obviously is. As a result we have to do potentially
blocking calls more carefully, specifically we need to handle
would-block-like errors when connecting the TcpSocket ourselves.
One benefit for this change is that adding more socket options to
TcpSocket is now merely a single function call away (in most cases
anyway).
PR #3881 factored out the spawning of local tasks on a `LocalSet` into a
function `spawn_local_inner`, so that the implementation could be shared
with the `task::Builder` API. But, that PR neglected to add a
`#[track_caller]` attribute to `spawn_local_inner`, so the `tracing`
spans for local tasks are all generated with `spawn_local_inner` as
their spawn location, rather than forwarding the actual spawn location
from the calling function.
This causes pretty useless results when using `tokio-console` with code
that spawns a number of local tasks, such as Actix
(https://reddit.com/r/rust/comments/snt5fq/can_tokioconsole_profile_actixrt/)
This commit fixes the issue by adding the missing `#[track_caller]`
attribute.
`Instant::duration_since`, `Instant::elapsed`, and `Instant::sub` may
panic. This is especially dangerous when `Instant::now` travels back in
time. While this isn't supposed to happen, this behavior is highly
platform-dependent (e.g., rust-lang/rust#86470).
This change modifies the behavior of `tokio::time::Instant` to prevent
this class of panic, as proposed for `std::time::Instant` in
rust-lang/rust#89926.
When backporting patches to LTS branches, we often run into CI failures due to
changes in rust. Newer rust versions add more lints, which break CI. We really
don't want to also have to backport patches that fix CI, so instead, LTS branches
should pin the stable rust version in CI (e.g. #4434).
This PR restructures the CI config files to make it a bit easier to set a specific rust
version in CI.
* Several of tokio's features (e.g. the channel implementation) do not
need a runtime to work, and can be compiled and used for
wasm32-unknown-unknown targets
* This change enables running tests for the `sync` and `macros` features
so that we can note any regressions there
This patch reduces the number of times worker threads wake up without having
work to do in the multi-threaded scheduler. Unnecessary wake-ups are expensive
and slow down the scheduler. I have observed this change reduce no-op wakes
by up to 50%.
The multi-threaded scheduler is work-stealing. When a worker has tasks to process,
and other workers are idle (parked), these idle workers must be unparked so that
they can steal work from the busy worker. However, unparking threads is expensive,
so there is an optimization that avoids unparking a worker if there already exists
workers in a "searching" state (the worker is unparked and looking for work). This
works pretty well, but transitioning from 1 "searching" worker to 0 searching workers
introduces a race condition where a thread unpark can be lost:
* thread 1: last searching worker about to exit searching state
* thread 2: needs to unpark a thread, but skip because there is a searching worker.
* thread 1: exits searching state w/o seeing thread 2's work.
Because this should be a rare condition, Tokio solves this by always unparking a
new worker when the current worker:
* is the last searching worker
* is transitioning out of searching
* has work to process.
When the newly unparked worker wakes, if the race condition described above
happened, "thread 2"'s work will be found. Otherwise, it will just go back to sleep.
Now we come to the issue at hand. A bug incorrectly set a worker to "searching"
when the I/O driver unparked the thread. In a situation where the scheduler was
only partially under load and is able to operate with 1 active worker, the I/O driver
would unpark the thread when new I/O events are received, incorrectly transition
it to "searching", find new work generated by inbound I/O events, incorrectly
transition itself from the last searcher -> no searchers, and unpark a new thread.
This new thread would wake, find no work and go back to sleep.
Note that, when the scheduler is fully saturated, this change will make no impact
as most workers are always unparked and the optimization to avoid unparking
threads described at the top apply.
Re-applies #4377 and fixes the bug resulting in Hyper's double panic.
Revert: #4394
Original PR:
This PR does some refactoring to the current-thread scheduler bringing it closer to the structure of the
multi-threaded scheduler. More specifically, the core scheduler data is stored in a Core struct and that
struct is passed around as a "token" indicating permission to do work. The Core structure is also stored
in the thread-local context.
This refactor is intended to support #4373, making it easier to track counters in more locations in the
current-thread scheduler.
I tried to keep commits small, but the "set Core in thread-local context" is both the biggest commit and
the key one.
Under the hood, the watch channel uses a RwLock to implement reading
(borrow) and writing (send). This may cause a deadlock if a user has
concurrent borrows on the same thread. This is most likely to occur due
to a recursive borrow.
This PR adds documentation to describe the deadlock so that future users
of the watch channel will be aware.
This patch does some refactoring to the current-thread scheduler bringing it closer to the
structure of the multi-threaded scheduler. More specifically, the core scheduler data is stored
in a Core struct and that struct is passed around as a "token" indicating permission to do
work. The Core structure is also stored in the thread-local context.
This refactor is intended to support #4373, making it easier to track counters in more locations
in the current-thread scheduler.
This adds the following methods:
- UdpSocket::set_send_buffer_size
- UdpSocket::send_buffer_size
- UdpSocket::set_recv_buffer_size
- UdpSocket::recv_buffer_size
Currently, the docs.rs documentation for tokio is built without
--cfg tokio_unstable set. This means that unstable features are not shown in
the API docs, making them difficutl to discover. Clearly, we do want to
document the existence of unstable APIs, given that there's a section in
the lib.rs documentation listing them, so it would be better if it was
also possible to determine what APIs an unstable feature enables when
reading the RustDoc documentation.
This branch changes the docs.rs metadata to also pass --cfg tokio_unstable
when building the documentation. It turns out that it's
necessary to separately pass the cfg flag to both RustDoc and rustc,
or else the tracing dependency, which is only enabled in
target.cfg(tokio_unstable).dependencies, will be missing and the build
will fail.
In addition, I made some minor improvements to the docs for unstable
features. Some links in the task::Builder docs were broken, and the
required tokio_unstable cfg was missing from the doc(cfg(...))
attributes. Furthermore, I added a note in the top-level docs for
unstable APIs, stating that they are unstable and linking back to the
section in the crate-level docs that explains how to enable unstable
features.
Fixes#4328
For now, this is only allowed on TcpStream. This is a problem when one
want to disable lingering (i.e. set it to Duration(0, 0)). Without being
able to set it prior to the connect call, if the connect future is
dropped it would leave sockets in a TIME_WAIT state.
Co-authored-by: Fabien Gaud <[email protected]>
Reads and buffered reads from a `tokio::io::empty` were always marked
as ready. That makes sense, given that there is nothing to wait for.
However, doing repeated reads on the `empty` could stall the event
loop and prevent other tasks from making progress.
This change uses tokio's coop system to yield control back to the
executor when appropriate.
Note that the issue that originally triggered this PR is not fixed
yet, because the `timeout` function will not poll the timer after
empty::read runs out of budget. A different change will be needed to
address that.
Refs: #4291
The implicit elided lifetimes of the `AsyncFd` references in return
types of methods on `AsyncFdReadyGuard` resolved to that of `&self`.
However that lifetime is smaller than `'a` since `self` contains an `&'a
AsyncFd` reference. This will not change so the change also does not
lessen future proofing.
## Motivation
This allows `StreamMap` to be used with [`futures::stream::StreamExt::collect`][collect].
My use case is something like this:
```rust
let stream_map: StreamMap<_, _> = things
.into_iter()
.map(|thing| make_stream(thing)) // iterator of futures
.collect::<FuturesUnordered<_>>() // stream of streams
.collect::<StreamMap<_, _>>() // combine all the inner streams into one
.await;
async fn make_stream(thing: Thing) -> impl Stream { ... }
```
[collect]: https://docs.rs/futures/0.3.17/futures/stream/trait.StreamExt.html#method.collect
## Solution
Add `Extend` impl that delegates to the inner `Vec`.
Depends on #4226
## Motivation
Currently, the safety invariants and synchronization strategy used in
`tokio::sync::oneshot` are not particularly obvious, especially to a new
reader. It would be nice to better document this code to make these
invariants clearer.
## Solution
This branch adds `SAFETY:` comments to the `oneshot` channel
implementation. In particular, I've focused on documenting the
invariants around when the inner `UnsafeCell` that stores the value can
be accessed by the sender and receiver sides of the channel.
I still want to take a closer look at when the waker cells can be set,
and I'd like to add more documentation there in a follow-up branch.
Signed-off-by: Eliza Weisman <[email protected]>
The Tokio project team welcomes security reports and is committed to providing prompt attention to security issues. Security issues should be reported privately via [[email protected]](mailto:[email protected]). Security issues should not be reported via the public Github Issue tracker.
The Tokio project team welcomes security reports and is committed to providing prompt attention to security issues. Security issues should be reported privately via [[email protected]](mailto:[email protected]). Security issues should not be reported via the public GitHub Issue tracker.
## Vulnerability coordination
Remediation of security vulnerabilities is prioritized by the project team. The project team coordinates remediation with third-party project stakeholders via [Github Security Advisories](https://help.github.com/en/github/managing-security-vulnerabilities/about-github-security-advisories). Third-party stakeholders may include the reporter of the issue, affected direct or indirect users of Tokio, and maintainers of upstream dependencies if applicable.
Remediation of security vulnerabilities is prioritized by the project team. The project team coordinates remediation with third-party project stakeholders via [GitHub Security Advisories](https://help.github.com/en/github/managing-security-vulnerabilities/about-github-security-advisories). Third-party stakeholders may include the reporter of the issue, affected direct or indirect users of Tokio, and maintainers of upstream dependencies if applicable.
Downstream project maintainers and Tokio users can request participation in coordination of applicable security issues by sending your contact email address, Github username(s) and any other salient information to [[email protected]](mailto:[email protected]). Participation in security issue coordination processes is at the discretion of the Tokio team.
Downstream project maintainers and Tokio users can request participation in coordination of applicable security issues by sending your contact email address, GitHub username(s) and any other salient information to [[email protected]](mailto:[email protected]). Participation in security issue coordination processes is at the discretion of the Tokio team.
## Security advisories
The project team is committed to transparency in the security issue disclosure process. The Tokio team announces security issues via [project Github Release notes](https://github.com/tokio-rs/tokio/releases) and the [RustSec advisory database](https://github.com/RustSec/advisory-db) (i.e. `cargo-audit`).
The project team is committed to transparency in the security issue disclosure process. The Tokio team announces security issues via [project GitHub Release notes](https://github.com/tokio-rs/tokio/releases) and the [RustSec advisory database](https://github.com/RustSec/advisory-db) (i.e. `cargo-audit`).
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Carl Lerche
dtolnay